Network data corruption after upgrade

[brucet622]

Sorry but I have no idea if this is a qnap problem or not but I am running a Ubuntu 20.04 virtual machine on a qNAP server and am seeing data corruption problems when accessing it across the network to a windows machine. (Samba, winscp, etc.) I have two other nearly identical setups that don't show this issue but the Ubuntu VMs are not on a domain. This setup was running just fine until qNAP automatically updated itself to the latest version but the Ubuntu OS was also updated at that time so I can't really tell what caused it. I'm in the process of getting a domain fired up to see if I can reproduce the problem.

For all the gory details of what we have and what we have tried, see the following link...

https://askubuntu.com/questions/1390111 ... ws-clients

Like I said, I have no concrete evidence that this is caused by qNAP, its latest update, or the phase of the moon. I wanted to bring this up and maybe someone might be able to point me in the right direction (or any direction for that matter!)

Thanks!

Bruce

P.S. Will post a response if I find out what is going on....that is if I'm not fired before then!

[brucet622]

Well, after 4 days on site with the customer, 1500 miles of driving, and attempting all sorts of different configurations, I can now say with certainty that the network data corruption is a qNAP problem. Note that it is not on ALL of our qNAP servers since I setup as close to an identical setup in our lab with a smaller, lower-end qNAP server, and could not reproduce the issue. I brought back the customer's qNAP server after moving the system and their data to local hardware, and hooked it up onto our test system, replacing just the qNAP server. The file corruption was reproduced.

This was qNAP version software version that was shoved down everyone throats a couple of weeks ago. I see there is yet another code update available but I can't justify trying that version after all the time and expenses incurred on this issue. If someone from qNAP wants to diagnose it, I suppose I'll help but I can't justify any more work on this disaster.

Our company was considering using these devices as a way to easily deploy our complex software on a customer site. There are lots of options obviously but many of our target customers don't have servers, hypervisors, or the expertise to manage them. After this expensive incident where the customer was down for two weeks, qNAP is no longer being considered.

Good luck everyone and I hope you never have to deal with a problem like this one.

Bruce

[dosborne]

This was qNAP version software version that was shoved down everyone throats a couple of weeks ago.

I don't have any insight on the rest, but my systems have autoupdate disabled (including the check for new versions) and have never been updated unless I apply one manually. There is quite a dispute as to whether or not there really was a "forced" update or not. Many people didn't notice the check box and I think the default setting was change a while ago, so if you didn't go look then autoupdate was enabled. In any case, a small point, but not necessarily "forced".

Are you sure the issue is firmware related? Have you tried going back to a firmware version that worked in the past as a test? Could this just be a hardware failure?

I've never worked in any production environment or company (from small to 100,000+ people) that ever turned on autoupdate features in any live production infrastructure.

[brucet622]

OK, I make it a habit of turning off automatic updates but I can't prove I had it off. So, I'll give you that point. The problem started the day that the new code was upgraded so I have hard time believing it is/was a hardware problem. We did swap the Ethernet ports, cables, switch ports, etc. Submitted a service ticket so we'll see what comes of that.

[dosborne]

Assuming you have a backup, then rolling back the firmware would be an easy test to isolate or rule out the issue.

An internal Ethernet port issue would be harder to diagnose unless you had another box to test with.

[brucet622]

Well, I might be able to attempt a firmware downgrade now that I have it in our lab. The customer was "very resistant" to downgrading software with a known vulnerability during an active attack.

[brucet622]

Working with qNAP support. Sent them the logs and the wireshark traces of the two conditions (Corruption of the file during transfer and Unexpected network error on very large transfers). Will post any responses from them. Leaving the firmware at its current (failing) level for now until they suggest any up or downgrades.

[jarohex]

Hi Bruce, any news from qnap support?

[brucet622]

Nothing yet. After a few back and forth messages to clarify the condition, this is the last response I received last Friday.

>I haven't heard of this situation before, so let me go ahead and escalate this to our RD team for further review. Hopefully they can identify and resolve the issue.
>If they request a remote session, I will communicate that with you and try to schedule something. Thanks.

As you can se, I've told them I have the system available that can duplicate the problem and is available for remote access should that be useful.

Will post with updates, assuming I get any....

Bruce

[brucet622]

After about a month and a half of back and forth, I did get a sort of conformation that I'm not the only with this file corruption when reading data from Ubuntu VM to a PC. Gave them remote access to the machine. They did say that others are getting errors in their logs but I don't seem to see anything. We'll see what they come up with .

Stay tuned...

[FSC830]

There are some major issues in this case.
At first, QNAP is not a server! I know, its advertised as a one-4-all, can be easily managed, have nice and smooth functions, provides a lot of useful apss, etc. etc. etc.
But it is NOT!
QTS is a proprietary OS, based on Linux, but still proprietary with some undocumented modifications.
Is was designed as a network attached storage, not as a network attached server! That is, what people are always believe and what QNAP wants to make them believing!
But over the years it shows so often that QNAP cant fullfill their promises.
So I would never decide to use such a device as basic solution for my products. A real small server, running uBuntu, Debian or an other Linux distribution would be much better!
There have been so many issues running VS in the last months, you are on the lucky side if this now is the first issue with that.

Next: customer was attacked by ransomware? So customer uses an unsecured internet access? Thats not a reputation for customer and his IT staff! In a company how can this be that security is not in focus?

And customer has had an outage of two weeks? No backups? No desaster recovery scenario?
And yes, the update was not really "forced", there was an new option in a firmware v4.5.x (around April 2021) which was enabled by default and nearly no one noticed that.
The option is "update to recommended firmware" and is independent from the "auto update" option , another pretty surprise by QNAP!

Sorry, but this seems really that here are some homebrew issues, beside the annoying bugged firmware!

Regards

[brucet622]

Well, there is a lot more to this than you are assuming (SC830) and I don't see a need to go into them here, but your main point is now taken by our management. We have dumped the use of qNAP devices completely. I guess that is what you were getting at. What I was hoping was some type of conclusion as to what is going on but today I got a notice from support that they have no idea what is happening and have no fix for it. But, we are not the only ones seeing data corruption and network errors.

Bruce