QNAP firmware issues : Major file system bug

[schumaku]

D would be "Uninterruptible sleep" (typically waiting for I/O), W would be "wait for memory" (paging) ... which is obviously I/O to the page file, which could be RAID (paging is done on a RAID1 (ie. md4 or md5 typically). So the issue might be not on the file system level, more on the process memory usage side - which QNAP has tuned towards perfromance.

[tmt]

Processes in D-wait are completely normal when anything is doing disk i/o. Especially smbd, which is doing i/o to service your remote systems, and kjournald, which is maintaining the EXT3/4 filesystem metadata.

In other words, in itself this isn't the bug. Keep us posted on what QNAP reports.

[schumaku]

Processes in D-wait are completely normal when anything is doing disk i/o. Especially smbd, which is doing i/o to service your remote systems, and kjournald, which is maintaining the EXT3/4 filesystem metadata.

Of course - however these are supposed to be short waits - not things taking that long as it happens for some users.

Further on, in combination with the W (wait for memory) [PFW in my favorite OS...] it appears the process(es) are also try to allocate more memory, what potentially might fail in the physical memory and things start to require swapping...

[Don]

I have also had this happen to me and reported on a number of threads going back many many months.

[ALFo]

The same things happens to me..... (419P, 4x WD 2TB in Raid 5, EXT3). Mostly happens, if i will create a Folder. If the Folder exists, no problem with copying files.
I'm using Asus G71 Laptop with Win7 x64 and also MacMini (mid 2010, Snow Leo 10.6.. No problems with MacMini, but Windows makes this timeout all the time, since fw. 3.4.x .... previously firmwares were OK.....
On my 419P i don't have any QPKGs installed, and only necessary services are running (NFS, SAMBA, AFP) ... Should i post also some logs ? which ones ?

Edit: After upgrade to firmware 3.4.4 all these problems are gone THANKS !

[missileman]

I'm sure I'm getting this issue as well. QNAP TS-639 and TS-459 Pro+

[SuperMario]

This is getting ridiculous... It's now been several months that this problem has existed and our QNAP NAS units are completely useless.

The last email communication from QNAP I had was late last week and they requested more logs and claimed that they "couldn't replicate the problem". They requested that I accept a NAS sent from them and fill it with all my data to replicate the problem... WTF?

What an utterly ridiculous request to ask for many TB of proprietary data from your customers in order to prove a problem exists. When a simple perusal of this very forum going back months will show that many other people have the same issue.

I have the same issue on different versions of their NAS hardware with completely different data sets. So I don't know what their engineers or testers are doing if they "can't re-create the issue". Re-creating the issue is not hard, just fill the unit up past 70% on a large EXT3 RAID5 volume and ensure that you have several million files of sizes ranging upto ~5Gb of so.

I would now suggest that everyone here who has this problem go and email their support people directly and inform them. Maybe someone else is willing or able to send them a complete copy of their NAS with the problem, I sure as ** am not.

Email addresses: support@qnap.com (*email addressed edited for privacy reason)

Maybe a torrent of other ** off customers with the same issues to their email will be harder to ignore. Grrr...

Mario.

[schumaku]

Re-creating the issue is not hard, just fill the unit up past 70% on a large EXT3 RAID5 volume and ensure that you have several million files of sizes ranging upto ~5Gb of so.

I have severeal NAS - some for testing repeatedly flooded up to 99.9% or 100% and don't have the problem occuring here. Disagree - it's obviously not that easy as you say.

[SuperMario]

Ok, let's see what the differences are between us, then:

What is the largest volume size you have?
What filesystem (EXT?) and any caching?
What NAS version and FW?
RAID version?
How many files in total on the volume?
How many directories on the volume?
What percentage of the volume is used?

As I told QNAP, on the NAS units we have the problem on, it only becomes evident after >70% of 10TB is in use, with millions of files and hundreds of thousands of directories, with nesting levels of sub-directories up to 10 deep, using long file-names. File sizes range up to ~5Gb.

I am not the only person to have these issues, either, so it's obviously not that hard to reproduce the problem. I'd be interested to see what your answers are to the questions to compare.

Mario.

[schumaku]

What is the largest volume size you have? ~19.5 TB
What filesystem (EXT?) and any caching? EXT4 with cahcing
What NAS version and FW? 809, 859, ... to stick with the big ones...
RAID version? 5 and 6 (10 only on smaller units)
How many files in total on the volume? Millions
How many directories on the volume? up to around 250'000
What percentage of the volume is used? up to 100% as mentioined

Don't misatek me: I don't say the issue does not exist - I had the chance to login to some where this is certainly a big problem.

For me, it appears very similar to the problems we have seen ion the wralier v3.4 builds on ARM...also there on the smaller units the issue exists. It looks ot me like the issue "boundary" was just moved far up....

[SuperMario]

Schumaku,

You didn't mention what FW you are using? I find it strange that you don't see the issue? Maybe your use of caching with EXT4? What else could it be?

The issue you mention on the ARM builds, is there any further information on it? You make it sound like it is a similar (if not the same) issue.

Is there any more information? I don't for a second believe that QNAP is not aware of the problem, it seems to me that they are attempting to do their best to pretend they can't replicate it, as to why, who knows, maybe their lawyers have given them advice, I can only speculate.

The end result is that we have 5 of these ** NAS units of varying types (and not to mention the useless abortion that is the NMP-1000 that I have at home) that simply don't work as they should. They are costing me money due to lost productivity every day, and I can't keep going on like this!

Mario.

[schumaku]

Production NAS are on v3.4.3, a few already on v3.4.4. Test NAS are on v.3.5.0 Beta 1.

QNAP never unveiled what went mad when reaching sosme VM and/or buffer and/or cache size limitaitons on the early v3.4 ARM units.

[SuperMario]

Latest update on Friday evening from Tech Support, via email.

They have confirmed that they can now replicate the issue on some test NAS units.

They are apparently working on it and promised to get back to me with an update. We'll see.

I still can't believe that it's taken them more than 2 months to get to this stage, and that the problem has existed for more than twice that long... It's ridiculous.

Mario.

[AdrianW]

They have confirmed that they can now replicate the issue on some test NAS units.

They are apparently working on it and promised to get back to me with an update. We'll see.

Well, at least that sounds promising.

In my case the pauses as increasing in length, it used to be about 20 seconds, but its up at around 40 now.

[SuperMario]

Well, contrary to assertions from QNAP, we are starting to get data corruption issues on or NAS units now.

Maybe I should put a warning up as a new thread for all others, but we had a situation this morning where the stall occurred during a write operation (another file was being copied to the server at the same time).

It turns out that the stall stopped the other guest task during the write and now the filesystem is in a bad state. We can see the file that was stopped during the copy, but no longer access that file - any attempt will just stall for over 2 minutes and then eventually the host will return with an error "file is maybe being accessed by another user".

We have rebooted the host machines and the server... Thanks QNAP, so not only do we have useless NAS units, now even our data is at risk!

Grrrr.

I've just about had it. I know this is not to forum for it, but if someone can suggest a more reliable NAS vendor, please do so in a private PM. This is unacceptable.

Mario.