QNAP firmware issues : Major file system bug

[Badmac]

Just a note to say thanks to those who are working with QNAP to solve this issue.

I have been having it for over 3 months now, slowly got worse and worse. (419P+, 3.5.0, 4x2TB Seagates, Raid5, ~300gb free from 6+TB, large number of files and directories). Stalls now for over 2 minutes on EVERY write, can read, but sometimes even the read stalls. Device is now pretty much useless other than as an archive tool. I hope QNAP fixes this VERY soon. We have been selling various models to customers and I have one reporting stalling issues already.

[SuperMario]

Another update:

Over the weekend I did a test with an automated copy op to the worst affected NAS and added a few hundred more files to it. At several points, I recorded the timeout stall and there is a very direct correlation between the amount of files and the timeout delay, which pretty much confirms the suspicion that whatever is causing the stall is to do with the filesystem accessing the allocation tables/journals.

I added another 100 files, after the first 50 the delay was at 2 minutes, 35 seconds. After the full 200 files were added, the delay is now 2 minutes, 55 seconds.

So as the file count on the volume increases, so does the delay - so that begs the question, what services are handling the filesystem and how much system RAM do they have access to?

Watching the LEDs on the front of the NAS shows that some sort of stripped read/write operation is happening during the stall - the LEDs display a repeating pattern, striping from left to right of accesses whilst the timeout is taking place. I doubt that such a long delay is being caused by a simple read, so it is probably trying to buffer way more data than it can hold and is then forced to flush it out to some kind of pagefile/VM on disc for every access.

It would be sensible of QNAP if they could at least respond and let everyone know when we can expect a fix? The radio silence on their part on this issue doesn't instil much confidence.

Mario.

[SuperMario]

I had another email exchange with QNAP last night, the result thus far is that they still don't know how to fix the problem, nor can they even give an estimate time line for an expected fix.

Mario.

[sh4d0wt3ch]

My QNAP NAS has become as good as tape backup at the moment, write once and just used it as an archive. Nearly full at the moment.

[sh4d0wt3ch]

There's a reply from another user in one of the treads that I have created. It discusses about a swap file test they did. Linkin it for reference:

http://forum.qnap.com/viewtopic.php?f=45&t=48592#p221766

[ahallam]

Just wanted to throw my own 2 cents in on this discussion. I've also been having issues with the NAS (TS-410 with three 1.5TB drives and the 4th bay empty, about 300GB space left free), where the NAS would appear to lock up preventing reads and writes. The flush-9:0 process would sit eating up enormous amounts of CPU, and I would have no option but to quit the file operation I was attempting.

After reading this thread, I decided to clean out some backup folders and delete a number of things that I no longer needed stored on the drive. Free space is back to 827GB, and I am no longer experiencing these lockups on file operations. The flush-9:0 process is maxing out at maybe 2% CPU at the most.

Hardly a scientific process, but the evidence certainly indicates to me that the NAS does start running into serious issues with large numbers of files and/or low drive space. For now I'm going to keep the number of files on the NAS relatively lean, but it would be nice to see a resolution to this problem sooner rather than later.

[sh4d0wt3ch]

Hi ahallam, I did the same thing, I deleted alot of files and made my space to about 10% free (500GB+ of 5.6TB / 2TB x 4 in RAID5 settings) and it went back to the same performance. I'm keeping as it is at the moment until this issue is resolved.

One more thing, make sure to empty out your Network Recycle Bin if it is enabled to free up some space from it.

[AdrianW]

I've been having this problem for months, and my Qnap is only 50% full with 5142GB available space. But I do have a *lot* of files on the thing (653,527).

[SuperMario]

Not much to update, but still nothing from QNAP apart from a request for a logfile which one of our guys sent to them earlier in the week.

As we have "spread the load" to reduce writes to the most severely affected NAS units, we are now experiencing this issue on all of our QNAP NAS units.

The worst affected unit, the TS-859Pro+ now stalls for over 3 and a half minutes, that's ridiculous.

Some other people mentioned free space, from what we see here, that isn't the issue - it's more likely that deleting the files has freed up space in the problem area (journals, directory structures, file allocation tables, your guess is as good as mine). Like someone else posted, some of the NAS units with this issue have over 1TB free, so we're pretty certain that the amount of free space on the volume is not a contributing factor; it's just the amount of directories and files (and their length) that is the root cause of the problem.

It's now more than 9 months since this problem was initially reported... I don't want to swear so there isn't much else to say right now.

Mario.

[AdrianW]

It's now more than 9 months since this problem was initially reported... I don't want to swear so there isn't much else to say right now.

Here's the first post I made when I started experiencing the problem: Copying files via samba slow to start (which went totally unnoticed) - that was back in November last year.

[SuperMario]

Well, that's torn it.

The TS-859Pro+ has regressed to a point where it appears to have completely killed the file system. It dropped one of the drives (drive #1) and the NAS reset itself. In the log it shows the "did not shut down" correctly error and then errors about "the file system is not clean" and "RAID in degraded mode".

I removed drive 1, did a full check with the Seagate tools, SMART etc, and put it back in. Left it rebuilding and came back some 4 hours later to check, and the log shows "start rebuilding" and then it drops drive #8.

I then have "error occured while accessing the devices of the volume in degraded mode", yet even after showing an error on drive #8, it then says "drive 1 removed" and the red LED is on above drive 1.

Next entry is "Mount the file system read-only failure".

So, it appears that the entire array is screwed and we have now lost all the data. As all the other QNAPs are having the same issue, even the ones supposed to act as backup units, we appear to have lost ~10TB of our data.

QNAP can change the "Q" in their name from Quality to Questionable from now on.

Mario.

[sl1000]

You could manually readd drive 8 to the array, and let the system rebuild drive 1 again. My guess is you are using consumer grade drives, not bussiness grade?

[SuperMario]

It appears that this issue is another firmware issue, a simple search of this forum shows a large number of other people with drives dropping out and the same error. No idea of these issues are related at this time, but the fact many other people are having issues with different drive types and manufacturers doesn't bode well. Some links below...

Here: viewtopic.php?f=25&t=14246&start=330
Here: http://pop3.qnap.com/viewtopic.php?f=18 ... d4fd5632d0

Right now the NAS won't let us access anything on it, I won't do anything with it until I hear back from QNAP if there is some way of accessing our most recent data.

Mario.

[AlexKe]

No, the HDD drop out issue is not related to the used disk capacity over 75% issue. Please try not to mix them. We are digging on it.

The following two symptom will occur when used disk space is over 75%.
- the SAMBA performance will drop from 30Mb/s to about 1 Mb/s
- late response about 30 ~ 40 seconds when create a share folder

[florian.baumert]

Hi QNAPAlex,

from my experiences with my NAS, these are not the only symptoms. I would add

* slow response (up to a minute) with every read done over a shared connection (even a directory read), not only creations of files or folders (which are writes).
* extreme slow write rates (2MB every 40 seconds) even with a local copy (cp/dd from /dev/zero to a file)
* sometimes data written through samba is garbled (content of the files is zeroed out)

Greetings
Florian