QNAP firmware issues : Major file system bug

[grocovid]

A very quick note: I have had this problem twice (once when I was in a single-disk setup 90% full, now with my 3-disk raid-5 setup also 90% full - hoping that adding a 4th disk will solve the problem, until I get 90% full again, but then i'll have to buy a second nas anyhow). I wondered if it might be related to the way I transfer file to the NAS, using samba. So I disabled smbd, and transferred files using FTP. Same problem (even worse, since I had FTP timeout errors, samba seems more resilient here !).

Linux file copies from the shell seems to be just as bad (but I can't say whether it's bursty or not - it's just really slow for big files )

Grocovid

[Don]

I think 25 pages of me too postings is enough.We get it, there is a major problem with the firmware. What we need is some communication from QNAP as to what the problem exactly is and when it will be fixed. Oh and when will the samba security bug will be addressed as well.

QNAP needs to improve their communication with their customers. They also need to improve the amount of time it takes them to provide a fix. Very bad business practice IMHO.

[grocovid]

Sure, Don. Something I don't quite understand, though, is why some users relate this issue to "data loss". For me, this issue may induce timeouts during transfers, but I've never encountered (or read any evidence in this thread) of actual data loss due to this bug. (Not to say this isn't serious, of course, but not as serious as data loss, since in my experience read speed is not affected)

Grocovid

[P3R]

Something I don't quite understand, though, is why some users relate this issue to "data loss".

The exact prerequisites to be affected aren't even established yet so the problem is certainly complex and maybe in some special cases it could also lead to data loss? In addition to that there are most likely in this thread also a number of reports that have been incorrectly diagnosed as being this particular problem when something else was the real issue. Many are very eager to write yet another "me too" and are not carefully checking if the problem acually fits.

When this problem is finally resolved (but when?) there probably will be some people that are surprised to find that they still have a problem...

[Don]

Since QNAP has not been forthcoming as to what the problem is it is hard to say why some people experience a problem with different number of files, different free space, and on rare occassions data loss which may or may not be related to the actual problem.

[heroheroz]

I sadly add myself in this loop as I found myself suffer in the same problem.
First I found my 410 perform very slowly, from the usual 5Mb/s down to 80kb/s and sometimes non-responseive.
I keep rebooting the nas but it doesnt help, problems appear again and again.
Now I finally notice it is QNAP 's problem, I found they are irresponsible for hiding their problem, I can plan ahead to avoid the problem if it is related to the disk space / number of files in NAS.
Anyway, I will wait and see if they gonna fix this.
For now, I guess I have to remove as much file as I can from the NAS to see if I can reciver it's performance to a usable state.

[Liperty]

I'm in the market for a bigger NAS (Raid 5) and thought it was worth checking the forums for any 'gotchas' and came across this thread.
I really didn't want my important data to be stored on a device with what seems to be described as a serious degradation of performance. This does seem to be a known issue - so I contacted QNAP to ask how / if the issue was resolved.

My email

I would like to purchase a 6 / 8 disk RAID NAS and the Turbo Nas II look like great models according to specs. I did a bit of internet research and found many users complaining of dramatic slow down of performance when the array is nearing a ''full'' state. Apparently this is a firmware issue and I am not confident it has been resolved. Can you please let me know if the issues related to NAS access slowdown have been fixed ? viewtopic.php?p=259148 Thank you.

The response from 'Roy Chen' was as follows

Dear customer,
Thanks for choosing QNAP.
Problem already fixed by firmware 3.6.1

To which I replied

Dear Roy,

According to forum members the 3.6.1 firmware still exhibits this issue
http://pop3.qnap.com/viewtopic.php?f=25 ... &start=315

I checked the release notes for the firmware and it didn't describe a file system performance fix.
I need a level of confidence that I can purchase a turbo NAS without suffering the issues described in your forums. I don't want to recommend the QNAP NASs until these assurances have been given.

Kind regards,

And finally received....

Hi ,
In firmwre 3.6.1 we reserved 500MB space for system can running smoothly to prevent the system halt issue when the volume is full.
Regarding the performance drop down issue, firmware 4.x will fix it.

So....
I can only hope that firmware 4.x will be released in the near future - it seems the latest firmware does not address the issue and a yet to be released firmware (should?) will. Not very reassuring for those sys admins currently relying on the performance of their NAS devices.

[AdrianW]

Not very reassuring for those sys admins currently relying on the performance of their NAS devices.

It's also troubling that Qnap staff haven't posted any updates to this thread for months and months.

Considering this forum is part of their site, you'd think they'd be a little more pro-active in keeping the many, many unhappy customers informed about their progress on this issue.

[Goldtooth]

Not very reassuring for those sys admins currently relying on the performance of their NAS devices.

It's also troubling that Qnap staff haven't posted any updates to this thread for months and months.

Considering this forum is part of their site, you'd think they'd be a little more pro-active in keeping the many, many unhappy customers informed about their progress on this issue.

I agree!

[grocovid]

New observation: I've had this problem for a couple of weeks now. I get only about 140 GB free now, and it seems to be getting worse. But (now comes the interesting part) what I did tonight was this : I copied a 4.3 GB file onto my nas (took more than 30 minutes, a nightmare), then deleted a 700 MB file from the NAS (in fact, I replaced a low-res version with a high-res version of the same movie), then copied a 730 MB file onto the nas. What surprised me is that the first 700 MB of the 730 MB were copied without any freeze.
Apparently, any space that has just been freed can be reused without any problem. This seems to confirm the hypothesis that the NAS has trouble "finding free space". If a block of contiguous free space has just been freed, it will just use it and there won't be any freeze.
I think that's interesting. If there's any ext3 / ext4 linux file system specialist out there, can he please tell us how these FSs manage fragmentation, and if that may not be the root of the problem, I mean maybe the NAS is moving blocks around so that the new file is not created in too many chunks, reducing fragmentation.
Well, that's just a hunch.

grocovid

PS: It really ** me off that QNAP is not reacting to this thread at all. I now need to increase my free space (I would need to even without this awful issue). Going from 3-disk raid 5 to 4-disk raid 5 involves detaching the file system for a couple of days while the raid is being extended, which I do NOT wish to do. So I will probably buy a second NAS with 4 3-TB disks (they are less expensive now), copy everything onto it and reformat my first NAS. So, do I buy a second TS-412 or do I switch over to the competition? Please QNAP I don't want to be rude and discuss the competition on your forum, but if you are not doing anything about this issue, we'll need to. Maybe if we do discuss competing products here, this will make QNAP react ???

[forkless]

In all fairness, I've had two TS-809 units one of which had the symptoms, one did not. Since upgrading to 3.6.1 build 0302 (a week ago for me) the problem seems to be gone on the unit suffering from it. Too early to make any definite statements but let's hope it stays stable that way.

What is totally annoying though is the total lack of communication on the matter. I don't want to make too many assumptions but QNAP really needs to ask themselves if stonewalling their customers on such important issues is a good move. Maybe its an approach I don't understand but acting like the problem does not exist is not the mature/professional way to go about it.

@grocovid: You can expand the RAID5 set offline as well obviously there will be some downtime but you greatly reduce it by tweaking the extreme throughput cap that has been put on rebuilding the array. For more info check out this excellent thread on the topic; viewtopic.php?f=11&t=10268

[pmw]

Hi.

I have same problem here. I got QNAP TS-419P+, after using +65% of disk capasity, smbd started to be extremely slow and I did notice some network problems.
After reading this whole thread... I managed to remove some files and free some space, so now I got 20% usage and everything works fine.

I don't know for how long this problem have been there, never used that much space of my NAS. But I really want to see some patch for this, and soon. This is really big problem and I can't really buy more qnaps or tell anyone to buy qnaps if we can't get patch soon.

If theres any news or good advices to fix this problem, I'm listening.

[forkless]

I think you are confusing this issue for another

The samba performance drop is as far as we know not related. It only had some resemblance to this issue as both symptoms appeared to happen at the 60-70% disk utilization mark as some people observed.

The problem described in this thread however refers to system freezes varying in length from several seconds to up to a minute when copying files/creating new folders. This behavior can be observed when copying files within a shell session from physical location to physical location (ie. excluding any network transport).

[pmw]

Hm, maybe. I tested and mkdir took 3 mins for me. It sounds pretty long time, but not sure is it (and with smbd) related to this one.
Do you have any link for samba-topic? I wonder is that solved already. Sorry if this was pretty offtopic.

[forkless]

I would have to search too. Anyway, no worries you're not the first one and you probably won't be the last one either

However the freezing mkdir does sound symptomatic for this particular issue, have you upgraded to 3.6.1 build 0302 yet?