QNAP firmware issues : Major file system bug

[QNAPSimon]

Update on 2012 June 19
If you have similar issues, please first upgrade to firmware 3.7.1 or later, and then check if you are still having performance issue. If you do, please do not hesitate to contact QNAP support or open a new discussion, because in real life operations, other known facts could also cause slow performance, including bad blocks on the hard drives, very little free space left on the disk volume, or fragmented data. . For more information on this topic, please visit here for more info.

--------------------------------------------------------------------
SuperMario, gonna have to hijack your thread to make an announcement.

This problem has been identified as a kernel problem. It's related to how the kernel handles I/O and memory usage. All I can say is that after some testing, the same problem shows up in Another Popular Linux distribution with the same kernel version we are using. (2.6.33.2) However, in a later kernel, 2.6.39, the problem is gone. The tests were done on the same file system.

We will be working on kernel upgrade. However, the entire effort including porting to Marvell platform and proper testing could take several weeks. In the mean time, please contact our tech support if you have the symptom. As we have discovered, a lot of people with similar symptoms did not actually have the same problem. Our tech support will be able to verify for you and maybe resolve your issue. Please do not just assume you have the same issue.

---------------------------------------------------------------------------------------------------------------------------------------------------
Symptoms: User experience dramatic performance degradation when the volume reaches certain percentage usage. However, the problem goes away after some files are removed.
Confirmed affected firmware: 3.3.0~3.5.2
Please contact: support@qnap.com

[SuperMario]

First off the bat, let me say I am a long time user and supporter of QNAP. However, their inaction and incompetence over the past ~6 months has left me with no option but to prove to them that ignoring your customers and burying your head in the sand isn't the way to do business in this day and age.

We have 5x QNAP NAS units, from TS-209 all the way up to a couple of TS-859Pro+ units. There is a major bug in the newer firmware of the units that causes the guest SAMBA process to stall when the main volume gets to above ~75% and there are several million files on the unit.

We have proven to QNAP that this problem is to do with the latest firmwares, it was not present in the older 0408T builds (a downgrading test proved this). After upgrading to the newer FW versions all our larger units (TS-639Pros and TS-859Pro+ units) now exhibit the same problem.

For clarification, the issue exhibits itself as a stall on the QNAP as the file system is waiting for a semaphore that doesn't occur when an attempt to create a directory is being made. This causes timeouts on the host, as the SAMBA guest process blocks until the host times out. This is the same issue that I and many people on this forum are referring to in various other threads on this support forum.

I have provided QNAP (a whole host of them are CC'd into the private email thread that has gone on for months now) and yet, after months there has been no action or even acknowledgement on their part. I had suggested that they make their users aware of the issue publicly, but again, they are trying to keep this under wraps.

Any NMP-1000 owners will realise that this appearing as the much the same as their inaction on the Youtube playback issue - and that *still* doesn't work and they haven't even bothered to inform their customers of even if they can fix it, let alone when they can expect it to work. I do note that they're quite happy to continue selling those units to new customers in the meantime.

Well, not any longer. I've had enough and I think it's high time for a public shame campaign, as maybe QNAP will finally bother to acknowledge these bugs and finally inform their users of any actions on the issues. More importantly, they need to provide a fix ASAP as right now, we and other users have unusable units due these bugs. For me personally, every single QNAP product I have, both NAS and NMP units, doesn't work as advertised - and that's completely unacceptable.

Please, if you have an issue, please post or link back to this thread: It will hopefully spur QNAP to take some sensible action and take note of their customer base, instead of pretending that the problems (and maybe their customers) will quietly go away.

Mario.

[TonyPh12345]

Yeah, that'll be helpful... Since they're using SAMBA GPL, have you looked for that issue in Bugzilla?

[SuperMario]

What? I am not their developer, I am not their beta tester. I am their customer and as such it's *THEIR* problem to trouble-shoot and to fix, not mine - nor is it the responsibility of any customer of theirs to fix, either.

As customers, we have in good faith paid for products and it is their updates that have broken the functionality of said products. You seem by your comments as if are trying to pass the buck onto either the open source community and/or QNAP's customers? It was QNAP that took our money for products that they produced, products that don't work, so it's QNAP's problem to fix - not their customers.

Mario.

[AdrianW]

Maybe this is the same issue as I'm having (see: Copying files from windows pauses entire NAS for 20 seconds).

Although my TS-859 is less than 50% full, it does contain 100's of thousands of jpeg images as well as thousands of video files. Any time I start a copy to the NAS, the thing will stall and sit there for ages, eventually the copy will start - but it's too late as it's already interrupted other people in the house trying to stream video from it.

I'd really like a fix for this, as it's very annoying not being able to update data on the NAS until no-one else is using it.

[SuperMario]

AdrianW, yes this is the same issue.

Sadly, from my experience here, I can tell you that it will only get worse as more files and directories are added. Our NAS units have become completely unusable due to this. They now stall for over 60 seconds and it is impossible to use them for their intended purposes. They are *supposed* to be the backup units for various parts of my network; yet for the last 3 months they're useless lumps draining power, much like the NMP-1000 that I bought wanting to watch Youtube videos on... and that is the reason I am upset, because of the way QNAP are handling that issue (ie: they're not) I fear that the exact same thing will happen with the NAS units now.

Seeing as QNAP is quite proud of displaying all these various "Award" logos and links to favorable reviews on their main page, I have started to contact many of the news and review websites I frequent and I suggest to others do the same. I think QNAP's attitude and response up until now is unforgivable and allowing them free reign to continue to rip off other unsuspecting consumers, whilst treating their existing user base as they do, shouldn't be permitted to continue any longer than it already has.

Hopefully doing so will give them some form of incentive to actually fix the issues and treat their paying customers with some respect; instead of letting them continue to sell unfit for purpose products in silence.

Mario.

[TonyPh12345]

What? I am not their developer, I am not their beta tester. I am their customer and as such it's *THEIR* problem to trouble-shoot and to fix, not mine - nor is it the responsibility of any customer of theirs to fix, either.
Mario.

That's true. But you can be proactive/helpful in coming to a solution, or you can hold your breath until you pass out. Your choice!

[QNAPJason]

Hi Mario,
Thanks for raising the issues up.
We have found your support case and are now tracing it.
currently we're trying to replicate your issue here (NAS stalling when NAS is >75% full or with large files in RAID 5 while copying files via Samba).
Either Alex or I will update the result as soon as we find clue.

Regarding the NMP YouTube playback issue, we have been using the correct YouTube API (http://code.google.com/apis/youtube/overview.html) since March.
However, it still gets disconnected for unknown reason. This is one area we still need to overcome or find alternative solution/provider for Youtube videos.

Jason

[schumaku]

Jason, The issue is not caused by the storage usage level of >75% ... muhc more likely by a large number of files.

[tmt]

For clarification, the issue exhibits itself as a stall on the QNAP as the file system is waiting for a semaphore that doesn't occur when an attempt to create a directory is being made. This causes timeouts on the host, as the SAMBA guest process blocks until the host times out.

Sounds like a great analysis, can you provide the details here? What semaphore, exactly? Also, what filesystem are you using on the NAS, ext3 or ext4?

[AdrianW]

Well, I don't know if this helps - but in my case I have 8 x 2TB drives in RAID 6 as a single EXT4 volume.

The volume is just a touch less than 50% full. I have 583,604 files in the user shares, over 500,000 of them are smallish jpg and png files, most of the rest are video files.

[AlexKe]

@ AdrianW

It does help for us to identify the issue. Thanks for your information.
In our experience, the file system might contain some incorrect info when you encounter the I/O error or slow performance.
It will be advised to run the "check file system", as it can fix some file system issue by the checking process. However, the Disk Volume will become unmounted and data cannot be accessed when the check file system procedure is running, which might take a few hours or days.

@ SuperMario

the issue exhibits itself as a stall on the QNAP as the file system is waiting for a semaphore that doesn't occur when an attempt to create a directory is being made. This causes timeouts on the host, as the SAMBA guest process blocks until the host times out.

- when the main volume gets to above ~75% and there are several million files on the unit. (SuperMario)

We are trying to reproduce the issue by filling the disk volume with instant allocation LUN data, but not success. We are not sure if the root cause is caused by the used percentage of disk volume or not.

- The volume is just a touch less than 50% full. I have 583,604 files in the user shares, over 500,000 of them are smallish jpg and png files. (AdrianW)

Could you provide the info as AdrianW mentioned, about how many files/folders in the user shares? Do you use EXT3 as your file system? Firmware 3.4.2 ? RAID 5 ? Any installed QPKG on the NAS? If the symptom can be reproduced easily by specific steps? Do you use the same disks on TS-639 and TS-859 Pro+ for testing when both units get the same problem? For example, every time when you create a share folder on the NAS via SAMBA/FTP/WFM... you will encounter the no response issue and the no response will remain for how long?

It will be also helpful if you collect these log via SSH connection to NAS and post here for us. Thanks
# df
# demsg
# mount
# cat /proc/mdstat

[AdrianW]

It will be also helpful if you collect these log via SSH connection to NAS and post here for us. Thanks
# df
# demsg
# mount
# cat /proc/mdstat

I previously posted those details for my NAS here, but shortly after that the thread died.

I have 3.4.3 firmware on mine, but have had the same issue for at least the previous two releases.

For me the pause problem doesn't happen every time I copy data to the NAS, but still quite regularly. Usually it will happen when nothing has been copied to the NAS for a while (at least a few minutes) and the file being copied is quite large (say 100MB or larger). After the pause has ended and I've copied a file, I can usually initiate a few more copies without the pause. But then waiting a couple of minutes (sometimes as little as a few seconds) and starting another copy the pause will occur again.

On a previous version of the firmware (3.3.4) (as I noted here), when the pause was happening there was a flush-9:0 process using around 20% of the CPU. But, since moving to 3.4.3 I haven't seen that process showing up when running top.

[SuperMario]

I have posted other logs and information in the private email thread that's been running for months between myself and a bunch of people from tech support - but I will post the information here for clarity.

Here is the information as requested. The first is from one of the TS-859Pro+ and second from one of the TS-639Pro units.

The units are configured solely as SAMBA file servers only; nothing else is enabled/running on them.

****************************************
QNAP-859Pro+ FW: 3.4.3 0520T

157694 files in 15255 directories
7.34 TB (8,078,452,877,158 bytes)

Using 8x 1.5TB drives in RAID5 as single volume EXT3


[~] # df
Filesystem Size Used Available Use% Mounted on
/dev/ramdisk 139.5M 116.5M 22.9M 84% /
tmpfs 32.0M 124.0k 31.9M 0% /tmp
/dev/sda4 310.0M 166.5M 143.4M 54% /mnt/ext
/dev/md9 509.5M 48.3M 461.2M 9% /mnt/HDA_ROOT
/dev/md0 9.4T 7.4T 2.0T 78% /share/MD0_DATA
tmpfs 32.0M 0 32.0M 0% /.eaccelerator.tmp

[~] # demsg
-sh: demsg: command not found

[~] # mount
/proc on /proc type proc (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
sysfs on /sys type sysfs (rw)
tmpfs on /tmp type tmpfs (rw,size=32M)
none on /proc/bus/usb type usbfs (rw)
/dev/sda4 on /mnt/ext type ext3 (rw)
/dev/md9 on /mnt/HDA_ROOT type ext3 (rw)
/dev/md0 on /share/MD0_DATA type ext3 (rw,usrjquota=aquota.user,jqfmt=vfsv0,user_xattr,data=ordered,noacl)
tmpfs on /.eaccelerator.tmp type tmpfs (rw,size=32M)
none on /sys/kernel/config type configfs (rw)

[~] # cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath]
md0 : active raid5 sda3[0] sdg3[7] sdh3[6] sdf3[5] sde3[4] sdd3[3] sdc3[2] sdb3[1]
10244987200 blocks level 5, 64k chunk, algorithm 2 [8/8] [UUUUUUUU]

md8 : active raid1 sdh2[2](S) sdg2[3](S) sdf2[4](S) sde2[5](S) sdd2[6](S) sdc2[7](S) sdb2[1] sda2[0]
530048 blocks [2/2] [UU]

md13 : active raid1 sda4[0] sdh4[7] sdg4[6] sdf4[5] sde4[4] sdd4[3] sdc4[2] sdb4[1]
458880 blocks [8/8] [UUUUUUUU]
bitmap: 0/57 pages [0KB], 4KB chunk

md9 : active raid1 sda1[0] sdh1[7] sdg1[6] sdf1[5] sde1[4] sdd1[3] sdc1[2] sdb1[1]
530048 blocks [8/8] [UUUUUUUU]
bitmap: 0/65 pages [0KB], 4KB chunk

unused devices: <none>

****************************************
QNAP-639Pro FW: 3.4.3 0520T

1676972 files in 111022 directories
8.01 TB (8,808,477,617,775 bytes)

Using 6x 2TB drives in RAID5 as single volume EXT3


[~] # df
Filesystem Size Used Available Use% Mounted on
/dev/ramdisk 139.5M 112.0M 27.5M 80% /
tmpfs 32.0M 108.0k 31.9M 0% /tmp
/dev/sda4 310.0M 129.4M 180.5M 42% /mnt/ext
/dev/md9 509.5M 76.4M 433.1M 15% /mnt/HDA_ROOT
/dev/md0 8.9T 8.0T 944.1G 0% /share/MD0_DATA
tmpfs 32.0M 0 32.0M 0% /.eaccelerator.tmp

[~] # demsg
-sh: demsg: command not found

[~] # mount
/proc on /proc type proc (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
sysfs on /sys type sysfs (rw)
tmpfs on /tmp type tmpfs (rw,size=32M)
none on /proc/bus/usb type usbfs (rw)
/dev/sda4 on /mnt/ext type ext3 (rw)
/dev/md9 on /mnt/HDA_ROOT type ext3 (rw)
/dev/md0 on /share/MD0_DATA type ext3 (rw,usrjquota=aquota.user,jqfmt=vfsv0,user_xattr,data=ordered,noacl)
tmpfs on /.eaccelerator.tmp type tmpfs (rw,size=32M)
none on /sys/kernel/config type configfs (rw)

[~] # cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath]
md0 : active raid5 sda3[0] sdf3[5] sde3[4] sdd3[3] sdc3[2] sdb3[1]
9759728000 blocks level 5, 64k chunk, algorithm 2 [6/6] [UUUUUU]

md6 : active raid1 sdf2[2](S) sde2[3](S) sdd2[4](S) sdc2[5](S) sdb2[1] sda2[0]
530048 blocks [2/2] [UU]

md13 : active raid1 sda4[0] sdf4[5] sde4[4] sdd4[3] sdc4[2] sdb4[1]
458880 blocks [6/6] [UUUUUU]
bitmap: 0/57 pages [0KB], 4KB chunk

md9 : active raid1 sda1[0] sdf1[5] sde1[4] sdd1[3] sdc1[2] sdb1[1]
530048 blocks [6/6] [UUUUUU]
bitmap: 1/65 pages [4KB], 4KB chunk

unused devices: <none>


I agree with some of the posters above in that the issue could appear to be more related to the number of files and directories, rather than to actual volume size. As both are related it's difficult to tell which is really is. Although as more new files and directories are added to the volume, the problem is exacerbated and it then starts to exhibit itself on the creation of files, not only creation of directories. Although at the moment file creation causes the issue much less often.

As things stand with both units in the state they are in now (the logs of which are posted above) they will constantly cause the ~60 second timeout whenever an attempt is made to create a directory. Less often the same issue is starting to arise during attempts to create a new file. Both of the the problems are most definitely getting worse as the volume size and/or file/directory count grows, though.

Mario.

[onlyalex]

Might be irrelative but what type of harddrives are you using and rev / firmware version ?

Cheers.