QNAP firmware issues : Major file system bug

[heroheroz]

BTW, not necessary to post it in an IT related forum, put that into the joker's forum as well.
7 months for setting up test environment, really funny to me, and I am still waiting LOL

[BloodShed]

What I'm thinking here is that some of us are transferring their frustration of running out of disk space on a fauly NAS software (and again, it IS annoying, it shouldn't take so long, ... - I agree with you guys), since we run out of space sooner than we expected. When choosing my new NAS, I compared with the competition - correcting for the fact that with QNAP I will only be able to use 90% of my disk space - and still QNAP is the best deal.

I want to believe that you're baiting a response. This is the most ridiculous argument I've yet to see in this thread.

First, I have no frustration with running out of disk space. Frankly, it's inconsequential considering the defective QNAP software doesn't allow someone to achieve the capacity that was paid for. I purchased 9TB of capacity for a reason; not for QNAP to indirectly regulate my usage.

Secondly, I can't imagine what mysterious statistical data could lead you to an absurd claim of "QNAP is the best deal". As an unfortunate owner of the TS-659 Pro, I can assure you that it does not ship with hard drives. Obviously, the opinionated "value" of QNAP hardware cannot be quantified through usable storage capacity because you have no basis for measuring the total cost.

I should also add that 90% is not even accurate for your unfounded statistics. There is no specific percentage at which the problem occurs. Based on the evidence throughout this thread, it could be related to file fragmentation, but that would be an assumption. Furthermore, the loss of usable storage (and therefore the loss of value) increases exponentially with larger NAS units. You’ll notice that the original poster has five QNAP NAS units including a couple of the TS-859 Pro+. You should also have noticed that the originally reported usage percentage at which failure occurs is 75%; not 90%. I suspect you've done little more than skim through the posts in this thread as evidenced by your ignorance of the situation.

Frankly, if you’re satisfied with your purchase of a faulty product with substandard communication and support from the manufacturer, I urge you to move on from this thread. You can sing your song of admiration for QNAP somewhere else. You will not succeed in defusing the mess that QNAP has created.

When the problem occurs, it certainly causes loss of performance and functionality. But I've yet find a reasonable explanation on how it can cause data corruption or loss. It seems to be something users post to scare QNAP.

So it seems you've successfully discovered the specific cause of this problem. I believe everyone here, including QNAP, would love to know the results of your findings. I'm pleased to hear that you've confirmed there is no risk of file, file system, or journal corruption due to write delays.

I suggest to people who suffer from the same problem simply find a few review forums and put the fact there, the only way to make them care is let their potential customer know about it and make qnap lose their business.

Absolutely agree!

[2000wolf]

... You should also have noticed that the originally reported usage percentage at which failure occurs is 75%; not 90%. ...

I think I' will hit that 75 to 90% limitation in 2 to 3 years. So for me that issue is more academic ...
Nevertheless I'd like to get a sort of tested confirmation that the Synology NAS don't suffer from a similar bug with the same disk usage!

[maeldiyan]

First it was a failed mainboard. Alright, that stuff happens and it was (thankfully) replaced under warranty. Then came the surprise that the "certified" 2Tb drives were no longer "certified." Umm, what's the point of having a drive certification program if your certifications don't mean anything? 5x$140 = $700 of "incompatible" drives, super.....

Now there's the data transfer issue. I'm at ~3Tb free out of an ~8Tb RAID5 setup and my transfer rates are in the 6Mbps-9Mbps range, when they used to be 30Mbps+. It's apparently a firmware issue (I have NO data x-fer issues within the rest of my environment) that QNAP has been unwilling to address for over a year now. Effectively, I have an $800+ paperweight because QNAP can't get around to finding a fix for this issue. Completely unacceptable!

For a company that aspires to be used in large corporate IT environments (they were hawking their wares at Citrix Synergy next to the likes of EMC and NetApp), it's comical that they are ignoring a critical bug in their core business. Perhaps QNAP doesn't care about the little guy, but, I'm always more than happy to share my own QNAP experiences with everyone I encounter.

[grocovid]

Come on, BloodShed, I'm not interested in name-calling, and I won't call your arguments "ridiculous". Please review this thread carefully, and you will see that I'm one of those people who posted repeated observations, possible hypotheses (including fragmentation issues), new remarks (like the fact that when a 700 MB file is deleted under the "freezing" condition and you try to reuse that same place immediately afterwards, this space can be reused without any freeze). And I think I know this bug, I've been using my NAS in this terrible condition for months, copying files at 50 KB/sec on a gigabit line, without any corruption, ever (ext4, no write cache).

Saying that I did not read this thread carefully is dishonest; You may disagree with me. But I've yet to find evidence of data loss caused by this issue.

I'm not saying there is no problem. I'm saying that, in MY experience (and, again, there may be different configurations, and differing severities for this bug), it happens around 90-95% (for 6 - 9 TB total capacity) and is a pain in the ** but doesn't cause data loss. In MY experience, freeing up disk space makes it more easy to reuse than the first time it was used (strange, but true and reproducible).

Moreover, I've been carefully comparing prices in my country, for equivalents to the TS-412, and even removing 25% capacity, Synos are more expensive.

This kind of argument gets us nowhere. What I'm just saying is that such issues may arise in any product, it's the lack of timely fix and even more seriously the lack of communication from QNAP that's the biggest problem, in my opinion.

And, no, I'm not moving on from this thread. Although I'm mostly satisfied with the product, I'm not satisfied at all with QNAP support and communication.

Grocovid

[BloodShed]

Come on, BloodShed, I'm not interested in name-calling

Name calling? I should hope not, since I certainly haven't participated.

Please review this thread carefully, and you will see that I'm one of those people who posted repeated observations, possible hypotheses (including fragmentation issues), new remarks (like the fact that when a 700 MB file is deleted under the "freezing" condition and you try to reuse that same place immediately afterwards, this space can be reused without any freeze).

Your first contribution to this thread was on January 31 of this year. At that point, this topic already had 21 pages of replies. The theory of file fragmentation, the fact that deleting files temporarily corrects the problem, and even the fact that disk space is not a direct factor were already discussed well before then. You'd have realized this if you actually read the thread.

And I think I know this bug, I've been using my NAS in this terrible condition for months, copying files at 50 KB/sec on a gigabit line, without any corruption, ever (ext4, no write cache).

I honestly am happy that you have not personally experienced data loss.

I've been dealing with this issue on my hardware for over a year. Again, if you had followed the thread, supermario had reported data loss as a result of this issue. Was it related? I can't say for sure. I know that I'm not willing to risk 7TB of my data to find out. I also know that it's completely reasonable to suspect that write delays certainly can cause corruption. So, you'll forgive me if your self-proclaimed expertise is not reassuring enough.

Saying that I did not read this thread carefully is dishonest;

Dishonest suggests that I already knew you read this thread when I said you didn't. That makes no sense. I am claiming you have not as evidenced by the content of your posts. Your reply continues to support this suspicion.

You may disagree with me. But I've yet to find evidence of data loss caused by this issue.

What you mean to say is, you have not personally experienced data loss. This fact means nothing.

I'm saying that, in MY experience (and, again, there may be different configurations, and differing severities for this bug), it happens around 90-95% (for 6 - 9 TB total capacity) and is a pain in the ** but doesn't cause data loss. In MY experience, freeing up disk space makes it more easy to reuse than the first time it was used (strange, but true and reproducible).

Exactly my point. This is your experience, while using your specific hardware, with a specific configuration, in a specific way. The fallacy in your logic was suggesting your experience somehow translates to everyone else. You advocated that there is no danger of data loss and even implied other victims were liars when you said:

It seems to be something users post to scare QNAP.

Moreover, I've been carefully comparing prices in my country, for equivalents to the TS-412, and even removing 25% capacity, Synos are more expensive.

I don't think you understood the point I was making. Comparing prices in your specific country, for your specific hardware does not directly correlate to anyone else. It's no secret that QNAP supplies multiple products worldwide. More importantly, you can not remove "25% capacity" on the value of a device that does not ship with hard drives. Last, it doesn't matter if you want to apply a loss of 10%, 25%, or whatever number you want to switch your figures to. There is no percentage at which this problem occurs. Therefore, your claim that "QNAP is the best deal" is invalid and absurd.


I agree with the rest of your post and, frankly, I'm done arguing about this. Especially since I am basically repeating myself. My problem is that I don't appreciate someone jumping into the middle of this exhausting discussion with misinformation and praise for QNAP. As I've said before, my patience is at its limit on this.

QNAP has shown no real evidence that they have done any work on this issue or that they even care to prioritize it. Because you skipped the majority of this thread, I'll catch you up... QNAP has already claimed (in this very thread) that they had recreated the problem and they're working on a fix, only to come back later to clarify that they never actually recreated it at all. So I've quickly learned that, until I see a fix, I can't trust a single thing QNAP says.

[Flerbizky]

Never buying another QNap product. ** because I like them, but this is unacceptable..

BR,
Steffen - Left with an empty partition table :-/

[P3R]

Steffen - Left with an empty partition table :-/

But most likely that is unrelated to the bug in this thread.

As I've already answered in your own thread, I'd recommend that you contact Qnap support for assistance.
'

[grocovid]

Again, if you had followed the thread, supermario had reported data loss as a result of this issue. Was it related? I can't say for sure.

I'm just saying that there is no evidence of data loss related to this issue, as you indicate yourself that the relationship isn't proved. And yes, it's reasonable to suspect that write delays can sometimes cause corruption, although I trust ext4 to be fairly reliable.

Moreover, I've been carefully comparing prices in my country, for equivalents to the TS-412, and even removing 25% capacity, Synos are more expensive.

I don't think you understood the point I was making. Comparing prices in your specific country, for your specific hardware does not directly correlate to anyone else. It's no secret that QNAP supplies multiple products worldwide. More importantly, you can not remove "25% capacity" on the value of a device that does not ship with hard drives. Last, it doesn't matter if you want to apply a loss of 10%, 25%, or whatever number you want to switch your figures to. There is no percentage at which this problem occurs. Therefore, your claim that "QNAP is the best deal" is invalid and absurd.

Indeed, my wording was not accurate. In my specific situation and in my specific country, assuming that I would remove 25% capacity from the hard drives i would buy, the total cost of a QNAP-based solution was still lower than the competition. "QNAP is the best deal" was an unacceptable shortcut, for which I apologize.

My problem is that I don't appreciate someone jumping into the middle of this exhausting discussion with misinformation and praise for QNAP. As I've said before, my patience is at its limit on this.

I've repeated over and over again, as have others in this thread, that QNAP's communication on this issue in unacceptable. But, again, I repeat my call for ext4 specialists who know more about fragmentation in this file system that I do. Maybe the answer lies somewhere there.

QNAP has shown no real evidence that they have done any work on this issue or that they even care to prioritize it. Because you skipped the majority of this thread, I'll catch you up... QNAP has already claimed (in this very thread) that they had recreated the problem and they're working on a fix, only to come back later to clarify that they never actually recreated it at all. So I've quickly learned that, until I see a fix, I can't trust a single thing QNAP says.

I do remember the last fix supposedly coming Q1 2012, and (some of) their inconsistent posts. I can't really believe that they were unable to reproduce the issue whereas it would be a simple matter if they cared to involve any of their dissatisfied customers here.

grocovid

[AlexKe]

Dear Folks,

The issue regarding potential performance drop when the system is about to be full (< 85%) has been continuously reproduced, verified, tested, and experimented in QNAP's lab. As you've also noted, providing a reliable system should be our only concern on designing a product.

Currently, we know the performance drop issue might be caused by the data fragment and it is a global issue. We can reproduce it on standard Linux platform and other NAS system. The Linux kernel upgrade or modification might help to improve the performance drop a little bit, but not a solution. We have been surveying some defragment tools, but they are not mature enough. We will continue on finding a solution to solve the issue, as this is one thing I'm positive is that we do put the verification into our high priority list.

/Alex
===============================
Edit: 2012/5/26

For the performance drop caused by short of RAM allocation, now we have a kernel patch which can solve the issue. The solution is under testing now and will have beta firmware soon.

[chrisqncl]

Dear Alex,

I think you are confusing issues here, please check with QNAPSimon. The problem here is not a "performance drop" but a crash due to out of memory e.g. when doing a simple "mkdir" in SSH terminal causing the system to reboot (at least on TS-410 with 256MB RAM), which indeed causes a MASSIVE performance drop.

And the problem does not occur with 3.1.2 firmware (I've switched beetween 3.1.2 and 3.4.* two times with same results: works fine with 3.1.2, problem 100% reproducable with 3.4.* doing a mkdir command) so I'm failing to see how disk space fragmentation can be the root cause.

Finally, it seems to have to do with the number of files in the file system and not so much with the usage degree.

QNAP feels no sense of urgency? Well, with the demonstrated quality of software engineering, QA and service, I will never ever buy a QNAP device again. I will never ever recommend a QNAP device to anyone. I'm stuck with 3.1.2 firmware unless I switch to Debian. I've switched my drives to "supported ones" (=> $). I've lost quite a few hours in diagnosing and looking for answers in the Internet and up-/downgrading the firmware. Thank you very much, but I'd rather done some meaningful work instead.

-chris

[sl1000]

Chris: That is really different issue, please open a seperate post on it.

i'm closing this one for now. It has become one big list of general issues which very often are _not_ related to the real issue discussed here.

People _please_ do your homework. A none performing nas is in 99,9% of the cases not related to the bug discussed here, but a network setup issue, a failing drive, hanging qpkgs that eat cpu cycles, antivirus scan, etc.

in general
- if your nas is new, and you just setup raid. wait until it is fully synced. this is a background process that can take upto 24 hrs. during the syncing time you'll see a lot less performance.
- check your network setup. disable all fancy loadbalancing features, and connect only one cable to check if it is a setup issue. if using jumboframes, reset it to 1500 and check if it solves the issue. (also check for errors on the network interfaces, you might have a dodgy cable, router, etc)
- check the cpu load when idle. an idle nas should hardly use any cpu cycles. if it is heavely used first fix that issue (disable the offending qpkg/feature and reboot the nas)
- do a smart test on each of your drives. check for bad blocks, etc
- check if the issue also exists on another machine. it might be host related ( a host based virusscanner, etc)
- check the systemlog for any errors.
- last but not least (re)load the latest firmware

oh for what it's worth: cleaning your network recycle bin (if enabled) is always a good idea to start with. you'll need to empty it manually, by pushing the button in the admin interface!

If after the above steps you still see issue's, make a new post with a clear discription, the nas used, the firmware used, and the steps you've undertaken to solve it.

[sl1000]

As QNAP has added some modifications in firmware 3.7.0/3.7.1 which should fix this issue, i've unlocked this thread again.

Everybody is invited to share their experience with these fixes.

[AdrianW]

As QNAP has added some modifications in firmware 3.7.0/3.7.1 which should fix this issue, i've unlocked this thread again.

Some interesting changes listed for that firmware, most notably:

Fixed the slow write speed when the system would take a long time to find free blocks with large amount of data stored on the NAS

Also, the Network Recycle Bin Automatic cleanup sounds very useful to me.

But, then there's this problem being experienced by multiple users: hard disk standby mode no longer works

So, do I upgrade now, or wait until the standby issue is addressed?

[schumaku]

@AdrianW: This thread was closed to avoid more and more unrelated posts - now re-opened to undestand if this latent performance loss is addresed fofr the users experienced it on their NAS installations. What is more esential: for you: Operational reliability or HDD hiberation?