Crash when writing more than 200GB in one batch

iSCSI related applications
mr-auh
Starting out
Posts: 17
Joined: Thu Nov 29, 2018 5:44 pm

Crash when writing more than 200GB in one batch

Post by mr-auh »

Hello,
we recently bought two TS-877-1700-16G (upgraded to 64GB RAM, Firmware 4.3.5.0728) in order to transfer data from customers to our datacenter.
However whenever we start copying files to the QNAP (manual copy with windows explorer or backupy copy job with Veeam) the cache usage starts to rise (screenshot attached) and when hitting 100% the QNAP crashes and reboots after a few minutes. When I cancel the copying the cache gets freed after a couple of seconds and I can copy files once again. This happens when using CIFS or iSCSI. Sadly the support has not answered my ticket since 7 Nov.

Best regards
Alexander
You do not have the required permissions to view the files attached to this post.
Last edited by mr-auh on Fri Nov 30, 2018 4:07 pm, edited 1 time in total.
User avatar
storageman
Ask me anything
Posts: 5507
Joined: Thu Sep 22, 2011 10:57 pm

Re: Crash when writing more than 200GB in one batch

Post by storageman »

Wow! 7th Nov is bad!

Would help if you itemised your pool and volume and disk setup.
I assume you have an SSD cache so for now I would turn it off.
Also who's RAM if upgraded?
User avatar
Trexx
Ask me anything
Posts: 5393
Joined: Sat Oct 01, 2011 7:50 am
Location: Minnesota

Re: Crash when writing more than 200GB in one batch

Post by Trexx »

I would also try updating to the latest firmware (4.3.5.0760) to see if that makes any difference as well as I believe there were some fixes for the 10Gb Aquinta cards in it.
Paul

Model: TS-877-1600 FW: 4.5.3.x
QTS (SSD): [RAID-1] 2 x 1TB WD Blue m.2's
Data (HDD): [RAID-5] 6 x 3TB HGST DeskStar
VMs (SSD): [RAID-1] 2 x1TB SK Hynix Gold
Ext. (HDD): TR-004 [Raid-5] 4 x 4TB HGST Ultastor
RAM: Kingston HyperX Fury 64GB DDR4-2666
UPS: CP AVR1350

Model:TVS-673 32GB & TS-228a Offline[/color]
-----------------------------------------------------------------------------------------------------------------------------------------
2018 Plex NAS Compatibility Guide | QNAP Plex FAQ | Moogle's QNAP Faq
mr-auh
Starting out
Posts: 17
Joined: Thu Nov 29, 2018 5:44 pm

Re: Crash when writing more than 200GB in one batch

Post by mr-auh »

storageman wrote: Thu Nov 29, 2018 9:07 pm Wow! 7th Nov is bad!

Would help if you itemised your pool and volume and disk setup.
I assume you have an SSD cache so for now I would turn it off.
Also who's RAM if upgraded?
Thank you for your reply. SSD cache is disabled. Disabling System Cache also did not help.
What do you mean with "Also who's RAM if upgraded?"?
Trexx wrote: Thu Nov 29, 2018 11:57 pm I would also try updating to the latest firmware (4.3.5.0760) to see if that makes any difference as well as I believe there were some fixes for the 10Gb Aquinta cards in it.
Thank you for your reply as well! When I ran the automatic firmware check yesterday it reported no new firmware available. When manually checking, I found the version you're referencing and installed it. Sadly the behavior did not change. Switching from the 10G NIC to an internal 1G NIC made no difference. If it is of any help, here is my configuration:

Code: Select all

6x 6TB HDD RAID 50
2x 1TB SSD RAID 1 for Qtier
Thick Volume
User avatar
storageman
Ask me anything
Posts: 5507
Joined: Thu Sep 22, 2011 10:57 pm

Re: Crash when writing more than 200GB in one batch

Post by storageman »

You could have said all this up front. I don't like drip drip drip of information.
Now you're saying it's Qtier with RAID 50 (which not many people use).
What drive models?
Are they on the compat list?
mr-auh
Starting out
Posts: 17
Joined: Thu Nov 29, 2018 5:44 pm

Re: Crash when writing more than 200GB in one batch

Post by mr-auh »

storageman wrote: Fri Nov 30, 2018 4:33 pm You could have said all this up front. I don't like drip drip drip of information.
Now you're saying it's Qtier with RAID 50 (which not many people use).
What drive models?
Are they on the compat list?
As this looks like a software problem / bug (memory leak) to me I did not think these details are very necessary. If you tell me what more info could be useful I am happy to provide it!
HDD: WD60EFRX (on compat list)
SSD: MZ-76P1T0BW (on compat list)
User avatar
storageman
Ask me anything
Posts: 5507
Joined: Thu Sep 22, 2011 10:57 pm

Re: Crash when writing more than 200GB in one batch

Post by storageman »

This is why I was asking about the RAM, is it all Qnap RAM?
mr-auh
Starting out
Posts: 17
Joined: Thu Nov 29, 2018 5:44 pm

Re: Crash when writing more than 200GB in one batch

Post by mr-auh »

storageman wrote: Fri Nov 30, 2018 4:43 pm This is why I was asking about the RAM, is it all Qnap RAM?
Thanks, didn't quite get your question.
RAM: 4x Crucial 16GB DDR4-2400 UDIMM (CT16G4DFD824A.M16FB)
User avatar
storageman
Ask me anything
Posts: 5507
Joined: Thu Sep 22, 2011 10:57 pm

Re: Crash when writing more than 200GB in one batch

Post by storageman »

So you know what I'm going to say...
Take the Crucial RAM out (although the Crucial model does look right), put the Qnap RAM back and test it again.

Seriously, you need to think what information is relevant if you want help. Otherwise talk to Qnap.,
mr-auh
Starting out
Posts: 17
Joined: Thu Nov 29, 2018 5:44 pm

Re: Crash when writing more than 200GB in one batch

Post by mr-auh »

storageman wrote: Fri Nov 30, 2018 4:55 pm So you know what I'm going to say...
Take the Crucial RAM out, put the Qnap RAM back and test it again.

Seriously, you need to think what information is relevant if you want help. Otherwise talk to Qnap.,
I would love to talk to QNAP, sadly they do not want to talk to me. I don't see how a memory leak could come from a certain make of RAM, as the system itself is running perfectly stable and also running a VM (which uses the RAM) does not make it crash. The crash only occurs when transferring data to the qnap, accessing the data or using the RAM for Apps / VMs does not crash it.
I will put back the original RAM (which is labelled to be from ADATA) and report back. Thank you for your help so far!
User avatar
storageman
Ask me anything
Posts: 5507
Joined: Thu Sep 22, 2011 10:57 pm

Re: Crash when writing more than 200GB in one batch

Post by storageman »

If you don't want to remove the RAM for now, run the RAM test in the Diagnostic Tool.
At least if you go back to the Qnap/Adata RAM and it still crashes you can rule out RAM as the issue.

My suspicion is the Samsung SSDs is the problem.
mr-auh
Starting out
Posts: 17
Joined: Thu Nov 29, 2018 5:44 pm

Re: Crash when writing more than 200GB in one batch

Post by mr-auh »

storageman wrote: Fri Nov 30, 2018 5:19 pm If you don't want to remove the RAM for now, run the RAM test in the Diagnostic Tool.
At least if you go back to the Qnap/Adata RAM and it still crashes you can rule out RAM as the issue.

My suspicion is the Samsung SSDs is the problem.
On my way to swap the RAM (even though I actually would need the 64GB to use it for its purpose), then I will report back. The Samsung SSDs should be fine as they are on the compat list? I sadly don't have any spare ones of another manufacturer on hand so I can't test that. After that I will disable Qtier and see if that fixes the problem, even though without Qtier the whole point of buying these quite expensive models is somewhat missed.
User avatar
storageman
Ask me anything
Posts: 5507
Joined: Thu Sep 22, 2011 10:57 pm

Re: Crash when writing more than 200GB in one batch

Post by storageman »

mr-auh wrote: Fri Nov 30, 2018 5:25 pm
storageman wrote: Fri Nov 30, 2018 5:19 pm If you don't want to remove the RAM for now, run the RAM test in the Diagnostic Tool.
At least if you go back to the Qnap/Adata RAM and it still crashes you can rule out RAM as the issue.

My suspicion is the Samsung SSDs is the problem.
On my way to swap the RAM (even though I actually would need the 64GB to use it for its purpose), then I will report back. The Samsung SSDs should be fine as they are on the compat list? I sadly don't have any spare ones of another manufacturer on hand so I can't test that. After that I will disable Qtier and see if that fixes the problem, even though without Qtier the whole point of buying these quite expensive models is somewhat missed.
Experience has taught me being on the compat list is no absolute guarantee. Don't be so sure. This is why you need to test each component in the build.
FYI Synology have removed 860 Pros from some high end Synology models due to crashing issues.
Micron 5200s are better drives.

Unless something has changed you cannot disable Qtier only start again.
mr-auh
Starting out
Posts: 17
Joined: Thu Nov 29, 2018 5:44 pm

Re: Crash when writing more than 200GB in one batch

Post by mr-auh »

storageman wrote: Fri Nov 30, 2018 5:35 pm
mr-auh wrote: Fri Nov 30, 2018 5:25 pm
storageman wrote: Fri Nov 30, 2018 5:19 pm If you don't want to remove the RAM for now, run the RAM test in the Diagnostic Tool.
At least if you go back to the Qnap/Adata RAM and it still crashes you can rule out RAM as the issue.

My suspicion is the Samsung SSDs is the problem.
On my way to swap the RAM (even though I actually would need the 64GB to use it for its purpose), then I will report back. The Samsung SSDs should be fine as they are on the compat list? I sadly don't have any spare ones of another manufacturer on hand so I can't test that. After that I will disable Qtier and see if that fixes the problem, even though without Qtier the whole point of buying these quite expensive models is somewhat missed.
Experience has taught me being on the compat list is no absolute guarantee. Don't be so sure. This is why you need to test each component in the build.
FYI Synology have removed 860 Pros from some high end Synology models due to crashing issues.
Micron 5200s are better drives.

Unless something has changed you cannot disable Qtier only start again.
Alright so swapping the RAM did not do the trick. I will now reconfigure the storage without Qtier, leaving out the Samsung 860 Pro SSDs. Will report back after the weekend when the RAID rebuild is done. Thank you all for your help so far!
User avatar
storageman
Ask me anything
Posts: 5507
Joined: Thu Sep 22, 2011 10:57 pm

Re: Crash when writing more than 200GB in one batch

Post by storageman »

Also not sure why you'd want to use Reds in a top end box like this.
3Yr warranty, No vibration control and 54-5900rpm.

Not the right drive in my view.
Locked

Return to “iSCSI – Target & Virtual Disk”