Speed up dir listing or best m.2 SATA SSD for Cache Acceleration?
-
- Starting out
- Posts: 12
- Joined: Fri Nov 23, 2018 11:56 pm
Speed up dir listing or best m.2 SATA SSD for Cache Acceleration?
I have a TS-1635AX configured with 7+2+1 (RAID 6+Hot Spare) WD Gold drives, 4 Samsung 860 Pro SSD in RAID 10 (4 TB capacity). These are part of a storage pool using qtier. I use dual 10GbE. Drives are used via NFS.
I have found that when I need to iterate over the files in my system with the linux find command (I have many small files) -- the results take longer than I would like.
On our old SAN, such a search might take 2 minute. With the qnap, it takes more than 20 minutes. The network isn't saturated by this operation.
I have tried the noatime setting on the NFS client but this didn't help.
Is this just a consequence of NFS instead of iSCSI? I would be interested in any opinions on the matter, but NFS is critical for my application.
My next idea: Adding SSD Read-Only Cache might help with the directory reads for files housed on the HDDs.
Along these lines, I have some questions that I think the community might be able to help with:
#1: Will SSD cache also cache my existing SSD RAID Array?
- Presumably caching SSD confers no benefit, and because there would be fewer drives, a potential 50% performance penalty.
#2: Can the cache be set up only for Random, not sequential, IO?
- Sequential performance is already "okay" and I am not seeking to make any big gains here.
#3: I consider to purchase 2 SSDs for the Read-only Cache. Should Raid1 (my default) or RAID0 be used?
#4: What m.2 SATA SSD model is recommended? Considerations:
- Might be most intensively written SSD in the system.
- Intention is to increase speed of directory listings: has to be fast. Do larger capacities with more chips offer greater speed?
- Not looking for capacity particularly. If smaller sizes work and I can save a buck -- then I am happy.
- The Samsung 860 EVO seems like a good compromise in the m.2 form factor, but what about endurance? PM871a -- higher endurance but slower -- do I want slower in a cache?
#5: If a drive in an SSD cache configuration fails how does the QNAP behave? Will it keep working ignoring the failed drive? Will I have to reboot? Anyone experience this?
Thanks!
Aaron
P.S. I also have off-site backup, I know: RAID != Backup.
I have found that when I need to iterate over the files in my system with the linux find command (I have many small files) -- the results take longer than I would like.
On our old SAN, such a search might take 2 minute. With the qnap, it takes more than 20 minutes. The network isn't saturated by this operation.
I have tried the noatime setting on the NFS client but this didn't help.
Is this just a consequence of NFS instead of iSCSI? I would be interested in any opinions on the matter, but NFS is critical for my application.
My next idea: Adding SSD Read-Only Cache might help with the directory reads for files housed on the HDDs.
Along these lines, I have some questions that I think the community might be able to help with:
#1: Will SSD cache also cache my existing SSD RAID Array?
- Presumably caching SSD confers no benefit, and because there would be fewer drives, a potential 50% performance penalty.
#2: Can the cache be set up only for Random, not sequential, IO?
- Sequential performance is already "okay" and I am not seeking to make any big gains here.
#3: I consider to purchase 2 SSDs for the Read-only Cache. Should Raid1 (my default) or RAID0 be used?
#4: What m.2 SATA SSD model is recommended? Considerations:
- Might be most intensively written SSD in the system.
- Intention is to increase speed of directory listings: has to be fast. Do larger capacities with more chips offer greater speed?
- Not looking for capacity particularly. If smaller sizes work and I can save a buck -- then I am happy.
- The Samsung 860 EVO seems like a good compromise in the m.2 form factor, but what about endurance? PM871a -- higher endurance but slower -- do I want slower in a cache?
#5: If a drive in an SSD cache configuration fails how does the QNAP behave? Will it keep working ignoring the failed drive? Will I have to reboot? Anyone experience this?
Thanks!
Aaron
P.S. I also have off-site backup, I know: RAID != Backup.
- dolbyman
- Guru
- Posts: 35272
- Joined: Sat Feb 12, 2011 2:11 am
- Location: Vancouver BC , Canada
Re: Speed up dir listing or best m.2 SATA SSD for Cache Acceleration?
how is the CPU usage during all of this ?.. Those ARM units are not very fast
Box Zelin had disappointing results with a first gen x31 ARM (annapurna)
viewtopic.php?t=142194#p676730
Can't even find any speed stats for the TS-1635AX on QNAP's page
Box Zelin had disappointing results with a first gen x31 ARM (annapurna)
viewtopic.php?t=142194#p676730
Can't even find any speed stats for the TS-1635AX on QNAP's page
- storageman
- Ask me anything
- Posts: 5506
- Joined: Thu Sep 22, 2011 10:57 pm
Re: Speed up dir listing or best m.2 SATA SSD for Cache Acceleration?
How slow is Qsirch?
If SSH "find" is slow GUI searches unlikely to be faster.
Seems like maybe you should have picked a faster box.
If you're talking millions of files ISCSI will search faster on this type of box.
If SSH "find" is slow GUI searches unlikely to be faster.
Seems like maybe you should have picked a faster box.
If you're talking millions of files ISCSI will search faster on this type of box.
Last edited by storageman on Sat Nov 24, 2018 1:11 am, edited 2 times in total.
-
- Starting out
- Posts: 12
- Joined: Fri Nov 23, 2018 11:56 pm
Re: Speed up dir listing or best m.2 SATA SSD for Cache Acceleration?
Looking at Resource Monitor during the find -- I see system processes hovering around 37%.
In top on the command line, I see about 50% idle during the find.
Find running or not doesn't seem to impact the idle number shown in top.
Aaron
In top on the command line, I see about 50% idle during the find.
Find running or not doesn't seem to impact the idle number shown in top.
Aaron
-
- Starting out
- Posts: 12
- Joined: Fri Nov 23, 2018 11:56 pm
Re: Speed up dir listing or best m.2 SATA SSD for Cache Acceleration?
I haven't tested Qsirch yet.
Doing a search in FileStation 5-- I am looking for all files under a certain mount point owned by myself -- takes a few minutes. Utilization goes up, top on the qnap shows 5.1% to 8.7% idle. I started this query just now (12:11) and it finished by (12:15) which seems reasonable.
Aaron
Doing a search in FileStation 5-- I am looking for all files under a certain mount point owned by myself -- takes a few minutes. Utilization goes up, top on the qnap shows 5.1% to 8.7% idle. I started this query just now (12:11) and it finished by (12:15) which seems reasonable.
Aaron
-
- Starting out
- Posts: 12
- Joined: Fri Nov 23, 2018 11:56 pm
Re: Speed up dir listing or best m.2 SATA SSD for Cache Acceleration?
I should point out that some of my directories contain about 250,000 files -- from 8k to 100k in size each.
-
- Starting out
- Posts: 12
- Joined: Fri Nov 23, 2018 11:56 pm
-
- Starting out
- Posts: 12
- Joined: Fri Nov 23, 2018 11:56 pm
Re: Speed up dir listing or best m.2 SATA SSD for Cache Acceleration?
Another question:
#6: Is there a way to adjust the amount of "equilibrium" free space on the qtier SSDs? (My utilization seems low to me.)
Aaron
#6: Is there a way to adjust the amount of "equilibrium" free space on the qtier SSDs? (My utilization seems low to me.)
Aaron
-
- Starting out
- Posts: 12
- Joined: Fri Nov 23, 2018 11:56 pm
Re: Speed up dir listing or best m.2 SATA SSD for Cache Acceleration?
To speed up the copy of a particularly large directory, I ssh'd into the box and ran cp -R from/. to/ -- this seemed to cause my NAS to become unresponsive (no web nor ssh) -- only ping-able. I powered down and upon reboot my qtier's Samsung Pro SSDs needed to be resync'd.
I have no idea if the cp command caused the failure, or if the disks spontaneously needed to be resync'd, or if there was another background job, e.g., qBoost or qTier, that ruined thing. Maybe "cp /share/directory" doesn't play nicely with qtier...
I am not a happy camper.
I have no idea if the cp command caused the failure, or if the disks spontaneously needed to be resync'd, or if there was another background job, e.g., qBoost or qTier, that ruined thing. Maybe "cp /share/directory" doesn't play nicely with qtier...
I am not a happy camper.
- Trexx
- Ask me anything
- Posts: 5388
- Joined: Sat Oct 01, 2011 7:50 am
- Location: Minnesota
Re: Speed up dir listing or best m.2 SATA SSD for Cache Acceleration?
I have QTier enabled, and have done CP commands before without issue. Could be due to the more limited CPU resources of the Arm boxes.
QTier you can do some "tuning" via the option at the shared folder level with ("Enable Auto-Tiering").
QTier you can do some "tuning" via the option at the shared folder level with ("Enable Auto-Tiering").
Paul
Model: TS-877-1600 FW: 4.5.3.x
QTS (SSD): [RAID-1] 2 x 1TB WD Blue m.2's
Data (HDD): [RAID-5] 6 x 3TB HGST DeskStar
VMs (SSD): [RAID-1] 2 x1TB SK Hynix Gold
Ext. (HDD): TR-004 [Raid-5] 4 x 4TB HGST Ultastor
RAM: Kingston HyperX Fury 64GB DDR4-2666
UPS: CP AVR1350
Model:TVS-673 32GB & TS-228a Offline[/color]
-----------------------------------------------------------------------------------------------------------------------------------------
2018 Plex NAS Compatibility Guide | QNAP Plex FAQ | Moogle's QNAP Faq
Model: TS-877-1600 FW: 4.5.3.x
QTS (SSD): [RAID-1] 2 x 1TB WD Blue m.2's
Data (HDD): [RAID-5] 6 x 3TB HGST DeskStar
VMs (SSD): [RAID-1] 2 x1TB SK Hynix Gold
Ext. (HDD): TR-004 [Raid-5] 4 x 4TB HGST Ultastor
RAM: Kingston HyperX Fury 64GB DDR4-2666
UPS: CP AVR1350
Model:TVS-673 32GB & TS-228a Offline[/color]
-----------------------------------------------------------------------------------------------------------------------------------------
2018 Plex NAS Compatibility Guide | QNAP Plex FAQ | Moogle's QNAP Faq
-
- Starting out
- Posts: 12
- Joined: Fri Nov 23, 2018 11:56 pm
Re: Speed up dir listing or best m.2 SATA SSD for Cache Acceleration?
An update on this:
- I installed 2x Samsung EVO 860 m.2 in RAID 1 as a RW cache. I think this has improved throughput. In resource monitor I can see r/w speeds around 50MB/s typical to CPU, peak to CPU around 150MB/s and peak during backup around 300MB/s. Still very curious/anxious what will be the failure mode of this cache.
- I switched from NFS to iSCSI for mounts with huge number of files. This helped enormously. The find command which took minutes now takes seconds. Sadly, these now iSCSI-mounted files were intended to be shared across several boxes -- so my client will have to re-export, which seems like it will invite the same issue. I am unsure if I want to share the iSCSI target across Debian machines (e.g., one RW and others RO). Anyone do this?
- I suspect the lockup issue experienced during the copy was due to concurrent local (over ssh) and NFS access to the same files.
- I have also switched from using NFSv4 to NFSv3 with the noatime,nodiratime,nolock,noacl,udp options. This has eliminated the huge number of NFSv4 DELEG locks that were building up on the QNAP. I suspect this DELEG lock accumulation may have contributed to the issue during cp.
- I installed 2x Samsung EVO 860 m.2 in RAID 1 as a RW cache. I think this has improved throughput. In resource monitor I can see r/w speeds around 50MB/s typical to CPU, peak to CPU around 150MB/s and peak during backup around 300MB/s. Still very curious/anxious what will be the failure mode of this cache.
- I switched from NFS to iSCSI for mounts with huge number of files. This helped enormously. The find command which took minutes now takes seconds. Sadly, these now iSCSI-mounted files were intended to be shared across several boxes -- so my client will have to re-export, which seems like it will invite the same issue. I am unsure if I want to share the iSCSI target across Debian machines (e.g., one RW and others RO). Anyone do this?
- I suspect the lockup issue experienced during the copy was due to concurrent local (over ssh) and NFS access to the same files.
- I have also switched from using NFSv4 to NFSv3 with the noatime,nodiratime,nolock,noacl,udp options. This has eliminated the huge number of NFSv4 DELEG locks that were building up on the QNAP. I suspect this DELEG lock accumulation may have contributed to the issue during cp.
-
- Starting out
- Posts: 12
- Joined: Fri Nov 23, 2018 11:56 pm
Re: Speed up dir listing or best m.2 SATA SSD for Cache Acceleration?
I think this may be a factor as well, particularly when dealing with lots of small files.
I had a look here:
https://openbenchmarking.org/result/180 ... -180506462
The link suggests to me that the 1635AX performs about 25% the speed of an i7-7700k; a CPU mark of about 3,000.