Speed up dir listing or best m.2 SATA SSD for Cache Acceleration?

Printers, HDDs, USB/eSATA drives, 3rd-party programs
Post Reply
aschweig
Starting out
Posts: 11
Joined: Fri Nov 23, 2018 11:56 pm

Speed up dir listing or best m.2 SATA SSD for Cache Acceleration?

Post by aschweig » Sat Nov 24, 2018 12:33 am

I have a TS-1635AX configured with 7+2+1 (RAID 6+Hot Spare) WD Gold drives, 4 Samsung 860 Pro SSD in RAID 10 (4 TB capacity). These are part of a storage pool using qtier. I use dual 10GbE. Drives are used via NFS.

I have found that when I need to iterate over the files in my system with the linux find command (I have many small files) -- the results take longer than I would like.
On our old SAN, such a search might take 2 minute. With the qnap, it takes more than 20 minutes. The network isn't saturated by this operation.

I have tried the noatime setting on the NFS client but this didn't help.

Is this just a consequence of NFS instead of iSCSI? I would be interested in any opinions on the matter, but NFS is critical for my application.

My next idea: Adding SSD Read-Only Cache might help with the directory reads for files housed on the HDDs.

Along these lines, I have some questions that I think the community might be able to help with:

#1: Will SSD cache also cache my existing SSD RAID Array?
- Presumably caching SSD confers no benefit, and because there would be fewer drives, a potential 50% performance penalty.

#2: Can the cache be set up only for Random, not sequential, IO?
- Sequential performance is already "okay" and I am not seeking to make any big gains here.

#3: I consider to purchase 2 SSDs for the Read-only Cache. Should Raid1 (my default) or RAID0 be used?

#4: What m.2 SATA SSD model is recommended? Considerations:
- Might be most intensively written SSD in the system.
- Intention is to increase speed of directory listings: has to be fast. Do larger capacities with more chips offer greater speed?
- Not looking for capacity particularly. If smaller sizes work and I can save a buck -- then I am happy.
- The Samsung 860 EVO seems like a good compromise in the m.2 form factor, but what about endurance? PM871a -- higher endurance but slower -- do I want slower in a cache?

#5: If a drive in an SSD cache configuration fails how does the QNAP behave? Will it keep working ignoring the failed drive? Will I have to reboot? Anyone experience this?

Thanks!

Aaron

P.S. I also have off-site backup, I know: RAID != Backup.

dolbyman
Guru
Posts: 11010
Joined: Sat Feb 12, 2011 2:11 am
Location: Vancouver BC , Canada

Re: Speed up dir listing or best m.2 SATA SSD for Cache Acceleration?

Post by dolbyman » Sat Nov 24, 2018 12:57 am

how is the CPU usage during all of this ?.. Those ARM units are not very fast

Box Zelin had disappointing results with a first gen x31 ARM (annapurna)
viewtopic.php?t=142194#p676730

Can't even find any speed stats for the TS-1635AX on QNAP's page

User avatar
storageman
Experience counts
Posts: 4030
Joined: Thu Sep 22, 2011 10:57 pm

Re: Speed up dir listing or best m.2 SATA SSD for Cache Acceleration?

Post by storageman » Sat Nov 24, 2018 1:07 am

How slow is Qsirch?
If SSH "find" is slow GUI searches unlikely to be faster.
Seems like maybe you should have picked a faster box.
If you're talking millions of files ISCSI will search faster on this type of box.
Last edited by storageman on Sat Nov 24, 2018 1:11 am, edited 2 times in total.

aschweig
Starting out
Posts: 11
Joined: Fri Nov 23, 2018 11:56 pm

Re: Speed up dir listing or best m.2 SATA SSD for Cache Acceleration?

Post by aschweig » Sat Nov 24, 2018 1:08 am

Looking at Resource Monitor during the find -- I see system processes hovering around 37%.
In top on the command line, I see about 50% idle during the find.
Find running or not doesn't seem to impact the idle number shown in top.

Aaron

aschweig
Starting out
Posts: 11
Joined: Fri Nov 23, 2018 11:56 pm

Re: Speed up dir listing or best m.2 SATA SSD for Cache Acceleration?

Post by aschweig » Sat Nov 24, 2018 1:16 am

I haven't tested Qsirch yet.

Doing a search in FileStation 5-- I am looking for all files under a certain mount point owned by myself -- takes a few minutes. Utilization goes up, top on the qnap shows 5.1% to 8.7% idle. I started this query just now (12:11) and it finished by (12:15) which seems reasonable.

Aaron

aschweig
Starting out
Posts: 11
Joined: Fri Nov 23, 2018 11:56 pm

Re: Speed up dir listing or best m.2 SATA SSD for Cache Acceleration?

Post by aschweig » Sat Nov 24, 2018 2:40 am

I should point out that some of my directories contain about 250,000 files -- from 8k to 100k in size each.

aschweig
Starting out
Posts: 11
Joined: Fri Nov 23, 2018 11:56 pm

Re: Speed up dir listing or best m.2 SATA SSD for Cache Acceleration?

Post by aschweig » Sat Nov 24, 2018 2:47 am

Brainstorming -- I saw this link:

viewtopic.php?t=29780

aschweig
Starting out
Posts: 11
Joined: Fri Nov 23, 2018 11:56 pm

Re: Speed up dir listing or best m.2 SATA SSD for Cache Acceleration?

Post by aschweig » Sat Nov 24, 2018 5:11 am

Another question:

#6: Is there a way to adjust the amount of "equilibrium" free space on the qtier SSDs? (My utilization seems low to me.)

Aaron

aschweig
Starting out
Posts: 11
Joined: Fri Nov 23, 2018 11:56 pm

Re: Speed up dir listing or best m.2 SATA SSD for Cache Acceleration?

Post by aschweig » Mon Nov 26, 2018 6:49 pm

To speed up the copy of a particularly large directory, I ssh'd into the box and ran cp -R from/. to/ -- this seemed to cause my NAS to become unresponsive (no web nor ssh) -- only ping-able. I powered down and upon reboot my qtier's Samsung Pro SSDs needed to be resync'd.

I have no idea if the cp command caused the failure, or if the disks spontaneously needed to be resync'd, or if there was another background job, e.g., qBoost or qTier, that ruined thing. Maybe "cp /share/directory" doesn't play nicely with qtier...

I am not a happy camper.

User avatar
Trexx
Experience counts
Posts: 4445
Joined: Sat Oct 01, 2011 7:50 am
Location: Minnesota
Contact:

Re: Speed up dir listing or best m.2 SATA SSD for Cache Acceleration?

Post by Trexx » Tue Nov 27, 2018 1:58 am

I have QTier enabled, and have done CP commands before without issue. Could be due to the more limited CPU resources of the Arm boxes.

QTier you can do some "tuning" via the option at the shared folder level with ("Enable Auto-Tiering").
Paul

Model: TS-877-1600 FW: 4.3.5.x
QTier (HDD): [RAID-5] 6 x 3TB HGST DeskStar NAS QTier (SSD): [RAID-1] 2 x 525GB Crucial MX300 m.2's
(SSD): [RAID-1] 2 x 500GB Evo 860
RAM: Kingston HyperX Fury 32GB Kit DDR4-2666
GPU: EVGA GTX 1060, ACX 2.0(1 Fan), 6GB
UPS: CyberPower AVR1350 Ext. Backup: USB 3.0 Seagate 5TB
Media Boxes: Nvidia ShieldTV Pro, AppleTV 4, Roku Stick

Model: TVS-673 32GB FW: 4.3.5.x Test/Backup Box
-----------------------------------------------------------------------------------------------------------------------------------------
NAS RAID Rebuild Times | Live QTS Videos | | QNAP NAS Guide | Information needed when you ask for HELP | QNAP Links, Tutorials, etc.
2018 Plex NAS Compatibility Guide | QNAP Plex FAQ | Moogle's QNAP Faq

aschweig
Starting out
Posts: 11
Joined: Fri Nov 23, 2018 11:56 pm

Re: Speed up dir listing or best m.2 SATA SSD for Cache Acceleration?

Post by aschweig » Fri Dec 07, 2018 1:00 am

An update on this:

- I installed 2x Samsung EVO 860 m.2 in RAID 1 as a RW cache. I think this has improved throughput. In resource monitor I can see r/w speeds around 50MB/s typical to CPU, peak to CPU around 150MB/s and peak during backup around 300MB/s. Still very curious/anxious what will be the failure mode of this cache.

- I switched from NFS to iSCSI for mounts with huge number of files. This helped enormously. The find command which took minutes now takes seconds. Sadly, these now iSCSI-mounted files were intended to be shared across several boxes -- so my client will have to re-export, which seems like it will invite the same issue. I am unsure if I want to share the iSCSI target across Debian machines (e.g., one RW and others RO). Anyone do this?

- I suspect the lockup issue experienced during the copy was due to concurrent local (over ssh) and NFS access to the same files.

- I have also switched from using NFSv4 to NFSv3 with the noatime,nodiratime,nolock,noacl,udp options. This has eliminated the huge number of NFSv4 DELEG locks that were building up on the QNAP. I suspect this DELEG lock accumulation may have contributed to the issue during cp.

aschweig
Starting out
Posts: 11
Joined: Fri Nov 23, 2018 11:56 pm

Re: Speed up dir listing or best m.2 SATA SSD for Cache Acceleration?

Post by aschweig » Fri Dec 07, 2018 6:08 am

Trexx wrote:
Tue Nov 27, 2018 1:58 am
I have QTier enabled, and have done CP commands before without issue. Could be due to the more limited CPU resources of the Arm boxes.

QTier you can do some "tuning" via the option at the shared folder level with ("Enable Auto-Tiering").
I think this may be a factor as well, particularly when dealing with lots of small files.
I had a look here:
https://openbenchmarking.org/result/180 ... -180506462

The link suggests to me that the 1635AX performs about 25% the speed of an i7-7700k; a CPU mark of about 3,000.

Post Reply

Return to “Hardware & Software Compatibility”