Speed up dir listing or best m.2 SATA SSD for Cache Acceleration?

[aschweig]

I have a TS-1635AX configured with 7+2+1 (RAID 6+Hot Spare) WD Gold drives, 4 Samsung 860 Pro SSD in RAID 10 (4 TB capacity). These are part of a storage pool using qtier. I use dual 10GbE. Drives are used via NFS.

I have found that when I need to iterate over the files in my system with the linux find command (I have many small files) -- the results take longer than I would like.
On our old SAN, such a search might take 2 minute. With the qnap, it takes more than 20 minutes. The network isn't saturated by this operation.

I have tried the noatime setting on the NFS client but this didn't help.

Is this just a consequence of NFS instead of iSCSI? I would be interested in any opinions on the matter, but NFS is critical for my application.

My next idea: Adding SSD Read-Only Cache might help with the directory reads for files housed on the HDDs.

Along these lines, I have some questions that I think the community might be able to help with:

#1: Will SSD cache also cache my existing SSD RAID Array?
- Presumably caching SSD confers no benefit, and because there would be fewer drives, a potential 50% performance penalty.

#2: Can the cache be set up only for Random, not sequential, IO?
- Sequential performance is already "okay" and I am not seeking to make any big gains here.

#3: I consider to purchase 2 SSDs for the Read-only Cache. Should Raid1 (my default) or RAID0 be used?

#4: What m.2 SATA SSD model is recommended? Considerations:
- Might be most intensively written SSD in the system.
- Intention is to increase speed of directory listings: has to be fast. Do larger capacities with more chips offer greater speed?
- Not looking for capacity particularly. If smaller sizes work and I can save a buck -- then I am happy.
- The Samsung 860 EVO seems like a good compromise in the m.2 form factor, but what about endurance? PM871a -- higher endurance but slower -- do I want slower in a cache?

#5: If a drive in an SSD cache configuration fails how does the QNAP behave? Will it keep working ignoring the failed drive? Will I have to reboot? Anyone experience this?

Thanks!

Aaron

P.S. I also have off-site backup, I know: RAID != Backup.

[dolbyman]

how is the CPU usage during all of this ?.. Those ARM units are not very fast

Box Zelin had disappointing results with a first gen x31 ARM (annapurna)
viewtopic.php?t=142194#p676730

Can't even find any speed stats for the TS-1635AX on QNAP's page

[storageman]

How slow is Qsirch?
If SSH "find" is slow GUI searches unlikely to be faster.
Seems like maybe you should have picked a faster box.
If you're talking millions of files ISCSI will search faster on this type of box.

[aschweig]

Looking at Resource Monitor during the find -- I see system processes hovering around 37%.
In top on the command line, I see about 50% idle during the find.
Find running or not doesn't seem to impact the idle number shown in top.

Aaron

[aschweig]

I haven't tested Qsirch yet.

Doing a search in FileStation 5-- I am looking for all files under a certain mount point owned by myself -- takes a few minutes. Utilization goes up, top on the qnap shows 5.1% to 8.7% idle. I started this query just now (12:11) and it finished by (12:15) which seems reasonable.

Aaron

[aschweig]

I should point out that some of my directories contain about 250,000 files -- from 8k to 100k in size each.

[aschweig]

Brainstorming -- I saw this link:

viewtopic.php?t=29780

[aschweig]

Another question:

#6: Is there a way to adjust the amount of "equilibrium" free space on the qtier SSDs? (My utilization seems low to me.)

Aaron

[aschweig]

To speed up the copy of a particularly large directory, I ssh'd into the box and ran cp -R from/. to/ -- this seemed to cause my NAS to become unresponsive (no web nor ssh) -- only ping-able. I powered down and upon reboot my qtier's Samsung Pro SSDs needed to be resync'd.

I have no idea if the cp command caused the failure, or if the disks spontaneously needed to be resync'd, or if there was another background job, e.g., qBoost or qTier, that ruined thing. Maybe "cp /share/directory" doesn't play nicely with qtier...

I am not a happy camper.

[Trexx]

I have QTier enabled, and have done CP commands before without issue. Could be due to the more limited CPU resources of the Arm boxes.

QTier you can do some "tuning" via the option at the shared folder level with ("Enable Auto-Tiering").

[aschweig]

An update on this:

- I installed 2x Samsung EVO 860 m.2 in RAID 1 as a RW cache. I think this has improved throughput. In resource monitor I can see r/w speeds around 50MB/s typical to CPU, peak to CPU around 150MB/s and peak during backup around 300MB/s. Still very curious/anxious what will be the failure mode of this cache.

- I switched from NFS to iSCSI for mounts with huge number of files. This helped enormously. The find command which took minutes now takes seconds. Sadly, these now iSCSI-mounted files were intended to be shared across several boxes -- so my client will have to re-export, which seems like it will invite the same issue. I am unsure if I want to share the iSCSI target across Debian machines (e.g., one RW and others RO). Anyone do this?

- I suspect the lockup issue experienced during the copy was due to concurrent local (over ssh) and NFS access to the same files.

- I have also switched from using NFSv4 to NFSv3 with the noatime,nodiratime,nolock,noacl,udp options. This has eliminated the huge number of NFSv4 DELEG locks that were building up on the QNAP. I suspect this DELEG lock accumulation may have contributed to the issue during cp.

[aschweig]

I have QTier enabled, and have done CP commands before without issue. Could be due to the more limited CPU resources of the Arm boxes.

QTier you can do some "tuning" via the option at the shared folder level with ("Enable Auto-Tiering").

I think this may be a factor as well, particularly when dealing with lots of small files.
I had a look here:
https://openbenchmarking.org/result/180 ... -180506462

The link suggests to me that the 1635AX performs about 25% the speed of an i7-7700k; a CPU mark of about 3,000.