10G Write Speeds slow TS-1635

[Bob Zelin]

Hi -
boy, I thought I was a 10G expert, but I guess I am not.
Brand new TS-1635, QTS 4.3.3.0154 build 20170413. Drives are 6TB HGST 7200 RPM. No SSD in TS-1636. Connection is via SFP+ Twinax cable to Netgear XS708T 10G switch SFP+ port.
Tried both Jumbo Frames MTU enabled and disabled. Clients are all Apple MAC computers (SMB Signing disabled - I do this all the time, so yes, its done correctly) with Promise SanLink2
thunderbolt 2 to 10G adaptors. All Mac OS is macOS Sierra 10.12.3

Using AJA System Test, my READ speeds are wonderful - over 600 -700 MB/sec, but my WRITE speeds to the QNAP TS-1635 is only about 100 MB/sec - the speed of a 1G connection. I do this all
the time, mostly with TVS-871T, TVS-1282T, TS-1685, and TVS-EC1680U, without issue - wonderful speeds of 800 MB/sec read and write with Netgear 10G switches, and Promise or Sonnet
thunderbolt to 10G adaptors.

And unfortunately I know it's the QNAP now, because if I hookup a Promise Thunderbolt RAID to one of the Mac workstations, and make that a "server" and connect via 10G over the same 10G network
(same switch, same Promise adaptors), I get about 300 MB/sec to the PROMISE RAID array (acting as a server) WRITE speed. Back to the QNAP, only about 100 MB/sec WRITE - but again - READ is fantastic
on all 7 Mac's - between 600 - 700 MB/sec.

Any ideas on why only the WRITE speed would be so slow ?
Thanks -
Bob Zelin

[dolbyman]

any CPU spikes while writing ?

[storageman]

What do you get as direct 1/10GbE connection to a Mac?
Can we rule out the switch to start?
What is the internal write speed on the 1635 (e.g. copying within Filestation)?

[Bob Zelin]

Hi - sorry for the delay in my reply.
All clients are Macs running macOS Sierra 10.2.3 with Promise SanLink2 thunderbolt 2 to 10G adaptors.
The QNAP connects to a Netgear XS708T switch via the SFP+ 10G port using a Twinax cable.
Because all Mac's are on the system, last night I put a small SSD RAID on one of the Macs, and did a speed test
(AJA System Test) and got over 400 MB/sec Read and Write. When I test the TS-1635, I get about 100 MB/sec Write
and 600 MB/sec read from each station. So something on the QNAP in inhibiting the write speed.
I did not do a 1G connection for any of my tests. And yes, SMB Signing is disabled on each Mac ( I am well versed
in the process). I did not test the internal write speed on the 1635, but will do so. What I have NOT done is to
back rev the firmware to QTS 4.2.4 or 4.2.5 - I have only used the new QTS 4.3.3 (which has no issues on other
QNAP systems) - but hey - you never know. I did not look at the performance graph for CPU spikes while writing.
I will do this as well.
Thank you for your input so far

Bob Zelin

[tombulum]

Not being familiar with the TS-1635, I'll ask a basic question: Is write caching enabled on the NAS?
Please run the internal speed test and let us know the result.

This is what my TS-879U-RP gets, for reference:

[~] # qcli_storage -p
Enclosure Port Sys_Name Size Type RAID RAID_Type Pool TMeta Volume_Type Volume
NAS_HOST 1 /dev/sdc 9.10 TB data /dev/md1 RAID 6,512 1 64 GB flexible DataVol_1,SecureVol_1
NAS_HOST 2 /dev/sdd 9.10 TB data /dev/md1 RAID 6,512 1 64 GB flexible DataVol_1,SecureVol_1
NAS_HOST 3 /dev/sdb 9.10 TB data /dev/md1 RAID 6,512 1 64 GB flexible DataVol_1,SecureVol_1
NAS_HOST 4 /dev/sda 9.10 TB data /dev/md1 RAID 6,512 1 64 GB flexible DataVol_1,SecureVol_1
NAS_HOST 5 /dev/sde 9.10 TB data /dev/md1 RAID 6,512 1 64 GB flexible DataVol_1,SecureVol_1
NAS_HOST 6 /dev/sdf 9.10 TB data /dev/md1 RAID 6,512 1 64 GB flexible DataVol_1,SecureVol_1
NAS_HOST 7 /dev/sdg 9.10 TB data /dev/md1 RAID 6,512 1 64 GB flexible DataVol_1,SecureVol_1
NAS_HOST 8 /dev/sdh 9.10 TB data /dev/md1 RAID 6,512 1 64 GB flexible DataVol_1,SecureVol_1

[~] # qcli_storage -T
fio test command for physical disk: /sbin/fio --filename=test_device --direct=1 --rw=read --bs=1M --runtime=15 --name=test-read --ioengine=libaio --iodepth=32 &>/tmp/qcli_storage.log
fio test command for RAID: /sbin/fio --filename=test_device --direct=0 --rw=read --bs=1M --runtime=15 --name=test-read --ioengine=libaio --iodepth=32 &>/tmp/qcli_storage.log
Start testing!
Performance test is finished 100.000%...
Enclosure Port Sys_Name Throughput RAID RAID_Type RAID_Throughput Pool
NAS_HOST 1 /dev/sdc 212.03 MB/s /dev/md1 RAID 6 897.25 MB/s 1
NAS_HOST 2 /dev/sdd 217.22 MB/s /dev/md1 RAID 6 897.25 MB/s 1
NAS_HOST 3 /dev/sdb 198.96 MB/s /dev/md1 RAID 6 897.25 MB/s 1
NAS_HOST 4 /dev/sda 200.30 MB/s /dev/md1 RAID 6 897.25 MB/s 1
NAS_HOST 5 /dev/sde 212.17 MB/s /dev/md1 RAID 6 897.25 MB/s 1
NAS_HOST 6 /dev/sdf 203.85 MB/s /dev/md1 RAID 6 897.25 MB/s 1
NAS_HOST 7 /dev/sdg 218.56 MB/s /dev/md1 RAID 6 897.25 MB/s 1
NAS_HOST 8 /dev/sdh 237.27 MB/s /dev/md1 RAID 6 897.25 MB/s 1
[~] #
[~] # qcli_storage -t
fio test command for LV layer: /sbin/fio --filename=test_device --direct=0 --rw=read --bs=1M --runtime=15 --name=test-read --ioengine=libaio --iodepth=32 &>/tmp/qcli_storage.log
fio test command for File system: /sbin/fio --directory=test_device --direct=0 --rw=read --bs=1M --runtime=15 --name=test-read --ioengine=libaio --iodepth=32 --size=128m &>/tmp/qcli_storage.log
Start testing!
Performance test is finished 100.000%...
Vol_ID Vol_Name Pool_ID Mapping_Name Throughput Mount_Path Throughput
1 DataVol_1 1 /dev/mapper/cachedev1 521.63 MB/s /share/CACHEDEV1_DATA 815.29 MB/s
2 SecureVol_1 1 /dev/mapper/ce_cachedev2 288.51 MB/s /share/CE_CACHEDEV2_DATA 438.36 MB/s
[~] #

[kd6icz]

Bob,

I have been troubleshooting this issue with QNAP on my 1635 since March. I have been online for countless hours with the engineers in Tiawan letting them have full access to run internal diagnostics on my unit.

The last session they had identified the bottleneck and are currently working on a fix. However, it wouldn't hurt for you to open a ticket with them so they have another confirmed case.

In my case, mine would be blazing fast (400-600 Mbps write in RAID6) for the first 48 to 96 hours after reboot. Then it would slow​ to a gawd awful 15-40Mbps and remain that way until I reboot.

I'm using the 10Gbps port with a Mikrotik SFP and Mikrotik switch.

Sent from my XT1650 using Tapatalk

[tinyark]

Hi kd6icz,

The TS-1635 in mention is mine.

Currently we have engineers working on the NAS and will let you know if they come to a conclusion. Personally my WRITE speeds are much worse and have never reached much above 120MB/s. This is unacceptably low and as this is a brand new machine I expect QNAP to come up with a solution shortly.

Thanks for your input.

Alex

[kd6icz]

Hi kd6icz,

The TS-1635 in mention is mine.

Currently we have engineers working on the NAS and will let you know if they come to a conclusion. Personally my WRITE speeds are much worse and have never reached much above 120MB/s. This is unacceptably low and as this is a brand new machine I expect QNAP to come up with a solution shortly.

Thanks for your input.

Alex

Well.... With the introduction of the 64bit OS, we now have new issues to address in addition to this one!

I just want to be clear, you're using 10Gbps Ethernet on both the 1635 and the computer?

Sent from my XT1650 using Tapatalk

[tinyark]

Hi,

Yes we were using 10gbe.

We found the fault. I installed 3rd party RAM which the ARM processor didn't like. QNAP RAM only!!!

Since my purchase was less than 30 days I have now sent back and exchanged for TS1685 with Intel architecture.

It got speeds of around 400 read and write when I changed the RAM back which still isn't enough for our business.

[schumaku]

We found the fault. I installed 3rd party RAM which the ARM processor didn't like.

Do you remember what SO-DIMM modules were installed from the factory, and what 3rd party modules you tried to install? This would be very useful for the community.

[tinyark]

Sure,

The RAM I installed was pretty standard - Micron Crucial 16 GB Kit (8 GB x 2) DDR3L 1600 MT/s (PC3L-12800) SODIMM 204-Pin Memory

I'll check tomorrow when back in the office what the RAM that QNAP install is.

[tinyark]

To be more specific the Crucial RAM I was using:
CT2KIT102464BF160B

Which QNAP are now testing and trying to find a solution for.

The RAM they provided with the TS-1635 was ADATA 4GB RAM:
ADDS1600W4G11B


I have been advised now by the engineers that due to the ARM processor there is compabtability issues and they only recommend QNAP installed RAM. Despite this I was advised that with Intel architecture there is better compatibility...although they still only reccomend QNAP RAM on all devices. (I'd say you'll probably be fine changing the RAM yourselves on Intel based machines though.)

Nonetheless I managed to find out by the enginners that the TS-1635 version for 8GB RAM is Transcend TS1GSK64V6H.

Hope that gets some of you out of trouble!

[Bob Zelin]

Hi -
I have been working with TinyArk and with QNAP directly on this issue. We were completely baffled as to what
was going on. I personally have a TS-531X with only 3 1TB drives in it, and my performance via 10G was better than
the TinyArk TS-1635. QNAP engineering finally asked the "fatal" question - "Is this QNAP RAM" - and of course, the answer
was NO. Crucial RAM was the culprit of the problem, and even when the original 4G of RAM was reinstalled in the
TS-1635, the Write speeds dramatically increased. SO, the moral of the story, is don't try to save a few bucks on your
accessories. Buy certified QNAP approved or branded parts for your QNAP (this means 10G cards, SAS expanders,
RAM, etc), and you will not suffer with your installation. As TinyArk has stated above, they are still proceeding
with the purchase of the TS-1685 (to replace the 1635). I have personally used the TS-1685, and it's great.
More money than the 1635 ? YES, but it's worth it - and it expands with the purchase of a QNAP 12G miniSAS card
to four external expander chassis, so it's really a great value.

Thanks to everyone here. This is a very good community.

Bob Zelin