TS-1635, Slow disk r/w speeds via iSCSI

QNAP NAS solution for server virtualization and clustering/HA/FT
Post Reply
msattler
New here
Posts: 7
Joined: Mon Jul 31, 2017 3:20 pm

TS-1635, Slow disk r/w speeds via iSCSI

Post by msattler »

Hello, I am running the following:

Quanta LB6M Switch
Dell R610, Dual Xeon Hex Core, 64GB Memory connected to the Quanta via TwinAx 10Gbe, Running ESXi 6.5 U1
TS-1635 connected to the Quanta via TwinAx 10Gbe

TS-1635 has configured:
8 x 2TB Disks, RAID-10, and 2 additional Spare disks
2 x 3TB Disks, RAID-1, for the initial volume that was created
4 x 250GB Samsung 850 eVO SSD drives

I have several VM's created on the large RAID 10 volume with the spinning disks.

The SSD LUN contains my single Ubuntu Plex VM.

ESXi, and the Quanta switch are setup for Jumbo Frames.



Using Crystal Disk MArk with a 5 x 100MiB test I got the following:


Sequential Read (Q= 32,T= 1) : 573.987 MB/s
Sequential Write (Q= 32,T= 1) : 16.253 MB/s
Random Read 4KiB (Q= 32,T= 1) : 65.813 MB/s [ 16067.6 IOPS]
Random Write 4KiB (Q= 32,T= 1) : 12.941 MB/s [ 3159.4 IOPS]
Sequential Read (T= 1) : 394.475 MB/s
Sequential Write (T= 1) : 11.744 MB/s
Random Read 4KiB (Q= 1,T= 1) : 10.804 MB/s [ 2637.7 IOPS]
Random Write 4KiB (Q= 1,T= 1) : 1.232 MB/s [ 300.8 IOPS]


Is it normal for the write throughput to be that low?

I also run a PlexVM on the SSD Raid 10 Lun, and the output can really vary:

sudo hdparm -Tt /dev/sda

/dev/sda:
Timing cached reads: 13518 MB in 2.00 seconds = 6763.25 MB/sec
Timing buffered disk reads: 2 MB in 7.83 seconds = 261.41 kB/sec


The highest result I've seen was 300MB/sec, but it's rare. VM is running Ubuntu 16.04.

Disk Information:

Disk /dev/sda: 320 GiB, 343597383680 bytes, 671088640 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x6622dbc3

Device Boot Start End Sectors Size Id Type
/dev/sda1 * 2048 999423 997376 487M 83 Linux
/dev/sda2 1001470 335542271 334540802 159.5G 5 Extended
/dev/sda3 999424 1001469 2046 1023K 8e Linux LVM
/dev/sda4 335542272 671088639 335546368 160G 8e Linux LVM
/dev/sda5 1001472 335542271 334540800 159.5G 8e Linux LVM


I've made sure to upgrade to ESXi 6.5 U1. I've even shut other VM's down, to reduce load but still cannot seem to get better write speeds, and the overall speeds on the Linux VM are abysmal.

I did find a post that mentioned running some internal QNAP tests and here are those results. On the qcli_storage -T test I do find it interesting that one of the SSD's had much lower throughput than the others.

[~] # qcli_storage -p
Enclosure Port Sys_Name Size Type RAID RAID_Type Pool TMeta VolType VolName
NAS_HOST 1 /dev/sdl 232.89 GB data /dev/md2 RAID 10,64 2 16 GB flexible LUN_0
NAS_HOST 2 /dev/sdm 232.89 GB data /dev/md2 RAID 10,64 2 16 GB flexible LUN_0
NAS_HOST 3 /dev/sdn 232.89 GB data /dev/md2 RAID 10,64 2 16 GB flexible LUN_0
NAS_HOST 4 /dev/sdo 232.89 GB data /dev/md2 RAID 10,64 2 16 GB flexible LUN_0
NAS_HOST 5 /dev/sdg 2.73 TB data /dev/md1 RAID 1 288 -- Static DataVol1
NAS_HOST 6 /dev/sdf 1.82 TB data /dev/md3 RAID 10,64 1 16 GB flexible BigLun
NAS_HOST 7 /dev/sde 1.82 TB data /dev/md3 RAID 10,64 1 16 GB flexible BigLun
NAS_HOST 8 /dev/sdd 1.82 TB data /dev/md3 RAID 10,64 1 16 GB flexible BigLun
NAS_HOST 9 /dev/sdc 1.82 TB data /dev/md3 RAID 10,64 1 16 GB flexible BigLun
NAS_HOST 10 /dev/sdb 1.82 TB data /dev/md3 RAID 10,64 1 16 GB flexible BigLun
NAS_HOST 11 /dev/sda 1.82 TB spare /dev/md3 RAID 10,64 1 16 GB flexible BigLun
NAS_HOST 12 /dev/sdp 1.82 TB spare /dev/md3 RAID 10,64 1 16 GB flexible BigLun
NAS_HOST 13 /dev/sdh 1.82 TB data /dev/md3 RAID 10,64 1 16 GB flexible BigLun
NAS_HOST 14 /dev/sdi 1.82 TB data /dev/md3 RAID 10,64 1 16 GB flexible BigLun
NAS_HOST 15 /dev/sdj 2.73 TB data /dev/md1 RAID 1 288 -- Static DataVol1
NAS_HOST 16 /dev/sdk 1.82 TB data /dev/md3 RAID 10,64 1 16 GB flexible BigLun
[~] # qcli_storage -T
fio test command for physical disk: /sbin/fio --filename=test_device --direct=1 --rw=read --bs=1M --runtime=15 --name=test-read --ioengine=libaio --iodepth=32 &>/tmp/qcli_storage.log
fio test command for RAID: /sbin/fio --filename=test_device --direct=0 --rw=read --bs=1M --runtime=15 --name=test-read --ioengine=libaio --iodepth=32 &>/tmp/qcli_storage.log
Start testing!
Performance test is finished 100.000%...
Enclosure Port Sys_Name Throughput RAID RAID_Type RAID_Throughput Pool
NAS_HOST 1 /dev/sdl 517.99 MB/s /dev/md2 RAID 10 486.60 MB/s 2
NAS_HOST 2 /dev/sdm 530.27 MB/s /dev/md2 RAID 10 486.60 MB/s 2
NAS_HOST 3 /dev/sdn 270.41 MB/s /dev/md2 RAID 10 486.60 MB/s 2
NAS_HOST 4 /dev/sdo 444.16 MB/s /dev/md2 RAID 10 486.60 MB/s 2
NAS_HOST 5 /dev/sdg 154.29 MB/s /dev/md1 RAID 1 154.29 MB/s 288
NAS_HOST 6 /dev/sdf 138.32 MB/s /dev/md3 RAID 10 297.16 MB/s 1
NAS_HOST 7 /dev/sde 133.89 MB/s /dev/md3 RAID 10 297.16 MB/s 1
NAS_HOST 8 /dev/sdd 125.64 MB/s /dev/md3 RAID 10 297.16 MB/s 1
NAS_HOST 9 /dev/sdc 138.18 MB/s /dev/md3 RAID 10 297.16 MB/s 1
NAS_HOST 10 /dev/sdb 112.31 MB/s /dev/md3 RAID 10 297.16 MB/s 1
NAS_HOST 11 /dev/sda 144.77 MB/s /dev/md3 RAID 10 297.16 MB/s 1
NAS_HOST 12 /dev/sdp 178.47 MB/s /dev/md3 RAID 10 297.16 MB/s 1
NAS_HOST 13 /dev/sdh 133.87 MB/s /dev/md3 RAID 10 297.16 MB/s 1
NAS_HOST 14 /dev/sdi 114.02 MB/s /dev/md3 RAID 10 297.16 MB/s 1
NAS_HOST 15 /dev/sdj 172.12 MB/s /dev/md1 RAID 1 154.29 MB/s 288
NAS_HOST 16 /dev/sdk 183.42 MB/s /dev/md3 RAID 10 297.16 MB/s 1
[~] # qcli_storage -t
fio test command for LV layer: /sbin/fio --filename=test_device --direct=0 --rw=read --bs=1M --runtime=15 --name=test-read --ioengine=libaio --iodepth=32 &>/tmp/qcli_storage.log
fio test command for File system: /sbin/fio --directory=test_device --direct=0 --rw=read --bs=1M --runtime=15 --name=test-read --ioengine=libaio --iodepth=32 --size=128m &>/tmp/qcli_storage.log
Start testing!
Performance test is finished 100.000%...
VolID VolName Pool Mapping_Name Throughput Mount_Path FS_Throughput
1 DataVol1 288 /dev/mapper/cachedev1 147.16 MB/s /share/CACHEDEV1_DATA 140.04 MB/s

What else can I look for to try and determine why my throughput is so low?

Thanks,
Marcus
User avatar
Trexx
Ask me anything
Posts: 5393
Joined: Sat Oct 01, 2011 7:50 am
Location: Minnesota

Re: TS-1635, Slow disk r/w speeds via iSCSI

Post by Trexx »

To start with I don't see the TS-1635 being shown on the VMware certified platform.
https://www.qnap.com/solution/virtualization/en-us/
http://files.qnap.com/news/pressresourc ... 3-RS_A.pdf

My initial guess would be ultimately you are hitting CPU bottlenecks, but those write #'s do seem low.

Which QTS build are you on? Have you opened a helpdesk ticket with QNAP? What specific drive models are you using for your HDD?
Paul

Model: TS-877-1600 FW: 4.5.3.x
QTS (SSD): [RAID-1] 2 x 1TB WD Blue m.2's
Data (HDD): [RAID-5] 6 x 3TB HGST DeskStar
VMs (SSD): [RAID-1] 2 x1TB SK Hynix Gold
Ext. (HDD): TR-004 [Raid-5] 4 x 4TB HGST Ultastor
RAM: Kingston HyperX Fury 64GB DDR4-2666
UPS: CP AVR1350

Model:TVS-673 32GB & TS-228a Offline[/color]
-----------------------------------------------------------------------------------------------------------------------------------------
2018 Plex NAS Compatibility Guide | QNAP Plex FAQ | Moogle's QNAP Faq
msattler
New here
Posts: 7
Joined: Mon Jul 31, 2017 3:20 pm

Re: TS-1635, Slow disk r/w speeds via iSCSI

Post by msattler »

Trexx wrote:To start with I don't see the TS-1635 being shown on the VMware certified platform.
https://www.qnap.com/solution/virtualization/en-us/
http://files.qnap.com/news/pressresourc ... 3-RS_A.pdf

My initial guess would be ultimately you are hitting CPU bottlenecks, but those write #'s do seem low.

Which QTS build are you on? Have you opened a helpdesk ticket with QNAP? What specific drive models are you using for your HDD?
I really did not expect the TS-1635 to be on the certified platform, especially since it is just out.

I'm using 2TB Hitachi and HGST 7200 rpm drives. While I would prefer to have them all be the same make the reality is 2TB HGST cannot be found anymore brand new. The SSD's are Samsung 850 EVO 250GB drives.

I actually came across several things last night that seemed to have helped.

1) Upgraded ESXi 6.5 to 6.5 U1 which apparently solves several latency issues.

2) I switched over the Linux VM to the para-virtual scsi driver and that seems to have helped it a ton. As you will see below it's a definite improvement.

/dev/sda:
Timing cached reads: 13910 MB in 2.00 seconds = 6959.53 MB/sec
Timing buffered disk reads: 1048 MB in 3.01 seconds = 348.68 MB/sec

One thing I'm confused on is what kind of throughput I should get here since this is on 4 x 250GB Raid 10 SSD's. I would expect to see all 4 disks with the same throughput? Or would it be expected that this test may not show the same speeds for the mirrored array? Assuming 1 and 2 are paired and 3 and 4 are paired.

NAS_HOST 1 /dev/sdl 517.99 MB/s /dev/md2 RAID 10 486.60 MB/s 2
NAS_HOST 2 /dev/sdm 530.27 MB/s /dev/md2 RAID 10 486.60 MB/s 2
NAS_HOST 3 /dev/sdn 270.41 MB/s /dev/md2 RAID 10 486.60 MB/s 2
NAS_HOST 4 /dev/sdo 444.16 MB/s /dev/md2 RAID 10 486.60 MB/s 2

No ticket opened yet, I wanted to feel like I got to a point where I had something substantial I could pinpoint before going that route.

Thanks!
msattler
New here
Posts: 7
Joined: Mon Jul 31, 2017 3:20 pm

Re: TS-1635, Slow disk r/w speeds via iSCSI

Post by msattler »

Trexx wrote:To start with I don't see the TS-1635 being shown on the VMware certified platform.
https://www.qnap.com/solution/virtualization/en-us/
http://files.qnap.com/news/pressresourc ... 3-RS_A.pdf

My initial guess would be ultimately you are hitting CPU bottlenecks, but those write #'s do seem low.

Which QTS build are you on? Have you opened a helpdesk ticket with QNAP? What specific drive models are you using for your HDD?
Forgot to ask, CPU bottlenecks on the VM's or on the TS-1635? The TS-1635's cpu utilization never seems to rise above 30%.

As for build 4.3.3.0238 20170703


Thanks!
User avatar
Trexx
Ask me anything
Posts: 5393
Joined: Sat Oct 01, 2011 7:50 am
Location: Minnesota

Re: TS-1635, Slow disk r/w speeds via iSCSI

Post by Trexx »

For your HDD's, are they HGST Desktar NAS drives on QNAP's compatibility list or just standard desktop drives? That makes a huge difference.

In terms of SSD's, you SATA bus is likely your main bottleneck so I don't think RAID-10 really is going to give you much of any benefit and you do have some risk if the wrong 2 drives failed at once.
Different throughput #'s could be from different workload profiles (random/sequential, block size, etc.). Also would check to make sure all drives are on same Samsung FW version, etc.

One good test is to install QNAP diagnostic apps, and then run HDD Analyzer, Throughput test. That will test the individual drives at the QNAP level they should be similar in terms of performance in that test.

For ESX, there is a lot of tuning via the VM config, VM Tools, etc. that will impact performance, so make sure you are following the standard best practices for those.
Paul

Model: TS-877-1600 FW: 4.5.3.x
QTS (SSD): [RAID-1] 2 x 1TB WD Blue m.2's
Data (HDD): [RAID-5] 6 x 3TB HGST DeskStar
VMs (SSD): [RAID-1] 2 x1TB SK Hynix Gold
Ext. (HDD): TR-004 [Raid-5] 4 x 4TB HGST Ultastor
RAM: Kingston HyperX Fury 64GB DDR4-2666
UPS: CP AVR1350

Model:TVS-673 32GB & TS-228a Offline[/color]
-----------------------------------------------------------------------------------------------------------------------------------------
2018 Plex NAS Compatibility Guide | QNAP Plex FAQ | Moogle's QNAP Faq
msattler
New here
Posts: 7
Joined: Mon Jul 31, 2017 3:20 pm

Re: TS-1635, Slow disk r/w speeds via iSCSI

Post by msattler »

Trexx wrote:For your HDD's, are they HGST Desktar NAS drives on QNAP's compatibility list or just standard desktop drives? That makes a huge difference.

In terms of SSD's, you SATA bus is likely your main bottleneck so I don't think RAID-10 really is going to give you much of any benefit and you do have some risk if the wrong 2 drives failed at once.
Different throughput #'s could be from different workload profiles (random/sequential, block size, etc.). Also would check to make sure all drives are on same Samsung FW version, etc.

One good test is to install QNAP diagnostic apps, and then run HDD Analyzer, Throughput test. That will test the individual drives at the QNAP level they should be similar in terms of performance in that test.

For ESX, there is a lot of tuning via the VM config, VM Tools, etc. that will impact performance, so make sure you are following the standard best practices for those.

Thanks, that was really helpful! One thing I noticed is that the SSD speeds, 2 of them are almost brand new, 2 are older. And it looks like when it comes to SSD's that age becomes more of an issue than physical drives?

I also see lot's of variances in the 2TB drives I have.

I don't need a ton of space so if I start looking at buying supported 2TB drives would you recommend the WD Pro 2TB's?

Here are the results, and as such I guess it speaks volumes to the results. My 2TB drives are not on the supported list. I believe Disk 1-4 should be the SSD's and the rest the spinning disks.

Thanks for the help so far!

-Marcus


Name: Disk 5
Path: /dev/sdg
Performance: 150.21 MB/sec
Name: Disk 6
Path: /dev/sdf
Performance: 111.82 MB/sec
Name: Disk 7
Path: /dev/sde
Performance: 87.90 MB/sec
Name: Disk 8
Path: /dev/sdd
Performance: 71.00 MB/sec
Name: Disk 9
Path: /dev/sdc
Performance: 101.78 MB/sec
Name: Disk 10
Path: /dev/sdb
Performance: 63.64 MB/sec
Name: Disk 11
Path: /dev/sda
Performance: 145.62 MB/sec
Name: Disk 13
Path: /dev/sdh
Performance: 107.70 MB/sec
Name: Disk 14
Path: /dev/sdi
Performance: 102.69 MB/sec
Name: Disk 3
Path: /dev/sdn
Performance: 263.84 MB/sec
Name: Disk 4
Path: /dev/sdo
Performance: 442.26 MB/sec
Name: Disk 12
Path: /dev/sdp
Performance: 181.69 MB/sec
Name: Disk 16
Path: /dev/sdk
Performance: 139.04 MB/sec
Name: Disk 15
Path: /dev/sdj
Performance: 172.31 MB/sec
Name: Disk 2
Path: /dev/sdm
Performance: 513.12 MB/sec
Name: Disk 1
Path: /dev/sdl
Performance: 507.27 MB/sec
User avatar
Trexx
Ask me anything
Posts: 5393
Joined: Sat Oct 01, 2011 7:50 am
Location: Minnesota

Re: TS-1635, Slow disk r/w speeds via iSCSI

Post by Trexx »

Non-Nas rated drives will tend to have issues especially with WRITE behavior. WD Greens & Seagate DM's have been particularly bad due to how they handle bad blocks/block relocations/etc.

Right now the sweet spot in terms of GB/$ is 4TB. I am a big fan of the HGST Deskstar NAS drives as they are similar price to WD RED's, but you get 7200 RPM drives vs. 57(?)00 in the WD Red. Higher RPM = Higher IOPs (per drive). If you look at BackBlazes reliability reports over the past couple years Hitachi has been a consistent leader in reliability there as well.

Age as well as internal technology for SSD makes a huge difference. Just because they both say Samsung Evo 850 doesn't mean they are identical inside.
http://www.tomshardware.com/news/consum ... 34631.html
Paul

Model: TS-877-1600 FW: 4.5.3.x
QTS (SSD): [RAID-1] 2 x 1TB WD Blue m.2's
Data (HDD): [RAID-5] 6 x 3TB HGST DeskStar
VMs (SSD): [RAID-1] 2 x1TB SK Hynix Gold
Ext. (HDD): TR-004 [Raid-5] 4 x 4TB HGST Ultastor
RAM: Kingston HyperX Fury 64GB DDR4-2666
UPS: CP AVR1350

Model:TVS-673 32GB & TS-228a Offline[/color]
-----------------------------------------------------------------------------------------------------------------------------------------
2018 Plex NAS Compatibility Guide | QNAP Plex FAQ | Moogle's QNAP Faq
msattler
New here
Posts: 7
Joined: Mon Jul 31, 2017 3:20 pm

Re: TS-1635, Slow disk r/w speeds via iSCSI

Post by msattler »

Trexx wrote:Non-Nas rated drives will tend to have issues especially with WRITE behavior. WD Greens & Seagate DM's have been particularly bad due to how they handle bad blocks/block relocations/etc.

Right now the sweet spot in terms of GB/$ is 4TB. I am a big fan of the HGST Deskstar NAS drives as they are similar price to WD RED's, but you get 7200 RPM drives vs. 57(?)00 in the WD Red. Higher RPM = Higher IOPs (per drive). If you look at BackBlazes reliability reports over the past couple years Hitachi has been a consistent leader in reliability there as well.

Age as well as internal technology for SSD makes a huge difference. Just because they both say Samsung Evo 850 doesn't mean they are identical inside.
http://www.tomshardware.com/news/consum ... 34631.html
The WD Reds I was looking at were the 2TB Red Pro's which are 7200 rpm as well.

I have been a huge hitachi fan for years but didn't think hitachi disks were being manufacturer/sold still under that label.

The Hitachi Deskstar NAS are $139 for 3TB and $149 for 4TB. The 2TB Red Pro I found on sale for $135. The Red Pro do come with a 5 year warranty.
msattler
New here
Posts: 7
Joined: Mon Jul 31, 2017 3:20 pm

Re: TS-1635, Slow disk r/w speeds via iSCSI

Post by msattler »

Screw it, I grabbed 4 of the 0S03660 Hitachi 3TB Deskstar NAS's and they will be here tomorrow.

I was still planning on setting these up 4 x 3TB RAID 10 to get better throughput. Granted these VM's are not anything that are too terribly difficult to rebuild, and I am using Free Veeam to back them up as well.

As for my Plex VM, it's running on the 4 x 850 Pro's right now. But two of them are slow so I may replace them as well.

I'll give an update tomm.

Thanks!
User avatar
Trexx
Ask me anything
Posts: 5393
Joined: Sat Oct 01, 2011 7:50 am
Location: Minnesota

Re: TS-1635, Slow disk r/w speeds via iSCSI

Post by Trexx »

For 4 drives go raid-5. If you then migrate to 8 drives down the road then I would move to Raid6. I have 6 HGST drives in a Raid-5 right now and have plenty of performance.


Sent from my iPhone using Tapatalk
Paul

Model: TS-877-1600 FW: 4.5.3.x
QTS (SSD): [RAID-1] 2 x 1TB WD Blue m.2's
Data (HDD): [RAID-5] 6 x 3TB HGST DeskStar
VMs (SSD): [RAID-1] 2 x1TB SK Hynix Gold
Ext. (HDD): TR-004 [Raid-5] 4 x 4TB HGST Ultastor
RAM: Kingston HyperX Fury 64GB DDR4-2666
UPS: CP AVR1350

Model:TVS-673 32GB & TS-228a Offline[/color]
-----------------------------------------------------------------------------------------------------------------------------------------
2018 Plex NAS Compatibility Guide | QNAP Plex FAQ | Moogle's QNAP Faq
msattler
New here
Posts: 7
Joined: Mon Jul 31, 2017 3:20 pm

Re: TS-1635, Slow disk r/w speeds via iSCSI

Post by msattler »

Trexx wrote:For 4 drives go raid-5. If you then migrate to 8 drives down the road then I would move to Raid6. I have 6 HGST drives in a Raid-5 right now and have plenty of performance.


Sent from my iPhone using Tapatalk
Will set the 4 disk up tomorrow and play around and see what I end up with. Honestly if I go Raid 5, I think if I were to add more disks I'd just got with 2 Raid 5 arrays, and just split the iops vm wise between the 2 arrays.

Thanks again!
Post Reply

Return to “Server Virtualization & Clustering”