iSCS very, very slow...

iSCSI related applications
Post Reply
Jbertrand1
New here
Posts: 2
Joined: Tue Feb 02, 2021 11:18 pm

iSCS very, very slow...

Post by Jbertrand1 » Tue Feb 02, 2021 11:41 pm

Hello,

I have connected a TS-431P2 directly on a NetBSD server (without switch between server and NAS). I have configured only one LUN (four disks in raid6 configuration). This NAS runs with firmware 4.4.3.1439.

I a first time, adapter on NetBSD side was an old Realtek adapter that randomly works with jumbo frames. iSCSI throughput doesn't exceed 10 MB/s. I though that this old adapter was responsible for these poor performances.

Thus, I have replace last week this adapter by an new Intel ethernet adapter. Now, mtu is set on link to 9000 bytes. But iSCSI throughput still does not exceed 10 MB/s !

I have checked interface throughput with iperf3 :

Code: Select all

-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.10.103, port 40986
[  5] local 192.168.10.128 port 5201 connected to 192.168.10.103 port 40988
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   110 MBytes   919 Mbits/sec                  
[  5]   1.00-2.00   sec   111 MBytes   932 Mbits/sec                  
[  5]   2.00-3.00   sec   110 MBytes   925 Mbits/sec                  
[  5]   3.00-4.00   sec   111 MBytes   934 Mbits/sec                  
[  5]   4.00-5.00   sec   112 MBytes   936 Mbits/sec                  
[  5]   5.00-6.00   sec   112 MBytes   936 Mbits/sec                  
[  5]   6.00-7.00   sec   111 MBytes   935 Mbits/sec                  
[  5]   7.00-8.00   sec   111 MBytes   935 Mbits/sec                  
[  5]   8.00-9.00   sec   111 MBytes   928 Mbits/sec                  
[  5]   9.00-10.00  sec   111 MBytes   934 Mbits/sec                  
[  5]  10.00-10.00  sec  33.9 KBytes   785 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  1.08 GBytes   931 Mbits/sec                  receiver

hilbert:[~] > iperf3 -c 192.168.10.128
Connecting to host 192.168.10.128, port 5201
[  6] local 192.168.10.103 port 40988 connected to 192.168.10.128 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  6]   0.00-1.00   sec   110 MBytes   922 Mbits/sec    0   69.3 KBytes       
[  6]   1.00-2.00   sec   111 MBytes   932 Mbits/sec    0   69.3 KBytes       
[  6]   2.00-3.00   sec   110 MBytes   925 Mbits/sec    0   69.3 KBytes       
[  6]   3.00-4.00   sec   111 MBytes   934 Mbits/sec    0   69.3 KBytes       
[  6]   4.00-5.00   sec   112 MBytes   936 Mbits/sec    0   69.3 KBytes       
[  6]   5.00-6.00   sec   112 MBytes   936 Mbits/sec    0   69.3 KBytes       
[  6]   6.00-7.00   sec   111 MBytes   935 Mbits/sec    0   69.3 KBytes       
[  6]   7.00-8.00   sec   111 MBytes   935 Mbits/sec    0   69.3 KBytes       
[  6]   8.00-9.00   sec   111 MBytes   928 Mbits/sec    0   69.3 KBytes       
[  6]   9.00-10.00  sec   111 MBytes   935 Mbits/sec    0   69.3 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  6]   0.00-10.00  sec  1.08 GBytes   932 Mbits/sec    0             sender
[  6]   0.00-10.00  sec  1.08 GBytes   931 Mbits/sec                  receiver
I think I can affirm that network runs as expected.

I have tried to write a huge file in iSCSI volume to check NAS CPU load. This load doesn't exceed 15%. Bottleneck doesn't come from CPU. Same constatation about NAS memory.

Code: Select all

[~] # qcli_storage -p
Enclosure Port Sys_Name      Size      Type   RAID        RAID_Type    Pool TMeta  VolType      VolName 
NAS_HOST  1    /dev/sda      5.46 TB   data   /dev/md1    RAID 6,512   1    64 GB  flexible     euclide_0
NAS_HOST  2    /dev/sdb      5.46 TB   data   /dev/md1    RAID 6,512   1    64 GB  flexible     euclide_0
NAS_HOST  3    /dev/sdc      5.46 TB   data   /dev/md1    RAID 6,512   1    64 GB  flexible     euclide_0
NAS_HOST  4    /dev/sdd      5.46 TB   data   /dev/md1    RAID 6,512   1    64 GB  flexible     euclide_0
[~] # qcli_storage -T
fio test command for physical disk: /sbin/fio --filename=test_device --direct=1 --rw=read --bs=1M --runtime=15 --name=test-read --ioengine=libaio --iodepth=32 &>/tmp/qcli_storage.log
fio test command for RAID: /sbin/fio --filename=test_device --direct=0 --rw=read --bs=1M --runtime=15 --name=test-read --ioengine=libaio --iodepth=32 &>/tmp/qcli_storage.log
Start testing!
Performance test is finished 100.000%...
Enclosure  Port  Sys_Name      Throughput    RAID        RAID_Type    RAID_Throughput   Pool  
NAS_HOST   1     /dev/sda      210.50 MB/s   /dev/md1    RAID 6       310.78 MB/s       1     
NAS_HOST   2     /dev/sdb      200.62 MB/s   /dev/md1    RAID 6       310.78 MB/s       1     
NAS_HOST   3     /dev/sdc      207.33 MB/s   /dev/md1    RAID 6       310.78 MB/s       1     
NAS_HOST   4     /dev/sdd      206.52 MB/s   /dev/md1    RAID 6       310.78 MB/s       1     
[~]
If I understand, Raid6 throughput on md1 is greater than 300 MB/s. Why not. But in this case, why my NetBSD server writes to the NAS (or read), the throughput does not exceed 10 MB/s ? Of course, I have tried to tuned sysctl parameters without any success. I have tested also that with another iSCSI target (Linux or xBSD) on the same NetBSD server and throughput reaches 120 MB/s...

Help will be welcome,

JB

Jbertrand1
New here
Posts: 2
Joined: Tue Feb 02, 2021 11:18 pm

Re: iSCSI very, very slow...

Post by Jbertrand1 » Wed Feb 03, 2021 3:18 pm

Some news.

legendre# dd if=/dev/zero of=/opt/bacula/test.dd count=10 bs=10m
10+0 records in
10+0 records out
104857600 bytes transferred in 0.537 secs (195265549 bytes/sec)
legendre# dd if=/dev/zero of=/opt/bacula/test.dd count=10 bs=100m
10+0 records in
10+0 records out
1048576000 bytes transferred in 53.396 secs (19637725 bytes/sec)
legendre# dd if=/dev/zero of=/opt/bacula/test.dd count=10 bs=1000m
10+0 records in
10+0 records out
10485760000 bytes transferred in 1026.927 secs (10210813 bytes/sec)
legendre#

186 MB/s for the first file. 18,7 MB/s for the second one and about 10 MB/s fort the third one.

And I don't understand. If iSCSI target or raid6 subsystem on qNAP were the bottleneck, CPU load should be greater than 1 and it's not the case. Maybe I will try to add a SSD disk as cache (but I'm not sure that this NAS supports cache over USB3).

Best regards,

JB

DP-Wolf
New here
Posts: 3
Joined: Fri Jan 22, 2021 3:47 am

Re: iSCS very, very slow...

Post by DP-Wolf » Thu Feb 04, 2021 11:18 am

Hi! I'm new to the Form and New to Qnap not to mention NAS in general.
I'm a videographer who uses the Adobe workflow land decided to implement a NAS system so I can make use of the Productions workflow from 3 Work Stations.

**PROBLEM:
When I first installed everything and ran tests using Blackmagic - Disk Speed Test utility. I achieved over 1000MB/s Write and 1600MB/s Read.
I started using the system, but was having trouble with Multi Cam. I wondered if it would help if I turned the cache into a Raid 0 (Read Only) array and did so. There after my read/write speeds are the SAME AS WITHOUT CACHE (220MB/s Read 260MB/s Write)!!! I feel like I should be getting faster than this with 4 disks in a raid 10 without cache, given that each individual disk is supposed to have Read/Write speed of 250MB/s.
I've reset up the cache many times in different configurations with no improvement! In fact, on my iMac Thunderbolt 2, I get repeated speeds of 200MB/s Write and 90MB/s Read.

Something is very wrong with this. This thing is practically useless in its current state. Admittedly, I've been messing around with the features on this NAS, and it's possible I messed some setting up. I've already uploaded and organized 7TB worth of data on this thing, and would like not to have to start from scratch again, as this took forever to setup!

PLEASE HELP!!! I really need to get this thing working.


Here's my setup:
TVS-472XT (with Thunderbolt card)
Firmware: 4.5.2.1566
Upgraded RAM to 32GB: 2 x Crucial 16GB DDR4 2666 MT/s (CT16G4SFD8266)
Installed 4 x 10TB Raid 10 array: 4 x Seagate Exos X10 10TB 7200 RPM (ST10000NM0086)
Installed 2 x 1TB Raid 1 (Read/Write)(CACHE): 2 x Samsung 970 EVO SSD 1TB (MZ-V7E1T0BW)

The 3 computers I use are:
iMac Mini (M1) - Thunderbolt 3.0 (6') 20gbps
iMac 2017 - Thunderbolt 2.0 (6' using a 3.0 to 2.0 adapter) 20gbps
MacBook Pro - 10GbE via Cat7 through a CalDigit 10GbE to Thunderbolt converter

- Snapshots are Disabled
You do not have the required permissions to view the files attached to this post.

Bob Zelin
Experience counts
Posts: 1105
Joined: Mon Nov 21, 2016 12:55 am
Location: Orlando, FL.
Contact:

Re: iSCS very, very slow...

Post by Bob Zelin » Thu Feb 04, 2021 11:35 pm

why on earth are you using iSCSI ? You will never get full video editing with Adobe to work on 3 workstations with only a 4 bay NAS. IT's too slow - not the QNAP - the four 7200 RPM SATA drives. You will never exceed
450 - 500 MB/sec, which is not enough for 3 workstations all doing 4K with Premiere.

Thunderbolt 2 is a terrible interface. You will never get consistant results with this. Your SSD or M.2 Cache will do nothing for you. I do not know the CalDigit 10GbE adapter, but with a QNAP QNA-T310G1T, or Promise SanLink3, or
Sonnet Solo 10G T3 on your Mac Mini M1 (there is no such thing as an iMac M1) - you get about 450 MB/sec with a 4 drive system in a single RAID group (no iSCSI, no Cache on the QNAP - turn that off)
Your 2017 iMac is Thunderbolt 3 - not Thunderbolt 2 - so get a QNAP QNA-T310G1T ($183 on Amazon) or a Sonnet Solo 10G T3 ($149 on Amazon) and you will get the same speeds.

You are NEVER going to get all three workstations to work off of a TVS-472XT. You should have purchased the TVS-872XT (or one of the new models like the TVS-h1288X).
You take the single 10G port, and go into a small 10G switch like a QNAP QSW-M1208-8C ($599) or a QNAP QSW-M408-4C ($299).

SO - how do you get 800 MB/sec on the TVS-472XT ? Simple - YOU DONT - whoever told you that you could is an idiot, and has no experience with this product.
I do this stuff for video editing professionals all day long.

Bob Zelin
Bob Zelin / Rescue 1, Inc.
http://www.bobzelin.com

DP-Wolf
New here
Posts: 3
Joined: Fri Jan 22, 2021 3:47 am

Re: iSCS very, very slow...

Post by DP-Wolf » Fri Feb 05, 2021 6:05 am

Bob Zelin wrote:
Thu Feb 04, 2021 11:35 pm
why on earth are you using iSCSI ? You will never get full video editing with Adobe to work on 3 workstations with only a 4 bay NAS. IT's too slow - not the QNAP - the four 7200 RPM SATA drives. You will never exceed
450 - 500 MB/sec, which is not enough for 3 workstations all doing 4K with Premiere.

Thunderbolt 2 is a terrible interface. You will never get consistant results with this. Your SSD or M.2 Cache will do nothing for you. I do not know the CalDigit 10GbE adapter, but with a QNAP QNA-T310G1T, or Promise SanLink3, or
Sonnet Solo 10G T3 on your Mac Mini M1 (there is no such thing as an iMac M1) - you get about 450 MB/sec with a 4 drive system in a single RAID group (no iSCSI, no Cache on the QNAP - turn that off)
Your 2017 iMac is Thunderbolt 3 - not Thunderbolt 2 - so get a QNAP QNA-T310G1T ($183 on Amazon) or a Sonnet Solo 10G T3 ($149 on Amazon) and you will get the same speeds.

You are NEVER going to get all three workstations to work off of a TVS-472XT. You should have purchased the TVS-872XT (or one of the new models like the TVS-h1288X).
You take the single 10G port, and go into a small 10G switch like a QNAP QSW-M1208-8C ($599) or a QNAP QSW-M408-4C ($299).

SO - how do you get 800 MB/sec on the TVS-472XT ? Simple - YOU DONT - whoever told you that you could is an idiot, and has no experience with this product.
I do this stuff for video editing professionals all day long.

Bob Zelin

Hi Bob,

Thanks for the reply.
I obviously posted in the wrong place. I didn't know what iSCSI was. Did a little digging and I'm definitely not using it. Also typo on my part, yes Mac Mini (M1), and I meant iMac 2015 (not 2017). Sorry. I know my setup isn't ideal and I spent the total budget I had on this system, hoping I could get at least 2 stations working at the same time. Is there a way I can configure this to run 2 stations?

So let me see if I can unpack your reply. Sounds like your saying if I turned the four 7200rpm drives into a RAID 0, that I would max out at 450MB/s? Is that correct? (I was under the impression that it multiplies the speed so Raid 0 - 4 x 250MB/s = 1000MB/s & Raid 10 - 2 x 250MB/s = 500MB/s) Is this wrong?

Are you saying Cache is worthless? I thought even with the TVS-872XT in Raid 5 that guys were using cache acceleration to get 3 stations working.

A set the NVMe M.2 SSDs into their own Pool and Volume and tested the speeds from the 3 computers. Here's my results:
Mac Mini (M1): Write: 896 MB/s Read: 1091 MB/s - over Thunderbolt 3
iMac 2017: Write: 357 MB/s Read: 889 MB/s - over Thunderbolt 2
Macbook Pro: Write: 829 MB/s Read: 505 MB/s - 10 GbE

So I think if I'm not mistaken, that each of the connections are fast enough (could be faster, but I think they should work). So it's really down to how I have my Qnap configured?

How would you setup the TVS-872XT from scratch? Which Raid array. Would you add cache? Would you upgrade any components (RAM)? Do you recommend Snapshots? 10GbE setup vs Thunderbolt? What's the best backup software option? Is there a comprehensive tutorial somewhere for setup specifically for Videographers? Do's and don'ts?

What if I took my TVS-472XT and installed 4 x "1TB Samsung SSD 860 EVO 1TB 2.5 Inch SATA II" (MZ-76E1T0B/AM)(that I have on hand) + the 2 NVMe M.2 Samsung 970 EVOs and configured them into a Raid 0 - 6TB storage pool. Then if I got a QNAP TR-004 ($270 on Amazon) and configured the Seagate 7200 SATAs into a Raid 0 (40TB) and connected it via USB 3.2 to the TVS-472XT. Would this setup work for 3 stations? Is there a better solution (using what I have, without spending a ton of money)?

Bob Zelin
Experience counts
Posts: 1105
Joined: Mon Nov 21, 2016 12:55 am
Location: Orlando, FL.
Contact:

Re: iSCS very, very slow...

Post by Bob Zelin » Fri Feb 05, 2021 7:52 am

Hi DP Wolf - I will try to answer all of your questions. A 2015 iMac is a thunderbolt 2 computer. I don't care what QNAP you are running, no matter how much money you spent. If your 10G adapter is a Sonnet Solo 10G T2, you are not going to exceed 350 - 400 MB/sec. I don't care if you are using a TS-453BT3, a TVS-472XT or a new TS-h686 - with 4 drives, you are not going to get 800 MB/sec. It's NEVER going to happen. I don't care what SSD or M.2 cache you add in there. I don't care if it's a Synology - if it's 4 drives, you are limited to the speed of the drives. You want fast - you buy an 8 bay or a 12 bay, from ANY manufacturer.
YES - for the TVS series with QTS and video editing applications - yes, I am telling you from experience that Cache is worthless. Yes - even with a TVS-872XT and eight 7200 RPM drives, adding the 2 M.2 as Qtier or Cache is worthless, not unless you have a huge photo library, or huge sound effects library. For conventional 4K editing - it's worthless.

At the beginning of this post, you said you have a 2015 iMac, and now once again, you are saying you have a M1 Mac Mini. then you say you have a 2017 iMac (which is thunderbolt 3) - and then you say over thunderbolt 2 - so WHICH IS IT - you are making no sense in your own post.

In 2021, since you said you had a TVS-472XT, and not a TVS-872XT - you would not buy a TVS-872XT - you would purchase a TS-h1288X if you wanted an 8 bay.
So once again - do you have a TVS-472XT or a TVS-872XT ? With a TVS-872XT for professional video editing, you install all 8 drives in a single static RAID group - either RAID 5 or RAID 6 (I prefer RAID 6 for protection). No Qtier, no caching, no M.2 drives. Everything over 10G. No snapshots. No thunderbolt - all 10G. Have 3 editors - get a QSW-M1208-8C 10G switch, and thunderbolt to 10G adpaters for all your workstations.
For thunderbolt 3 computers, you use the QNAP QNA-T310G1T. For thunderbolt 2 computers - you are now stuck with the Sonnet Solo 10G T2. IF you have a thunderbolt 2 computer and a Sonnet Twin 10G or Promise SanLink2, you disable Apple SIP to get these to run, and now you will get 800 MB/sec (but not with a Solo 10G T2).

There is no comprehensive tutorial for videographers. You hire someone like me to set it up for you. Just like I do for hundreds of professional video editors.

You keep saying "what if" - and you want to keep throwing money at your TVS-472XT to get it to do 800 MB/sec by spending money with NVMe drives, and a QNAP TR-004. ITS NOT GOING TO HAPPEN.
You should have purchased a TVS-872XT. Today, you would purchase a TVS-h1288X (with eight matching drives, 2 480 Gig SSD's for the OS) and a 10G switch like the QSW-M1208-8C ($599) and the 10G adapters (all under $200 each).

And you should have hired me. Everything would be working, and you would be an expert in all of this. SO now - how do you get your TVS-472XT to do 800 MB/sec for 3 workstations so that 3 guys can do 4K editing at the same time - simple answer - YOU DONT. ITS NOT GOING TO HAPPEN. Not even if you hire me.

Bob Zelin
Bob Zelin / Rescue 1, Inc.
http://www.bobzelin.com

DP-Wolf
New here
Posts: 3
Joined: Fri Jan 22, 2021 3:47 am

Re: iSCS very, very slow...

Post by DP-Wolf » Fri Feb 05, 2021 10:19 am

Bob,

I don’t know why I kept typing 2017, cuz the iMac is 2015.
Thanks for the information. How much do you charge, and can it be setup remotely?

Bob Zelin
Experience counts
Posts: 1105
Joined: Mon Nov 21, 2016 12:55 am
Location: Orlando, FL.
Contact:

Re: iSCS very, very slow...

Post by Bob Zelin » Sat Feb 06, 2021 3:43 am

you email me at
bobzelin@icloud.com

yes, everything is done with www.teamviewer.com
Just like QNAP uses.
Bob Zelin
Bob Zelin / Rescue 1, Inc.
http://www.bobzelin.com

m_00_m
Starting out
Posts: 10
Joined: Tue Apr 20, 2021 5:32 pm
Location: FRANCE

Re: iSCS very, very slow...

Post by m_00_m » Tue Oct 12, 2021 10:17 pm

Hi,

My iSCSI is not slow but it's not symmetrical.
I have a TVS-h1288X with 64 GB memory running on QTS 4.5.4.1800. I have a LUN like this:
Network.PNG
Sorry for the interface language ;-)
My PC (Windows 10) is connected to the NAS via a QSW-M1208-8C with copper Ethernet. My PC Network interface card is an Intel Ethernet Converged Network Adapter X550-T2.
Here are the result of a test with AJA:
AJA.png
Here is the result with Blackmagic Disk Speed Test:
BlackMagic.png
I did the same test with a direct connection between PC and NAS, the result are quite the same.
Can someone tell me if it's normal to have a so big difference between read and write?

Maurice.
You do not have the required permissions to view the files attached to this post.
Thank you for sharing

Post Reply

Return to “iSCSI – Target & Virtual Disk”