2008R2 Hyper-V slow iSCSI and TS-419u

[RedPlanet]

Hi,
I am having trouble getting a good throughput on my Hyper-V iSCSI system.

The system hosts 8 VMs for software deployment testing, a mix of Windows 7 x86 & x64 and Windows Server 2008R2

The system is getting an average iSCSI network speed of 1 to 3 MBps with bursts of 5-7 MBps and the NICs are showing under 1% utilization most of the time.
All NICs are running at (link speed) 1gbps.

The QNAP and the server are not being stressed by the load and I cannot find out why

I have attached screen-shots to help visualize the issue.


Any help with optimizing the configuration would be great.


Hardware:
QNAP: TS-419u / 4 x 2TB 7800rpm RAID 6
Switches: 2 x Netgear GS108
Host: 1 x HP DL165 G7 - Windows Server 2008R2 Enterprise

Network Setup:
4 ports via HP NC362i Integrated DP Gigabit Server Adapter
192.168.1.19 - LAN Management - Switch 1
192.168.1.20 - iSCSI - Switch 1
192.168.1.21 - iSCSI - Switch 2
4th Port used in Hyper-V Virtual Network - Switch 1

4 ports via Intel(R) Ethernet Server Adapter I340-T4
192.168.1.22 - iSCSI - Switch 1
192.168.1.23 - iSCSI - Switch 2
3rd Port unused
4th Port used in Hyper-V Virtual Network - Switch 2

192.168.1.1 - QNAP Port 1 - Switch 1
192.168.1.2 - QNAP Port 2 - Switch 2

All NICs have Jumbo Frames to 9000 and all tasks are Offloaded


iSCSI Network:
192.168.1.20 to QNAP Port 1 - Switch 1
192.168.1.22 to QNAP Port 1 - Switch 1
192.168.1.21 to QNAP Port 2 - Switch 2
192.168.1.23 to QNAP Port 2 - Switch 2

iSCSI is running MPIO with policy "Least Queue Length"

Windows Task Manager and Perf Mon

iSCSI Slow.png

QNAP NIC Usage

QNAP NIC Usage.png

QNAP CPU Usage

QNAP CPU Usage.png

QNAP RAM Usage

QNAP RAM Usage.png

[storageman]

I would never put 8 VMs on a 419, far too slow. These are backup/file storage NAS. VMs, you need to think about 459s and above.

... you get what you pay for.

[RedPlanet]

The TS-419u was originally used for File Shares but recently had been unused and re-tasked for testing.

We usually only operate one or two of the VMs at once as this is only for software deployment testing and not our main system.

I was just hoping to get a bit more out to the unit rather than just leaving it unused.

[storageman]

Does seem incredibly slow.
Did you go back to basics to see if it's a networking issue, run direct to host, turn off jumbo frames, test with iometer on vm, etc

[RedPlanet]

I have just tried a direct connection to the QNAP and the speed was an average of 3MBs and bursts of 10MBs which is faster as the average was 2MBs with bursts of 7MBs

The NICs showed a usage of 2%

I have tried Jumbo Frames on and off and all Offloads on and off.

and I have dropped the NICs down to a 1-to-1 with the QNAP.


I reactivated the Windows File Sharing and copied a 500MB file to test the connection and got an average speed of 21MBs so I tried to copy the same 500MBs file direct to the iSCSI drive and got a speed of 31MBs and the QNAPs CPU hit 60%
NOTE: the 31MBs was bursting - 1 second at 31MBS and 1 second as 0MBS according to "Resource Manager" in Windows Server 2008 R2

The speed issue only effects the Hyper-V VHDs and the AV is excluding that drive
Right now I have 2 VMs running for these QNAP tests and I am getting 2MBs I/O and 100% Highest Active Time and Disk Queue Length of 8.3