QNAP with Veeam getting slow

FTP Server, File Server, DDNS, SAMBA, AFP, NFS
Post Reply
bouda05
New here
Posts: 5
Joined: Mon Jan 07, 2019 2:19 pm

QNAP with Veeam getting slow

Post by bouda05 » Mon Jan 07, 2019 8:09 pm

Hi,

I have QNAP TS-873U-RP with Veeam Backup 9.5 U3. QNAP is connected with 1x LAN to management and 2x 1 Gbps teamed LAN for file transfer. When I freshly installed QNAP and Veeam Backup, backup jobs run around 130 - 140 MB/s which was great. Unfortunately, after some days speed down crease to 20-30 MB/s, which is bad. Veeam logs shows, that bottleneck is on backup target. I tried reboot QNAP and speed was again around 120 MB/s. But after some days, speed drops to 20 MB/s.
I made complete smart test of HDD, test network connectivity, but everything seems well.
QNAP is filled by 8x 10 TB Seagate IronWolf in RAID 5 without SSD cache. Veeam accessing storage via fileshare, not iSCSI.

Do you have any clue what is wrong and why is speed dropping?

User avatar
storageman
Experience counts
Posts: 4195
Joined: Thu Sep 22, 2011 10:57 pm

Re: QNAP with Veeam getting slow

Post by storageman » Mon Jan 07, 2019 9:08 pm

What read speeds are you getting, test in SSH/Putty?

qcli_storage -t -> filesystem performance test
qcli_storage -T -> disk/raid performance test

What kind of Veeam backups/status of jobs?
How full is the unit?
Can you run a direct test between Veeam server and Qnap just to see if there is any difference from running through switch?
Also if using Jumbo frames, turn off as a test.

bouda05
New here
Posts: 5
Joined: Mon Jan 07, 2019 2:19 pm

Re: QNAP with Veeam getting slow

Post by bouda05 » Mon Jan 07, 2019 10:09 pm

Now is the Qnap before reboot, so it is slow in backups. I did test on Qnap in this state.
Filesystem performance: 390MB/s
RAID performance: 712 MB/s
Disk performance: 212 MB/s

We are using incremental backup with 30 day retention and full backup every week. Unit has 25 % of free space.
Jumbo frames are turned off.
When I try sequential copy from Windows, I can get speed around 70 MB/s.

If I copy something between backup server and other part of environment, it works on highest speed, so the switch is not the problem.

User avatar
storageman
Experience counts
Posts: 4195
Joined: Thu Sep 22, 2011 10:57 pm

Re: QNAP with Veeam getting slow

Post by storageman » Mon Jan 07, 2019 10:53 pm

SSD cache will not help.

Possible the read speed is too slow to work out the incrementals.
If the Veeam server has fast processor try doing a small test using ISCSI LUN block.

bouda05
New here
Posts: 5
Joined: Mon Jan 07, 2019 2:19 pm

Re: QNAP with Veeam getting slow

Post by bouda05 » Mon Jan 07, 2019 11:07 pm

Veeam server is virtual on VMware with 16vCPU and 16 GB RAM. It is weird, because I have this Qnap almost one year and this issue start in December 2018. Every time, when I reboot QNAP, I get full speed, but after some days, data transferring getting slower and slower. I tried different settings in Veeam, also I have latest firmware on QNAP.
I have same box in different location with Veeam backup and backups are really fast. Speeds around 160 MB/s. I really do not understand, what is wrong with this. I was suspect HDD is bad, so I check latency of every HDD, but all drives have same latency around 30 ms. I am stucked in this time

User avatar
storageman
Experience counts
Posts: 4195
Joined: Thu Sep 22, 2011 10:57 pm

Re: QNAP with Veeam getting slow

Post by storageman » Tue Jan 08, 2019 12:31 am

Veeam in a VM bad idea, it will affect performance in doing compares.
Much better in bare metal server.

bouda05
New here
Posts: 5
Joined: Mon Jan 07, 2019 2:19 pm

Re: QNAP with Veeam getting slow

Post by bouda05 » Tue Jan 08, 2019 12:42 am

Disagree, I can set better HW parameters than on bare metal. I have many installations in VM (it must be, because of Dell VRTX) and all installation exceed 400 MB/s with 10 Gbps NAS. In other cases I can get around 160 - 180 MB/s which is enough for backup.
Unfortunately, I do not know, what to check more. It looks like software problem in NAS. I opened case in Qnap, we will see-

User avatar
storageman
Experience counts
Posts: 4195
Joined: Thu Sep 22, 2011 10:57 pm

Re: QNAP with Veeam getting slow

Post by storageman » Tue Jan 08, 2019 1:10 am

bouda05 wrote:
Tue Jan 08, 2019 12:42 am
Disagree, I can set better HW parameters than on bare metal. I have many installations in VM (it must be, because of Dell VRTX) and all installation exceed 400 MB/s with 10 Gbps NAS. In other cases I can get around 160 - 180 MB/s which is enough for backup.
Unfortunately, I do not know, what to check more. It looks like software problem in NAS. I opened case in Qnap, we will see-
Yes but NAS SMB shares are not great for Veeam. Veeam needs fast performance NAS, especially if large image files

I would:
1)run a Iometer sequential write test on VM to Qnap
2)run a Iometer sequential write test from bare metal server to Qnap
Repeat with ISCSI LUN on NAS
Leave it running for at least 30 mins and see if performance drops over times.

Sorry but, I'd be surprised if Qnap come back to you with anything sensible so good luck there.

bouda05
New here
Posts: 5
Joined: Mon Jan 07, 2019 2:19 pm

Re: QNAP with Veeam getting slow

Post by bouda05 » Fri Jan 11, 2019 7:44 pm

I run test for sequential Read/Write on QNAP with 1 TB file.
Write started on 130 MB/s and after some minutes drops to 42 MB/s
Read was simillar.

Test for small 4 KB files give result around 1200 IOPS for read and 700 IOPS for write.

I also check network infrastructure during the test and I did not notice any latency.

User avatar
storageman
Experience counts
Posts: 4195
Joined: Thu Sep 22, 2011 10:57 pm

Re: QNAP with Veeam getting slow

Post by storageman » Fri Jan 11, 2019 9:42 pm

bouda05 wrote:
Fri Jan 11, 2019 7:44 pm
I run test for sequential Read/Write on QNAP with 1 TB file.
Write started on 130 MB/s and after some minutes drops to 42 MB/s
Read was simillar.

Test for small 4 KB files give result around 1200 IOPS for read and 700 IOPS for write.

I also check network infrastructure during the test and I did not notice any latency.
Tested from where, inside a VM?
IOPs test is not really relevant to Veeam
Tested it again from a bare metal server just to humour me.

Post Reply

Return to “File Sharing”