QNAP with Veeam getting slow

FTP Server, File Server, DDNS, SAMBA, AFP, NFS
Post Reply
bouda05
New here
Posts: 7
Joined: Mon Jan 07, 2019 2:19 pm

QNAP with Veeam getting slow

Post by bouda05 »

Hi,

I have QNAP TS-873U-RP with Veeam Backup 9.5 U3. QNAP is connected with 1x LAN to management and 2x 1 Gbps teamed LAN for file transfer. When I freshly installed QNAP and Veeam Backup, backup jobs run around 130 - 140 MB/s which was great. Unfortunately, after some days speed down crease to 20-30 MB/s, which is bad. Veeam logs shows, that bottleneck is on backup target. I tried reboot QNAP and speed was again around 120 MB/s. But after some days, speed drops to 20 MB/s.
I made complete smart test of HDD, test network connectivity, but everything seems well.
QNAP is filled by 8x 10 TB Seagate IronWolf in RAID 5 without SSD cache. Veeam accessing storage via fileshare, not iSCSI.

Do you have any clue what is wrong and why is speed dropping?
User avatar
storageman
Ask me anything
Posts: 5507
Joined: Thu Sep 22, 2011 10:57 pm

Re: QNAP with Veeam getting slow

Post by storageman »

What read speeds are you getting, test in SSH/Putty?

qcli_storage -t -> filesystem performance test
qcli_storage -T -> disk/raid performance test

What kind of Veeam backups/status of jobs?
How full is the unit?
Can you run a direct test between Veeam server and Qnap just to see if there is any difference from running through switch?
Also if using Jumbo frames, turn off as a test.
bouda05
New here
Posts: 7
Joined: Mon Jan 07, 2019 2:19 pm

Re: QNAP with Veeam getting slow

Post by bouda05 »

Now is the Qnap before reboot, so it is slow in backups. I did test on Qnap in this state.
Filesystem performance: 390MB/s
RAID performance: 712 MB/s
Disk performance: 212 MB/s

We are using incremental backup with 30 day retention and full backup every week. Unit has 25 % of free space.
Jumbo frames are turned off.
When I try sequential copy from Windows, I can get speed around 70 MB/s.

If I copy something between backup server and other part of environment, it works on highest speed, so the switch is not the problem.
User avatar
storageman
Ask me anything
Posts: 5507
Joined: Thu Sep 22, 2011 10:57 pm

Re: QNAP with Veeam getting slow

Post by storageman »

SSD cache will not help.

Possible the read speed is too slow to work out the incrementals.
If the Veeam server has fast processor try doing a small test using ISCSI LUN block.
bouda05
New here
Posts: 7
Joined: Mon Jan 07, 2019 2:19 pm

Re: QNAP with Veeam getting slow

Post by bouda05 »

Veeam server is virtual on VMware with 16vCPU and 16 GB RAM. It is weird, because I have this Qnap almost one year and this issue start in December 2018. Every time, when I reboot QNAP, I get full speed, but after some days, data transferring getting slower and slower. I tried different settings in Veeam, also I have latest firmware on QNAP.
I have same box in different location with Veeam backup and backups are really fast. Speeds around 160 MB/s. I really do not understand, what is wrong with this. I was suspect HDD is bad, so I check latency of every HDD, but all drives have same latency around 30 ms. I am stucked in this time
User avatar
storageman
Ask me anything
Posts: 5507
Joined: Thu Sep 22, 2011 10:57 pm

Re: QNAP with Veeam getting slow

Post by storageman »

Veeam in a VM bad idea, it will affect performance in doing compares.
Much better in bare metal server.
bouda05
New here
Posts: 7
Joined: Mon Jan 07, 2019 2:19 pm

Re: QNAP with Veeam getting slow

Post by bouda05 »

Disagree, I can set better HW parameters than on bare metal. I have many installations in VM (it must be, because of Dell VRTX) and all installation exceed 400 MB/s with 10 Gbps NAS. In other cases I can get around 160 - 180 MB/s which is enough for backup.
Unfortunately, I do not know, what to check more. It looks like software problem in NAS. I opened case in Qnap, we will see-
User avatar
storageman
Ask me anything
Posts: 5507
Joined: Thu Sep 22, 2011 10:57 pm

Re: QNAP with Veeam getting slow

Post by storageman »

bouda05 wrote: Tue Jan 08, 2019 12:42 am Disagree, I can set better HW parameters than on bare metal. I have many installations in VM (it must be, because of Dell VRTX) and all installation exceed 400 MB/s with 10 Gbps NAS. In other cases I can get around 160 - 180 MB/s which is enough for backup.
Unfortunately, I do not know, what to check more. It looks like software problem in NAS. I opened case in Qnap, we will see-
Yes but NAS SMB shares are not great for Veeam. Veeam needs fast performance NAS, especially if large image files

I would:
1)run a Iometer sequential write test on VM to Qnap
2)run a Iometer sequential write test from bare metal server to Qnap
Repeat with ISCSI LUN on NAS
Leave it running for at least 30 mins and see if performance drops over times.

Sorry but, I'd be surprised if Qnap come back to you with anything sensible so good luck there.
bouda05
New here
Posts: 7
Joined: Mon Jan 07, 2019 2:19 pm

Re: QNAP with Veeam getting slow

Post by bouda05 »

I run test for sequential Read/Write on QNAP with 1 TB file.
Write started on 130 MB/s and after some minutes drops to 42 MB/s
Read was simillar.

Test for small 4 KB files give result around 1200 IOPS for read and 700 IOPS for write.

I also check network infrastructure during the test and I did not notice any latency.
User avatar
storageman
Ask me anything
Posts: 5507
Joined: Thu Sep 22, 2011 10:57 pm

Re: QNAP with Veeam getting slow

Post by storageman »

bouda05 wrote: Fri Jan 11, 2019 7:44 pm I run test for sequential Read/Write on QNAP with 1 TB file.
Write started on 130 MB/s and after some minutes drops to 42 MB/s
Read was simillar.

Test for small 4 KB files give result around 1200 IOPS for read and 700 IOPS for write.

I also check network infrastructure during the test and I did not notice any latency.
Tested from where, inside a VM?
IOPs test is not really relevant to Veeam
Tested it again from a bare metal server just to humour me.
bouda05
New here
Posts: 7
Joined: Mon Jan 07, 2019 2:19 pm

Re: QNAP with Veeam getting slow

Post by bouda05 »

I made second test from our spare server R720 with test backup jobs. Results were quite similar. Write started at 144 MB/s and drops to 47 MB/s. It is the same situation like in VM.
QNAP support ticket does not bring any advices.
I think, that I will try to buy some 10 Gbps twinax cables and connect directly QNAP to server card. I do not have any ideas, what to set more on QNAP to speed up.
bouda05
New here
Posts: 7
Joined: Mon Jan 07, 2019 2:19 pm

Re: QNAP with Veeam getting slow

Post by bouda05 »

I finnaly summarize logs from QNAP and find this repeating issues:

Code: Select all

<6>[2416862.860327] ata9: hard resetting link
<4>[2416862.864166] ata9: SERR_PHYRDY_CHG SERR_DEV_XCHG. Clear the bit/burst error record
<4>[2416862.871781] ata9: SERR_PHYRDY_CHG SERR_DEV_XCHG. Clear the bit/burst error record
<3>[2416873.244610] ata9: softreset failed (1st FIS failed)
<6>[2416873.249641] ata9: hard resetting link
<3>[2416996.702734] ata5: exception Emask 0x10 SAct 0x0 SErr 0x4000000 action 0xe frozen
<3>[2416996.710266] ata5: irq_stat 0x80000040, connection status changed
<3>[2416996.716432] ata5: SError: { DevExch }
<6>[2416996.720267] ata5: hard resetting link
<4>[2416996.724102] ata5: SERR_PHYRDY_CHG SERR_DEV_XCHG. Clear the bit/burst error record
<4>[2416996.731730] ata5: SERR_PHYRDY_CHG SERR_DEV_XCHG. Clear the bit/burst error record
<3>[2417007.522483] ata5: softreset failed (1st FIS failed)
<3>[   95.666793] ata8: exception Emask 0x10 SAct 0x0 SErr 0x4000000 action 0xe frozen
<3>[   95.674176] ata8: irq_stat 0x80000040, connection status changed
<3>[   95.680157] ata8: SError: { DevExch }
<6>[   95.683811] ata8: hard resetting link
<4>[   95.687466] ata8: SERR_PHYRDY_CHG SERR_DEV_XCHG. Clear the bit/burst error record
<4>[   95.694911] ata8: SERR_PHYRDY_CHG SERR_DEV_XCHG. Clear the bit/burst error record
I am not sure, if this SATA errors points to issue with backplane or HDD. Smartctl of HDD does not show any issue.
hansssklosss
First post
Posts: 1
Joined: Sat Dec 10, 2016 7:11 am

Re: QNAP with Veeam getting slow

Post by hansssklosss »

Hi.
I had similar issue, 10Gb Ethernet, CIFS/SMB repository on QNAP 531X, 8MB Ram, Raid 6, backup from Veem starts with 300MB/s, after 20 min. 80 MB/s, finally (after about 2h) 15MB/s. Backup image was more than 3TB. Incrementals small backups were fast. Ive tested every setting both on veem side and QNAP side. No significant improvements. Yesterday removed static volume with 8K stripe and made volume on storage pool with max 64k stripe. After synchro., etc., today backup from veem is creating with 240MB/s without problems. It is same speed am achieving backuping to direct connected disk. Looks like problem was in stripe size, earlier QNAP processor was max. 11% busy now is 45%. It took some time, hope will help, sorry for my english.
MarcOStn
First post
Posts: 1
Joined: Fri Dec 13, 2019 10:20 pm

Re: QNAP with Veeam getting slow

Post by MarcOStn »

Hi All,
i'm experiencing a similar issue on my QNAP TS-1679U-RP...
the backup job is really fast with 400 MB/S data transfer on the network and 150 MB/S of write speed on disks.
the write speed suddenly drops to 5/8 MB/s during the backup merge feature.
any suggestions?
thanks
User avatar
storageman
Ask me anything
Posts: 5507
Joined: Thu Sep 22, 2011 10:57 pm

Re: QNAP with Veeam getting slow

Post by storageman »

Probably if you're server is fast I would try ISCSI and thick LUN
Post Reply

Return to “File Sharing”