Daily NAS to NAS Backup Slowing Down

[saltworkstech]

Hi all,

I've seen similar questions posted in the past but I don't think any of them apply to my exact needs.

We have two NAS that we use for storing data and work at work. The backup NAS is placed at our secondary site away from our primary site.

Main NAS: TS-879U-RP
8GB RAM
Firmware 4.1.1 Build 20141101
8x Western Digital red 4TB HDDs in RAID6
Site speedtest results: 35.92 Mbps down, 4.21 Mbps up

Backup NAS: TS-1279U-RP
4GB RAM
Firmware 4.1.4 Build 20150820
12x Western DIgital red 4TB HDDs in RAID6
Site speedtest results: 12.39 Mbps down, 5.43 Mbps up

We set up schedules for the main NAS to initiate NAS-to-NAS file-level backups on several folders to the backup NAS at different times during a day, outside of working hours. (ie. folder 1 at 8pm, folder 2 at 9pm, etc.)

The configuration have worked well for a while, but over the past two weeks I have noticed that the backup have slowed down significantly - Larger folders (5TB each) are showing >5000 hours remaining, while smaller folders (466 GB) showed 300 hours to complete. Note that all these backups are jobs executed weekly so there should not be significant differences in files. Also some of the scheduled backups have failed halfway with the error "target folder is missing", even though the folder is there on the target backup NAS.

This is the second time we had the same issue. The last time this happened, we had to move the backup NAS to the same site as the main NAS and leave them connected in the same network for a week before the backups were up to speed. Is there a different solution that does not involve transporting the backup NAS to the same site as the main NAS?

[dolbyman]

how are the sites connected ?

[saltworkstech]

how are the sites connected ?

Both sites are connected to Internet through the same local business ISP.

If it helps, straight-line distance between the two sites is about 12km.

We have VLAN set up to include both sites. Main site is in the 10.70.XX.XX network while the backup is in the 10.72.XX.XX network.

[dolbyman]

just making sure that there is no vpn tunnel, where cpu gets pummeled etc.

[saltworkstech]

just making sure that there is no vpn tunnel, where cpu gets pummeled etc.

No VPN tunnel.

As of right now (6 backup jobs ongoing):

Main NAS is at 20% CPU 65% RAM used
Backup NAS is chillin' with 1% CPu and 22% RAM usage

[dolbyman]

i meant the cpu of whatever is creating the tunnel

[saltworkstech]

If by that you meant the Untangle server that essentially acts as the router upstream of the entire building... The load has been consistently at <1%.

[dolbyman]

I have no idea how your network infrastructure looks like, I am just trying to find a cause for the "slowdown"

There is a LOT that can go wonky in interconnecting sites and it might not have anything to do with the QNAP systems

Did you test site to site transfer speeds ? I only see line speed-test results in your first test. If your ISP peers internally the speeds might be different (higher) than the internet speeds

[P3R]

The last time this happened, we had to move the backup NAS to the same site as the main NAS and leave them connected in the same network for a week before the backups were up to speed.

This indicates that your very slow up link of 4.21 Mbps is a bottleneck and that it simply can't cope with the volumes of changed data that you're having a need to transfer.

Is there a different solution that does not involve transporting the backup NAS to the same site as the main NAS?

None that I can think of when in this state.

I would move it again and allow them to catch up. Then you could try to enable the option Activate file compression if you don't use that already. Depending on what type of data you have, it may make things better but it's also possible that you need to rethink or redesign this solution completely if it's not possible to upgrade the bandwidth on the site-site connection.

Not that I think it's the issue here but the QTS firmware you use is very old. Personally I would get it up to date and also have them on synchronized firmware and versions and build dates.