Automatic file long distance transfer - limited bandwidth - prioritized folders

[arnemeister]

Hi
Most likely it’s possible to read a lot and try a lot and figure this out myself, but I’ve learned the hard way to ask the adults for some help, and not try and fail to much.
I want to use two Qnap TS-432XU-RPs for a automatic file transfer of low res audio/video files from a remote location (Mexico) to editing facilities at home (Norway).
I see that there are many ways of doing this, rsync, RTRR etc, and several apps for QNAP that can do this.
I have a total of six incoming video streams in a 800kbps codec, usually 10 to 60 min recordings, and I’m planning to writing them directly to separate folders on the NAS on site in Mexico, and set up a automagic file transfer to the NAS at home, so logging and editing can start as soon as possible without manual file handling via ftp or torrents as we have used before.
Bandwidth in Mexico is limited, and the connection is not very stable. Usually it will be plenty for these files, but not always, so I wish to prioritize one of the folders, as it will contain the files we really need to start logging as soon as possible.
Can anyone suggest a simple to set up, and rugged way to do this?
Simple, as sorry, I’m no specialist. Used to program 6502s and Z80s in hex and binary code in the early nineties, but I’m terribly outdated, and more at an advanced but a bit outdated user level, and not geek level anymore, as I have moved from computers and networking to video solutions years ago.
Rugged, as yes – I want this to just work, without too much hassle, and auto restart transfer if it should fail when the connection is down.
So far I’ve looked at Hybrid Backup Sync, and it seems to work on my limited tests on a local network here at home, but I have not found a way to prioritize my main folder.
Some additional information:
I have no fixed IP on location in Mexico.
I’m not even sure I will get a fixed IP here at home (working on it, but need to plan for a dynDNS solution)
At home I can set up port forwarding and manage the firewall/router, but in Mexico I’m afraid it will be a local DSL with 4G failover, and I’m afraid I cannot configure the routers/modems at all.
I want a strict one way replication, where whatever is on the NAS in Mexico is replicated to the one at home, unless it’s already there. If removed form Mexican NAS, it should not be deleted from the one at Home.
Any good suggestions on how to set up a good teleporter?

[Bob Zelin]

Hi -
6502, Z80 - you are old (it's ok - I am old) -
you do not need a fixed IP in Mexico, you do not need a fixed IP at your home in Norway.
Hybrid Backup Sync is what you want, as you have discovered. You do not need to know any programming.
Because you said "if removed from Mexican NAS, it should NOT be deleted from the one at home" - this means you need a BACKUP or one way sync, and not a two way sync.
You can select Real Time Replication, (RTRR) or Rsync.
The TRICK to making this work is not in Mexico on the main unit. the TRICK is on your BACKUP system in NORWAY. If you do NOT know how to open ports on your router, then you will fail.
The ports on the router have to be opened on the remote BACKUP end (in Norway). The default port for RTRR is 8899. The default port for Rsync is 873.
On your QNAP at home, you setup MyQnapCloud (do this in Mexico too). You need to create a free account to do this. If no one in Mexico can do this for you, you use a remote program like Teamviewer to access
their computers, and you set it up there for them.
At home, you go into my QNAP Cloud and you see what your DDNS address is (that is the IP that your Internet Service Provider gave you - it does not need to be a static IP address). You can also use
whatismyip.com to get this information. On your backup QNAP in Norway you enable RTRR or Rsync. Make the password simple, so it's not an ordeal to do this.
Because I don't know all routers, and many of them are confusing, I use a simple cable modem with no blocked ports, and a Synology RT2600ac router, which is a super easy UPnP router, that even if you want to do
your own port forwarding, it's easy. But the QNAP MyQNAPCloud will do this for you if you enable Rsync or RTRR and click on those services in MyQnapCloud and click on APPLY TO ROUTER. It will open those ports
for you. (that is why I love the Synology Router). I guess a Netgear Nighthawk will do the same thing.


On the system in Mexico (if I did this for you, I would not rely on them, and use Teamviewer to remote into their system) - you setup your MyQNAPCloud, and setup Hybrid Backup Sync.
Create a REMOTE JOB and enter the DDNS address (the one at your home, or that you got from the QNAP or "whatismyip") - and your name and password for your QNAP. You select the source folder (the one in Mexico) and the destination folder (the one in Norway) and tell it to sync. You can schedule this, or you can select BACKUP NOW. the QNAP part is easy. The part that is not easy is getting the ports on the router opened in Norway to make this work.
I do this ALL THE TIME, and I just did a system yesterday, that uses a Frontier FIOS box, and a Linksys AC1750 router. I configured the router properly, but apparently the Frontier FIOS box has BLOCKED PORTS so I can't get anything to work. Without the cooperation of Frontier (who gets terrible reviews for support as per the US websites) - there is nothing I can do to get this to work.

I can go into details about the setup, but even if you did everything correctly, if your internet service provider box is blocking ports, and you can't open up 873 and 8899 - then this will never work, and you will be frustrated.

As far as prioritizing the folders - I have no idea. You tell it to backup, and it backs up. You can do one folder, or all the folders, and you can setup schedules on when each folder backs up. \
In my opinion, your #1 concern is getting your router ports opened (873 and 8899) because if you can't get this to happen, and everything else is setup 100% correct - this is never going to work.
And this is not a QNAP issue - this is an issue with the ISP box. My ideal situation is to provide a simple cable or DSL modem that blocks no ports, and then use a Synology Router (like the RT2600AC) which the QNAP
will auto configure for this process. but I know that in REAL LIFE, you may not be allowed to provide your own cable box, and you have to deal with the aggravation of the cable company.

In the US, because the Arris Surfboard modem/routers are so popular, I put a LOT of effort and suffering out, to figure out how to get these to work. But even though some people consider me an "expert" now, I failed just the other day with the Frontier FIOS box.

There are a lot of variables, that may be out of your control (and QNAP can't address for you).

Bob Zelin

[arnemeister]

Thanks Bob

Old - nah, I'm younger than ARPANET

And being Norwegian, I might express myself a bit different from native English speakers, so maybe I should clarify a bit more.

I'm not entirely lost with ports and the joy of modern network, after all I wrote some fun papers on the OSI model once, and I did design and build my own first routers and switches. For video we also play with stuff like the nice little Nimbra 688 160Gbps switches, so as long as the Mexican cable or 4G modems don't block ALL outgoing traffic, or route it through a nasty contraption to inspect what I'm sending on what ports, and block on service /content level, I'm pretty sure I can route a RTRR or Rsync all the way home, even if I will have to use a VPN tunnel on port 80.
If I still can't get it all the way, then I will push it to some cloud from the Mexican Nas, and pull it off the cloud at home. Should be doable with or without control of port forwarding, but it will complicate all, and I really prefer simple stupid.

I just don't know the tools the Qnap has to offer.
Seems to be many quite similar sync /backup / replicate possibilities, and I'm pretty sure some of them are more suited than others, and that there are a lot of small details that can and will affect stability and performance.

[mangoon2008]

I would recommend to give resilio sync a try for your Problem...
I use it for years now and it JUST WORKS!
Without opening ports and exposing your NAS to the world, without hassle its fast very robust (technologically based on Bittorrent) and very very User friendly....