Realistic usage cost per month? ~9TB

[binarypower]

I'd like to have a one way sync from my QNAP to Amazon S3 to prevent a "holycrapIlosteverything" situation.

I know that the IN costs are free, the volumes are limited to 500GB each (and cost money per disk?) and if there was a "holycrapIlosteverything" situation, I know would have to pay an arm and a leg to restore the data (which I don't mind for the assurance that my data is safe).

Can anyone provide me with examples of how much you are paying, how much data you are storing... any tips or alternatives for cheap backups for less than 10TB of data?

[shadowsports]

http://calculator.s3.amazonaws.com/index.html

I'm using AWS with JungleDisk at work. Glacier might be a better option given the amount of data you are looking to protect.

[Evadman]

S3:
9 TB data backup from you to Amazon: $0 (over internet)
9 TB data backup from you to Amazon using Import/Export: $157.19 + Drives (3TB for $80ish) + 1 way shipping. Total: $412ish
9 TB full restore from Amazon to a local (to you) backup device such as another NAS using regular download: $1214.54
9 TB full restore from Amazon to a local (to you) backup device such as another NAS using Import/Export: $157.19 + Drives (3TB for $80ish)+ 1 way shipping. Total: $412ish
9 TB Storage cost per month (magnetic): $460.80

So basically S3 cost is $400 for initial, $450 a month, and $400 for Disaster Recovery.

Glacier:
9 TB data backup from you to Amazon: $0 (over internet)
9 TB data backup from you to Amazon using Import/Export: Don't do this. It will be imported to Glacier as a single file per hard drive; meaning you can only download an entire drive worth of data at a time. This is important for the restore cost.
9 TB Storage cost per month: $92.16

Restore cost on Glacier is hugely dependent on how fast you want to restore it and the total amount of data you have stored. The max to 'retrieve' for free is 5% per month of all data you have in Glacier; spread across each hour of the month. with each retrieval request being the total data requested divided by 4 because the assumption that amazon makes is that retrieval takes 4 hours. That means you can 'retrieve' %0.17 per day if evenly spread throughout each hour of each day of a 30 day calendar month. Amazon does some of the spreading for you automatically since each request is billed at a 4 hour timeframe (though they can be made concurrently). That means every 4 hours (if perfectly timed) you can 'retrieve' 2.61 GB of data and fit within the free tier (no cost to 'retrieve') if you have 9 TB of data in Glacier. There is still the cost to transfer the data to you, which is defendant on availability zone, but is usually 9 cents per GB (Amazon just reduced the cost by 25%, see here: http://aws.amazon.com/blogs/aws/aws-dat ... reduction/ ). If perfectly timed, you can do a full restore over 20 months for a total cost of $829. If you go outside the free 5%, the retrieval is a penny per GB, but it isn't a straight penny because you are charged per hour for the busiest hour in the month. That means if you go over by 1 GB in any given hour you are actually charged that penny for every hour in the month even if you only requested data once. So that 1 GB actually cost you $0.01 * 30 days * 24 hours = $7.20. Read on for a better example.

If you use the import/export process to load the data, you will have three 3 TB files. That means a single request for a file will be 3 TB divided by 4 hours for 768 GB. You get a free 2.61 GB of that, which leaves 765 GB that was retrieved in that spike. Here's the kicker. If you go over the free tier, the remaining is charged for every single hour of the month. Meaning requesting that 3TB file is the same as requesting 765GB every single hour for the month. That puts the cost of that 3TB request to be 765GB * 30 days * 24 hours * $0.01 per GB putting the total cost at $5508 to retrieve that 3 TB file. It doesn't matter that you can't download it that fast, what matters is the request to retrieve the file. It would then cost an additional 9 cents per GB to transfer to you ($276.48). After that retrieval, your peak has been set for the month at 768 GB/hour so you can keep requesting that file (or the others) all month long and the cost would be 'free' since it was still equal to your peak hour usage. you would pay only the transfer cost of $276. It should be quite evident that spreading out the recovery time has a gigantic cost savings for Glacier.

A realistic recovery that more approximates a S3 data load timing would be doing the restore over a week since that would be closer to the time it would take to ship drives to Amazon, have Amazon load them, and ship them back to you. A week would be 9 TB requested over 168 hours (and require a 200 Megabit connection). That is 54.8 GB an hour for a peak cost of 52 cents. That puts the cost of retrieval at $374 then the 9 cents data transfer would put the total cost for recovery at $1193.

A realistic recovery that approximates a average home connection would be around 20 Megabits per second. That puts the total time for recovery at 1154 hours or 8 GB per hour. The retrieval for that is 5.4 cents/hr or $38.80 + bandwidth download at 9 cents ($829) for the full recovery cost of $868. Remember, since the cost is by peak of a request divided by 4 the file size has to stay under that magic number. Meaning a file size of 8 GB * 4 is the maximum before the peak is artificially inflated due to file size.

For this 9 TB size and a recovery time of 1154 hours, the max file size is 32 GB. If you have virtual machines or something similar (such as a 250 GB virtual hard drive image) no matter how you do it your cost is going to be the biggest file size divided by 4. So that 250GB file will cost ((250 GB / 4) - 2.61 GB) * $0.01 * 720 = $611 no matter how or when you download it from Glacier if you request the whole file at once. There are ways to download part of a file which is called a Range. Basically, you provide a byte count and offset to the Glacier API and you can download the file in pieces to spread it across time so you can stay within the magic free tier boundary. I don't know if free tools such as FastGlacier support Range downloads.

Glacier is cheap, but you have to know how to use it. Thankfully, there is a setting in AWS to set the max cost for Glacier requests. You can give it a maximum dollar amount (in $7.20 increments) or the best checkbox which is 'free tier only'. In that way, any request that would force you out of the free tier (in this case, more than 2.61 GB/hr) is rejected automatically. For each additional GB/hr, the cost is $7.20 per month (the increment mentioned earlier). So if you want 4.61 GB/Hr instead of 2.61 GB/hr, cost is $14.40 for the month which can be set in the AWS console as maximum. This limiter was added recently since a PEBKAC error can cause a MASSIVE bill. Imagine if a programmer was testing and accidentally put their code in a loop on the API and requested the entire set of all archives in a 1 PB company-wide backup within a few seconds. Actually on second thought, the cost of that mistake is about 1.7M.

Source: I use Amazon Web Services way too much for work and personal (including Glacier) and a quick check on their calculators. Check online for their current cost calculators to verify because I am bad at math.

[dsmithdewarcom]

Some 3 or 4 TB USB drives you keep off site would be cheaper, faster and more secure.

[Evadman]

Some 3 or 4 TB USB drives you keep off site would be cheaper, faster and more secure.

The same redundancy as glacier (2:1) can be achieved a few ways. If you use a straight up 2:1, the cheapest drive cost right now I can find is 3 TB for $85 or 2.8 cents per GB. Backing up the 9 TB 2:1 would be a total cost of $510. Then, you need to store the drives in a secure offsite location. 'Real' offsite would be something like Iron Mountain, which has cost dependent upon volume of the owner. For a one off like this, it would probably be around $50 a month to store one of their lock boxes, for a per-month-per-GB cost (over 3 year life expectancy of drives) of (($510 + ($50*36))/9216)/36 = 0.6 cents per GB per month. A restore would cost you a pull/ship from Iron Mountain which is pretty cheap (comparatively).

The cost can be lowered if you use something not meant for storing data/drives such as a lock box at a bank. Cost on those is around $1 per square inch, and you would need around 100 square inches, so the annual cost of a lock box would be in the $100 range. That puts the cost at (($510 + ($100*3))/9216)/36 = 0.25 cents per GB per month (about 75% less than Glacier). Recovery is free, but there is a better likelyhood on drives being dead when stored in a metal lock box in a bank that was designed to hold paper, jewelry and such; not specialized to hold hard drives. This can be remediated by buying a hard drive case from iron mountain or related for a hundred or so bucks. Better than losing a drive, and it only adds fractions of a cent to the data storage cost over the expected life of the drives.

The cost can be lowered even more if you remove 'secure' from the list of requirements. Put the drives in that storage box and store them at a friends house. That's basically free, or perhaps a case of beer or similar. Storing them in your office at work (if you have one) is also a possibility, though a lot of work places have issues with outside media being brought in due to security concerns.

What you lose in those cases isn't too much compared to glacier and/or S3. S3 allows you to restore that file you deleted on accident nearly instantly, and Glacier can do it in 4 hours, both for (basically) zero cost. However, that isn't really DR, that's just a normal backup process which you should be using your on-site backup for in the industry standard 3-2-1 backup strategy. You also lose the ability (or it gets much harder) to do incremental backups. After all, you don't want to restore your data from 3 years ago, you want to restore it from just before the crash. The harder it is to get the drives back and update them, the more likely that you won't do it. If it is a backup job that sends data offsite automatically, it is much more likely to be done at the frequency it needs to be done. Again, this can be remediated just like most other things. Discipline is a thing.

I'm not endorsing Glacier, S3, iron Mountain, safety deposit boxes, or a friends house; I'm just the math guy. Each person should review their needs and their specific requirements (including risk acceptance level, overall cost, and annoyance level) and make a decision based on their situation.

[dsmithdewarcom]

"If you use a straight up 2:1"

What is that? Two separate backups? Why?

[Evadman]

"If you use a straight up 2:1" What is that? Two separate backups? Why?

Drives die. If you are comparing apples to apples, Glacier stores 2 copies of everything (to the best of my knowledge) uploaded while Amazon S3 stores 3. So storing 1 GB of data on Glacier causes 1 GB to actually be written to 2 disks. For S3, data is stored 3 times in 3 data centers of Amazon's. S3 Reduced Redundancy is stored in 2 data centers. Apples to apples comparison would be the same redundancy of 2:1.

I included all the math and calculations, so you are free to calculate your own cost/benefit analysis using your own variables if you wish. If you want to use an external enclosure in RAID 5, Raid 6, RAID 0, or JBOD feel free to recalculate the cost for your use case.

[dsmithdewarcom]

Of course cloud vendors promise the moon on a stick: that's what they do.

The OP is looking for a low-cost backup solution that offers a measure of data security. Clouds don't do that. There are too many variables and failure points and, as you have amply demonstrated, it ain't cheap, even when it's "free". So (they tell you) there are "two" copies. Why only two, you may well ask? Well how many do you want? What if we say ten? Sure, ten. Whatever.

Cloud storage makes sense when geographically diverse access is required. Local access, not so much.

I think one RAID and one backup set will suffice for most. I also put more stock in physical possession than in cloud storage. Always fast access to data, regardless of what the network is doing/the cloud is down 'today', zero risk of hacks.

3 x 4TB drives for ~$300, make the copy, take the drives home. Done and done.

[supercool]

That is possibly the best thread response i have ever read. Thanks for cutting to the chase.

[marshalleq]

Most people on this list left one point out - Changed data. The reason GFS backups were invented. So depending on if you have regularly changing data or not, and if you can easily update that data onto these 'off-site' hard drives depends on your solution.

Here's my solution in case it helps.

Glacier for the smaller document backups (10-20GB) because it's just SOOOO reliable - in fact I'd consider it the only reliable QNAP backup solution at present.
An offiste NAS IF and only IF your data is painful to update and changes often
Some kind of GFS style solution such as rsnapshot (local or remote depending on your requirements) to allow you to 'go back'.

Going back is very important, especially when you want that document, financial database or whatever that corrupted 30 backups ago....

Anyway, that's my 2c.