S3 app vs Glacier app

[NAS_hans]

TS-253A / QTS 4.2.2

I've viewed many AWS tutorials, and they all seem to indicate that S3 is required to be able to use Glacier (via lifecycle rules). In other words, S3 is the interface to approach Glacier.

Now, in the QNAP app library, I see a dedicated S3 app and dedicated Glacier app. Looking at the release notes for both apps, I can't work out what the difference is, and if I need S3 app PLUS Glacier app, or if only Glacier app is sufficient.

What is your experience?

[rjspring]

TS-253A / QTS 4.2.2
What is your experience?

You don't need both. You can upload to S3 and set the lifecycle there for it to automatically push your data to glacier or you can use the glacier app to push directly to glacier.

Just be aware the two systems are different, and restores will be much different.

I've been using glacier for over a year as my offsite storage. Luckily I've never had to restore from it, but it is there just in case.

[NAS_hans]

Thanks for that reply, clear.

I am now using only the Glacier backup, so without S3. Just because it's cheaper. I would only need these files in case of disaster recovery, and am fine to wait a week or so for restore from Glacier.

However I have 2 additional questions, curious to see answer from forum users:

1. If I select the option to use SSL, does that encrypt the data between NAS and Glacier, even if I have not purchased a SSL certificate? How safe is that SSL option in Glacier backup app? Do I need to take additional security measures?

2. How can I browse files in Glacier? I tried using FastGlacier and Cloudberry (windows version, installed on my laptop) but they don't seem to be able to retrieve the inventory (files + folders with original file names) when files were uploaded using Glacier backup app. What is your experience on this? Suggestions?

Thanks!

[lehrblogger]

@NAS_hans, I only have a limited understanding of these things, but I'll try to answer as best I can.

1. The SSL option does encrypt the transfer of data between your NAS and Glacier. I'd assume it works like a webpage that uses HTTPS rather than plain HTTP, and would encrypt the data with Amazon's SSL certificate prior to transport. The encryption means more computation work for your NAS, which is presumably why it's an optional setting.

Note that there is a separate setting for encrypting the files themselves before they are sent to, and stored in, Glacier. That would theoretically prevent someone from reading them even if they had access to your Amazon account. I've had trouble decrypting data stored that way, however, so be sure to test that before relying on it.

2. I've used the Freeze app for macOS with some success. I don't think you can actually browse the files on Glacier – you can only look to see what you have in the inventory, and then download specific files. This is pretty cumbersome, but the constraints make sense if Glacier is indeed using magnetic tape.

[NAS_hans]

@lehrblogger, thanks for the info.

Regarding point 2, all I want is to see the inventory in my Glacier vault, so I can check at any time which files and folders exist in my vault. Since Amazon does not provide that feature from the standard Glacier interface, I need a 3rd party tool (indeed like Freeze as you mention). Issue is that the 2 tools I have tested (Cloudberry and FastGlacier) are not able to list the inventory for files that are uploaded using the Glacier app offered by QNAP.

I've contacted QNAP support and they suggested I try Hybrid Backup app instead of the Glacier app.

So I will try that and post the outcome here later.

[lehrblogger]

@NAS_hans, ah, I see the problem now with Cloudberry and FastGlacier. One thing I noticed when trying to get Freeze to work was that the QNAP Glacier app was using what appeared to be a non-standard JSON format for specifying the file paths. For example, here's an item I copied from the ArchiveList object in the retrieved JSON, after the Freeze developer told me where to look for it on my hard drive:

Code: Select all

{
   "ArchiveId": "4zEwUv9j2OyAOwIUJV-MxPkGK3nB4R_yg3n8WKWVYLXkePMh_WzMx8Kef-BdB3RmUoHDrZC6yDEqWAb7QDO53K3CZasza2uXcms5O5yIA8TdemlqWf3uNdcXoKk7BlyQAxRPSub0Lw",
   "ArchiveDescription": "{\"path\": \"/Backups/Macintosh HD.sparsebundle/bands/3280\", \"type\": \"file\"}",
   "CreationDate": "2016-05-01T10:00:59Z",
   "Size": 8388640,
   "SHA256TreeHash": "2f8b7e6f86ed0faba09527a738f00c2a5097f166c7d3de6cc6a1599b39f28527"
 },

The unescaped ArchiveDescription is:

Code: Select all

{"path": "/Backups/Macintosh HD.sparsebundle/bands/3280", "type": "file"}

The Freeze developer was responsive and nice enough to add support for this format, but it's possible the escaped JSON is confusing Cloudberry and FastGlacier. Perhaps one/both of them has a raw inventory file downloaded and stored on your machine somewhere, that you could inspect manually.

This is the first I've heard of the Hybrid Backup app, so I'm very curious to hear how it goes!

[NAS_hans]

Hi,

Just a short status update:

I did the following since my last post:

1. I have contacted QNAP support and sent my diagnostic logs + screenshots of the issue at hand. Developers are working on this issue.

2. Tested the current version of Hybrid Backup app. It does not resolve the issue of garbled filenames / lack of folder structure etc. when viewing Vault inventory in FastGlacier / Cloudberry. I see no change compared to using the Glacier app.

3. Tested the Cloudberry for QNAP app (via manual install). That app does provide correct folderstructure and filenames. However it adds a system-created folder above the lowest level (i.e. above each individual file) so the Vault inventory becomes quite confusing. Not what I'm looking for.

4. Tested Hybrid Backup app with compression set to yes + only backup modified files. That gives me yet another issue, which is that Chinese (or Taiwanese?) characters are included in the file names when listing Vault inventory in FastGlacier. I will leave that for now.

In summary, no workable solution yet, looking forward to the solution from QNAP dev's.

[Moogle Stiltzkin]

TS-253A / QTS 4.2.2

I've viewed many AWS tutorials, and they all seem to indicate that S3 is required to be able to use Glacier (via lifecycle rules). In other words, S3 is the interface to approach Glacier.

Now, in the QNAP app library, I see a dedicated S3 app and dedicated Glacier app. Looking at the release notes for both apps, I can't work out what the difference is, and if I need S3 app PLUS Glacier app, or if only Glacier app is sufficient.

What is your experience?

found an article here for you
http://cloudacademy.com/blog/amazon-s3- ... the-cloud/

[NAS_hans]

Thanks @Moogle Stiltzkin

Maybe, looking at the issues I have described in this post, and the article from cloudacademy, the best option would be to use S3 as target for QNAP Hybrid Backup app, instead of Glacier. Then, using LifeCycle rules, move my files directly to Glacier after upload to S3.

From this article I understand that S3 objects with Glacier storage class will be more expensive than Glacier archives. So it will cost more in terms of data storage and transfer S3 to Glacier. On the other hand I haven't seen all these annoying issues yet when uploading to S3.

I would then need to test if an inventory retrieval using S3 with Glacier storage class is better than inventory retrieval using Glacier archives directly.

Has anybody used that setup, and what is your experience?