S3 app vs Glacier app

Post Reply
NAS_hans
Getting the hang of things
Posts: 50
Joined: Mon Oct 03, 2016 9:19 pm

S3 app vs Glacier app

Post by NAS_hans » Sat Oct 15, 2016 9:16 pm

TS-253A / QTS 4.2.2

I've viewed many AWS tutorials, and they all seem to indicate that S3 is required to be able to use Glacier (via lifecycle rules). In other words, S3 is the interface to approach Glacier.

Now, in the QNAP app library, I see a dedicated S3 app and dedicated Glacier app. Looking at the release notes for both apps, I can't work out what the difference is, and if I need S3 app PLUS Glacier app, or if only Glacier app is sufficient.

What is your experience?

User avatar
rjspring
Starting out
Posts: 21
Joined: Tue Jul 29, 2008 2:53 am

Re: S3 app vs Glacier app

Post by rjspring » Sun Oct 23, 2016 4:34 am

NAS_hans wrote:TS-253A / QTS 4.2.2
What is your experience?


You don't need both. You can upload to S3 and set the lifecycle there for it to automatically push your data to glacier or you can use the glacier app to push directly to glacier.

Just be aware the two systems are different, and restores will be much different.

I've been using glacier for over a year as my offsite storage. Luckily I've never had to restore from it, but it is there just in case.
QNap - TS-453 Pro

NAS_hans
Getting the hang of things
Posts: 50
Joined: Mon Oct 03, 2016 9:19 pm

Re: S3 app vs Glacier app

Post by NAS_hans » Fri Nov 11, 2016 7:32 pm

Thanks for that reply, clear.

I am now using only the Glacier backup, so without S3. Just because it's cheaper. I would only need these files in case of disaster recovery, and am fine to wait a week or so for restore from Glacier.

However I have 2 additional questions, curious to see answer from forum users:

1. If I select the option to use SSL, does that encrypt the data between NAS and Glacier, even if I have not purchased a SSL certificate? How safe is that SSL option in Glacier backup app? Do I need to take additional security measures?

2. How can I browse files in Glacier? I tried using FastGlacier and Cloudberry (windows version, installed on my laptop) but they don't seem to be able to retrieve the inventory (files + folders with original file names) when files were uploaded using Glacier backup app. What is your experience on this? Suggestions?

Thanks!

lehrblogger
Starting out
Posts: 20
Joined: Sun Jul 24, 2016 11:45 pm
Location: New York, I love you but
Contact:

Re: S3 app vs Glacier app

Post by lehrblogger » Sat Nov 12, 2016 1:29 pm

@NAS_hans, I only have a limited understanding of these things, but I'll try to answer as best I can.

1. The SSL option does encrypt the transfer of data between your NAS and Glacier. I'd assume it works like a webpage that uses HTTPS rather than plain HTTP, and would encrypt the data with Amazon's SSL certificate prior to transport. The encryption means more computation work for your NAS, which is presumably why it's an optional setting.

Note that there is a separate setting for encrypting the files themselves before they are sent to, and stored in, Glacier. That would theoretically prevent someone from reading them even if they had access to your Amazon account. I've had trouble decrypting data stored that way, however, so be sure to test that before relying on it.

2. I've used the Freeze app for macOS with some success. I don't think you can actually browse the files on Glacier – you can only look to see what you have in the inventory, and then download specific files. This is pretty cumbersome, but the constraints make sense if Glacier is indeed using magnetic tape.

NAS_hans
Getting the hang of things
Posts: 50
Joined: Mon Oct 03, 2016 9:19 pm

Re: S3 app vs Glacier app

Post by NAS_hans » Mon Nov 14, 2016 9:10 pm

@lehrblogger, thanks for the info.

Regarding point 2, all I want is to see the inventory in my Glacier vault, so I can check at any time which files and folders exist in my vault. Since Amazon does not provide that feature from the standard Glacier interface, I need a 3rd party tool (indeed like Freeze as you mention). Issue is that the 2 tools I have tested (Cloudberry and FastGlacier) are not able to list the inventory for files that are uploaded using the Glacier app offered by QNAP.

I've contacted QNAP support and they suggested I try Hybrid Backup app instead of the Glacier app.

So I will try that and post the outcome here later.

lehrblogger
Starting out
Posts: 20
Joined: Sun Jul 24, 2016 11:45 pm
Location: New York, I love you but
Contact:

Re: S3 app vs Glacier app

Post by lehrblogger » Mon Nov 14, 2016 9:39 pm

@NAS_hans, ah, I see the problem now with Cloudberry and FastGlacier. One thing I noticed when trying to get Freeze to work was that the QNAP Glacier app was using what appeared to be a non-standard JSON format for specifying the file paths. For example, here's an item I copied from the ArchiveList object in the retrieved JSON, after the Freeze developer told me where to look for it on my hard drive:

Code: Select all

{
   "ArchiveId": "4zEwUv9j2OyAOwIUJV-MxPkGK3nB4R_yg3n8WKWVYLXkePMh_WzMx8Kef-BdB3RmUoHDrZC6yDEqWAb7QDO53K3CZasza2uXcms5O5yIA8TdemlqWf3uNdcXoKk7BlyQAxRPSub0Lw",
   "ArchiveDescription": "{\"path\": \"/Backups/Macintosh HD.sparsebundle/bands/3280\", \"type\": \"file\"}",
   "CreationDate": "2016-05-01T10:00:59Z",
   "Size": 8388640,
   "SHA256TreeHash": "2f8b7e6f86ed0faba09527a738f00c2a5097f166c7d3de6cc6a1599b39f28527"
 },


The unescaped ArchiveDescription is:

Code: Select all

{"path": "/Backups/Macintosh HD.sparsebundle/bands/3280", "type": "file"}


The Freeze developer was responsive and nice enough to add support for this format, but it's possible the escaped JSON is confusing Cloudberry and FastGlacier. Perhaps one/both of them has a raw inventory file downloaded and stored on your machine somewhere, that you could inspect manually.

This is the first I've heard of the Hybrid Backup app, so I'm very curious to hear how it goes!

NAS_hans
Getting the hang of things
Posts: 50
Joined: Mon Oct 03, 2016 9:19 pm

Re: S3 app vs Glacier app

Post by NAS_hans » Tue Dec 06, 2016 7:34 pm

Hi,

Just a short status update:

I did the following since my last post:

1. I have contacted QNAP support and sent my diagnostic logs + screenshots of the issue at hand. Developers are working on this issue.

2. Tested the current version of Hybrid Backup app. It does not resolve the issue of garbled filenames / lack of folder structure etc. when viewing Vault inventory in FastGlacier / Cloudberry. I see no change compared to using the Glacier app.

3. Tested the Cloudberry for QNAP app (via manual install). That app does provide correct folderstructure and filenames. However it adds a system-created folder above the lowest level (i.e. above each individual file) so the Vault inventory becomes quite confusing. Not what I'm looking for.

4. Tested Hybrid Backup app with compression set to yes + only backup modified files. That gives me yet another issue, which is that Chinese (or Taiwanese?) characters are included in the file names when listing Vault inventory in FastGlacier. I will leave that for now.

In summary, no workable solution yet, looking forward to the solution from QNAP dev's.

User avatar
Moogle Stiltzkin
Ask me anything
Posts: 8381
Joined: Thu Dec 04, 2008 12:21 am
Location: Around the world....
Contact:

Re: S3 app vs Glacier app

Post by Moogle Stiltzkin » Tue Dec 06, 2016 8:36 pm

NAS_hans wrote:TS-253A / QTS 4.2.2

I've viewed many AWS tutorials, and they all seem to indicate that S3 is required to be able to use Glacier (via lifecycle rules). In other words, S3 is the interface to approach Glacier.

Now, in the QNAP app library, I see a dedicated S3 app and dedicated Glacier app. Looking at the release notes for both apps, I can't work out what the difference is, and if I need S3 app PLUS Glacier app, or if only Glacier app is sufficient.

What is your experience?


found an article here for you
http://cloudacademy.com/blog/amazon-s3- ... the-cloud/
NAS
[Main Server] QNAP TS-877 w. 4tb [ 3x HGST Deskstar NAS (HDN724040ALE640) & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 + 16gb ddr4 Crucial + QWA-AC2600 wireless adapter.
[Backup] QNAP TS-653A w. 5x 2TB Samsung F3 (HD203WI) EXT4 Raid5
[^] QNAP TS-659 Pro II 1x 4TB HGST Deskstar NAS
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-228 w. 1x 1TB WD RE3 (WD1002FBYS)
[^] QNAP TS-128
Mobile NAS TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1

Network
Asus AC68U Router|100dl/50ul MBPS FTTH Internet | Windows 10, WC PC-Intel i7 920 Ivy bridge desktop (1x 512gb Samsung 850 Pro SSD + 1x 4tb HGST Ultrastar 7K4000)


Guides & articles
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review

https://www.patreon.com/mooglestiltzkin

NAS_hans
Getting the hang of things
Posts: 50
Joined: Mon Oct 03, 2016 9:19 pm

Re: S3 app vs Glacier app

Post by NAS_hans » Fri Dec 09, 2016 9:28 pm

Thanks @Moogle Stiltzkin

Maybe, looking at the issues I have described in this post, and the article from cloudacademy, the best option would be to use S3 as target for QNAP Hybrid Backup app, instead of Glacier. Then, using LifeCycle rules, move my files directly to Glacier after upload to S3.

From this article I understand that S3 objects with Glacier storage class will be more expensive than Glacier archives. So it will cost more in terms of data storage and transfer S3 to Glacier. On the other hand I haven't seen all these annoying issues yet when uploading to S3.

I would then need to test if an inventory retrieval using S3 with Glacier storage class is better than inventory retrieval using Glacier archives directly.

Has anybody used that setup, and what is your experience?

Post Reply

Return to “Amazon Glacier”