Support for S3 Glacier

#1

Does rclone support S3 Glacier? I know that an S3 remote supports the storage class of Glacier as discussed in #923 - https://github.com/ncw/rclone/issues/923. I’m curious though if rclone can sync with Amazon’s dedicated glacier service which uses vaults as opposed to buckets.

0 Likes

#2

rclone doesn’t talk directly to the vaults service.

I think you can retrieve glacier objects via the S3 interface though can’t you?

0 Likes

#3

It looks like both S3 Glacier and S3 with storage class glacier both store objects in the Amazon glacier service. The difference is the interface with which you access the files, at least in the web, not sure if the APIs present other options. Files stored in S3 with storage class glacier are not accessible from the S3 Glacier interface and vice versa.

I think S3 with storage class glacier will work just fine in my scenario. I just wanted to confirm rclone couldn’t interface with S3 Glacier directly.

0 Likes

#4

I hadn’t heard of there being another interface until today!

0 Likes

#5

I had the same question and same problem.

Rclone allows you to interact with S3 buckets and store the data in the “glacier” class but does not allow to sync the data to an S3 Glacier Vault which is exactly what I need.

What did you end up doing? This is something that hopefull rclone will add in the near future.

https://docs.aws.amazon.com/amazonglacier/latest/dev/working-with-vaults.html

0 Likes

#6

It would be useful if you could describe the workflow you want. I don’t really understand how glacier vaults fit in with s3 buckets…

0 Likes

#7

Nick one of the advantages of Vaults is that they let you define storage policies to meet compliance rules and therefore creating a compliant storage archive.

More info in case you are interested:
https://aws.amazon.com/getting-started/projects/set-up-compliant-archive/services-costs/

For my particular scenario, the data storage needs to comply with some SEC rules so I have to use a Vault instead of an S3 bucket with storage class Glacier. However, because I noticed there is software out there that interacts directly with the vault interface, I wondered if there was any reason for rclone to not support that as well.

I will build my own solution using the AWS Boto SDK for interacting with Glacier but maybe this is something that could be added to rclone in the future to make it even better!

1 Like

#8

My use case allowed for storage in S3 with storage class Glacier. The ability to access individual files from the management interface simplifies 1 off restores. Something not possible with Glacier Vaults.

0 Likes

#9

I found the docs for the Go SDK

https://docs.aws.amazon.com/sdk-for-go/api/service/glacier/

It looks like this would be the same complexity as building a new backend, so not trivial, but something that is well understood.

Interestingly the pricing is still above backblaze b2 which is $5/TB/month and I know they have various compliance features but quite possibly not what you need.

0 Likes

#10

For future reference if anyone has the same problem, I ended up landing on Wasabi which is also SEC Rule 17a-4 compliant and supported by rclone!

:ok_hand:

1 Like