Rclone and AWS Glacier Deep Archive request costs

[SnowySailor]

I am considering using AWS Glacier Deep Archive to back up my NAS (n*10k files eventually) and I am curious about how rclone/AWS handles LIST requests to determine what needs to be uploaded. I see it's $0.05 per 1000 PUT, COPY, POST, LIST requests which might get expensive pretty quickly if I'm trying to sync every day (lot of LIST requests).

Do the LIST requests just list the contents of a directory? Could I lower the number of LIST requests by reducing the total number of directories I have? I know I'll have to bit the bullet and pay for a PUT request for each file I upload but that isn't really a concern of mine since it'll only happen once per file. My main concern is the LIST requests or GET requests that'll be happening. Would love to understand how many times those occur for each file/directory.

Thanks

[Animosity022]

This would give you a few pointers and general idea. Less directories would be better as a general rule as the more you have, the more times you would have to list them out.

Why so many transactions on S3 using SYNC?

Im using this command manually to sync about 600GB on files to S3 every day. rclone sync -v --log-file=/logs/sync-log-date +%Y-%m-%d-%H%M%S.log --fast-list /Data/ avirom-s3:Data & I uploaded the first backup at the beginning of June and daily sync every day until today (June 30) Amazon billing says that I made: PUT, COPY, POST, or LIST requests 650,875 Requests GET and all other requests 5,794,450 Requests That sounds like a lot. Only about 100MB worth of files change every day, probably …

[SnowySailor]

So it sounds like there's no real way to know exactly what calls will be made (and thus how many). So I should use --fast-list and --checksum to try to minimize the number of requests. I understand how --fast-list can minimize the number, but how does --checksum lower the number of required requests? Does --fast-list come back with the checksum included for each file or something so we don't have to go and make a separate request for modification time?

Thanks

[Animosity022]

@SnowySailor

So it sounds like there's no real way to know exactly what calls will be made

Sure, the code can be stepped through and figured out exactly as it's not random by any means. If you want to share the layout and file number structure, I'm sure someone can help calculate it.

@SnowySailor

Does --fast-list come back with the checksum included for each file or something so we don't have to go and make a separate request for modification time?

I'm not 100% sure on this so hopefully someone else can chime in.

[ncw (Nick Craig-Wood)]

@SnowySailor

So I should use --fast-list and --checksum to try to minimize the number of requests. I understand how --fast-list can minimize the number, but how does --checksum lower the number of required requests? Does --fast-list come back with the checksum included for each file or something so we don't have to go and make a separate request for modification time?

That is exactly right. Modification time is stored as metadata on the object which doesn't come back in the object listing. You can also use --size-only or you could use the --update flag along with --use-server-modtime in place of --checksum as --checksum will md5sum your local objects which can take quite a long time.

Another way of reducing the requests is to do a partial sync so

rclone copy --max-age 24h --no-traverse /path/to/local s3:bucket

This will make rclone find any files newer than 24h and copy those to the bucket without listing the remote bucket.

You could run this a couple of times a day then once a week you could run the full rclone sync which will delete any files that have been deleted from the archive.

You might like to check out this blog: https://noellh.com/blog/rclone-to-s3-glacier/ which I found which has some costs in.