Im using this command manually to sync about 600GB on files to S3 every day.
rclone sync -v --log-file=/logs/sync-log-date +%Y-%m-%d-%H%M%S.log --fast-list /Data/ avirom-s3:Data &
I uploaded the first backup at the beginning of June and daily sync every day until today (June 30)
Amazon billing says that I made:
PUT, COPY, POST, or LIST requests 650,875 Requests
GET and all other requests 5,794,450 Requests
That sounds like a lot.
Only about 100MB worth of files change every day, probably about 40 files.
What is Rclone SYNC doing under the hood? it is listing all my files every day?
Amazon will charge for all that,, is there a way to make it cheaper?
There is a section about this in the s3 docs, but maybe I need to put a how to make it cheaper section in...
If you have enough memory you can use --fast-list to exchange transactions for memory.
Using --size-only or --checksum (which will md5sum all your local files so may take some time) will speed the sync up as reading the modified time takes another transaction.
And finally, if you just want to do a top-up sync, then you can use --update and --no-server-modtime instead of --size-only or --checksum.
I do backups to swift which has a similar problem, and for that I use --fast-list and --size-only. Periodically I run rclone check to make sure all the files are uploaded.
What is the 'sync' command doing under the hood? (regarding AWS S3)
How many "list" "get" "put" etc during a regular folder-to-bucket syncing
(I know is on the source, but Im not a Dev so it will take me long to understand it)
It is actually easier to do some tests, so run with -vv --dump headers and rclone will show the HTTP headers for all requests it does. That will give you a good idea.
Your question isn't easy to answer directly because the sync routine only uses the backend abstraction, not the S3 API directly.
If you want to minimise transactions then try --fast-list and --size-only in a sync. Or --fast-list --checksum if you want to be sure of detecting modifications and you don't mind doing a bit more work locally.