As Amazon S3 pricing is a bit complicated and related not only to the size of the storage, but also to the API calls, is that possible during a sync or copy operations, just for test, list the API calls so we can count the overall number and:
realize if we can optimize some API calls using different flags for our transfer
do an estimate of the cost of each sync/copy operation
I was thinking to some flag in rclone to enable logging of API calls.Probably this could be very helpful.
What about pricing of the monitor service in AWS? I saw the log is registered to S3 so it has a cost and probably also the logging has a cost for entries. Did you have the chance to try this servise?
for most recent backup files, such as veeam backup files and for recent backups with lots of small files, they goto wasabi, s3 clone known for hot storage, does not charge for api calls.
for long term storage of backups, they goto aws s3 deep glacier using rclone copy --immutable.
if there is a folder with lots of small files, they are zipped, and then that single zip is copied.
about a flag, not sure the value of it, as aws offers detailed logging.
if that is something you want, start a new topic using the feature category, make the request and see how other rcloners respond.
Sorry for my poor attention to te S3 documentation. Yes this is that API call and the requested http header is exactly what is described in the documentation. So the --dump-headers options is perfect for this debug purpose! Thanks a lot again.
I was worried when doing a sync that there was an API call for every object in the bucket. Now I know there is just one API call every 1000 object, so I can count the frequency of sync operations I can take
in a month considering that Amazon will charge 5cents every 1000 ListObjects API calls/month
I'm just using --size-only without --fast-list and from my understanding looking at the HTTP request (I posted before) it should be only one API call every 1000 objects also without --fast-list. Is that true, or perhaps am I missing something?.
I realized looking at the log that in some cases there was one API call every 1000 entries, but in some (few in my cases because most of the directories contains lots more than 1000 entries) cases there were extra API calls. Now reading to your comment I see that really it happens on directory changes!
Now everything is clear and I got the advantage of using --fast-list flag