I have an Amazon S3 bucket that has about 3 million files (with new files being added to it every day).
I currently have an rclone job that copies this bucket to a Wasabi bucket. I run this sync every week.
So far this works. The copy works fine, but takes hours. I also worry about the number of API calls/data transfer that it uses.
I use both the --fast-list option as well as --max-age.
I've noticed rclone spends a lot of time initially upfront when the job starts. My guess is because it is enumerating the files to copy.
I'd like to know what best command-line options to use to do the following:
- Minimize the number of API calls to S3
- Minimize the amount of data transfer
With the --max-age options it seems like when you use it, rclone still enumerates every file from S3 but just skips over it locally on the client if it's older than --max-age. Can this be optimized so that this age/date is sent to the S3 API when fetching the list of files?
Is it possible to have --max-age value be sent to the S3 API so that the API only returns items less than certain age (thus making the enumeration faster)?
Thanks for the help!