Most efficient way to sync new files to a S3-compatible iDrive e2

Hi Rodrigo,

The files transferred before you cancelled the task will be compared and then skipped the next time you sync, so no worries.

You didn't tell how much was synced when you cancelled so let us assume it was 50% (1TB) in 16 hours. That would correspond to 17MB/s (1TB/16/60/60s) which roughly corresponds to a 170 mbps upload speed.

This sounds reasonable fast, but may not fully saturate your connection. If not, then try doubling the number of concurrent transfers by adding --transfers=8 to your command. Note: You may meet the limitations of your external drive (or iDrive) before the limitations of your internet connection.

I suggest you add --log-file=photo_uploads.log -v to you command to keep track on the uploads. You may also want to add --progress --stats=3s to see the progress.

Your log shows that it only took 2 minutes to identify the changes, so I would initially focus on the upload speed discussed above.

You may be able to make your sync quicker by using concurrent checks and starting the transfers before the checks have finished. You can do this by removing --fast-list, you can find the detailed description in the docs. Note: I assume you have free API calls at iDrive.

You may be able to increase the speed of the checks by using more checkers, e.g. --checkers=16; except if using --fast-list.

If at some point the checks still take to long then you may consider switching to the top-up approach which may be quicker when the number of files to be transferred are considerable less than the number of files in the target. I cannot predict when, so consider it when you start to notice a considerable delay (e.g. 10 minutes) before the transfers start, or long pauses in your transfers where rclone only performs checks.

The top-up approach is explained in this post:

In your case the --max-age should be a duration of at least twice the maximum length between syncs (e.g. 240h) and the full syncs could then be every week or month.