Syncing many files (5M+)

[asdffdsa (jojothehumanmonkey)]

@ncw

wasabi?

that is a good suggestion. also, idrive works well with rclone.

i did some testing with wasabi with a bucket con tainting 1,000,000 files.
rclone sync --dry-run took just 33 seconds

Fastest way to check for changes in 2.5+ Million files?

for what it is worth, using wasabi, a s3 clone, known for hot storage for 1,000,000 files this command took 33 seconds. rclone sync d:\folder wasabi01:folder --size-only --transfers=64 --checkers=64 --dry-run --progress --stats-one-line --log-level=DEBUG --log-file=log.txt