Copy to S3 incrementally

Help and Support
1

I have a lot of objects (millions) that are already in both the local and the destination (some in glacier and some standard). What's the best/fastest way to make sure the destination stays up to date with the newest files?

Previous files will NEVER need to be updated.

My current command is
rclone copy /media/storage/Events/01/ gfevents:gfevents/Events/01/ -P -v --log-file=/var/log/rclone.log

And since I also don't need the modified date updated on previous files should I use the
--update & --use-server-modtime flags?

How about --fast-list?

Maybe once I ensure that everything is in the destination moving forward I can use a max-age flag. My issue with that is, if for some reason the cron job doesn't run, now we can get behind.

How about, --ignore-existing? If I truly know my previous files won't change, would that be safe? Only possibility would be if an upload failed perhaps a file could be missed.

2

That is a good optimization.

If the files are never updated you can use --size-only which might be better than those two flags.

That will speed up the listing times greatly at the cost of memory.

Once everything is in sync then using rclone copy --max-age --no-traverse --size-only) will probably be the most efficient way of doing a top-up sync.

If files never get deleted then copy will work fine.

I'd be generous with the --max-age so if you crontab runs every hour, I might use --max-age 24h.

You can always do a topup sync without --max-age - with rclone copy --size-only --fast-list

I wouldn't bother with --ignore-existing - rclone will skip them equally well with --size-only

3

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.