I know there have been previous discussions on the topic of metadata cache.
I'm trying to sync a moderate number of files to S3 compatible cloud storage. 12GB comprised of 17k files takes almost 5 minutes to check every time the sync is run (I'm using 16 checkers and transfers).
If we know this is the only process uploading to that destination, we should be able to keep a local cache of the metadata and check against that instead, perhaps doing the full (and resource intensive) remote check periodically (e.g., once a week).
Based on the suggestion here, I wrote a quick console app that caches the metadata before calling rclone, and produces a list of the differences. Not surprisingly, this multithreaded process took only 12 seconds to run, compared to over 4 minutes for remote checks.
As suggested in the link above, I am trying to feed that list into --files-from, which works for new and updates files. Is there a way to also do deletes in the same command, or do I need to call rclone delete with a separate deleted files list?
I hope the local cache can be considered, as it reduces resource usage significantly for large syncs.