Hello, just working on optimizing a rather large daily one way backup of a file server to box.com, total of maybe 20TB and 5-6 million files. Using copy instead of sync due to the quirks of how box deals with versions / deleted items,
What has helped immensely, since this is a one way backup, is to cache the box side of things with a very long age time (I think I set it to 6 months) , and doing size only. I was wondering if everything is running on one box if there would be any benefit to caching the local FS (with obviously somewhat shorter max age) the daily delta isn’t too bad but of course scanning the entire data set for changes takes a very long time. I have played around with say doing an rclone check and generating a specific list of files to copy, but I guess in practice it would be a wash?
EDIT: perhaps a check with time since last change locally? So stuff that hasn’t been touched in ages is skipped wholesale?