I'm evaluating rclone as a backup solution for diverse file types.
As a test I used rclone to backup a directory to Dropbox, with 1.6GB, about 60k files in 27k directories.
The initial upload took about 8h, which I expected to.
I then ran a new sync, without any changes and after 2h looking at the log, just checking the metadata from Dropbox, I gave up.
Given this, I would like to know if the following behaviour is already possible in rclone:
On first sync, save metada in local database, for uploaded files.
On subsequent syncs, check which operations must be done against local database, and then perform those operations (uploads and deletions) and update local database.
This way there would be no need to check metadata from the remote. I am assuming that the remote, in this case, is untouchable.
If at any point I would be unsure of the sanity of the remote then I would just do a full sync again.
If this is not possible I will have to implement it myself outside rclone (I'll be using rclone for the uploads anyway). Or, I might even try to have a "go" a it and look at rclone code to see if could implement it.
I'm pretty sure the cache backend can do this but I don't remember how to configure it off hand - check the parameters to the cache backend here: https://rclone.org/cache/#standard-options
I will of course try it with the "cache" solution with chunk size zero (or very small) but from the discussion I was not convinced it would implement the behaviour I want. Also, you say, in the documentation that the "cache" will be phased out in favour of VFS, which led me to think that a better, more reliable, alternative could be available.
I thought cache backend can not cache metadata only.
You can not use chunk_size = 0 it won't work. Tried that multiple times even with the latest version. You may try with smaller chunk size but I think it will defeat the purpose and of course server side copy won't work either.
Still... when you do the full sync, spending 4 or more hours, just checking metadata on the remote, when it could be avoided, is not a good solution. For this specific case (when you are sure the remote is sane) using a local database, keeping the metadata from the last sync, is the optimal solution.
Anyway, I'll be curious to see how is it going to work with VFS.
In the meanwhile, the cache remote, is working for my case.
