Creating this feature request based on my experience with Google Photos as per Detecting incomplete file download from Google Photos
While downloading from Google Photos I notice that my backlog or transfer queue is normally quite large. This queue rarely gets completed before Google Photos starts throwing 404s or other errors requiring me to kill
rclone sync. Building this queue used up some of the limited API quota so it seems pretty unfortunate that it gets thrown away and re-built on next
rclone sync run.
I think it would be amazing if we could someone save & reuse the queue instead of having to rebuild from scratch every time. I think there are multiple ways to solve this problem and all of them will have one major drawback obviously: we will be working with a stale or outdated queue. But in my case this is a risk I'll happily make given my data set doesn't change as often.
Some ideas on how to solve are below.
--interruptible flag optionally take a filename or dirname which can be used to cache the transfer queue. If no filename or dir is supplied then use something in local current dir and/or /tmp (or its OS equivalent).
--resume flag would similarly take a filename or dirname from where it will read the previously cached transfer queue and continue from there.
These flags might make sense for
rclone copy as well as for
rclone lsf even. There are probably other commands that might be benefit from these flags.
2. Allow providing transfer file list to
rclone sync by flag.
This would add a flag like
--transfer-queue [FILE]. Here FILE format could be the output of
rclone lsf --csv or
My thinking here is that for Google Photos I could run the expensive operation of
rclone lsjson or
rclone lsf once a day or or once a week and then supply the saved file from here to
rclone sync so it doesn't have to repeat those operations and it can operate from a pre-built list.
This option allows someone to build a list manually as well and supply it to
3. Add flag to
rclone sync to only traverse (no transfer) and save output.
--traverse-tree option which will only build the queue without doing any transfers. I feel that
rclone check already does this so I'm not sure if this option is the best option.
Those are my ideas so far. Will add if I think of more. Would appreciate any thoughts, suggestions.