Add way to interrupt and resume commands

Feature
1

Creating this feature request based on my experience with Google Photos as per Detecting incomplete file download from Google Photos

While downloading from Google Photos I notice that my backlog or transfer queue is normally quite large. This queue rarely gets completed before Google Photos starts throwing 404s or other errors requiring me to kill rclone sync. Building this queue used up some of the limited API quota so it seems pretty unfortunate that it gets thrown away and re-built on next rclone sync run.

I think it would be amazing if we could someone save & reuse the queue instead of having to rebuild from scratch every time. I think there are multiple ways to solve this problem and all of them will have one major drawback obviously: we will be working with a stale or outdated queue. But in my case this is a risk I'll happily make given my data set doesn't change as often.

Some ideas on how to solve are below.

1. Add --interruptible and --resume flags

The --interruptible flag optionally take a filename or dirname which can be used to cache the transfer queue. If no filename or dir is supplied then use something in local current dir and/or /tmp (or its OS equivalent).

The --resume flag would similarly take a filename or dirname from where it will read the previously cached transfer queue and continue from there.

These flags might make sense for rclone sync, rclone copy as well as for rclone lsf even. There are probably other commands that might be benefit from these flags.

2. Allow providing transfer file list to rclone sync by flag.

This would add a flag like --transfer-queue [FILE]. Here FILE format could be the output of rclone lsf --csv or rclone lsjson.

My thinking here is that for Google Photos I could run the expensive operation of rclone lsjson or rclone lsf once a day or or once a week and then supply the saved file from here to rclone sync so it doesn't have to repeat those operations and it can operate from a pre-built list.

This option allows someone to build a list manually as well and supply it to rclone sync.

3. Add flag to rclone sync to only traverse (no transfer) and save output.

Add --traverse-tree option which will only build the queue without doing any transfers. I feel that rclone check already does this so I'm not sure if this option is the best option.

Those are my ideas so far. Will add if I think of more. Would appreciate any thoughts, suggestions.

2

I have a plan for cached transfers which would work like this.

Watch this space!

You can do this already with --files-from or --files-from-raw. I'd use rclone lsf --files-only to generate the list and pass it straight back into --files-from-raw. I'd use it with rclone copy rather than rclone sync though.

You can either use rclone check for this which outputs files which are compatible with --files-from or use rclone sync --dry-run and parse the logs.

3

Awesome! :slight_smile:
I'll take a look at your other suggestions as well of using --files-from and --files-from-raw.

4

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.