Drive resume / resumable upload (implementation)

What is the problem you are having with rclone?

I thought that resumable uploads were supported by rclone. At least that's what I had in mind when I skimmed through the source (backend/drive/upload.go). However I wanted to test if resumable uploads actually works. So I started uploading a ~4GB file and killed the rclone command with CTRL-C. Using the exact same command just restarted uploading from start. Then I thought maybe it is designed to only work when the process is still active. So (unfortunately) by coincidence my connection dropped this night. I have no logs or so a I reused the terminals without copying the log. It seems there were some retries but apparently there is some limit for that?

Now I wonder how resumable uploads are actually implemented so I will have to look into the source in depth. I checked the drive documentation before. You just have to do a POST with query parameter


Then you will get some


which is valid for 1 week according to docs (no clue if that can be refreshed or so). Then you have to use PUT to update existing files. In my understanding using Content-Range (also used when doing multi-part uploads) google will concat that according to given value of Content-Range to the existing file?! So we just need to store that upload_id somewhere (e.g. in local folder and in the cloud drive at the specified position)

I would have tagged this thread with 'Help and Support' and 'Dev Discussions' but am not familiar with discourse.

What is your rclone version (output from rclone version)

rclone v1.52.2
- os/arch: windows/amd64
- go version: go1.14.4

Which OS you are using and how many bits (eg Windows 7, 64 bit)

see above

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone -vv copy <local_folder> <crypt_drive:/local_folder>

The rclone config contents with secrets removed.

Shouldn't be needed but can be provided on request.

A log from the command with the -vv flag

Stated in first section.

There are some issues about this and I've worked out how to make it general purpose, I just need some time to work on it.

Unless you wanted to help?

Sure I can lend you a hand. Maybe it's possible to create some kind of interface for resumable uploads that each provider could use with individual proper implementation.

What issues are there? Would be nice if you can elaborate this any further :slight_smile:

Here is the minimum interface I came up with - this won't make sense unless you study the backend interfaces in fs/fs.go

Optional interface for backends

type Resumer interface {
    // Resume checks whether the (remote, ID) pair is valid and returns
    // the point the file should be resumed from or an error.
    Resume(ctx context.Context, remote string, ID string) (Pos int64, err error)

A new fs.Option for Put/Upload for doing the resuming - OptionResume
should not be passed in unless the backend supports the optional
Resume method.

struct OptionResume {
    ID string             // resume this ID if set
    Pos int64             // and resume from this position
    SetID func(ID string) // called when and if the upload knows an ID for resumable upload

operations.Copy would initially set just the SetID function pointer -
it would then call Put/Update and it would get called back with an ID
if a multipart upload was in use.

On the callback it would persist the ID along with the Fs, path and
the Fingerprint of the source object.

If OptionResume is passed in then the backend should not cancel
multipart uploads on failure.

If Put/Upload returned OK then the upload was successful so the on
disk metadata can be removed.

So when being re-run operations.Copy looks on disk for uncompleted
uploads for this (Fs, remote, ID) and checks that the fingerprint of
the source object hasn't changed.

If the fingerprint is unchanged then it calls the Resume method
If not then it calls this optional

If this returns an error then Copy carries on as usual.

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.