I thought that resumable uploads were supported by rclone. At least that's what I had in mind when I skimmed through the source (backend/drive/upload.go). However I wanted to test if resumable uploads actually works. So I started uploading a ~4GB file and killed the rclone command with CTRL-C. Using the exact same command just restarted uploading from start. Then I thought maybe it is designed to only work when the process is still active. So (unfortunately) by coincidence my connection dropped this night. I have no logs or so a I reused the terminals without copying the log. It seems there were some retries but apparently there is some limit for that?
Now I wonder how resumable uploads are actually implemented so I will have to look into the source in depth. I checked the drive documentation before. You just have to do a POST with query parameter
uploadType=resumable
Then you will get some
upload_id=<STRING>
which is valid for 1 week according to docs (no clue if that can be refreshed or so). Then you have to use PUT to update existing files. In my understanding using Content-Range (also used when doing multi-part uploads) google will concat that according to given value of Content-Range to the existing file?! So we just need to store that upload_id somewhere (e.g. in local folder and in the cloud drive at the specified position)
I would have tagged this thread with 'Help and Support' and 'Dev Discussions' but am not familiar with discourse.
What is your rclone version (output from rclone version)
rclone v1.52.2
- os/arch: windows/amd64
- go version: go1.14.4
Which OS you are using and how many bits (eg Windows 7, 64 bit)
see above
Which cloud storage system are you using? (eg Google Drive)
Google Drive
The command you were trying to run (eg rclone copy /tmp remote:tmp)
Sure I can lend you a hand. Maybe it's possible to create some kind of interface for resumable uploads that each provider could use with individual proper implementation.
What issues are there? Would be nice if you can elaborate this any further
Here is the minimum interface I came up with - this won't make sense unless you study the backend interfaces in fs/fs.go
Optional interface for backends
type Resumer interface {
// Resume checks whether the (remote, ID) pair is valid and returns
// the point the file should be resumed from or an error.
Resume(ctx context.Context, remote string, ID string) (Pos int64, err error)
}
A new fs.Option for Put/Upload for doing the resuming - OptionResume
should not be passed in unless the backend supports the optional
Resume method.
struct OptionResume {
ID string // resume this ID if set
Pos int64 // and resume from this position
SetID func(ID string) // called when and if the upload knows an ID for resumable upload
}
operations.Copy would initially set just the SetID function pointer -
it would then call Put/Update and it would get called back with an ID
if a multipart upload was in use.
On the callback it would persist the ID along with the Fs, path and
the Fingerprint of the source object.
If OptionResume is passed in then the backend should not cancel
multipart uploads on failure.
If Put/Upload returned OK then the upload was successful so the on
disk metadata can be removed.
So when being re-run operations.Copy looks on disk for uncompleted
uploads for this (Fs, remote, ID) and checks that the fingerprint of
the source object hasn't changed.
If the fingerprint is unchanged then it calls the Resume method
If not then it calls this optional
method
If this returns an error then Copy carries on as usual.
