Chunker resume upload contribution

Dev Discussions
1

I recently posted a comment on a Github issue expressing my interest in adding support for resumable uploads to the Chunker backend. I'm working on accomplishing this by checking for and skipping existing chunks during an upload. https://github.com/rclone/rclone/issues/87#issuecomment-671506822

Since then, I came across a discussion that mentioned a Resumer interface that is being worked on to support resumable uploads. Drive resume / resumable upload (implementation)

It is my understanding that this would only work for cloud providers that have built-in support for resumable uploads. Would there still be interest in what I've been working on with adding resumes to Chunker? My thought is that even once the Resumer interface is added, having resumes supported by Chunker would still be useful because it would bring resumes to more cloud providers.

2

I think what you've suggested is very interesting - having a resume interface for chunker to wrap around any backend.

I think ideally this should implement the resume interface that we come up with as this will need support from other parts of rclone to be seamless.

You could potentially store the uploaded part names in the resume ID to save having to scan the folder on resume too.

I don't think anyone is working on the resume interface at the moment - are you interested in helping with that?

3

Yes, I am interested in helping with the resume interface.

I don’t think I fully understand how the resume interface is intended to work though. Is there somewhere I can look to get a better understanding of what the structure of the interface would be and how it should work? What is the resume ID meant to store? Where is this information being stored so that future operations can check for incomplete uploads?

4

The description I made was the result of thinking about it for several years so it has lots of assumptions in it which may not be obvious!

This is an opaque ID which belongs to the backend. When you upload a file with S3 (for example) when you start a multipart upload you are given an ID string. You need this ID string to query or resume the upload.

It might be that backends don't need the ID.

The chunker backend can use this ID string as it pleases, so storing file names in there might make sense.

As yet not decided! But probably under --cache-dir so in ~/.cache/rclone maybe in a directory called "resume". This will need storing for each (remote, file) tuple too - details yet to be determined.

5

I am sorry that I kinda ghosted away. Still being interested in working on a solution for this. Maybe there is some chat like communication platform (slack-like gitter, rocketchat etc?). I've been pretty busy with other projects. This would be my first time coding with go but that should be no issue.

Is there any branch already for this?

6

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.