Http upload contribution

Here is the full mount command used: rclone mount local:/home/nolan/Downloads/rclone-upload /home/nolan/go/src/tusd/data --vfs-cache-mode writes

Actualy I think I've fixed that in the latest beta - try that.

I decided to build from source and the issue is resolved. The go build process is weird..
The uploads seem to work flawlessly in the various modes.

That is good!

Do you think caching the local files first (which is what --vfs-cache-mode writes is doing) is acceptable? If so then that would make a rclone serve tusd remote: practical.

Local caching should be a fallback if the backend doesn't implement resumable uploads, otherwise it would be better for the backend to manage it. It would also need to be able to be disabled, returning a service error if the backend doesn't support it.

With tus supporting simple POST requests would it not be reasonable to just incorporate it into the existing front-ends? do they already have upload support? is there an alternative for upload support that you would want for them? I am still not convinced that naming tus explicitly in the interface is the way to go. tus is more an implementation detail than a free standing service.

I found that there is already code that maintains a metadata store in code. It uses boltdb which doesn't allow multiple connections. What do you think of replacing/substituting it with a standard database interface that will allow scaling?

Making an interface for every backend to support resumable uploads is a lot of work, hence my suggestion to split this into parts.

It would be great if rclone could support resumable uploads for eg rclone copy that is an often suggested feature.

rclone serve webdav supports uploads. rclone serve http doesn't. I'd suggest rclone serve tusd includes rclone serve http.

The other way would be to add a --tus-upload flag to rclone serve http.

I try not to keep state with rclone if at all possible! boltdb is a pain because it can't be accessed by more than one process at once. Also boltdb is verging on unsupported :frowning: I'd suggest files in the cache directory for simplicity's sake, one per file.

Agreed, I mean to create an interface with a cached fallback if the backend doesn't implement it (which initially none would). This is just a small amount of work to save from having to refactor when I get to modifying the backends.

In my mind this makes more sense, but I want this to fit into the bigger picture of rclone. Are you happy with using a flag or will this compromise an existing pattern?
My intended use case would have me running rclone rcd --tus-upload.

If the backend can't manage file locks I need some way to provide that functionality. Replacing boltdb with database/sql will allow ACID access patterns of the state. A free standing rclone can create a small sqlite db in the cache directory that manages metadata and locks. The code can be made to be tolerant of this file being deleted.

For my purposes I would configure rclone to connect to a PostGRES instance rather than a sqlite db as I want to be able to run replicated instances of rclone rcd in a highly available manner. Each chunk upload request would check the database for a lock meaning that the upload state would not belong to any one rclone instance and can therefore be load balanced.

The rclone serve rcd and rclone serve http are slightly different cases. rclone serve http can present one remote only whereas rclone serve rcd can present multiple remotes provided the --rc-serve flag is provided.

Adding a flag to rclone serve http will provide an rc prefixed version of it in rclone rcd

rclone can't use sqlite because it isn't pure go. I think using files for locks would work well enough without adding the extra complication of a database...

Interesting idea! I'm not sure that belongs in the core rclone code though. Rclone is trying to be a reasonably lightweight command line tool that doesn't need a lot of configuration or use local state (other than the config file).

The other place to store locks would be on the cloud storage itself which is what the tus s3 backend does.

What is your use case for a tus enabled rclone?

I am not sure I understood this.

nuts, you are right. That actually derails some of my other plans...
Looking into it, boltdb seems to be the only real solution for golang right now.

Ideally file locks will be handled by the backend. We can skirt around needing locks for caching. We use a folder to represent an uploading document, and each chunk is written to its own file with the name as the byte range. The most difficult situation is when the cache folder is on a mounted NFS. If (for whatever crazy reason) the same byte range is transmitted to another instance of rclone, the chunk file creation collision can happily be blocked for one process (or discarded depending on the NFS implementation). Once all chunks have been received, the data can be transmitted to the backend. If the backend absolutely doesn't provide a safe way to grant an rclone instance a lock, a lock is generated by following this algorithm in the cache folder.

My overall use for rclone is actually to use it both freestanding as a server and as a library in a larger project. The server takes rc commands from rclone clients or web clients and moves data around. The web requests can include uploaded data. I do need to add a way to add rules for user provided remotes and beef up the authentication model, but security comes last :wink: .

You'll see that all the http flags are duplicated with --rc- prefixes, that is what I mean.

Is quite like to think about how a resume interface might work. Have you any thoughts about that?

Ok, so I took some time to really read through the rclone code, it looks like the best way to do this is to implement it in the fs code. A backend will need to offer the OpenWriterAt() Feature and locking should be done in the Open() and Close() methods.
Ideally the interface should offer that OpenWriterAt() doesn't truncate the object. A boolean can be added to Features that allows discovery of if truncation will occur.

A fs wrapper is needed to provide the OpenWriterAt() function if it is not offered by the backend. This will cache the data until a complete fs.put() can be called on the Object.

The frontend calls the fs and uses fs.put() normally unless the upload is chunked.
I suggest we switch to the chi http multiplexer, and add middleware that checks for the tus request header. This way any frontend wanting to use tus doesn't need to provide a special endpoint dedicated to tus, such as webdav. This also allows the same endpoint to accept different upload protocols.
The middleware would be implemented very simmilar to content_type.

What do you think?

The OpenWriterAt interface is the generalised file interface for files. Unfortunately the only backend you can really implement that on is the local backend (maybe the SFTP backend).

It would be possible to make a much more limited resumable file interface where you can do something like

  • start a resumable upload getting an ID back
  • write data sequentially to that ID
  • query the upload position for that ID
  • finish with ID

This could be supported by most of the major cloud providers

That is the VFS with --vfs-cache-mode writes.

The chi multiplexer does look nice and light weight which is good. I have to say I haven't particularly found the standard library one lacking though... I think there would only need to be one place that the headers needed to be investigated though since there is common code already for the http listing.

The VFS puts a lot of effort in providing the os interface which is an uneccesary step for file uploads. Directly using the fs interface with its existing wrappable and discoverable features would be more robust. Yes, the VFS cache already exists, but it would require an overhaul to support backend and multiprocess caching (which is also outside of the VFSs scope). S3s multipart uploads is a great example of where we would just want tus to hand the chunks to the backend directly.

The OpenWriterAt interface does not need to implement random access. It can record the byte ranges passed to it and reassemble them on close. The close step is where the upload to the backend occurs if needed.

The multiplexer is where the service branching logic should be done. It makes it easier to integrate new functionality and it seems awkward to call the tus http.Handler explicitly in code. There are also signs of duplicated code in various places that should be handled by the multiplexer.

I'd like to put a bit of effort into refactoring some of the http server code to deduplicate functionality and make a better defined interface for future contribution. This would be built around the http multiplexer idom.

An OpenWriterAt interface would then need to keep chunks of file on disk regardless of backend support which defeats the point doesn't it?

The interface I outlined above is what the backends can actually do - if we can make it work with that then we won't have to store chunks on disk.

I'd be interested to see a pull request for that. If you are going to do it, try to keep each commit nice and small to make it easy for me to review!

Here is the PR for the refactor. I am working out the specifics of the the abstraction between the tus http.Handler and the fs backend. That should be ready soon as a separate PR.

Thanks! I've got a bit of a PR backlog, but I should get to it in the next day or two :slight_smile:

Here is a prototype for the upload handler. I created a layer that will allow easily handling multiple types up loads (tus, multipart, etc). The link is to the code that will interface with the fs.Object. Object creation and resolution is left to the service handler along with defining the url to object mapping.
The upload handler is responsible for finishing the Request and interacting with the fs.Object.

The service handler would look something like:

httplib.Mount("/", http.HandlerFunc(func(w http.ResponseWriter, r http.Request) {
   f := getFSFromRequest(r)
   create := func(...) fs.Object {
      return f.Put(...)
   }
   get := func(...) fs.Object {
      return f.NewObject(...)
   }
   if h := upload.HandleUpload(r, create, get); h != nil {
       h.ServeHTTP(w, r)
       return
   }
    ... Some other endpoint routing/response
}

it doesn't need to know anything about the upload, but has many opportunities to catch relevant information during the upload handling.

I had not understood part of the tus protocol surrounding upload chunks.
It has two modes: sequential, and fragmented.

  • sequential only requires that the backend can append to a object
  • fragmented is a bit of a beast in that it requires the backend to be able to concatenate existing objects.

These call for some sort of fs.Object.append(io.Reader) and fs.Object.concatentate(fs.Objects) interface. I should add a fs.Features argument to HandleUpload() to allow a preflight check before creating any objects.

It is starting to feel like caching should be done in the upload handler rather than wrapping the backend given the required access pattern is decided by the uploader. What do you think?

That sounds do-able :slight_smile:

Can we not support non-sequential? That would make this whole thing a lot easier!

None of the backends except local and sftp will be able to support directly appending to an object.

Quite a few of the backends can do this, take a set of existing objects and make a new object from them. It is quite a time consuming process though.

I think for the first version you want to cache the upload in the upload handler. However that doesn't require writing it to disk if all the writing is being done sequentially, it can be written straight to the backend. What will need to be cached for the upload handler is the handle used for writing data to the backend.

This will make the upload work in a reasonably straight forward way with the downside that if rclone is restarted any partial uploads will be lost.

To fix that we'd need to implement for each backend the resumable upload interface which will work with a good fraction of the backends (but not all of them - the others can fall back to the method above).

The resumable upload interface needs to be something like this to be supportable by the backends. All the main backends could support this I think.

// ResumableUploader describes how to start or resume a resumable upload
type ResumableUploader interface {
	ResumableUploadStart() (Uploader, error)         // start a new resumable upload
	ResumableUploadFind(ID string) (Uploader, error) // find an old resumable upload with ID
        // maybe list something about uploads - not sure - will need to think about it!
}

// Upload describes a resumable upload
type Uploader interface {
	io.WriteCloser
	ID() string         // a unique string to pass to ResumeUploadFind
	Pos() int64         // where we have got to in the upload
	Abort() (err error) // abort the upload
}

Do you mean all at once? or random writes?

Time consuming to implement? or time consuming to actually concatenate?

By caching I mean in the event that a backend doesn't support appending or concatenation, and the client is uploading using one of those patterns, then it is first written to a temporary fs. I was thinking of implementing caching by simply passing a fallback fs to the handler that does have full support. The fs can be local or whatever the server admin desires by just specifying a remote url.
Once the upload is complete, fs.Move(remote) is called on the fallback fs. This will allow me to specify more distributed, network friendly upload caches than the local filesystem.

This doesn't need to happen. Uploads are stateless in that the client always specifies the handle returned in the original POST during subsequent PATCH requests.

That interface looks about right, we also need to add a expires() getter and concatenate interface.
ResumableUploadStart can be discarded for ResumableUploadFind as I can't see a reason you would want ResumableUploadFind to fail. There is also no reason to manage separate ID's as the full remote path should uniquely identify the upload. This removes the need for the ID() getter.

How about:

// fs implements
type ResumableUploader interface {
	ResumableUpload(path) (Uploader, error) // Start or continue an existing upload to path
	ResumableCleanup() error // Clean up expired incomplete uploads
}

// fs implements
type Concatenator interface {
	Concat(fs.Objects, remote) error // Concatenate to remote path
}

// Upload describes a resumable upload
type Uploader interface {
	io.WriteCloser
	Pos() int64         // where we have got to in the upload
	Abort() (err error) // abort the upload
	Expires() *time.Time // the time the upload expires or nil if never
}

leaving this here for my own reference

I mean make our version of TUS upload not support non-sequential, so the user gets an error message if they try to upload in a non-sequential way?

I meant the concentration of the files. The remote object store has to copy all the blocks around for you which can take some time.

Yes Expires is a good idea

OK

I think that scheme would work fine for S3 and B2 for example.

However it wouldn't work for google drive: https://developers.google.com/drive/api/v3/manage-uploads#resumable

The problem there is that google drive has no "list resumable uploads in process" call that I can find.

Doing it like that (not exposing the ID) would mean that the drive backend would have to cache ID <-> path lookups on disk (assuming you want this to work after rclone is stopped and restarted which I do because I want this same interface to work for resuming uploads).

Rather than writing a cache lots of times for each backend, I think I'd rather the low level interface included the IDs but there was a higher level "resume cache" which could find a resume ID given an (fs, remote).

So there would be a high level function ResumeUpload(fs, remote) (Uploader, error) which would look in that resume cache, find the ID for the (fs. error) or make a new upload.

Unfortunately it is possible to have more than one multipart upload active per key in S3 so I'm not sure using the S3 list uploads function is going to be useful.