Http upload option

I'm looking for a bit of advice on whether rclone can achieve what we need in a project. We're designing a system where we will receive (up to 600MB) files from a remote location and needs to make them available via the bare filesystem for performance reasons (so no object store such as S3, not even with something like CephFS for file access).

The remote system that will push the files to us is planning to use rclone because it supports their existing need to push files to S3 and NAS. However, it's not clear that it will work for us. We want to avoid ftp- or samba-like solutions, so we're looking into http transport options. I noticed with interest the thread "http upload contribution" (13271), talking about using TUS to add write capability to http remotes but it looks like that effort petered out. I also saw the current PR #4221 but have I understood correctly that that's about pushing a file to a rclone running remotely (so the wrong way round for us)?

If there are options that I've missed or misunderstood I'd be very glad to hear about them. Thanks.

If you want to run a rclone 'serve' on your servers and allow a remote host to upload files to you via that rclone process, you could use webdav, sftp. The http serve doesn't currently allow to upload files to it. I currently run an sftp serve and stream large surveillance video data to it all day long. It works well. You mentioned you wanted to do it via http though. I believe PR 4221 would allow that as well when it it merged.

With rclone serve WebDAV the uploads are http PUTs I think. You can do them with curl provided you create the directory in advance.

1 Like

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.