Implement serving remote over S3

[pmjohann (Johann Polecsak)]

Hello guys,

I just started using rclone recently and I am amazed by the work of the creators & the community, it is an awesome piece of software!

The only thing I'm missing and came up with an ugly workaround hack for is serving a remote using S3. I saw this issue came up way back here but was closed without further comments.

My simple solution is to mount a filesystem using fuse, then start a minio instance using the fuse mountpoint as the data path.

I think this functionality could be relatively easily ported natively into rclone.
And while I know many things "could be done", the reason I think this also should be done is that S3 is one of the most widely used object store protocols used by many software in many different areas.

Thoughts?

[Animosity022]

Not sure I'm completely following but you would create a Minio remote and you can either mount it or server using rclone serve http already.

[pmjohann (Johann Polecsak)]

No, it goes the other way around.

The goal I am (and I'm pretty sure many others are) after is to serve e.g. a OneDrive backend and act as an S3 compatible server.

So the S3 client could connect to the rclone server, create buckets, upload objects etc. where the underlying storage media is a OneDrive (or any other rclone supported) backend.
Unfortunately my links were broken in my original post, I try to embed them again:

Rclone serve s3 <remote>

@ncw what are your thoughts on a feature like serving a remote over the s3 protocol? Use cases: Rclone as s3 proxy layer for other cloud storage backends. Quick s3 endpoint over a filesystem (local remote) for development purposes. I know this feature is something similar to what Minio does (basically an s3-like server), but I’m thinking of this as a lightweight solution that supports many more remote backends. Initially, the protocol implementation could be limited to just the most common …

minio/minio

There should be just some sanity checks, e.g. when issuing the command:
rclone serve s3 remote:/path/to/s3/root

We should apply some sanity checks whether there are only folders in /path/to/s3/root (there can only be buckets at the root) and whether there are only files inside those buckets (there is no nested structure in S3). If these sanity checks pass, the S3 server can be started, which implementation is excellently done by the minio team.

Hope I could clear this up a bit.

[Animosity022]

@pmjohann

edit: couldn't embed, so I had to base64 encode them. Guys, really??

Yes really as new users post spam links and it became a problem. I upgraded your account since you seem like a real human You can fix the links now.

@pmjohann

No, it goes the other way around.

Clear now, thanks!

[pmjohann (Johann Polecsak)]

Alright, links fixed, thanks for upgrading!

[Animosity022]

Here is the feature request issue to go with this:

serve s3: let rclone act as an S3 compatible server serving any backend

[pmjohann (Johann Polecsak)]

Oh well.

I wanted to open a feature request, but searched for the keyword S3 in the GitHub issues but somehow didn't find it. So I came here to look around this topic and here we are

Thank you!

[ncw (Nick Craig-Wood)]

I think this is a feature I really would like to have for rclone.

What I'd really like though is for someone else to make a go library I could import into rclone and use as the S3 protocol is quite tricky!

There are two possible ideas on that page. I don't think minio has moved to a library format though and is probably unlikely to so that leaves the gofakes library. It might be fun to build up a test with that library...