I'd like to implement a new rclone backend using a Rust library I wrote. That library is part of a demo for file upload and storage, plus publishing and serving websites on a new CAFS storage system (Autonomi).
The least disruptive way to do this seems to be, to create a Go package to wrap my Rust library and use that from within my rclone backend code.
I'm looking into using CGo or compiling the Rust library to WASM and calling that from Go.
A couple of questions:
one of my goals for this work is to have the Autonomi backend accepted into rclone at some point. Are there any reasons why the approach I'm looking at would be unacceptable to rclone? @ncw (BTW, all my code is GPLv3, as is is the Autonomi API implementation).
any other thoughts on the approach. I have called a Go lib from Rust in the past and for that project I did also write a little Go, but I'm not that familiar so any other comments or suggestions would be welcome!
You can use librclone from rust as that is just plain C bindings.
But you want to do the reverse...
We don't use CGO in rclone - it complicates the build too much.
Calling it via wasm could work. I was considering using wazero to allow rclone plugins so that could maybe be made to work.
Perhaps making a backend plugin system which could load wasm modules would be an interesting way to go.
We don't want to have CGO as part of the build unless it can be done like the cgofuse integration which loads it at runtime and does not make the binary depend on anything.
Rclone is MIT licensed, so maybe a plugin might be the way to go here?
Thanks Nick, these options are looking a bit tricky. I don't think WASM is an option here after all.
I've just had a discussion with an Autonomi dev who has been implementing their WASM target which for reasons requires access to WebSockets, which is fine in a browser but unlikely elsewhere. If I understand correctly they'd have to target WASI which is not happening at the moment. Short discussion here: Go lang support - current and future · Issue #2301 · maidsafe/safe_network · GitHub
Thanks for pointing to wazero which looks very nice. I'm not sure if it helps here or not so I've mentioned it on the above issue in case. If not I think WASM is a non-starter and I'll be looking at how go-fuse is used.
Dang, I thought it was GPL, must have been something else I was looking at in this area.
Thanks for your help. I'm quite keen to find a way to add an Autonomi backend to rclone for several reasons.
That would work. cgofuse dynamically loads the shared object which means we don't have to distribute it with rclone. No idea whether that satisfies the licensing or not!
Note that a backend which doesn't use rclone's http transport will miss a lot of features which is unfortunate.
The other alternative would be to build the backend as a Go plugin so a Go shim around your code. Plugins need an rclone built with CGO support anyway.
So anyone wanting to use the Autonomi backend would need to build (you could have a build script as you do for xgo) or download the go-autonomy binary and ensure it is somewhere rclone can find it.
Let me know if you agree? because if not this route can't work.
Phew, that's encouraging. Are you able to give a stronger commitment (in wording) because it would be awful for me to spend a lot of hours only to have it rejected?
I would get value from a fork that myself and others could use, but if it wasn't pretty certain to be merged into the main project I may want to think a bit more or look around for other projects. It's my fault I didn't spot this earlier so I appreciate you taking the time to answer my questions.
rclone is such a neat and powerful project I remain keen on this and bringing rclone and Autonomi together.
FYI I've been supporting Autonomi for over ten years. It was called Safe Network / MaidSafe in the past but rebranded early this year in the run up to launch. I'm a little disconcerted about some recent compromises, but it remains a unique potential addition to the 'decentralised cloud' storage space. And this is why I'd love to see it come together with rclone. They really will multiply the benefits for so many people if they can work together.
Bringing benefits to as many as possible is primary for me so becoming built into rclone would be a major plus.
I'm going to assume rclone+Autonomi will fly wrt licensing and begin working on a fork to test our ideas. So...
The docs say this doesn't work on Windows which rules it out. Much as I hate Windows I can't leave so many people without this feature! The plugin approach looks more user friendly though, so is that likely to support Windows any time?
In the mean time I'll look into the cgofuse approach.
I've never liked the idea of doing things by creating a local web server/service - which is probably why I'm late to think about it - but that seems a good option:
resolves issues over licensing (separate app download)
would I assume use rclone web transport
can start simple API extend to take advantage of rclone features over time
probably useful for things other than rclone wanting to use Autonomi
What do you think?
Can you suggest a good API to use as a template? I'm thinking of something that works like the Dropbox and Google Photos backends (using the batcher for metadata update) but which can maximise rclone functionality.
Do those APIs allow maximum rclone funcitonality or are their better models to follow?
I'm not sure I understand. Is this because those are file system interfaces, which means they have to handle metadata updates per object update or some other reason?
My plan was to write a normal rclone/backend (which can use your batcher) to talk to a local server which implements a new REST storage API (modelled on something appropriate) and behind the scenes talks to the Autonomi API.
Using FTP / WebDav makes it difficult or impossible for the service to handle metadata efficiently, so I think that is best done using your batcher in a new rclone/backend/autonomi-web which I can write in Go.
So I'm still wondering which RESTful API to use as a template in order to maximise rclone functionality and allow me to handle metadata using your batcher in rclone/backend/autonomi-web.
I can probably have something to prove the concept much more quickly via this route, but I think it important to understand how best to define the RESTful API to maximise usefulness with rclone.
Ah, I see. I was thinking of a more general purpose backend which could run other code.
I guess the batcher could be an option in that.
Rclone implements over 50 REST APIs so it doesn't really have a preference! The easy path is to use GET to read the data (with Range request support), PUT to upload the data with extra parameters as HTTP headers or as URL parameters and POST to send/receive JSON blobs to do everything else.
