Is there a "skeleton" template for adding new storage providers?

Hi there :wave: Newbie dev wannabe here. Well, that's actually pretentious, I'm just a humble amateur Go programmer.

I've been using rclone for a while now, and I'm quite happy at how it keeps everything in sync with my 12 or 13 different cloud accounts :smiley:

There are only two missing :slight_smile: and since they're not directly supported... I thought I'd try to write them from scratch. They have limited usage and reduced interest (contrasting with the 'big' providers and their hundreds of millions of potential users), so probably they will never be useful for anyone, but, alas... they will be useful for me. :laughing:

The first provider is something called "MEO Cloud", which is a cloud-based file service offered by Portugal's public telecom. They do have a well-established, stable, REST-based API for accessing their services โ€” which they claim to be strongly inspired/derived from Dropbox. Thus, in theory, you should be able to use a Dropbox connector to access the service (getting the minimalist set of functionality).

Unfortunately, what the authors of this API fail to explain is that they've based their API on version one of the Dropbox API. Bummer! Naturally enough, Dropbox now uses v2, and rclone diligently supports (only) v2. No wonder I wasn't having any success trying to connect...

The second provider is Pydio Cells. Pydio, originally known as "AjaXplorer", is a Go application allowing one to run their own self-hosted "cloud" service. It exposes a S3 API, and rclone can access it, but with lots of unknown errors, which, in turn, makes any mid-sized backup a real nightmare, taking eternities to complete. The alternative is to access its native API directly and forget all about S3 and its well-known bugs...

Anyway! These are my ideas. Now to put them in practice :slight_smile:

Naturally enough, I started to take a look at the many existing storage providers. These have varying degrees of complexity and are not immediately obvious to follow. Some rely on provider-published REST APIs to do their job; some rely on independent, third-party Go libraries; and some use various system calls to achieve their purposes (i.e., for local filesystems and such).

What they all implement is a minimum set of requirements โ€” but what are those? Is there some documentation on the functionality that needs to be implemented? Better still, is there a "skeleton" or template, demonstrating how the minimum set is implemented (with, say, a "null provider", which essentially does nothing but show how everything must be hooked together)? Or perhaps, out of the existing providers, can you point out to me which one is considered to be the "base" or inspiration for all of the others? I'm well aware that each provider has their own methods which only apply to them, but at the level above, all need to expose some common functionality which is what the rclone CLI shows to the user. Where can I find a list of that common functionality and what needs to be implemented?

Most of the documentation is focused on using rclone (in its many ways) but not really on developing rclone (in the sense of explaining how new storage providers are added), with a few exceptions โ€” such as how to use plugins as opposed to compiled-in modules, for instance. At least, as far as I could tell.

Or should I address the rclone community on Reddit instead? In fact, there is so much overlap between Reddit and the "official" rclone forums that it's hard to tell which one is best for these kind of questions...

Thanks in advance!

For high level overview start with reading this doc rclone/CONTRIBUTING.md at master ยท rclone/rclone ยท GitHub

1 Like

https://pydio.com/en/docs/cells/v4/webdav-server-access

That's exactly what I was looking for! Thank you very much! :blush:

Aye, WebDAV works, and does its job, but... it's soooooooooooooo paaaaaaaainfullyyyyyyyy sllllllooooooooooooooooow...

I'm pretty sure that there must be a way to get quick & fast WebDAV responsiveness, but, so far, it has eluded me :frowning: