Prototype for a nicer built-in web GUI


#1

Hello!

tl;dr: I wrote a web-based GUI which wraps the rclone binary. It works, but is pretty slow! If I incorporated my as part of the rclone binary ifself (using a similar technique as the current HTTP server) would this be a useful feature enhancement? Would you be willing to eventually merge it into the code base?

Basically, my existing code is a Flask app which uses the “subprocess” module to invoke rclone to list files, and serve them (including streaming media, and implementing slideshows.) It turns out (surprise!) that calling the command-line “cat” command repeatedly with --count and --offset works “technically” but is in fact incredibly slow (here is a link). I would be willing/able to port this as a new feature in Go. I’d be able to use the same semantics that “serve http” uses with Go’s built-in web server.

It sounds like there has been interest here: RClone GUI suggestions needed

Here are screenshots of my current implementation: https://imgur.com/a/RQ0wN25

For this, I’m not doing anything fancy with RESTful APIs, React, etc. (I would argue that this application does not demonstrate a clear need for extra frameworks - and also, javascript is not my expertise!) My prototype uses:

  • HTML templates for server-side rendering of the pages
  • semantic-ui as the front-end design
  • reveal.js for slide shows

If I started making PRs to incorporate this, would that be a welcome addition to the code base?


#2

Interesting!

On the rest-api branch I’ve been beefing up rclone’s API so that it can support a GUI.

https://beta.rclone.org/branch/v1.44-058-gd6bfda02-rest-api-beta/

You could use this rest API to interact with rclone which should be a lot quicker than using a subprocess.

I also made a new command rclone rcd which does nothing but serve the API.

So you can type

rclone rcd --rc-user user --rc-pass pass

or

rclone rcd --rc-noauth

to serve the remote control. Use rclone rc while it is running to see all the API. You’ll need authentication to do anything which involves looking at files (or use the --rc-no-auth flag).

If you want to read file data, you can use --rc-serve which is explained here: https://github.com/ncw/rclone/blob/rest-api/docs/content/rc.md#--rc-serve

There is no way to upload files right now, but you can use operations/copyfile to copy a local file to a remote file.

I’ve also made a proof of concept react app which talks to rclone via the API. You can serve this via rclone with the --rc-files flag.

My plan for this API is to enable the building of GUIs for rclone (and other scripting uses).

I can’t incorporate python code in the release as it would break the release model of a single binary so I think it would have to go into a separate repo with a separate release model.


#3

Hello! Thanks for the feedback! I’ll look at the REST API (though the link to your branch gives a 404?)

Also, let me clarify what I had in mind!

  • I agree it is a bad idea to shell out
  • Agree, would not want to ask you to merge any python code!

My proposal is specifically:

  • Implement a new command which is functionally equivalent to rclone serve http. Perhaps an rclone serve fancyHttp
  • That implementation would use the same mechanism (the rclone file descriptors, and Go’s built-in HTTP server) to serve files
  • This new fancy implementation would have a more “souped-up” GUI, similar to my prototype.
  • However, this would not actually use any of the code or functionality of my prototype, such as shelling out. I wouldn’t need to anyway!

#4

Try this build - it may not work yet though! You can always check out the branch from github which will compile, it just keeps failing the CI so the binary doesn’t get build!

https://beta.rclone.org/branch/v1.44-063-g64223bd3-rest-api-beta/ (uploaded in 15-30 mins)

Oh I see! That sounds interesting.

Could this be the http remote? I don’t think anyone would object if it looked a little prettier!


#5

Sure, I’d be happy - prefer even - for this to be the behavior of http remote! I had assumed that the current behavior was intentionally spartan :slight_smile:

Before I embark on this (does not seem too difficult, in principle), here are some tradeoffs I see:

  1. This will increase the “bloat” of the http remote - just more artifacts (CSS, styles, HTML, etc) for functionality
  2. Increases dependencies on other libraries. In my case, a dependency on front-end libs such as jquery or semantic-ui. These can be statically served, but still, its another lib that might need to be ugpraded
  3. Feature creep poetntial. For example, I’ve implemented an “image slideshows” feature. This would be an optional feature (and perhaps a welcome one) - I’m just not sure if your guiding principal is around minimalism for features like this.

If you’re relatively okay with #1 and #2, then I can take a crack at a first iteration of this. I think the bulk of the PR would be adding separate HTML and static content - but the existing templating logic would more or less be the same.

Point #3 is more of a question about future iterations - where do you see the future of the built-in web GUI? Something minimal, or something more feature-complete or media friendly?


#6

Spartan - that is a good word! I didn’t put any effort into styling it!

Because rclone is a single binary install these artifacts have to be embedded within rclone somehow. There is a stanard way of doing this with go-bin, but I don’t want to bloat the binary too much!

one possible ideas is that some of the artifacts could be served from https://rclone.org or a specialized subdomain.

Indeed…

They can also be served from a cdn such as unpkg.

I’m intending the serve http logic to become part of the rclone gui. I’m about to merge a branch rest-api which factors the serving of index pages and objects to its own directory cmd/httplib/serve so the logic can be reused in the remote control server.

My thinking behind this is a fancy GUI is probably still going to want to be able to pop up html pages with indexes or serve the objects from <a href=""> links.

I suggest you either start from the rest-api branch or wait until I’ve merged it (should be soon!)

i’d quite like rclone to have the option to do both!

I was thinking maybe that rclone could fetch its assets into a local cache for the rclone gui. Maybe serve html could work like this too.

serve html should work without any internet access, but if it degrades down into “spartan” mode that is fine!


#7

Thank you for your detailed response!

I haven’t yet had time to pull down your branch yet (I’ve looked at the code but not really looked at a diff). I will do so! Some thoughts from the above:

  1. If the goal is to have the (new) restful endpoints power a GUI, that implies some things about the GUI’s architecture. Namely that it will be more of a “single page app”, where the GUI primarily uses ajax calls to render the page, rather than a more traditional (old school?) web app where the server is responsible for rendering HTML. This is not how I myself would implement this but it’s certainly fine!

  2. If that is true, does this mean that the new serve http would actually be a superset of serve rcd - whereby we run both the REST endpoints along with additional static HTML routes for the GUI?

  3. In terms of static assets, if this were my project, I’d package the dependencies using go-bin (or whatever is in vogue these days.) This would add megabytes to the binary, but I see that as the lesser evil compared to relying on an external server (or CDN) for these libraries. After all, if I’m running a command dealing with my personal files, do I expect rclone to be making network calls to anywhere except for my remotes? Also, since this project is standalone, why introduce a dependency on some external server? I am reminded of the Lamport definition of a distributed system…


#8

I’ve merged it to master now :smile:

A REST API could power either type of server I think. It is just a more efficient was that shelling out to rclone for each thing.

I’m thinking serve http would remain without the REST endpoint, but if you want that then you have to use serve rcd (which I may rename at some point to serve gui if we ever get that far!)

A good point… I think it depends on how big the assets are really!