tl;dr: I wrote a web-based GUI which wraps the rclone binary. It works, but is pretty slow! If I incorporated my as part of the rclone binary ifself (using a similar technique as the current HTTP server) would this be a useful feature enhancement? Would you be willing to eventually merge it into the code base?
Basically, my existing code is a Flask app which uses the “subprocess” module to invoke rclone to list files, and serve them (including streaming media, and implementing slideshows.) It turns out (surprise!) that calling the command-line “cat” command repeatedly with --count and --offset works “technically” but is in fact incredibly slow (here is a link). I would be willing/able to port this as a new feature in Go. I’d be able to use the same semantics that “serve http” uses with Go’s built-in web server.
For this, I’m not doing anything fancy with RESTful APIs, React, etc. (I would argue that this application does not demonstrate a clear need for extra frameworks - and also, javascript is not my expertise!) My prototype uses:
HTML templates for server-side rendering of the pages
semantic-ui as the front-end design
reveal.js for slide shows
If I started making PRs to incorporate this, would that be a welcome addition to the code base?
You could use this rest API to interact with rclone which should be a lot quicker than using a subprocess.
I also made a new command rclone rcd which does nothing but serve the API.
So you can type
rclone rcd --rc-user user --rc-pass pass
or
rclone rcd --rc-noauth
to serve the remote control. Use rclone rc while it is running to see all the API. You’ll need authentication to do anything which involves looking at files (or use the --rc-no-auth flag).
There is no way to upload files right now, but you can use operations/copyfile to copy a local file to a remote file.
I’ve also made a proof of concept react app which talks to rclone via the API. You can serve this via rclone with the --rc-files flag.
My plan for this API is to enable the building of GUIs for rclone (and other scripting uses).
I can’t incorporate python code in the release as it would break the release model of a single binary so I think it would have to go into a separate repo with a separate release model.
Try this build - it may not work yet though! You can always check out the branch from github which will compile, it just keeps failing the CI so the binary doesn’t get build!
Sure, I’d be happy - prefer even - for this to be the behavior of http remote! I had assumed that the current behavior was intentionally spartan
Before I embark on this (does not seem too difficult, in principle), here are some tradeoffs I see:
This will increase the “bloat” of the http remote - just more artifacts (CSS, styles, HTML, etc) for functionality
Increases dependencies on other libraries. In my case, a dependency on front-end libs such as jquery or semantic-ui. These can be statically served, but still, its another lib that might need to be ugpraded
Feature creep poetntial. For example, I’ve implemented an “image slideshows” feature. This would be an optional feature (and perhaps a welcome one) - I’m just not sure if your guiding principal is around minimalism for features like this.
If you’re relatively okay with #1 and #2, then I can take a crack at a first iteration of this. I think the bulk of the PR would be adding separate HTML and static content - but the existing templating logic would more or less be the same.
Point #3 is more of a question about future iterations - where do you see the future of the built-in web GUI? Something minimal, or something more feature-complete or media friendly?
Spartan - that is a good word! I didn’t put any effort into styling it!
Because rclone is a single binary install these artifacts have to be embedded within rclone somehow. There is a stanard way of doing this with go-bin, but I don’t want to bloat the binary too much!
one possible ideas is that some of the artifacts could be served from https://rclone.org or a specialized subdomain.
I’m intending the serve http logic to become part of the rclone gui. I’m about to merge a branch rest-api which factors the serving of index pages and objects to its own directory cmd/httplib/serve so the logic can be reused in the remote control server.
My thinking behind this is a fancy GUI is probably still going to want to be able to pop up html pages with indexes or serve the objects from <a href=""> links.
I suggest you either start from the rest-api branch or wait until I’ve merged it (should be soon!)
i’d quite like rclone to have the option to do both!
I was thinking maybe that rclone could fetch its assets into a local cache for the rclone gui. Maybe serve html could work like this too.
serve html should work without any internet access, but if it degrades down into “spartan” mode that is fine!
I haven’t yet had time to pull down your branch yet (I’ve looked at the code but not really looked at a diff). I will do so! Some thoughts from the above:
If the goal is to have the (new) restful endpoints power a GUI, that implies some things about the GUI’s architecture. Namely that it will be more of a “single page app”, where the GUI primarily uses ajax calls to render the page, rather than a more traditional (old school?) web app where the server is responsible for rendering HTML. This is not how I myself would implement this but it’s certainly fine!
If that is true, does this mean that the new serve http would actually be a superset of serve rcd - whereby we run both the REST endpoints along with additional static HTML routes for the GUI?
In terms of static assets, if this were my project, I’d package the dependencies using go-bin (or whatever is in vogue these days.) This would add megabytes to the binary, but I see that as the lesser evil compared to relying on an external server (or CDN) for these libraries. After all, if I’m running a command dealing with my personal files, do I expect rclone to be making network calls to anywhere except for my remotes? Also, since this project is standalone, why introduce a dependency on some external server? I am reminded of the Lamport definition of a distributed system…
A REST API could power either type of server I think. It is just a more efficient was that shelling out to rclone for each thing.
I’m thinking serve http would remain without the REST endpoint, but if you want that then you have to use serve rcd (which I may rename at some point to serve gui if we ever get that far!)
A good point… I think it depends on how big the assets are really!
Hello! I haven’t forgotten about this! (Have been reading code, and of course other life happenings.)
I’m interested in whether you have a preference for how to structure code.
You currently have a lot of shared code in httplib/serve/dir.go which, interestingly, depends on this specific template for serving lists of directories. This code is shared by rcd and serve http, among others.
If I updated this to use a more interesting template, would you be okay with this template being used for any action which serves a GUI? If not, then I may have to do a bit of refactoring to separate the Serve() method from the code for rendering templates, or pass a template option into that method.
If that template looked nicer, I don’t think anyone will complain as long as it still works with rclone serve http etc. I think that would make an excellent first pull request.
Hello! I have made an initial PR! It does need to be rebased against master, and I need to clean up some of the vendor-related commits, but the tests do pass (no regressions) and here is what the code does:
I had started working on a PHP+MySQL solution. The site would let an admin create accounts/users and specify rclone config options (either a path to the config file or the config details themselves). The PHP could would just call rclone lsd or something similar to get the data and then present it up as HTML.
If you get a chance to implement I had one idea for letting user’s download encrypted files securely using password protected zip files. From the web a user could request to download a file on an encrypted remote and the user would provide a password. The PHP would download the file, pipe the file to 7z which would zip and password protect and write the file back to an unencrypted remote and create a public link which it would then provide the user.
I will say, it might be worth while to add the new web GUI as rclone serve http_new or something. The one that is there right now is slim for quick and dirty with minimal resource requirements – on both the front and back end.
Hello @ncw! Thank you for merging PR 2844, adding vfsgen for static items!
Here is a question: how should we deal with static assets?
For example, we may have a file called “styles.css”, and normally, that might be served via /static/styles.css. However, this would make it impossible for a user to browse a folder named /static (!!)
Here are some options:
Choose a sentinel value for static assets. For example, “/rclone-static-112358” or “/random-number-11235”. Downsides: now we have a hardcoded value which a user is never able to browse.
Inline everything. That is, rather than our code saying <link href="/rclone-static/styles.css">, simply inline it. The two disadvantages is that this does not work for images (for example, an icon depicting a folder), and this would increase the HTML payload size upon every request. (Technically, one could inline .svg files.)
Choose not to use any HTML features which rely on static assets, such as images. This could be very limiting.
Host assets externally (eg, <link href="http://example.com/styles.css">). I think this is the worst idea, since how rclone’s http feature would have an external dependency, which has both stability and security implications.
Run another local HTTP server, on a different port, whose sole job would be to serve static assets. Eg, <link href="http://localhost:9090/styles.css">. This would give the most flexibility, though probably has the most complexity.
If it were me, I would probably choose option 1, or possibly option 2, or option 5. But maybe you have an idea for a solution I haven’t thought of, or a tradeoff that you’d find acceptable.
A good question indeed! I was thinking that we could use a special directory like @rclonestatic@ and keep them all in there.
Another way of sorting this out would be to use a parameter string so append ?rclone=static to each URL and get the URL handler to separate them. This would have the advantage that there would be no namespace collision.
I’m not keen on options 2-4 and 5 would indeed make life more complicated. The URL parameter idea is probably the best one as it is “out of band” and all the URLs should be generated by us.
Okay, I have a work in progress branch that I would love your feedback on, if you have time. I’ve added the ability to:
Serve static files with the ?rclone=static URL param
Implemented a slightly richer GUI:
The ability to navigate “up” a directory
Show file sizes and modification times
This is important because, when we want to add even more features (yay!) to the GUI, we won’t be limited to the fields which are currently in the serve.DirEntry data structure.
