Introducing rclone.js

Off Topic
1

Hello everyone! Hope y'all have a great time with family and friends this season and be ready for the new year.

As we are getting back to work, I would like to take the time to introduce a side project that I have been working on, called rclone.js.

Homepage: https://rclone.js.org/
GitHub: GitHub - sntran/rclone.js
NPM: https://www.npmjs.com/package/rclone.js

As the name indicates, it's a Javascript wrapper around rclone, specifically in Node. Besides providing a Javascript API to integrate into your existing project, it also provides a CLI. Wait, I know what you are thinking. rclone is already a CLI, why having another CLI on top of it?

For my use case, it's easier for me to maintain rclone, as I work in Node.js a lot of time.

But as of version 0.6.0, the CLI allows us to use custom command that can be written in Javascript. This allows Javascript developers like me to have desired functionality faster than trying to figuring out Go.

For example, I have a custom command, aptly named rclone-archive, which will be published soon after ironing out some quirks. This command lets us archiving a remote path into another remote path, and can be used soon, while waiting for Streaming archive/unarchive capabilities · Issue #2815 · rclone/rclone · GitHub to be implemented.

The usage would be npx rclone archive source:path dest:path.

Besides supporting new functionality, rclone.js also allows us to override existing commands to add support for more backends or flags. Imagine adding a new cloud provider with copy or move.

Anyway, I hope it can be useful to some of us, and I really appreciate any feedback.

2 Likes
2

As mentioned before, here is the rclone-archive "plugin" for rclone.js:

This is the very first release, so bugs are expected. It is also a stop gap until Streaming archive/unarchive capabilities · Issue #2815 · rclone/rclone · GitHub is implemented in rclone itself.

3

I think I'm missing the point, help me out here.

Why not use an alias for rclone-archive instead of writing a node.js wrapper for it? What's the use case for your project? I love rclone, but I don't understand what your thing does/is for. I'm not saying your program is bad, just unclear.

4

The case is to make rclone embeddable in other applications and storage related workflows.
The core team has done a lot of steps in this direction: REST API, librclone for C and python bindings, permissive MIT license, open source.
I personally baked a simple wrapper to be used with python pandas to pull statistical data off storage providers.
So why not have a wrapper for node.js or other ecosystems?

5

What Ivan said.

Also, to clarify, rclone.js is a wrapper to call rclone API from within Node.js projects. For example, a Discord bot that needs to handle data, or a web server... While it's not that hard to just spawn a rclone process from within Node.js environment, having a wrapper reduces the boilerplate needed.

rclone-archive is a "plugin" for rclone.js that adds archive command to rclone. You can say it is one of the use cases for rclone.js, albeit small usefulness. rclone.js let JavaScript developers extend rclone with extra functionality using the languages they already know, not just Go.

6

Could you explain if or. how you support "special" flags such as _async ? As far as I can tell the module doesn't interpret these correctly, and treats them as normal flags, to which it adds '--'.

7

Hi there,

Can you give me a full example for such flag? I don't think I ever used one, so I looked into rclone documentation, and found a few used with RC call.

For example

rclone rc --json '{ "p1": [1,"2",null,4], "p2": { "a":1, "b":2 }, "_async": true }' rc/noop

In this case, "_async" is part of the JSON passed to the --json flag.

Admittedly, I did not expect that, so I can push a fix for this case today.

Is there an actual usage of _async as a flag?

9

@sntran Sorry, it's a special parameter rather than a flag - my mistake

10

No worries. I was confused myself reading rclone's doc.

Regardless, rclone.js did not parse that --json flag correctly anyway. I have pushed a commit to fix that if you want to try.

11

I need to keep rclone running even if its parent killed, So I need to set detached spawn option true, But it does not work

const upload = rclone.copy(srcRealPath, dstRealPath, {
              detached: true,
              'use-json-log': true,
              transfers: 100,
              'log-level' : 'INFO',
              'max-backlog': 999999,
              stats: '500ms'
            });