Introducing rclone.js

Off Topic
1

Hello everyone! Hope y'all have a great time with family and friends this season and be ready for the new year.

As we are getting back to work, I would like to take the time to introduce a side project that I have been working on, called rclone.js.

Homepage: https://rclone.js.org/
GitHub: GitHub - sntran/rclone.js
NPM: https://www.npmjs.com/package/rclone.js

As the name indicates, it's a Javascript wrapper around rclone, specifically in Node. Besides providing a Javascript API to integrate into your existing project, it also provides a CLI. Wait, I know what you are thinking. rclone is already a CLI, why having another CLI on top of it?

For my use case, it's easier for me to maintain rclone, as I work in Node.js a lot of time.

But as of version 0.6.0, the CLI allows us to use custom command that can be written in Javascript. This allows Javascript developers like me to have desired functionality faster than trying to figuring out Go.

For example, I have a custom command, aptly named rclone-archive, which will be published soon after ironing out some quirks. This command lets us archiving a remote path into another remote path, and can be used soon, while waiting for Streaming archive/unarchive capabilities · Issue #2815 · rclone/rclone · GitHub to be implemented.

The usage would be npx rclone archive source:path dest:path.

Besides supporting new functionality, rclone.js also allows us to override existing commands to add support for more backends or flags. Imagine adding a new cloud provider with copy or move.

Anyway, I hope it can be useful to some of us, and I really appreciate any feedback.

2 Likes
2

As mentioned before, here is the rclone-archive "plugin" for rclone.js:

This is the very first release, so bugs are expected. It is also a stop gap until Streaming archive/unarchive capabilities · Issue #2815 · rclone/rclone · GitHub is implemented in rclone itself.

3

I think I'm missing the point, help me out here.

Why not use an alias for rclone-archive instead of writing a node.js wrapper for it? What's the use case for your project? I love rclone, but I don't understand what your thing does/is for. I'm not saying your program is bad, just unclear.

4

The case is to make rclone embeddable in other applications and storage related workflows.
The core team has done a lot of steps in this direction: REST API, librclone for C and python bindings, permissive MIT license, open source.
I personally baked a simple wrapper to be used with python pandas to pull statistical data off storage providers.
So why not have a wrapper for node.js or other ecosystems?

5

What Ivan said.

Also, to clarify, rclone.js is a wrapper to call rclone API from within Node.js projects. For example, a Discord bot that needs to handle data, or a web server... While it's not that hard to just spawn a rclone process from within Node.js environment, having a wrapper reduces the boilerplate needed.

rclone-archive is a "plugin" for rclone.js that adds archive command to rclone. You can say it is one of the use cases for rclone.js, albeit small usefulness. rclone.js let JavaScript developers extend rclone with extra functionality using the languages they already know, not just Go.

6

Could you explain if or. how you support "special" flags such as _async ? As far as I can tell the module doesn't interpret these correctly, and treats them as normal flags, to which it adds '--'.

7

Hi there,

Can you give me a full example for such flag? I don't think I ever used one, so I looked into rclone documentation, and found a few used with RC call.

For example

rclone rc --json '{ "p1": [1,"2",null,4], "p2": { "a":1, "b":2 }, "_async": true }' rc/noop

In this case, "_async" is part of the JSON passed to the --json flag.

Admittedly, I did not expect that, so I can push a fix for this case today.

Is there an actual usage of _async as a flag?

9

@sntran Sorry, it's a special parameter rather than a flag - my mistake

10

No worries. I was confused myself reading rclone's doc.

Regardless, rclone.js did not parse that --json flag correctly anyway. I have pushed a commit to fix that if you want to try.

11

I need to keep rclone running even if its parent killed, So I need to set detached spawn option true, But it does not work

const upload = rclone.copy(srcRealPath, dstRealPath, {
              detached: true,
              'use-json-log': true,
              transfers: 100,
              'log-level' : 'INFO',
              'max-backlog': 999999,
              stats: '500ms'
            });
12

Hi! I'm a bit new to node.js so I might use some terminology wrong, but I hope you could still help me with my question :slight_smile:

I have an API route that I use in order to sync between several s3 buckets. I pass the name of the bucket using the controller, which calls the service and then the proxies.

In the proxies the actual call to the rclone.js package is happening. I need this whole process to be async, since I want to send back a response from my backend service about the status of the syncing (in case it fails or runs successfully with some info about the run)

The thing is the route call returns before the child process even starts running. How can i make sure that the request waits for the child process to start AND finish before sending a response and "making itself free" to take a new request?
(I want to return ONLY once the listener has recieved a "close" event)

From what I understand - there's a way to run this process asynchronously, but I'm not sure about:

  1. How to do it right
  2. If it will solve my issue

Excuse me for the long and detailed question :sweat_smile:

13

You said you wanted the whole process to be async, but you also said you want the request to wait for the child process to start AND finish before sending a response, which is conflicting.

async for me mean firing the request and do something else, not waiting for it. If you want to wait for it, then it's synchronous.

rclone.js has a Promise-based API, which provides async operations that you can await for them to finish (but not blocking). However, you can not listen to progress on that.

I don't know how your route is set up, but the idea is something like this:

function async onRequest(request) {
  const ls = rclone.ls("source:");

  const code = await new Promise((resolve, reject) => {
    ls.stdout.on("data", (data) => {
      console.log(data.toString());
    });

    ls.stderr.on("data", (data) => {
      console.error(data.toString());
    });

    ls.on("close", resolve);
  });

  // return Response.
  
}
14

Thank you very much for the quick reply!
I have a follow up question on the subject:
Shortly speaking: How can i kill a currently running process using the package?

I'll elaborate:
I'm using rclone.js for a route that syncs between different s3 buckets.
There are times when I might need to stop the syncing. Either because the user asked or for some other reason (exceptions or timeouts, for example), and once there is such a request - i want to "kill" the child-process. I have two questions about it:

  1. How can i do that in general?
  2. What is the safest way to do it? Meaning i won't jeopardize the info that is already synced or about to be synced?
15

rclone.js returns a ChildProcess for each command it starts, so you can use any of the methods documented at Child process | Node.js v22.1.0 Documentation.

For the example above, you would do ls.kill().

When you kill the child process like that, it's similar to do a Ctrl+C against the original rclone.