Using only one RCLONE instance for different commands?

[steinbohrer (steinbohrer steinbohrer)]

What is the problem you are having with rclone?

I would like to know if it is possible to start only 1 RCLONE instance and then run several different commands (like: lsjson, copy, etc) over it.

The documentation on the "--rc" flag ( Remote Control / API (rclone.org)) states that RCLONE will start an HTTP server.
But this does only work for a specific command. I find the docu about "--rc" "rclone rc", and "rclone rcd" really hard to understand. I do not get the actual purpose of each and what distinguishes one from the other and when to use one and when to use the other....

I would like to run several commands on the same remote, like:

• first run a "lsjson" to get the content.
• run a "copy" to copy specific files.

Do I have to start RCLONE each time I want to execute a specific command?

To read a specific file content I already use "rclone serve http ...".

What is your rclone version (output from rclone version)

rclone v1.57.0

• os/version: Microsoft Windows 10 Enterprise 2009 (64 bit)
• os/kernel: 10.0.19044.1415 (x86_64)
• os/type: windows
• os/arch: amd64
• go/version: go1.17.2
• go/linking: dynamic
• go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

I would like to use any or all RCLONE supports.

The command you were trying to run (eg rclone copy /tmp remote:tmp)

This question is not about a specific command.

The rclone config contents with secrets removed.

dito

A log from the command with the -vv flag

dito

[Animosity022]

Think of it like a webserver as you can run one webserver and do as many things as you like against that web server.

You start by spawning a rclone "webserver" in this example. You can do that by adding -rcd to your existing mount command.

I add:

--rc --rc-addr 127.0.0.1:5575

To my mount and that makes it remote controlable on the IP/Port listed above. You can run rclone rcd itself to spawn just a blank instance of rclone and do what you want there. You can run a rclone serve http with rc and do that. Options are quite large.

For controlling it, once you have remote control running, you can hit that "webserver" as many times / as many ways as you want but using the rclone rc command to hit against it.

So on my mounts when they load, I refresh their directory/file structure by using:

rclone rc vfs/refresh recursive=true --rc-addr 127.0.0.1:5573 _async=true

Which runs against a remote control mount I've started earlier.

[steinbohrer (steinbohrer steinbohrer)]

Hi Animosity022,

thanks for the quick reply.

I think the problem I have, and what the docu on the "--rc" flag misses, is, that the webserver only works on a specific RCLONE command, like "mount" in your example here.

I would like to use RCLONE on many different machines (Windows, Linux, etc.) and, at least on Windows, you first need to install some additional software before you can use the mount command.

I'm looking for a way to start only 1 RCLONE process and then use it's API to run any of the available commands in RCLONE.

But from what I understand now, this is not possible.

Thanks,
steinbohrer

[Animosity022]

The mount is just one example.

I'm not quite following.

On Machine A: run rclone rcd /rclone serve rcd / rclone mount /rcd / rclone anything with rc enabled
On Machine B: run rclone rc commands against Machine A
On Machine C: run rclone rc commands against Machine A
etc.

[steinbohrer (steinbohrer steinbohrer)]

Ok, then I will explain my use-case in a bit more detail:

I would like to integrate RCLONE in my product to give my customers access to files stored on any of the cloud providers RCLONE supports.
A customer would then specify the connection details (remote, bucket, credentials, etc.) to connect to his/her preferred cloud provider.
I would then run a "rclone.exe lsjson remote:bucket ..." command to get all files.
After that I want to copy one or more files (from the same remote), so I run a second RCLONE command:
"rclone.exe copy remote:bucket/filename destinationfolder --inlcude "some filter" ...".

Would it be possible to run both commands on only one RCLONE process/instance?

[Animosity022]

Sure, you can run 1 command, 10 commands or as many as you want.

[steinbohrer (steinbohrer steinbohrer)]

I'm really sorry, but I don't get it.

What exactly do I have to do to:

1. start RCLONE
2. run a "lsjson" command.
3. run a "copy" command.

I do not want to mount anything!

[Ole (Ole Frost)]

@Animosity022

Perhaps it would help if you are able to show a simple command/example starting rcd and then using curl (or similar) to control it?

(Outside my current expertise/experience)

[Ole (Ole Frost)]

@steinbohrer

I'm really sorry, but I don't get it.

This thread may give you an idea:

Rclone rc HTTP call with _config flag fails

OK, then let's try with curl. Assume I have started rclone rcd with default parameters in another terminal. What are the commands needed to perform my above example, I guess something like: $request='{…}' curl -H "Content-Type: application/json" -X POST -d $request http://localhost:5572/sync/sync

[Animosity022]

Start a RCD.

etexter@seraphite ~ % rclone rcd --rc-no-auth
2021/12/17 09:11:32 NOTICE: Serving remote control on http://localhost:5572/

Run something against it a few times.

etexter@seraphite ~ % rclone rc config/listremotes
{
	"remotes": [
		"DB",
		"UN",
		"TEST"
	]
}
etexter@seraphite ~ % rclone rc core/stats
{
	"bytes": 0,
	"checks": 0,
	"deletedDirs": 0,
	"deletes": 0,
	"elapsedTime": 99.984496291,
	"errors": 0,
	"eta": null,
	"fatalError": false,
	"renames": 0,
	"retryError": false,
	"speed": 0,
	"totalBytes": 0,
	"totalChecks": 0,
	"totalTransfers": 0,
	"transferTime": 0,
	"transfers": 0
}

[steinbohrer (steinbohrer steinbohrer)]

Thanks,

so the "rcd" command starts the remote control server and not the "--rc" flag (as stated in the docu).

Ok, I can run this and it's waiting for input, cool.
But now, I have to start a second RCLONE process to query that remote control?

That would not give me any advantage in running one RCLONE process for "lsjson" and another one for the "copy" command.

You see, the whole point of my question was/is to have only one RCLONE process running on the customers computer and to use this, for example, over HTTP, like the "serve http" comand enables me to do.

Because of it's size (around 42MB) the RCLONE.exe file always takes a couple of seconds to start and if you need that multiple times, it adds up.

[Animosity022]

@steinbohrer

so the "rcd" command starts the remote control server and not the "--rc" flag (as stated in the docu).

No, the remote control can be enabled on anything so I think it's just a bit confusing.

The RCD is the remote control daemon.

So on my mount, I don't start a RCD daemon, as I just start the remote control portion.

@steinbohrer

But now, I have to start a second RCLONE process to query that remote control?

That's how any server / client model works. You don't have to use rclone, you can use CURL or anything other method to access the API you want.

Remote Control / API (rclone.org)

@steinbohrer

You see, the whole point of my question was/is to have only one RCLONE process running on the customers computer and to use this, for example, over HTTP, like the "serve http" comand enables me to do.

This goes back to a client - server relationship. You can start a server, but need a client to interact with the server.

@steinbohrer

Because of it's size (around 42MB) the RCLONE.exe file always takes a couple of seconds to start and if you need that multiple times, it adds up.

Not sure what you are using as mine command takes:

rclone rc config/listremotes  0.04s user 0.02s system 112% cpu 0.051 total

[Ole (Ole Frost)]

@steinbohrer

so the "rcd" command starts the remote control server and not the "--rc" flag (as stated in the docu).

Both start the remote control server, seems like rcd is the correct choice in your situation.

--rc can be used when you e.g. start a mount (like @Animosity022) and then would like to remote control it for directory refreshed etc.

@steinbohrer

But now, I have to start a second RCLONE process to query that remote control?

I suggest your start rclone rcd in a terminal and then start another terminal and try some simple commands using curl.

When familiar with this then try creating and using a remote (e.g. GDrive) using curl.

When familiar with this then start sending commands from two terminals.

[steinbohrer (steinbohrer steinbohrer)]

I'm currently developing the stuff on my Windows PC.
When I type "rclone.exe ..." and hit the ENTER button, it takes roughly 2 to 3 seconds before I see any output.

Ok, if I got this right now, I start RCLONE with the "rcd" command, like:

rclone.exe rcd --rc-no-auth --config="myConfig.conf" --rc-addr :5025
2021/12/17 15:42:33 NOTICE: Serving remote control on http://[::]:5025/

to have a HTTP server running that I can then query via any HTTP client to run a "operations/list", which seems to be the equivalent to the "lsjson" command?

I will try that...

Thanks for your help Animosity022 and Ole.

[Animosity022]

@steinbohrer

I'm currently developing the stuff on my Windows PC.
When I type "rclone.exe ..." and hit the ENTER button, it takes roughly 2 to 3 seconds before I see any output.

On Windows, I get:

PS C:\Users\earlt\OneDrive\Downloads\rclone-v1.57.0-windows-amd64\rclone-v1.57.0-windows-amd64> Measure-Command {start-process rclone.exe --wait }


Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 0
Milliseconds      : 24
Ticks             : 242117
TotalDays         : 2.80228009259259E-07
TotalHours        : 6.72547222222222E-06
TotalMinutes      : 0.000403528333333333
TotalSeconds      : 0.0242117
TotalMilliseconds : 24.2117

So 24ms for me.

[steinbohrer (steinbohrer steinbohrer)]

Lucky you:

PS C:\Originale\rclone\Test> Measure-Command {start-process rclone.exe --wait }

Days : 0
Hours : 0
Minutes : 0
Seconds : 5
Milliseconds : 215
Ticks : 52158426
TotalDays : 6,03685486111111E-05
TotalHours : 0,00144884516666667
TotalMinutes : 0,08693071
TotalSeconds : 5,2158426
TotalMilliseconds : 5215,8426

[steinbohrer (steinbohrer steinbohrer)]

I think, it also heavily depends on the security stuff your IT has put on your machine, e.g. virus scanner(s) and other security software which can influence how fast an application starts.
And in larger enterprises, those stuff is a thing one cannot ignore.

[Animosity022]

If it takes 5 seconds for a 42MB binary to return, something else is going on.

I'm running on a super tiny cloud VM that has windows to test on.

@steinbohrer

I think, it also heavily depends on the security stuff your IT has put on your machine, e.g. virus scanner(s) and other security software which can influence how fast an application starts.

My background is in the IT landscape(Infra/Desktops/etc) and not a developer so if there are items on a machine that makes a 42MB binary take 5 seconds to open, something is not right.

I'd imagine you probably have an old machine with old spinning disk as that's just bonkers. If you run it twice, does it take 5 seconds?

[steinbohrer (steinbohrer steinbohrer)]

When I run that measurement command multiple times in PowerShell, it goes down to roughly 200 ms.
But if I run my full RCLONE command from CMD (or from inside my application via Process.Start()) it always takes 2 to 3 seconds.

Oh, and I have a quite good notebook with an SSD.

[Animosity022]

@steinbohrer

But if I run my full RCLONE command from CMD (or from inside my application via Process.Start()) it always takes 2 to 3 seconds.

I'm not sure what that means. I was just running the binary as if you are executing something else after it, the remote/command/etc all can feed into the time that the command takes to run so a transfer might take 1 minute to complete but that's not really an accurate test of running a binary.

@steinbohrer

Oh, and I have a quite good notebook with an SSD.

That would tell me something else is going on as if you have a SSD a read from a 42MB binary takes 5 seconds, there is something else afoot and that's beyond rclone support. Good luck!