Rclone in high availability mode

What is the problem you are having with rclone?####

We are using rclone in single instance mode right now.

But we have requirements to run Rclone in HA mode where we need to make sure atleast one instance of rclone must be running at any point of time.

We are fine with primary and secondary mode also, so when if one rclone instance goes down another should come up with the all the same remotes already configured..

Can any one please suggest the better way to achieve this?
As of now we have rclone installed on VM, but we would be moving docker and K8s.

Thanks
Mahendra

To do what? A mount? A copy? A sync? A what?

What's the use case that you are trying to make HA? Can you explain it a bit more.

Thanks for the reply @Animosity022 .

Apologies for missing that info -

We are using rclone in rcd mode.

And from tool called 'Apache Nifi' we are calling rclone api's to create remotes and then transfer files from cloud to cloud.

Invoking http://localhost:5572/config/create for remote creation &
invoking http://localhost:5572/sync/copy for transfer directory

Apache Nifi supports clustering (which enables HA) and we are seeing how we can run rclone in HA mode.

Since you aren't using the VFS cache, the rclone rcd is almost stateless and should load balance OK I think. Though if you are using async polling then that might not work so well unless you have persistent connections. It will certainly work Live and standby.

hi @ncw

I think I have not got you fully.
Sorry I am quite new to rclone, what do you mean by 'persistent connections' here ?

We might not need load balancing also in our case, but shared config (in case primary rclone server goes down then secondary should be able to use remotes created by primary etc)

Rclone isn't built to be 'stateful' so if you have 2 instances running, they can't act as a 'both on' type persisting across if one goes down like a stateful failover on a firewalll cluster/load balancer/etc.

You can load balance them and do a primary/standby but anything in flight on the primary would need to be replaying on the standby.

I'd imagine you can also have a VM that would failover from one host to another, but the same consideration I think would apply as you'd have to replay things.

That's a load balancer term. It means that connections from one IP keep going to the same instance if at all possible.

The rclone config file is just a text file so easy to sync and/or store on shared storage. For the oauth based backends, rclone writes to the config file so you have to be prepared for that.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.