Cluster mode for rclone rcd when running in kubernetes or docker compose

[shaunakv1 (Shaunak Vairagare)]

This feature request is to address the specific issue described in this post:

Running Rclone rcd in Kubernetes cluster in loadbalanced mode for HA

What is the problem you are having with rclone? We are running rclone rcd pods on our k8s deployment in internal load balanced mode. From other pods on the cluster we talk to rc HTTP API to start copy/sync jobs in async mode. Every job that starts the copy/sync request then makes subsequent HTTP calls to check for job status every 30 seconds. This works great if we have only one rclone pod. However if we have it scaled to multiple rclone pods, then internal load balancer routes the subseque…

When deploying rclone rcd to kubernetes cluster, the internal load balancer routes the http requests to one of the many available pods.

This works fine when queuing up jobs, but when checking for job status using http requests if the request is routed to a pod that's not handing the jobid in question , we get an error.

Feature Request

Add a "cluster mode" for rcd where various rcd instances on a network are made "aware" of each other's state data.

This can potentially be achieved by providing a specifc external volume attached to all rclone pods which they use to share state data and possible be aware of progress across instances.