Clustered rclone ops?

Hi,

I work with a community of storage engineers who work with large datasets. I notice that rclone is designed to work as 1 worker node (host) to transfer data from one cloud drive to another. Pretty cool. Is there any thought to having rclone work in a distributed mode? For example, have 10 nodes (hosts) transferring data from one cloud drive to another?

Maybe this already is a feature, I just need to learn how to use it?

Thanks,
Steve

There are various ways you could do this with rclone as it stands now but it isn't set up to do distributed copying at the moment.

Rclone is quite good at multithreading so you can increase the number of transfers with --transfers to speed things up.

I guess if you want to work faster than one machine could work then splitting the work up would be beneficial.

one thing you can do is make a list of files you want transferred from the source (with rclone lsf -R maybe) and then cut this into chunks and feed that into a number of rclone instances with --files-from-raw

Thanks Nick! Makes sense. I can take the list and feed it into a work queue like RabbitMQ, then feed chunks into the minion rclone instances.

I'll post results when I make some progress.

-Steve

1 Like