How to serve files with rclone to another server?

rclone seems designed to allow moving to lots of different storage end points but I don't seem to see a way to use it (as it's efficient) to serve local files and allow me to copy to another system (between servers).

There are examples of rsync and that's single threaded and various hacky ways to try to use it in parallel to gain efficiency. I've used rclone for talking to remote devices like google drive or dropbox, but wondered if it can also be used to "serve" files allowing copying between systems but haven't found examples of the appropriate invocation on each side to achieve this. I'm looking to copy tens of TB over a 10G/b network interface and want to do this as efficiently as possible. I've found other methods I have been using do not seem to allow me to do this as efficiently as I would like and think that maybe rclone can do this better.

If this is possible could someone provide an example? e.g. to allow me to sync or copy data from server1:/data to server2:/data using rclone.

Thanks,

Simon

1 Like

https://rclone.org/local/

The first example on that page shows the exact same behavior.

I think what is important to understand is that rclone isn't a server. it can connect to a whole bunch of different servers, be they cloud or other types, but you can't get around that you need a server on one of the machines to acomplish this. You need an endpoint to connect to on one of the sides at least.

I'm not quite sure why you need rclone to move data via just two servers, but if you have control over both then why not just set up a normal network share? (a network share is technically a file-server). If you want to use rclone over that network share for whatever reason, you can use the "local" remote as Darth points out to connect to that location. You also have other options for servers like FTP, webdav ect. but a network share is probably the most efficient and compatible I would bet.

You can use one of the rclone server commands with the corresponding backends, so

  • rclone serve sftp with the sftp backend (this is the best combo I think)
  • rclone serve ftp with the ftp backend
  • rclone serve webdav with the webdav backend.

Note that if you are running unix with ssh, then the sftp backend will likely work with your standard ssh daemon which will save running rclone serve sftp.

If you've got ssh agent set up you can use it without any config, eg

rclone copy --sftp-host example.com :sftp:path/to/dir /tmp/dir

What OS do the servers use?

1 Like

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.