Fast transfer of 1TB file from windows server to another windows server?

Help and Support
1

Hello, new user here. I've tried scanning through the help and forum already and I haven't found this directly.

Basically, can I skip the cloud-part and move a large file efficiently from one windows server to another across the internet?

We have a colocated facility with a 1gbit pipe (but limited to about 450mbit through our Meraki MX67). We have a backup that's a single file about 1TB in size, and are having trouble moving it t our DR offsite efficiently. Although it's likely the classic "long fat pipe" problem, the ping times between the two locations are 4-6ms typically. But despite having 450mbit to play with (confirmed with various speedtests), we're stuck around 20MB/sec for SCP and about 16MB/sec for FTP/SFTP. If I run 4x SFTP file transfers at once, I can get 35-40MB/sec. The problem is splitting the file takes 5+ hours, and the backup tool itself can't do it.

Using the VPN built into the meraki and simple robocopy with 16 threads gets us up to 30MB/sec. But I'm wondering if rclone can help with even more? We want to go directly site-to-site (otherwise going to and from cloud somewhat defeats the purpose of trying to get increased speed).

Thanks in advance for any suggestions.

2

hello and welcome to the forum,

best to do a test run with rclone first and compare to the other tests.
i always create a dummy test file and use that for testing.
might tweak https://rclone.org/docs/#multi-thread-streams-n

about the backup file, what software generated that?

have you tried fastcopy?

3

You could try rclone serve sftp at one and then use rclone copy from at sftp backend.

The rclone sftp client has been optimised with long pipes in mind.

You may get more performance out of rclone serve http and using the http backend to copy files.

Note that rclone copy to a local disk will automatically stream the file with 4 download streams.

5

rclone serve sftp does checksums. rclone serve http doesn't though.

1 Like
6

ok, i did a quick test, i see that you are correct about rclone serve sftp
i deleted that post.

what i should have written was that some sftp servers do not perform the checksum.

about the rclone serve sftp, how does rclone calculate the checksum of the file to be transferred.
--- before transfer
or
-- during transfer

7

Normally after the transfer. On a normal sftp server it would shell out to md5sum or sha1sum. Whereas rclone serve sftp has an internal version of sha1sum/md5sum it runs. Same idea though.

8

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.