Rclone serve question

could someone offer a little insight on the rclone serve feature. i'm wondering if i can use it to copy files between two windows computers using http. the reason is, they are 500 miles apart and smb on high latency is horrible. or, just tell me this is not how the feature is meant to be used and i'll understand. i haven't used it in our environment, but am wondering if it could solve this issue for me.

jared

hey @jared,

rclone has rclone serve sftp and that supports checksum verification.

i use tailscale in such cases

sftp is not exactly known for it's speed either :slight_smile: tailscale, which seems like an interesting tool, is essentially a mesh vpn network. wouldn't that take me back to smb again?

we've got a server in one state, and i'm just trying to send veeam backups to a one of our servers in a different state for backup purposes. it doesn't have to be hyper fast, but rclone copy is getting about 6mbps right now. more would be nice. just looking around.

jared

Hi jared,

You can work around high latencies by increasing/doubling --transfers on the rclone sync/copy command until you saturate bandwidth, CPU or disk - whatever comes first. I know this works on https and sftp, and expect the same on smb - and if using tailscale.

hey ole,
thanks for dropping by. there's been something i've been meaning to bring to your guys' attention, but haven't wanted to bug you with it. and that something is, rclone is really slow when it comes to single file transfers. i know i will probably get polite critiquing by certain monkeys in this forum (a) for using rclone for local transfers, but i just love the tool and it works really well, especially (among other things) because as mentioned, it allows parallel transfers, and is thus faster (than windows explorer) for copying large amounts of small files.

but for copying a single large file, for whatever reason, rclone is way slower than the windows explorer copy/paste feature. about 70% slower it seems. perhaps you guys were already aware of it; i'm certain you have this tool under more use cases than i ever will.

here is what an rclone transfer of a 2gb video file to a local server looks like. as you can see, the copy will jump right up to about 300mbps and remain there. completely stable.

here is what the same file with windows explorer looks like. as you can see, it immedialy pegs out the 1gb network connection for as long as the transfers takes.

the reason i mention this is, for the veeam backup that i started this post about, i am trying to copy single 50-100 GB virtual hard disk files over a high latency connection. when using windows explorer, i'll get about 30-40 mbps, but with rclone i get about 6 mbps. for some reason. that's why i was wondering about the http transfer question. just trying to see if there's yet another capability of this amazing tool that i'm missing. i also tried making the transfer with chunker, but chunker seems to also just transfer one chunk at a time and therefore doesn't offer me speed improvements beyond the 6mbps i was already getting.

as an aside, one of the reasons i use rclone for the transfers is because it does automatic retries if the 500 mile connection proves not stable enough for windows explorer. as is often the case, though i haven't gotten to the bottom of why. just some thoughts.

regards,
jared

Hi jared,

I cannot understand your situation well enough to comment. I don't have enough information and some of your observations also seems contradictory (to me).

I therefore suggest you try to reproduce the 6 mbps transfer rate with a smaller file placed in a testfolder with only this one file. Try finding a file size that can be transferred in 5-10 minutes to allow for easier test and troubleshooting. You can use --ignore-times to force (re)transfer of an unmodified file.

When reproduced make a new topic and fill out all the information in the help and support template, then I will be happy to see if the transfer can be optimized to reach the speed you see with Explorer. Feel free to ping me to catch my attention.

absolutely. here is a link to the topic. feel free to request additional information.

jared

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.