Hi!
Is it possible to let the rclone copy command copy a single file to amazon cloud drive at
full speed?
When i upload several files i get full speed. With a single file i am stuck at 1/4 of the performance
This happens with Ubuntu 16.04.1 in a VM and rclone 1.34
On MacOS the Upload is fine.
I was not able to figure out why.
Thanks!
Yea it weird, iam using rclone on ubuntu as well and performance for 1, 2 files is really slow compared to multiple files where I can easily get 50 to 80MB.
When you upload multiple files you will be using multiple TCP connections.
Is it possible something in your network is limiting transfer speeds of individual TCP connections?
I can also confirm this. I have used rclone on multiple VPSs (debian and ubuntu linux).
Transferring a single large file caps at ~25 mb/s most of the time. Transferring 8 or more files with -transfers=8 nearly maxes out a gigabit connection and results in constant ~100 mb/s for me.
I think this limitation originates from amazon rather than from rclone but this is just a guess.
While I am at it: Thanks for this awesome tool and keep up the superb work Nick! 
That would be my guess. Otherwise it would imply that rclone is limited in something… If you transfer a single file, how much CPU does rclone use? Is it maxing out a core? I doubt this is the problem, but it would be worth checking.
You are welcome!
It is a old topic but I’d like to say that I’m passing for the same problem. I’m using OneDrive and when I try to upload multiple files I get the maximum speed from my VPS, 100 Mbps or 12.5MBps, but when I try to upload one single file I get at most a constante upload speed of 32Mbps or 4MBps. Is there any way to tell rclone to create multiple TCP connections threads for one single file, just like how download managers works, example:
In this example, JDownloader2 is set to “split” the download in 20 different chunks which are then downloaded simultaneously using 20 TCP connections threads.
My suggestion is to create one rclone option (–threads=N) that works behaving like JDownloader2 but in particular, creating multiple “upload chunks”:
rclone copy pathToSourcefile onedrive:pathToDestination --transfers=3 --threads=20I don’t know if this is possible, or if there is a limitation from onedrive and the others cloud providers.
For single file, I have got max upload speeds at 65MBps on a 1Gbps network. Max download speed is around 70-72MBps. But yes, the speeds are not consistent enough. Sometimes it is steady and at other times it fluctuates. For multiple files, max upload and download speeds reach around 75-80 MBps.
Overall speeds are ok.
Isn’t there an option called onedrive-chunk-size which does exactly this?
Take a look at the documentation.
This option has the default value of 10MB, so I think that this is already a good value.
There is another option available:
[quote]
–onedrive-upload-cutoff int Cutoff for switching to chunked upload - must be <= 100MB (default 10M)[/quote]
But this option is already set by default to 10MB, that seems to be a good value as well.
I think that the rclone internal chunk system is not working the way I explained in my previous message.
My idea is like this example:
One file with 1GB of size is copied with rclone copy command configured with 20 threads of upload (–threads=20), so rclone will create 20 TCP connections to transfer 20 parts of 50MB of this 1GB file.
OneDrive probably doesn’t support this. Looking at the API documentation you can see that fragments have to be in order. I can’t test it but this either means that you can only start a new fragment when the old one is finished (this does not support multiple connections) or that you can start multiple fragments at once but only finish fragment X when fragment X-1 has finished (in this case your idea works).
Maybe someone who knows the API better can answer this.
I googled for "multithreaded upload of single file" and I found this:
That is what I am looking for.
Finalizing the object as described in the picture however has to be done on the server’s end. In your case OneDrive. If OneDrive does not support this, then there is nothing you can do about it.