Recently I had to download all the data I have uploaded to the cloud. I had more than 2tb of files and most of these files are very small. I know rclone has a bad time transfering small files, as there is a lot of overhead for each transfer. But I was bored by how slow it was. I had a internet connection speed which was very fast (around 1Gbps) and rclone was downloading the data at around 40Mbps (5MBbytes/s) speed.
I have tried making multiple transfers (up to 50), and I have created my own API token on google, I had some improvement but it was really not enough. It would take me like 2 weeks to download everything.
So I decided to try other apps. I found out that oDrive had a CLI that allow users to make as many simultaneous downloads as they want. I gave it a try and I was impressed/suprised by the fact that with odrive CLI I was getting download speeds around 5 times faster than I was getting with rclone. And I was even more astonished by the fact that odrive uses a CLI based in Python while rclone was created in Go. As we know Go is a very fast programming language, and should give speeds comparable to C or C++. But for some reason rclone still very slow in some tasks. So I was wondering if you guys know the reason why there is so much overhead for each file transfer? And is there any work in progress or ideas on how to optimize it. I am pretty sure that if we do some effort we will be able to make rclone much faster than what it is today.
How many transfers does odrive use do you know? How many did it use to get such impressive speeds? Have you tried with --transfers 250? You might want to raise --checkers too.
Any insight into how odrive is different from rclone would be much appreciated!
I have tried very high --transfers numbers, but after some 30/40 simultaneous transfers I didn’t get any more speed improvement. I also played with --checkers numebrs.
With odrive CLI we can also choose the number of simultaneous transfer. With only 30 simultaneous transfers it was already much faster than rclone. I didn’t tested with much higer numbers because I didn’t want to get banned. This is not a critic towards rclone. I am just wondering how to make it better and achieve the same results other software did. It’s a pitty that odrive has nothing open sourced. So we can only wonder what they do. But I suggest you to try odrive CLI with huge amount of small files, and you will see how big the difference is.
I am wondering the same thing. It might be because odrive has a higher rate limit for it’s client_id? I just checked and there isn’t a batch operation for upload in the google drive API.
I’ve never been able to upload files faster than about 2 per second - that is some kind of rate limiting at google and I expect that is the problem.
I have used my own client_id with rclone.
And also I don’t think it is a matter of how many files you can transfer per second, but the speed of each transfer. I am not sure about that though.
It might seem hard to believe, but I was able to download in 2 days, what was taking me 2 weeks with rclone. I didn’t spend much time tricking the odrive cli.
If you want to test it yourself I can send the commands I have used.