I’ve created a first version of multi threaded downloads here.
Multithreaded downloads are when rclone downloads a big file by using multiple download streams. In my testing downloads from drive can run twice as quickly with two download streams.
It could do with some more tests, but it is basically finished.
I’d really like some feedback on whether the interface is OK (the command line flags for controlling it) and what the defaults should be!
Long time since I logged here, and that is the first thing I see, a feature I’ve been wanting for a long time! Perhaps I will finally be able to stop using aria2c (which can do multithreaded downloads of single files, once I add an HTTP “intermediary” server somewhere) and can start using rclone for everything!
Great job, @ncw! Will certainly be testing that in the next few days/weeks.
PS: multithreaded uploads (ie, from local to cloud remotes) would be even better. Are they somehow being contemplated here?
Cheap and dirty way: run split on the file you want to upload and use --transfers=n to start n parallel transfers. But yeah, paralllel uploads would be great
Yeah, I know all about split. But the problem is, no way to re-join the files once they are uploaded (at least not server-side)... so it kinda defeats the purpose.
Multithreaded uploads depend on the the cloud provider. These are supported for quite a few providers such as google drive, b2, box, s3 etc. Dropbox annoyingly has a great interface for uploading parts of files but doesn't allow you to send the chunks out of sequence.
Thanks for the info @ncw, and it's (encrypted) GoogleDrive all the way for me, so I really look forward to putting that multithreaded code through its paces!
One question: would that multithreaded code work in rclone serve restic mode? From what I've seen so far about that mode, I'm guessing not (I think that, similarly to --transfers, this would not work because when rclone is serve'ing restic, it's the latter that 'takes the initiative' rergarding threads and I/O), but I thought better to ask anyway.
I think multithreaded uploads should work now with serve restic where the backend support them (eb B2).
Actually I looked at the Google drive code again and it does not do multithread uploads - sorry I looked at the drive upload API and I don't think the API is capable of it. I think it needs each chunk one at a time.
This will work for anything which calls the internal operations.Copy function. This is sync/copy/move and friends copyto/moveto. If you are using rclone mount with --vfs-cache-mode full it will be active. It won't be active for rclone serve though, but if you are using rclone serve http then you could use rclone to do multithreaded downloads from it.