Long time since I logged here, and that is the first thing I see, a feature I’ve been wanting for a long time! Perhaps I will finally be able to stop using aria2c (which can do multithreaded downloads of single files, once I add an HTTP “intermediary” server somewhere) and can start using rclone for everything!
Great job, @ncw! Will certainly be testing that in the next few days/weeks.
PS: multithreaded uploads (ie, from local to cloud remotes) would be even better. Are they somehow being contemplated here?
Multithreaded uploads depend on the the cloud provider. These are supported for quite a few providers such as google drive, b2, box, s3 etc. Dropbox annoyingly has a great interface for uploading parts of files but doesn’t allow you to send the chunks out of sequence.
Thanks for the info @ncw, and it’s (encrypted) GoogleDrive all the way for me, so I really look forward to putting that multithreaded code through its paces!
One question: would that multithreaded code work in rclone serve restic mode? From what I’ve seen so far about that mode, I’m guessing not (I think that, similarly to --transfers, this would not work because when rclone is serve’ing restic, it’s the latter that ‘takes the initiative’ rergarding threads and I/O), but I thought better to ask anyway.
I think multithreaded uploads should work now with serve restic where the backend support them (eb B2).
Actually I looked at the Google drive code again and it does not do multithread uploads - sorry I looked at the drive upload API and I don’t think the API is capable of it. I think it needs each chunk one at a time.
This will work for anything which calls the internal operations.Copy function. This is sync/copy/move and friends copyto/moveto. If you are using rclone mount with --vfs-cache-mode full it will be active. It won’t be active for rclone serve though, but if you are using rclone serve http then you could use rclone to do multithreaded downloads from it.