Multi threaded downloads - comments and testers needed

I’ve created a first version of multi threaded downloads here.

Multithreaded downloads are when rclone downloads a big file by using multiple download streams. In my testing downloads from drive can run twice as quickly with two download streams.

It could do with some more tests, but it is basically finished.

I’d really like some feedback on whether the interface is OK (the command line flags for controlling it) and what the defaults should be!

Thanks

3 Likes

Long time since I logged here, and that is the first thing I see, a feature I’ve been wanting for a long time! :wink: Perhaps I will finally be able to stop using aria2c (which can do multithreaded downloads of single files, once I add an HTTP “intermediary” server somewhere) and can start using rclone for everything!

Great job, @ncw! Will certainly be testing that in the next few days/weeks.

PS: multithreaded uploads (ie, from local to cloud remotes) would be even better. Are they somehow being contemplated here?

Cheap and dirty way: run split on the file you want to upload and use --transfers=n to start n parallel transfers. But yeah, paralllel uploads would be great :smiley:

Hello Chiraag,

Yeah, I know all about split. But the problem is, no way to re-join the files once they are uploaded (at least not server-side)… so it kinda defeats the purpose.

Cheers,
– Durval

Yeah, that’s fair :stuck_out_tongue_winking_eye: In my case, I’m simply backing up archives, so I don’t really care (and re-assemble the file when I download it).

Does the link in the GitHub thread not work for anyone else?

What isn’t working for you?

404 No found :confused:

2 Likes

The link in the GitHub to the download doesn’t work.

Excellent - thank you @durval

Multithreaded uploads depend on the the cloud provider. These are supported for quite a few providers such as google drive, b2, box, s3 etc. Dropbox annoyingly has a great interface for uploading parts of files but doesn’t allow you to send the chunks out of sequence.

@ncw, the download link you provided in the github issue doesn’t work and returns a 404.

I’m uploading a new one here

https://beta.rclone.org/branch/v1.47.0-028-g1de11062-fix-2252-multipart-download-beta/ (uploaded in 15-30 mins)

Thanks for the info @ncw, and it’s (encrypted) GoogleDrive all the way for me, so I really look forward to putting that multithreaded code through its paces!

One question: would that multithreaded code work in rclone serve restic mode? From what I’ve seen so far about that mode, I’m guessing not (I think that, similarly to --transfers, this would not work because when rclone is serve’ing restic, it’s the latter that ‘takes the initiative’ rergarding threads and I/O), but I thought better to ask anyway.

I think multithreaded uploads should work now with serve restic where the backend support them (eb B2).

Actually I looked at the Google drive code again and it does not do multithread uploads - sorry :frowning: I looked at the drive upload API and I don’t think the API is capable of it. I think it needs each chunk one at a time.

@ncw it is not included here?
https://beta.rclone.org/test/testbuilds-latest/

It won’t appear there until I have merged it. There should be an android build for it too, just not quite sure where!

thanks

also question:
this feature is only active when sync copy move or also with serve?

This will work for anything which calls the internal operations.Copy function. This is sync/copy/move and friends copyto/moveto. If you are using rclone mount with --vfs-cache-mode full it will be active. It won’t be active for rclone serve though, but if you are using rclone serve http then you could use rclone to do multithreaded downloads from it.

What about serve webdav?

rclone serve webdav supports Range requests so you could use multithreaded downloads from it just fine :slight_smile: