This is my first time trying to use rclone. I’m trying to make off-site copies or larger media files but am unsure what’s the most sensible way to upload those files.
Some cloud storage providers (like Google Drive for example) provide support for splitting larger files into pieces to make upload easier.
Can rclone access those APIs and use multi-threaded file uploads? Maybe also combined with resumable uploads in case the internet connection is unstable?
Yes it does, but it depends on the provider. Rclone does use multipart upload with google drive.
Rclone will do lots of retrying for multipart uploads to make them go through! It doesn’t do resume yet though.
Thanks for getting back to me.
Do you have to manually setup multipart uploads to google drive or is that something that rclone handles automatically?
In the case of Google Drive, when I use rclone and multipart uploads, does the Google server backend automatically put those parts back together into a single file?
Do you have a list of providers that support this feature?
Is that a feature you might consider adding in the future?
rclone does it automatically. You have some control over it with
--drive-chunk-size int Upload chunk size. Must a power of 2 >= 256k. (default 8M)
--drive-upload-cutoff int Cutoff for switching to chunked upload (default 8M)
Yes.
Errm, no. You’d have to look in the source. Most providers support some kind of multipart upload I’d say though. Some require the chunks uploaded in sequence (eg Dropbox) but some can have the chunks in any order (eg b2, s3, drive).
Yes - there is an issue for it #87. I’m planning to do a sort of half-way resume in #2018 too which will help a lot with downloads. There is also #2252 which you might find interesting about making the multipart upload capability visible to higher layers - this will help a lot with #87 for instance.
Perfect. And rclone handles this automatically when the provider supports multipart uploads? Or does it depend on the provider I’m using?
Where the provider supports multipart uploads, rclone will decide to use them automatically.