Hello,
When using the VFS cache set to either writes or full I’m able (by appropriately using --drive-chunk-size and --transfers) to reach over 500mbps upload to Google Drive (with a 1gbps NIC).
However, when disabling the VFS cache or setting it to minimal my uploading is limited to around 200mbps, seemingly regardless of the values of --drive-chunk-size and --transfers.
The above scenario is consistent. Is this expected behaviour? If not, how can I debug this?
transfers does nothing a mount.
drive-chunk-size works when you use writes or full, but if that is off, it doesn’t use it, hence the slower download. Not sure if that is on purpose or not as I’m guessing not.
If the goal is multiple transfers at a time, it’s best to use rclone copy/move instead.
If you turn the mount to debug, “-vv”, you can see all that info in the logs when it is writing.
Thanks for mentioning that transfers does nothing in a mount, as I forgot to mention I am indeed testing all this with a mount.
So currently, unless using vfs-cache writes or full, uploads are not chunked at all?
Technically speaking, do we know what’s the reason that drive-chunk-size doesn’t work unless using vfs-cache full or writes?
This is because the multipart uploader currently can’t upload files of unknown size (which is how mount works).
That is relatively easy to fix though - can you try this?
https://beta.rclone.org/branch/v1.47.0-035-g8f77a689-drive-multipart-stream-beta/ (uploaded in 15-30 mins)
Yep, that fixes it. It's now doing multipart uploads with a mount and without vfs-cache, and speeds are noticeably improved. Thanks
Great! I think uploading streamed files with multipart is a much better thing to do (the chunks can be retried on failure) so I'll merge this in a bit. Thanks for testing and taking the time to work out there was a problem because I hadn't noticed!
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.