Hi, if you work on rclone and are reading this, thanks so much for the great tool
I came across this closed thread and wanted to give an answer to the main question, because I didn't really see it mentioned in that thread or anywhere on this forum. (It is touched on here but not explicitly.)
Is there ever a good reason to consider chunk concurrency separately from file concurrency?
The reason this separation matters is the lack of "resume-file" functionality. I want concurrency in order to saturate my bandwidth, but I want to focus all my bandwidth on one file at a time, because finishing a file is the only "save point" available. (Correct me if I'm wrong -- I'm not sure if this is backend-dependent or not, but as far as I can tell there is no way to resume a partially uploaded file from local to B2, which is my use case.)
For example, right now I'm uploading a few dozen tar archives of 2-4GiB each. The best speed I'm able to get by playing with concurrency is ~300KiB/s, and on -vv
setting I see dropped connections every few minutes, so perhaps 1/20 chunks is getting retried (I also set chunk size to B2's minimum, 5MiB, for this reason). I'm using the copy
command to automatically find which files still need uploading, but since each file takes hours, I want to finish each one as quickly as possible to lock in the progress, which means being able to set --transfers 1
but also --b2-upload-concurrency 8
is important for my use case. (I was actually searching to understand the difference between these flags, and the linked thread is what clarified it for me )
I am new to rclone and maybe this was already obvious, but I just wanted to illustrate in case it wasn't. I imagine it's a common use case for folks who don't have very premium Internet plans