So I'm currently using rclone mount to do chunk > crypt > Backblaze (so that I can modify parts of large files without needing to re-upload the entire thing) and it seems that renaming a larger file results in hanging the operation until each chunk is server-side copied (since Backblaze has no move API, I know it will copy then delete as evidenced by the logs). When I inspect the process with -vv --dump headers I can see that each server-side copy operation runs in a sequential order. Is there any possibility that a feature could be added to offer parallel server-side copy operations? I imagine this would dramatically speed up the task from 60 seconds currently to around 4 (assuming the backend provider suffers no slowdown from concurrent calls). This improvement may also be something that could benefit the resulting delete operations afterwards too (specific to Backblaze).
Here's a dump of rclone version just in-case its already under works:
chunker does not support it. If you change any part of large file then all file will be uploaded again. Only chunker purpose today is to allow storing larger file than remote limit.
That's unfortunate. After looking into your comment, I see that another post has discussed making chunker more context-aware, although the idea seems to be too much work to implement (here).
Despite this, I would still like to keep the mv operations from hanging for so long and either have them work faster or act asynchronously (So if you happen to know a way to do this, let me know!). I did test some proof-of-concept changes and it seems that server-side copies are in fact faster when made parallel. The slow server-side copy does indeed affect non-chunker and chunker backends on larger files, but chunker has the possibility to improve this by working on multiple chunks in parallel. I still think this may be something to potentially consider for a feature given the difference in timings.
Before proof-of-concept:
localhost:/var/opt/test# time mv Archive.7z Archive-1.7z
real 1m 18.14s
user 0m 0.00s
sys 0m 0.00s
After proof-of-concept:
localhost:/var/opt/test# time mv Archive-1.7z Archive.7z
real 0m 14.82s
user 0m 0.00s
sys 0m 0.00s