I’m uploading large files like 1TB+ like once a week to Backblaze B2. But every time i upload such a file it stays at 0Bytes/s for like 20h. I assume that it is hashing the file at that time. As smaller files doesn’t take that long before they start uploading. (And I think I read somewhere, that that is what’s going on)
But now for my question. Can’t the byte stream that goes into the hashing function go both to the hashing function and to Backblaze (maybe other providers too?) at the same time? So that instead of reading the file twice from the hard drive it will only be read once and then split the stream into two. One for hashing and one for uploading. Se example below.
This is what I assume Rclone is doing atm:
This is what I’m suggesting it should do (if it is possible):
This way it would only be read once from the disk and hashing and uploading would be done in parallel. Which would increase performance radically for large files like this.
I do however not know if this is possible with the structure of Rclone. And I have never done something like this myself. So this might be impossible but in my head it should work. But as said, that is just done with pseudocode.