Each rclone transfer does its own disk read. I'm wondering if it could be more efficient to have a single thread do all the reads and fill buffers for each transfer.
i.e. imagine i'm doing 50 parallel transfers, I might say I want a 50MB buffer for each transfer (so I'm dedicating 2.5GB of memory to this).
so I set up 50 channels.
Each channel takes a message that contains data up to size block, so we make the channel size n = buffer_size / block_size.
each thread of transfer execution just reads form the channel.
the single buffer thread iterates over all transfers its processing and reads the length (l) of the channel and sees that we have n-l slots free so fills them, and proceeds to the next (when it finishes reading the file, it closes the channel, so the reader will see that)
idea is to reduce disk thrashing on large number of transfers in parallel, where we have large bandwidth but for whatever reason need large number of parallel threads to make use of it.