Recently a new feature got merged into
rclone beta, the
--vfs-read-chunk-size flag for the
Probably all cloud providers will limit the daily download volume and count the requested bytes therefore. The problem is, when a file is opened for reading,
rclone will request the whole file from the remote. This will add the whole file size to the download volume, even when only a few bytes are read.
seek call will request the file from the new offset until the end, which all add up to the download volume.
This is where
--vfs-read-chunk-size will help to reduce the download volume. When
--vfs-read-chunk-size 128M is given,
rclone will only request
128 MB from the remote at a time. Once a block is read til the end, the next one will be requested.
There is also a companion flag
--vfs-read-chunk-size-limit, which controls if the chunk size can grow when multiple blocks are read in a row.
--vfs-read-chunk-size-limit is greater than
--vfs-read-chunk-size, after every chunk read the chunk size will get doubled, until the limit is reached. A
seek call will reset the chunk size to the initial value and
--vfs-read-chunk-size-limit off will let the chunk size grow indefinitely.
Using the following mount command, I’m able to run Plex on a Google Drive remote without
cache and don’t get banned.
rclone mount \ --dir-cache-time 48h \ --buffer-size 64M \ --vfs-read-chunk-size 128M \ --vfs-read-chunk-size-limit 2G \ gsuite-crypt: /mnt/gsuite
I hope this clarifies the new feature and helps people to avoid bans in the future.
Here are some additional notes about the usage of this feature.
Using chunked reading only makes sense when used without a
cache remote, as the
cache itself uses chunks to retrieve and store data.
By using the term “requested bytes”, I tried to make clear, that there is a distinction between the requested bytes in a request for HTTP based remotes and the actual downloaded bytes. The “requested bytes” will probably be used for the quota calculation for most providers. At least Google Drive is using this value.
There is no additional caching involved when using these flags, neither on disk nor in memory. They only influence how rclone requests data from the remote. For HTTP based remotes, this means using HTTP range requests.