Vfs-cache, size & limit settings and interpreting logs

Help and Support
1

I’ve been debugging a speed issue (solved - it was my VPN settings) however in investigations I was reviewing the debug logs and I am confused regarding the content:

ChunkedReader.Read at 69632 length 8192 chunkOffset 65536 chunkSize 67108864
ChunkedReader.Read at 77824 length 16384 chunkOffset 65536 chunkSize 67108864
ChunkedReader.Read at 94208 length 32768 chunkOffset 65536 chunkSize 67108864
ChunkedReader.Read at 126976 length 65536 chunkOffset 65536 chunkSize 67108864
ChunkedReader.Read at 192512 length 131072 chunkOffset 65536 chunkSize 67108864
ChunkedReader.Read at 323584 length 262144 chunkOffset 65536 chunkSize 67108864
ChunkedReader.Read at 585728 length 524288 chunkOffset 65536 chunkSize 67108864
ChunkedReader.Read at 1110016 length 1048576 chunkOffset 65536 chunkSize 67108864
ChunkedReader.Read at 2158592 length 1048576 chunkOffset 65536 chunkSize 67108864
ChunkedReader.Read at 3207168 length 1048576 chunkOffset 65536 chunkSize 67108864
ChunkedReader.Read at 4255744 length 1048576 chunkOffset 65536 chunkSize 67108864
ChunkedReader.Read at 5304320 length 1048576 chunkOffset 65536 chunkSize 67108864
ChunkedReader.Read at 6352896 length 1048576 chunkOffset 65536 chunkSize 67108864
ChunkedReader.Read at 7401472 length 1048576 chunkOffset 65536 chunkSize 67108864
ChunkedReader.Read at 8450048 length 1048576 chunkOffset 65536 chunkSize 67108864
ChunkedReader.Read at 9498624 length 1048576 chunkOffset 65536 chunkSize 67108864

Note: This extract only includes the ChunkedReader.Read lines and the fields after the filepath.

For the above I had th following vfs options:
--vfs-read-chunk-size=64M --vfs-read-chunk-size-limit=2G

I noticed that changing the chunk-size parameter caused the chunkSize log field to alter, but it always indicated that as set in the config and it did not ‘ramp-up’ as described in the docs. However the length field in the logs does start at 4096, next to 8192 and continues to double (reflecting the behaviour described in the docs) until it reaches 1048576 - I interpret these values to be bytes.

How should I interpret these logs - I am expecting to see a field for the reads from the source with chunks starting at the vfs-read-chunk-size which doubles until it reaches the vfs-read-chunk-size-limit.

Am I looking at the correct fields (and interpreting correctly) and I have an issue as my chunks start at 4K and reach a maximum of 1M, or is it me? :slight_smile:

Regardless - is this information viewable somewhere - so I can see the chunks growing to their maximum based upon my config?

Thanks

E

2

You’d have to let it run a bit to start increasing.

Here an example of mine going up:

2019/01/08 15:20:03 DEBUG : Radarr_Movies/Loud Krazy Love (2018)/Loud Krazy Love (2018).mkv: ChunkedReader.Read at 134213632 length 1048576 chunkOffset 0 chunkSize 134217728
2019/01/08 15:20:03 DEBUG : Radarr_Movies/Loud Krazy Love (2018)/Loud Krazy Love (2018).mkv: ChunkedReader.Read at 134217728 length 1044480 chunkOffset 0 chunkSize 134217728
2019/01/08 15:20:04 DEBUG : Radarr_Movies/Loud Krazy Love (2018)/Loud Krazy Love (2018).mkv: ChunkedReader.Read at 135262208 length 1048576 chunkOffset 134217728 chunkSize 268435456
2019/01/08 15:20:04 DEBUG : Radarr_Movies/Loud Krazy Love (2018)/Loud Krazy Love (2018).mkv: ChunkedReader.Read at 136310784 length 1048576 chunkOffset 134217728 chunkSize 268435456
201

The chunk sizes do move up, but the reads come in the fuse 4k blocks. Don’t confuse the two as one the OS reading the blocks in and the chunks are coming from a call via rclone.

3

Perfect - Now I understand. It was me (being impatient) :slight_smile:

Makes sense now you have explained and, with a little less haste and a bit more patience I do see my chunkSize increasing, although not as rapidly as yours (taking a minute or two between each increment) - which i’ll put down to the vagaries of config, internet, bandwidth and other magic in the code which I don’t need to understand.

Thanks for your help!

E