Performance Issues with Rclone's VFS ChunkedReader

[Hazardsharex (Hazard)]

STOP and READ USE THIS TEMPLATE NO EXCEPTIONS - By not using this, you waste your time, our time and really hate puppies. Please remove these two lines and that will confirm you have read them.

What is the problem you are having with rclone?

Facing an issue with Rclone where VFS cache transactions are being held for 5 to 14 seconds when re-reading or rewinding old cached versions of the same file, while forward operations work fine. log time for issue in the file at 2024/07/15 23:29:38

Run the command 'rclone version' and share the full output of the command.

rclone v1.67.0

• os/version: ubuntu 22.04 (64 bit)
• os/kernel: 6.5.0-41-generic (x86_64)
• os/type: linux
• os/arch: amd64
• go/version: go1.22.4
• go/linking: static
• go/tags: none

YES

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount gc3: /home/ubuntu/test/ --allow-other -vv  --fast-list --vfs-cache-mode full --vfs-read-chunk-size 32M --vfs-read-ahead 128M --vfs-read-chunk-size-limit 256M --buffer-size=0 --vfs-cache-max-size 10G --vfs-cache-max-age 2h --tpslimit 8 --tpslimit-burst 100 --attr-timeout 9999h --poll-interval 5M --dir-cache-time 9999h  --drive-fast-list-bug-fix --no-checksum --no-modtime --read-only --log-file logs-rclone2.txt

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[gc3]
type = drive
scope = drive
service_account_file = /home/ubuntu/config/rclone/sa/service-account-1-key.json
service_account_file_path = /home/ubuntu/config/rclone/sa/
team_drive = XXX
root_folder_id = 

A log from the command that you were trying to run with the -vv flag

https://gist.github.com/khazardstudent/db8b7b0341aa22fc9684095a27a7ffd5

[asdffdsa (jojothehumanmonkey)]

welcome to the forum,

--fast-list does nothing on a mount, so can remove that and i would assume, to remove --drive-fast-list-bug-fix
service_account_file_path, what is that, where is it documented?

@Hazardsharex

Facing an issue with Rclone where VFS cache transactions are being held for 5 to 14 seconds when re-reading or rewinding old cached versions of the same file, while forward operations work fine

that debug log is huge, over 11,000 lines long.
really, need to post a very short, most minimal, concise summary of the issue, no extra lines or fluff.

[Hazardsharex (Hazard)]

Please ignore the service_account_file_path. Regarding the log, I don't know from which line I should start, but the issue begins at 2024/07/15 23:29:38. At that point, I rewinded the file, and the first cached data was re-read, imminently after cache stopped responding for a few seconds and started reading again at 2024/07/15 23:29:42. After furter testing i found that this only happens when the cache is being downloaded simultaneously. If the cache is not being modified or downloaded, then there is no issue

[ncw (Nick Craig-Wood)]

@Hazardsharex

After furter testing i found that this only happens when the cache is being downloaded simultaneously. If the cache is not being modified or downloaded, then there is no issue

That probably means it is a locking issue of some kind.

These VFS issues are very difficult to diagnose - if you want to give us a hand then please make a reproducible way of causing the problem. In a perfect world this would be a python/shell/whatever program we can run on an rclone mount with certain parameters to see the problem.

[Hazardsharex (Hazard)]

Señor, I was able to reproduce it with the command rclone mount gc3:anime x: --vfs-cache-mode full --drive-fast-list-bug-fix --vfs-read-chunk-size 16M --buffer-size 0 --network-mode -vv --log-file as3.txt --cache-db-purge. Every time I seek backward, the cache is being held if it was downloading. The 'openpending' message was the common output in all my testing and kept appearing at the start of the issue, although I'm not sure if it is the cause of the problem. Additionally, if this helps, I am using Plex (also tested with Jellyfin and Emby with ExoPlayer). Each time Plex seeks back, the cache gets held.
full log
short log
was able to reproduce it even on windows.

[ncw (Nick Craig-Wood)]

Try setting removing the --buffer-size setting above. Setting it to 0 might be causing the issue.

[Hazardsharex (Hazard)]

I tested it with 32MB, the default 16MB, and just for fun, 256MB and 512MB, but I encountered the same issue. If I wait for the file to cache (e.g., around 300MB+) and wait for rclone to stop/halt downloading, the file plays okay

[Hazardsharex (Hazard)]

Update: With the Chunker-Beta version, it is no longer an issue since files get downloaded quickly. Emby and Jellyfin are working smoothly now. Plex still sucks, probably due to something related to the ExoPlayer that Plex uses.