Best practices and setting for mounting gdrive and s3 services for streaming with plex

[ltty (Florian Lettner)]

What is the problem you are having with rclone?

S3 storages don't support polling, as a result, if you use long directory cache lifetimes, your remote content, if changed, is never refreshed in the mount.

Run the command 'rclone version' and share the full output of the command.

v1.64.0

Which cloud storage system are you using? (eg Google Drive)

Google Drive
S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount

I was wondering if somebody has put together and optimized caching settings for streaming media content from S3 and Google Drive to plex using rclone mounts and vfs caching, optimizing streaming for performance.

I am explicitly asking for dir-cache-time, vfs-cache-mode, vfs-cache-max-size and vfs-cache-max-age (and potentially other params I am not considering).

Currently, I have two issues:

1. if the dir-cache-time, which in most examples is set to 1000h for streaming, is too big, s3 content is not refreshed, so I have to set it to 24h so new content is refreshed at least once a day. Unfortunately, I do not understand well enough what short cache times mean for streaming.

2. Large caches become an issue on system reboot. If the cache is too large, the systemd service I use to create a mount runs into a timeout and I have to delete the caches first, before I can mount again.

With these two things in mind, I am looking for someone who actually made up his/her mind what actually makes sense for the streaming use case and can provide some guidance and elaborate on the chosen values.

Thanks,
Flo

[asdffdsa (jojothehumanmonkey)]

welcome to the forum,

first, check out my summary of the two rclone caches.
https://forum.rclone.org/t/status-about-using-rclone-for-music-storage-playback-in-2021-access-times-improved/27648/34

@ltty

Unfortunately, I do not understand well enough what short cache times mean for streaming.

does not matter that much, only applies when plex is scanning a library, not when streaming a media file.
i disable auto-scanning, and only manual scan when adding new media files.

@ltty

Large caches become an issue on system reboot. If the cache is too large, the systemd service I use to create a mount runs into a timeout and I have to delete the caches first, before I can mount again.

try --daemon-wait=0 - that has been discussed in the forum.

@ltty

if the dir-cache-time, which in most examples is set to 1000h for streaming, is too big, s3 content is not refreshed

for s3, i set --dir-cache-time=9999h and after adding files to the remote, i manually refresh the vfs dir cache.
to the mount command, add --rc --rc-no-auth
when i want to refresh the vfs dir cache, run rclone rc vfs/refresh recursive=true -vv

@ltty

vfs-cache-mode

sometimes i use that, sometimes i do not.
most users use --vfs-cache-mode=full

@ltty

vfs-cache-max-size and vfs-cache-max-age

that is up to you. how much space do you want rclone to use and how long do you want files to stay in the cache.

[kapitainsky]

@asdffdsa

@ltty

if the dir-cache-time, which in most examples is set to 1000h for streaming, is too big, s3 content is not refreshed

for s3, i set --dir-cache-time=9999h and after adding files to the remote, i manually refresh the vfs dir cache.
to the mount command, add --rc --rc-no-auth
when i want to refresh the vfs dir cache, run rclone rc vfs/refresh recursive=true -vv

@ltty

if the dir-cache-time, which in most examples is set to 1000h for streaming, is too big, s3 content is not refreshed, so I have to set it to 24h so new content is refreshed at least once a day. Unfortunately, I do not understand well enough what short cache times mean for streaming.

If you are happy with periodic cache refresh then trigger it from another systemd job - this is what I do. I also prefer --dir-cache-time=9999h. If something is already in cache I do not see any reason to evict it too early. It sort of assumes that you use --vfs-cache-max-size - otherwise all your free disk space can be consumed.