Emby Rclone workspace Very long launch

Jonathan_Audy · May 15, 2021, 6:02pm

The launch of files on emby link with rclone a workspace is very slow, about 15 to 30 seconds. To overcome the problem, I made a cache on another 4TB disk, but the metadata update takes a long time.

rclone v1.55.0

os/type: linux
os/arch: amd64
go/version: go1.16.2
go/linking: static
go/tags: cmount

Ubuntu 20.04

Workspace

rclone mount --allow-other --vfs-cache-mode full Gcache: /mnt/cryption --user-agent="ams"

[Gsuite]
type = drive
scope = drive
token =
root_folder_id =

[Gcrypt]
type = crypt
remote = Gsuitee:/Personnel
filename_encryption = standard
directory_name_encryption = true
password =
password2 =

[Gsuitee]
type = drive
scope = drive
token =
team_drive =

[Gcache]
type = cache
remote = Gcrypt:
plex_username =
plex_password =
chunk_size = 10M
info_age = 10y
chunk_total_size = 3.700T
db_path = /mnt/aeea3aba-b35e-4f7c-bf59-c3230b580a31
chunk_path = /mnt/aeea3aba-b35e-4f7c-bf59-c3230b580a31
workers = 10
writes = true

Animosity022 · May 15, 2021, 6:06pm

The cache backend is buggy/old/deprecated and really should not be used.

You should convert over to vfs cache mode full.

You should also increase dir-cache-time to a huge value since you are using Google Drive.

The one I use is here:

github.com

animosity22/homescripts/blob/master/systemd/rclone.service

[Unit]
Description=RClone Service
Wants=network-online.target
After=network-online.target

[Service]
Type=notify
Environment=RCLONE_CONFIG=/opt/rclone/rclone.conf
KillMode=none
RestartSec=5
ExecStart=/usr/bin/rclone mount gcrypt: /GD \
# This is for allowing users other than the user running rclone access to the mount
--allow-other \
# Google Drive is a polling remote so this value can be set very high and any changes are detected via polling.
--dir-cache-time 1000h \
# To log to syslog as well
--syslog \
# I reduce the poll internval down to 15 seconds as this makes changes appear fast the API quotas per day are huge
--poll-interval 15s \
# This is setting the file permission on the mount to user and group have the same access and other can read

This file has been truncated. show original

Also, what is Emby Workspace? You mean the Emby home screen?

Jonathan_Audy · May 15, 2021, 6:12pm

I would have a speed of execution using: rclone mount --allow-other --dir-cache-time 1000h --syslog --poll-interval 15s --vfs-cache-mode full Gcache: /mnt/cryption --user-agent="ams" ??

Animosity022 · May 15, 2021, 6:14pm

No, because that's still pointing to the cache remote in your config.

You want to point to the Gcrypt: remote.

Jonathan_Audy · May 15, 2021, 6:18pm

Yes of course, I forgot to remove cache in my example. Here is the current launch file without going through cache

[Unit]
Description=Gcrypt Daemon
After=syslog.target network.target

[Service]
User=jonathan
Group=jonathan
Type=simple
ExecStart=/usr/bin/rclone mount --allow-other --vfs-cache-mode full Gcrypt: /mnt/cryption --user-agent="ams"
TimeoutStopSec=20
KillMode=process
#Restart=on-failure

[Install]
WantedBy=multi-user.target

Jonathan_Audy · May 15, 2021, 6:22pm

so that way would it be good or do i have to change other things?

[Unit]
Description=Gcrypt Daemon
After=syslog.target network.target

[Service]
User=jonathan
Group=jonathan
Type=simple
ExecStart=/usr/bin/rclone mount --allow-other --dir-cache-time 1000h --syslog --poll-interval 15s --vfs-cache-mode full Gcrypt: /mnt/cryption --user-agent="ams"
TimeoutStopSec=20
KillMode=process
#Restart=on-failure

[Install]
WantedBy=multi-user.target

Animosity022 · May 15, 2021, 6:40pm

This should be notify.

This will lead to things not working well if the process is killed. I use none.

You probably want a place to put the cache files rather than the default.

--cache-dir=/cache \
# This is used for caching files to local disk for streaming

All in all, if you click on the file I linked, it gives you some examples on what each flag does and why I use it as there are a few more flags.

Jonathan_Audy · May 15, 2021, 6:52pm

You probably want a place to put the cache files rather than the default.

--cache-dir=/cache \
# This is used for caching files to local disk for streaming

Thanks for your quick response, for the cache destination does it take up a lot of space, and could putting the location elsewhere on another disk affect the speed of execution?

Animosity022 · May 15, 2021, 7:00pm

Yes, it's documented in the file I linked above:

# The local disk used for caching
--cache-dir=/cache \
# This is used for caching files to local disk for streaming
--vfs-cache-mode full \
# This limits the cache size to the value below
--vfs-cache-max-size 800G \
# This adds a little buffer for read ahead
--vfs-read-ahead 256M \
# This limits the age in the cache if the size is reached and it removes the oldest files first
--vfs-cache-max-age 1000h \

Jonathan_Audy · May 15, 2021, 8:08pm

Thanks that looks good, the file launches in 5-6 seconds!

[Unit]
Description=Gcrypt Daemon
After=syslog.target network.target

[Service]
User=jonathan
Group=jonathan
Type=notify
ExecStart=/usr/bin/rclone mount --allow-other --cache-dir=/mnt/aeea3aba-b35e-4f7c-bf59-c3230b580a31/cache --dir-cache-time 1000h --syslog --poll-interval 15s --vfs-cache-mode full --vfs-cache-max-size 3500G --vfs-read-ahead 256M --vfs-cache-max-age 1000h Gcrypt: /mnt/cryption --user-agent="ams"
TimeoutStopSec=20
KillMode=none
#Restart=on-failure

[Install]
WantedBy=multi-user.target

I sent the VFS cache to a 4TB disk

Jonathan_Audy · May 16, 2021, 8:40pm

Do you believe that --fast-list can improve large library scanning as well?

Thanks!

Animosity022 · May 16, 2021, 8:41pm

Fast-list does nothing on a mount.

Jonathan_Audy · May 16, 2021, 8:44pm

I look at the different flags and I confessed to be a little lost as there are some ...

Jonathan_Audy · May 20, 2021, 8:15pm

Hi, the vfs currently takes about 2 to, but strangely some movies start in 2 seconds and less and others in 20 seconds. Is there anything more I can do to avoid this slowness?

Thanks?

asdffdsa · May 20, 2021, 8:20pm

for what it is worth, i stream from emby on a raspberry pi4 and do not use vfs cache.

Jonathan_Audy · May 20, 2021, 8:23pm

I mounted a server under linux, it surprises me that it is also random when launching a file. How long on average on your side?

asdffdsa · May 20, 2021, 8:38pm

gdrive can be very slow at times.

your server, is it physical or virtual? - mine is a low-end raspberry pi4 running ubuntu.
your the internet connection, what is the result of a speedtest.net. i have a 300Mbps fiber optic.

for me -
if i stream from wasabi, a s3 clone known for hot storage.
usually less than one second to start playback and to seek within the media files, almost instantly

if i stream from a cheap seedbox, usally five seconds to start and less than one or two seconds to seek

Jonathan_Audy · May 20, 2021, 9:01pm

For my part I have a physical server and a quality graphics card, internet 500 mbps Internet speed is not the problem, I also have a ssd hard drive, an amd 3200 processor and 16 gb of ram 1660 super 6gb ram video card. My storage is google drive. vfs helps a lot just like hiding it, but it remains random

Jonathan_Audy · May 20, 2021, 9:22pm

I am also under emby and the transcoding is sent to the video card. I have noticed that when there is a file that needs to be transcoded for some xy reason it is slower to launch, but this is not always what causes the launch to be slow.

asdffdsa · May 20, 2021, 9:25pm

never transcode.
stream over wifi, never watch on the headless pi4.

i use vlc on android and windows pc.
what client do you use to watch the media?