Rclone Cache much slower than no cache download?

[Dulanic]

So I tried to swap from plexdrive to rclone cache and find that I can’t play 4K at all with rclone cache. I tried to identify why and during my testing I am finding that cache version is 1/3rd the speed. I have 1000/250 connection so it’s plenty fast.

I cleared both caches before hand to ensure it wasn’t using cached data.

Copying a file using rclone cache:

Transferred:   11.902 GBytes **(9.234 MBytes/s)**
Errors:                 0
Checks:                 0
Transferred:            1
Elapsed time:    21m59.8s

Copying a file using rclone /w plexdrive:

Transferred:   4.444 GBytes **(25.052 MBytes/s)**
Errors:                 0
Checks:                 1
Transferred:            0
Elapsed time:      3m1.6s

My rclone cache config:

[gd]
type = drive
client_id = *redacted*
client_secret = *redacted*
token = *redacted*

[crypt]
type = crypt
remote = cache:
filename_encryption = standard
password = *redacted*
password2 =
directory_name_encryption = true

[cache]
type = cache
remote = gd:abc
vfs-cache-dir = /chunks/rclone

My plexdrive rclone config:

[gd]
type = drive
client_id = *redacted*
client_secret = *redacted*
token = *redacted*

[gd-crypt]
type = crypt
remote = gd:
filename_encryption = standard
password = *redacted*
password2 =
directory_name_encryption = true

My plexdrive mount settings:

screen -dmS plexdrive /opt/cloud-media-scripts/plexdrive/plexdrive-linux-amd64 --config=/opt/cloud-media-scripts/plexdrive --temp=/chunks/plexdrive -o allow_other --clear-chunk-age=24h --chunk-size=20M --verbosity=3

/opt/cloud-media-scripts/rclone/rclone mount --config=/opt/cloud-media-scripts/rclone/rclone.conf --buffer-size 500M --checkers 16 --allow-non-empty --allow-other --max-read-ahead 30G local-crypt: /opt/cloud-media-scripts/.cloud-decrypt

My rclone cache mount settings

/opt/rclone mount gd-crypt: /local-media --config /home/dulanic/.config/rclone/rclone.conf --gid=1000 --uid=1000 --cache-workers=16 --cache-chunk-path=/rclone/chunks --buffer-size=0 --checkers 16 --cache-db-path=/rclone/cache --cache-writes --cache-tmp-upload-path=/rclone/uploads --cache-tmp-wait-time=30m --transfers=16 --allow-other --allow-non-empty --cache-total-chunk-size=250G --log-file=/opt/cloud-media-scripts/logs/rclone.log --log-level=INFO

[Animosity022]

Got a few typos in your rclone stuff.

Turn off cache writes if you are using cache-tmp-upload.
the buffer size needs 0M
transfers/checkers can be removed since it doesn’t do anything on a mount
I just leave cache-workers at 5, one up from the default at 4

I just use the following:

/usr/bin/rclone mount gmedia: /gmedia \
   --allow-other \
   --dir-cache-time=160h \
   --cache-chunk-size=10M \
   --cache-info-age=168h \
   --cache-workers=5 \
   --cache-tmp-upload-path /data/rclone_upload \
   --cache-tmp-wait-time 60m \
   --buffer-size 0M \
   --syslog \
   --umask 002 \
   --rc \
   --log-level INFO

I can play multiple 4K movies without issue.

[Dulanic]

No change

This is my updated mount, but same speed and everything.

/opt/torrent/rclone mount gd-crypt: /local-media --config /home/dulanic/.config/rclone/rclone.conf --gid=1000 --uid=1000 --cache-workers=5 --cache-chunk-path=/rclone/chunks --buffer-size=0M --cache-db-path=/rclone/cache --cache-tmp-upload-path=/rclone/uploads --cache-tmp-wait-time=24h --allow-other --allow-non-empty --cache-total-chunk-size=250G --cache-chunk-size=10M --dir-cache-time=160h --cache-info-age=168h

The first portion went faster as it was cached, but it’s averaging 9MBps vs 25.

Transferred: 3.562 GBytes (12.158 MBytes/s)
Errors: 0
Checks: 1
Transferred: 0
Elapsed time: 5m0s
Transferring:

• abc.mkv: 40% /8.735G, 9.257M/s, 9m32s

[Animosity022]

I’m not sure what you are trying to do. Are you playing back files or copying files?

[Dulanic]

Well the goal is playback, but I am copying to test the speed. It doesn’t play smoothly /w any 4K for sure /w this slower speed.

[Animosity022]

How are you testing the copy?

[Dulanic]

rclone --config rclone.conf copy "/local-media/Media/Movies/abc/abc.mkv" /mnt/test -v

[Animosity022]

So you are stopping the mount and testing a copy? Why not just play something back using the mount command and see if it works.

[Dulanic]

No, the /local-media is mounted with the post above. I have tried to play back and it skips like crazy.

[Animosity022]

You can’t share a cache though so something isn’t right.

Here is an example of my copy:

[felix@gemini Zootopia.(2016)]$ rsync --progress Zootopia.\(2016\).HDTV-1080p.mkv /data
Zootopia.(2016).HDTV-1080p.mkv
    377,257,984   6%   14.22MB/s    0:06:16

Here a snippet when I’m actually playing something 4k at super high bitrate:

What are you trying to play that’s skipped? Also, did you setup the plex integration as well?

[Dulanic]

I remounted again and so far it does seem smoother now. I’m still curious why the speed is significantly slower than non cached, but Ill give it some time and see how it handles things over the next few days.

[Animosity022]

Using the cache is definitely overhead compared to the mount, but not being banned makes it well worth it

If you are using plex, I’d recommend configuring the integrations as well as that helps out.

[darthShadow (Anagh Kumar Baranwal)]

This is totally unrelated, but which release of Black Panther is this? I don’t think I have ever seen a 220 mbps bitrate video.

[Animosity022]

Sorry, Tautilli does a bad job at estimating bandwidth. It’s this:

If it helps:

[camjac251]

I am running your config and tried rsync on a file but only get 6 MB/s on a 10gbps server for GDrive>Local. Is there something I can add to my config to hopefully speed it up since a 2160p Bluray remux can peak at 128mbps (16 MB/s) according to the format specifications

EDIT: I seem to get better results with a 50MB cache chunk size and upping the workers from 4 to 16

[darthShadow (Anagh Kumar Baranwal)]

That looks reasonable. Thanks.

[blim5001]

afaik Tautilli only shows what Plex reports and that bit is the “Plex Streaming Brain” estimate of required bandwidth, which is usually way higher than the Bitrate of the file. And in my experience, usually wrong

[darthShadow (Anagh Kumar Baranwal)]

Yeah, that’s why I usually ignore the required bandwidth. However, in this particular image even the “Quality” section was showing an insane value. Hence, my question.

[blim5001]

My bad, Sorry, I missed the “Quality” bit.

[seuffert]

Rclone copy starts a single download stream. I Am pretty sure that Rclone cache starts a new partial file download for every chunk. This will impact transfer speeds (tcp connection setup, slow start). In theory a larger chunk size and a higher number of workers will result in faster transfers. The disk type (SSD?) & speed will also have an effect on your result.

(a larger chunk size will likely result in a noticeable start delay in plex)