Can't get a decent speed with rclone + crypt + cache + plex


#1

Hello,

I have been 2 days straight with dozens of configs but cant get a decent speed adding my media to plex.

This is my rclone config

[GD]
type = drive
client_id =
client_secret =
scope =
root_folder_id =
service_account_file =
token = TOKEN
team_drive = TEAM DRIVE

[gcache]
type = cache
remote = GD:
plex_url = http://127.0.0.1:32400
plex_username = username
plex_password = password
chunk_size =
info_age =
chunk_total_size = 500M
plex_token = plex token

[gmedia]
type = crypt
remote = gcache:
filename_encryption = standard
directory_name_encryption = true
password = password
password2 =

This is my mount:

/usr/sbin/rclone mount gmedia: /gmedia --allow-other -cache-chunk-path /mnt/cache/rclone --cache-db-path /mnt/cache/rclone --cache-dir /mnt/cache/rclone --cache-tmp-upload-path /mnt/cache/rclone --dir-cache-time=160h --cache-chunk-size=10M --cache-info-age=168h --cache-workers=5 --cache-tmp-wait-time 60m --buffer-size 0M --syslog --umask 002 --rc --log-level INFO

Plex is taking forever to index a library with 35 tv shows.

And playback of all direct play files is like 10-15sec, backwards and forwards also 10-15sec

Please, any advice before i close my gsuite account :frowning: been days uploading 7.5TB od media :weary:

Btw, i use unraid, /mnt/cache is my SSD


#2

With the plex integration turned on, it’ll only use 1 worker so it’ll be slow to index, start times will be slow until more workers pick up. I’d probably turn that off for now.

I’d suggest doing a test from your gmedia and do something like “rsync --progress somefile /somelocaldirectory” and see what the speeds are.

That’s usually a good indication of how things will be.


#3

Thanks for the answer :slight_smile:

I removed workers parameter and did a transfer from /gmedia (thats my google drive) to local and this is the test:

image

My tests with upload and download from google drive with rclone are 70mb/s. Its way slower. Any tips in what to change in mount or config? Thanks


#4

Ok ive done some more testing.

this is speed i get from a simple mount from rclone cache

/usr/sbin/rclone mount gcache: /gmedia2 --allow-other &

that is 5mb/s

and this is the speed i get from rclone google drive without encrypting and no cache

edit: cant upload more that one image in a post

20mb/s with only 1 thread i believe.

So, is it posible that cache drive is slowing everything? What parameters should i change?


#5

More tests i have done.

I’ve tried gdrive > encrypt > cache and it gives me 5mb/s with default options

and gdrive > cache > encrypt and 2.4mb/s also with default.


#6

Is the cache disk for storage slow? I wonder if that might be causing you an issue.

You can try without cache at all just use vfs-chunks. Try something like:

/home/felix/go/bin/rclone mount gcrypt: /GD --allow-other -dir-cache-time 72h --vfs-read-chunk-size 64M --vfs-read-chunk-size-limit off --buffer-size 64M --umask 002 --bind 192.168.1.30 --log-level INFO --log-file /home/felix/logs/rclone.log

A default cache mount nets me:

felix@gemini:~$ rclone mount gmedia: /Test
felix@gemini:/Test/Radarr_Movies/Tomb Raider (2018)$ rsync --progress Tomb\ Raider\ \(2018\).mkv /data
Tomb Raider (2018).mkv
    931,954,688   1%   32.99MB/s    0:28:03

Where as on a vfs-chunk mount, I can get a lot more:

felix@gemini:/gmedia/Radarr_Movies/Tomb Raider (2018)$ rsync --progress Tomb\ Raider\ \(2018\).mkv /data
Tomb Raider (2018).mkv
  1,796,767,744   3%   92.65MB/s    0:09:50

#7

But are your tests with or with encryption?


#8

Yes, I use encryption not that it adds much overhead.

On your device, you aren’t maxing CPU or anything as well?


#9

No, cpu is fine. Ok without cache and with vfs-chunks i seem to get 35mb/s. Its not my max, but its much better.

I dont quite understand all this vfs or cache thing, is it safe to use vfs chunks, no ban?

Thanks for all your help Animosity022, your the boss!


#10

omg im indexing my tv shows now and plex logs are full with errors, is it normal? it goes really slow indexing…


#11

So that’s totally normal when you first index something. Plex is expecting the storage to be much faster as if it was local, it would return those items in milliseconds. Cloud storage by nature is slower so it’s going to generate slow items until it finishes that first pass of all your data.

Once the first pass is done, you should be fine. It only re-analyzes if you change the paths / location of the files.

I indexed 45TB of data in about a day and a half.


#12

Ok ill check tomorrow playback if its smooth.

Im still thinking about your 90mb/s transfer lol… I tried again this time i mounted gdrive in my windows machine and with vfs-chunks i top at 45mb/s, still far away from your speed. I gess ill have to play with parameters.


#13

Your mileage may vary. I’m running gigabit FIOS and am sure no one is using my bandwidth :slight_smile:


#14

Ok this is still scanning library.

Today i did another test to see if my isp was capping me. I got the free trial from Google cloud and created a vm instance with 8vcpus 4gig ram and i tried rsync from Google drive encrypyed to local and first 20 sec i was getting 160mb/s and then it dropped to stable 30mb/s. I repeated a few times with similar results. Did i hit a dead end? Im struggling to get your speeds lol


#15

Just to confirm, you are using rsync and seeing “MBs” and not “Mbs”? What is your download speed rated at?

If you did a Google VM, 30MB/s does seem slow as it should do much better than that. Are you using a .edu account or your own gsuite?


#16

MB. Yea its slow. It starts really fast like 160MB/s ifor like 15 sec thats amazing and than it drop to 20 or 40 like when i test it at my local server. Its my own gsuite account. Btw my fiber connection is 600mbit. Rclone copy nets me 70MB/s and mount copy with rsync 30 top. Just like Google vm with nice cpu and ssd. Dont know what to do


#17

Hmm.

Yeah, that’s strange as I let my copy run a bit more to see and it definitely does not drop as it remains very consistent.

felix@gemini:/gmedia/Radarr_Movies/1922 (2017)$ rsync --progress 1922\ \(2017\).mkv /data
1922 (2017).mkv
 28,634,124,013 100%   83.32MB/s    0:05:27

That is pretty good as my line is technically 940 down and I pulled a little over ~660 Mb/s the for the transfer.


#18

This is nuts… I have tried every posible way.

From google team drive to google drive from a google VM i get 20-40MB/s.

Today, i tried from an edu account and it was toping at 90MB/s.

Animosity022 which version of rclone are you using? Im on latest.


#19

My speeds really haven’t changed.

I’ve used 1.41 to 1.42 and I build my own version from source now so I can use cmount.

I can routinely max my gigabit link with a rsync copy from a mount.

./rclone -V
rclone v1.42-DEV
- os/arch: linux/amd64
- go version: go1.7.4

#20

Ok thats my version also.

I tried with 1.39 to check if any diference.

This is from google gsuite to google cloud VM

It should be like 150MB/s

It really sucks :frowning:

btw: i had to disable vfs parameters, it seems in this version it didnt have them already.