Speed disparity between mount and copy


#1

I have been troubleshooting plex playback issues for a couple weeks. Plex playback involves lots of pauses and slow starts. I have seen the same performance when using plexdrive or rclone cache.

Using rclone copy I can move data to and from googledrive usually between 12-25 MB/s

[root@ssdnodes-87861 Moana (2016)]# rclone copy gd-enc: /opt/docker/sabnzbd/downloads/complete/ --include "/Movies/Moana\ \(2016\)/**" -vu
2018/03/04 11:31:26 INFO : Local file system at /opt/docker/sabnzbd/downloads/complete: Modify window is 1ms
2018/03/04 11:31:27 INFO : Local file system at /opt/docker/sabnzbd/downloads/complete: Waiting for checks to finish
2018/03/04 11:31:27 INFO : Local file system at /opt/docker/sabnzbd/downloads/complete: Waiting for transfers to finish
2018/03/04 11:32:26 INFO :
Transferred: 1.348 GBytes (22.612 MBytes/s)
Errors: 0
Checks: 0
Transferred: 0
Elapsed time: 1m1s
Transferring:
 * ... (2016)/Moana (2016) Bluray-1080p.mkv: 10% /13.418G, 18.562M/s, 11m5s

2018/03/04 11:33:26 INFO :
Transferred: 2.925 GBytes (24.743 MBytes/s)
Errors: 0
Checks: 0
Transferred: 0
Elapsed time: 2m1s
Transferring:
 * ... (2016)/Moana (2016) Bluray-1080p.mkv: 21% /13.418G, 25.216M/s, 7m6s

Using rsync to measure speeds while copying from an rclone mount I see speeds in 1.2-5.5MB/s

[root@ssdnodes-87861 Moana (2016)]# rsync -avz --progress Moana\ \(2016\)\ Bluray-1080p.mkv /opt/docker/sabnzbd/downloads/complete/
sending incremental file list
Moana (2016) Bluray-1080p.mkv
304676864   2%    5.28MB/s    0:43:26

I am using rclone encryption. The system has 16GB RAM and 2 virtual CPUs. I have seen similar performance on different cloud provides (ssdnodes, chicagovps, digital ocean). The VMs do not seem resource constrained. The size of the google drive mount is about 11TB.

Rclone mount command being used

/usr/sbin/rclone --config /root/.config/rclone/rclone.conf mount --allow-other --stats 10s --buffer-size 1G --timeout 5s --contimeout 5s --cache-db-purge local-crypt: /mnt/media -vv

Rclone cache uses only 1 worker
#2

Could you try a test without the cache, just with the mount and crypt? It would be interesting to see if it is the cache layer or the mount layer which is causing the problem.


#3

Performance was better without using rclone cache or plexdrive. 7.5MB/s on the low end and 11.5MB/s on the high end.

[root@ssdnodes-87861 Moana (2016)]# rsync -avz --progress Moana\ \(2016\)\ Bluray-1080p.mkv /opt/docker/sabnzbd/downloads/complete/
sending incremental file list
Moana (2016) Bluray-1080p.mkv
4624252928 32% 7.62MB/s 0:20:53

Command used to mount google drive

 /usr/sbin/rclone --config /root/.config/rclone/rclone.conf mount --allow-other --stats 10s --buffer-size 1G --timeout 5s --contimeout 5s gd-enc: /mnt/gd-enc/ -vv

#4

Interesting… So removing the cache removes most, but not all the overhead.

It would be worth you trying the latest beta if you haven’t already


#5

I gave it another go with latest rclone rclone v1.39-246-g078d705dβ with using the rclone cache drive I experienced slightly worse results than I previously reported.

[root@ssdnodes-87861 Moana (2016)]# rsync -avz --progress Moana\ \(2016\)\ Bluray-1080p.mkv /opt/docker/sabnzbd/downloads/complete/Movies/
sending incremental file list
Moana (2016) Bluray-1080p.mkv
   391872512   2%    7.06MB/s    0:32:17
   655917056   4%    4.79MB/s    0:46:46
  1154056192   8%    6.85MB/s    0:31:28
1343193088   9%    9.65MB/s    0:22:02

The results using rclone copy appear about the same

root@ssdnodes-87861 Moana (2016)]# rclone copy gd-enc: /opt/docker/sabnzbd/downloads/complete/ --include "/Movies/Moana\ \(2016\)/**" -vu
2018/03/17 11:21:00 INFO  : Local file system at /opt/docker/sabnzbd/downloads/complete: Modify window is 1ms
2018/03/17 11:21:01 INFO  : Local file system at /opt/docker/sabnzbd/downloads/complete: Waiting for checks to finish
2018/03/17 11:21:01 INFO  : Local file system at /opt/docker/sabnzbd/downloads/complete: Waiting for transfers to finish
2018/03/17 11:22:00 INFO  : 
Transferred:   1.282 GBytes (21.673 MBytes/s)
Errors:                 0
Checks:                 0
Transferred:            0
Elapsed time:      1m0.5s
Transferring:
 *   ...ana (2016)/Moana (2016) Bluray-1080p.mkv:  9% /13.418G, 23.036M/s, 8m59s

#6

Just for added comparison. I spun up a m5.large ec2 (2 vcpu, 8GB memory) instance.

[root@ec2-107-23-146-18 Moana (2016)]# rsync -avz --progress Moana\ \(2016\)\ Bluray-1080p.mkv /tmp/
sending incremental file list
Moana (2016) Bluray-1080p.mkv
6205243392  43%    8.56MB/s    0:15:35

Here is an rclone copy

[root@ec2-107-23-146-18 Moana (2016)]# rclone copy gd-enc: /tmp --include "/Movies/Moana\ \(2016\)/**" -vu
2018/03/17 16:05:36 INFO  : Local file system at /tmp: Modify window is 1ms
2018/03/17 16:05:37 INFO  : Local file system at /tmp: Waiting for checks to finish
2018/03/17 16:05:37 INFO  : Local file system at /tmp: Waiting for transfers to finish
2018/03/17 16:06:36 INFO  : 
Transferred:   8.083 GBytes (136.303 MBytes/s)
Errors:                 0
Checks:                 0
Transferred:            0
Elapsed time:      1m0.7s
Transferring:
 *      ... (2016)/Moana (2016) Bluray-1080p.mkv: 60% /13.418G, 130.251M/s, 41s

I know speedtest aren’t everything, but it doesn’t look like rclone slowness is a network speed issue.

[mjmayer@ec2-107-23-146-18 ~]$ ./speedtest-linux-amd64-speed-issues 
github.com/zpeters/speedtest -- unofficial cli for speedtest.net
Server: 5132 - Ashburn, VA (Shentel Service Company)
2018/03/17 16:01:18 Testing download speed
..........
2018/03/17 16:01:19 Testing upload speed
.....
Ping (Lowest): 6.53 ms | Download (Max): 894.72 Mbps | Upload (Max): 296.48 Mbps

#7

What happens if you use a different tool other than rsync to do the copy?

Can you try something really simple like this

cat <source_file | pv | cat > destfile

instead of the rsync?


#8

About the same as rsync.

[root@ec2-18-232-175-103 Moana (2016)]# cat Moana\ \(2016\)\ Bluray-1080p.mkv | pv | cat > /tmp/Moana.mkv
14MiB 0:00:26 [8.16MiB/s] [                                                                                                          <=>                                                                                                   ]

However, I was reading through another post on the rclone forum as saw this post indicating that if plex integration is setup, the stream will start with a single worker until its confirmed that plex has started playing. I have plex integration setup in my config. Maybe it was not working right and was limiting rclone to a single worker. When I removed the plex config from the rclone config I saw these speeds.

SSDNodes

 root@ssdnodes-87861 downloads]# rsync -avz --progress /mnt/media/Movies/Moana\ \(2016\)/Moana\ \(2016\)\ Bluray-1080p.mkv /opt/docker/sabnzbd/downloads/
sending incremental file list
Moana (2016) Bluray-1080p.mkv
1889009664  13%   15.33MB/s    0:13:17

EC2

[root@ec2-18-232-175-103 ~]# rsync -avz --progress /mnt/media/Movies/Moana\ \(2016\)/Moana\ \(2016\)\ Bluray-1080p.mkv /tmp/
sending incremental file list
Moana (2016) Bluray-1080p.mkv
694714368   4%   25.52MB/s    0:08:44

#9

That is interesting, so without the plex integration it runs at full speed?

If you’ve tried with the latest beta then it would probably be worth making a new issue on github with your findings in.


#10

I’m not OP but I find similar result. Rclone mount with cache was around 10%-30% slower transfer speed for regular plex scans, analyzing, and playback. ‘rclone move’ via cache (not using mount) was about the same slower.
Disabling the plex integration via ‘rclone config’ gave me full speed for plex stuff, but bog down my minimalist 10watt cpu server with only 4gb RAM. Rclone would take up all the RAM and 2GB swaps for any plex scanning/analysis. I had to play a bit with the rclone options to get it the way i want. But ‘rclone move’ via cache was still around 30% slower compared to direct rclone move to GDrive:

In the latest beta, everything feels speedier without plex integration in rclone mount cache, so i left my old config just the same. Plex scans/analysis and playback is much faster but every few hours plex scans/analysis would do nothing like it was waiting for rclone to finish downloading chunks on a file, but rclone doesn’t download and doesn’t do anything.
My current workaround is to restart rclone cache mount with systemctl and right after that rclone would start downloading chunks again as plex scanner requested.
‘rclone move’ via cache also achieve the same speed as rclone move directly to GDrive. So thats solved.

When i enable plex integration in rclone config, plex scanning and analyzing is just way too slow that it never finish. Although to be fair i have rather big library around 45,000 files of movies and tv shows.

I believe it is written in rclone.org website that with plex integration rclone limit the number or thread or process it runs.


#11

The issue related to Plex scanning is discussed here: https://github.com/ncw/rclone/issues/2102
It was indeed affected by a change in the Plex API

The latest beta uses websockets and works with the updated API too: https://beta.rclone.org/v1.40-012-g0ed0d9a7/


#12

It might be worth tracking https://github.com/l3uddz/plex_autoscan/issues/20 that I’m discussing over there, if only for the partial library scanning on such a huge library!

Edit: I imagine if remusb handles the parent of a new directory being missing cache issue then my other feature request will simplify, that being said, I guess it’d be handy to check for validity and still hard purge the parent if it’s missing just for sanity/making sure it imports.


#13

I don’t have problem with plex auto scan for my big gdrive library with the latest stable 1.4 with plex integration disabled. Changes are immediately detected on plex whether it’s local or in gdrive, and i usually do a lot of changes in a day : renaming tv shows files, adding subs, uploading 25 movies, 50 episodes, etc all typically happen in a day. With api polling works, all of this are instantly picked by plex regardless i change it via cache or via gdrive webui etc.

However as soon as i use plex integration then plex scanning seems slow or stalled in a plex library section that’s why i left plex integration disabled. For example, plex would instantly recognize changes of all 50 episodes I recently uploaded in the past hour but plex scans would never finish tv shows section thus the 25 movies i just uploaded is never picked up. So with plex integration enabled, in this case i would have to cancel the tv shows section scan, and then initiate scan in movies section.


#14

Try the latest beta from the previous post for a better Plex integration. I posted the links earlier


#15

@remus.bunduc

I tested with beta below, i enabled plex integration and plex scanning does seem to be much faster and so far not stalled in one plex section.
However i found two possible issues:

  • on plex playback, the initial buffer is longer than before. And sometimes it looks like it finished downloading buffer but plex won’t play. The only remedy is to close the player and retry again.
    This seems to be similar bug in the past that reoccur again in this beta?
    Basically if it buffers ok, plex will play immediately. If it fails to buffer initially, then even when the buffer fully downloaded plex won’t play the file. I checked via netdata when playing file.
    So the initial buffer status somehow made big difference on plex playback. I’m guessing rclone is a bit slow on verifying actual plex playback as opposed to plex scanning?

  • i cant delete file from plex anymore. When i delete file via plex, say a movie, and i delete via plex web UI or plex android, it looks like its deleted but rclone doesn’t actually delete the file. So on plex service restart plex will find the same file.

root@jupiter:~# rclone -V
rclone v1.40-025-g1ba3ffdcβ

  • os/arch: linux/amd64
  • go version: go1.10

/system.slice/rclone.service
└─23134 /usr/sbin/rclone mount --cache-workers 4 --cache-db-purge --transfers 4 --cache-chunk-no-memory --config /home/aaa/.config/rclone/rclone.conf --allow-other --cache-writes GDCache: /mnt/Plexdrive

Mar 28 12:20:21 jupiter systemd[1]: Started Mount and cache Google drive to /mnt/Plexdrive.
Mar 28 12:25:30 jupiter rclone[23134]: 2018/03/28 12:25:30 ERROR : tv/Foyle’s War/Foyles War S07/Foyles War S07E01 The Enternity Ring x264 RB58.mp4: File.Remove file error: googleapi: Error 403: The user does not have sufficient permiss
Mar 28 12:34:09 jupiter rclone[23134]: 2018/03/28 12:34:09 ERROR : tv/Rosemary and Thyme/Rosemary & Thyme 2x06 The Gongoozlers.mp4: unexpected conditions during reading. current position: 303140864, current chunk position: 293601280, cu
Mar 28 12:34:09 jupiter rclone[23134]: 2018/03/28 12:34:09 ERROR : tv/Rosemary and Thyme/Rosemary & Thyme 2x06 The Gongoozlers.mp4: (303140864/303142847) error (unexpected EOF) response
Mar 28 12:34:09 jupiter rclone[23134]: 2018/03/28 12:34:09 ERROR : tv/Rosemary and Thyme/Rosemary & Thyme 2x06 The Gongoozlers.mp4: ReadFileHandle.Read error: low level retry 1/10: EOF
lines 1-16/16 (END)

I shouldn’t have permission error like above. Prior to rclone 1.40 i could delete files via plex ok.


VPS Plex Server with Cache not performing well