Best mount settings for streaming ( Plex )

Ajki · November 29, 2016, 12:39pm

I dropped acd_cli completely as with my library getting close to 30TB the acd_cli syncing was failing often and since I have mounts from 2 locations ( download/upload server & plex server ) it caused even more problems. The acd_cli cache (node.db was over 200MB and it was pain resyncing as in most cases it needed 2, 3 times before it was actually synced )

Now back on topic I decided to mount my amazon drive with rclone:
rclone mount --allow-non-empty --allow-other --max-read-ahead 200M acd:/ /storage/.acd/ &

With cron ( every minute ) iam running script that checks if drive is mounted and remount if its not.
` #!/bin/bash
if pidof -o %PPID -x “acd-check.sh”; then
echo “$(date “+%d.%m.%Y %T”) SKIP: acd-check.sh already running.” | tee -a “/var/log/acd-cron.log”
exit 1
fi

if [[ -f "/storage/.acd/encrypted/acd-check" ]]; then
   echo "$(date "+%d.%m.%Y %T") INFO: Check successful,drive mounted" | tee -a "/var/log/acd-check.log"
else
   echo "$(date "+%d.%m.%Y %T") ERROR: Drive not mounted remount in progress" | tee -a "/var/log/acd-check.log"
   fusermount -uz /storage/.acd
   rclone mount --allow-non-empty --allow-other acd:/ /storage/.acd/ | tee -a "/var/log/acd-check.log"

   if [[ -f "/storage/.acd/encrypted/acd-check" ]]; then
      echo "$(date "+%d.%m.%Y %T") INFO: Remount successful" | tee -a "/var/log/acd-check.log"
   else
      echo "$(date "+%d.%m.%Y %T") CRITICAL: Remount failed." | tee -a "/var/log/acd-check.log"
   fi
fi
exit

`

I usually have 10 to 15 concurrent streams on my Plex ( family and friends ) and with acdcli it worked flawlessly until one of the sync’s broke db and …

I was checking the docs http://rclone.org/commands/rclone_mount/ and have few questions for those who tested streaming
–max-read-ahead 200M ( this is something I set as i read it here on the forums )
what else could be added to speed up stream as much as possible like
–no-modtime
–write-back-cache
etc…

p.s. Also forgot to mention all content in my ACD drive is encfs encrypted so all mounts

rclone mount --allow-non-empty --allow-other acd:/ /storage/.acd/

ENCFS6_CONFIG=’/home/plex/data/encfs.xml’ encfs -o allow_other /storage/.acd/encrypted /storage/acd/ --extpass=“cat /home/plex/data/.encfs” --public

ENCFS6_CONFIG=’/home/plex/data/encfs.xml’ encfs -o allow_other /storage/.media-local /storage/media-local/ --extpass=“cat /home/plex/data/.encfs” --public

unionfs-fuse -o cow, -o allow_other /storage/media-local=RW:/storage/acd=RO /storage/media-unionfs

And everything is being uploaded with

/usr/bin/rclone move /storage/.media-local/ acd:/encrypted -c --transfers=500 --checkers=500 --delete-after --min-age 15m --log-file=/var/log/rclone-cron.log

ncw · November 29, 2016, 4:28pm

I’m not 100% sure setting the --max-read-ahead above 128k actually does anything… It can’t hurt though.

I’d probably set --read-only. --no-modtime won’t make any difference on ACD as the modfication time is delivered with the other info. If you set read only you don’t need --write-back-cache.

Ajki · November 29, 2016, 4:33pm

would it work faster with --read-only, iam not really writing anything to it, however on my download/upload server sometimes i delete files directly.

How much --write-back-cache should i set and what about checkers is default value good enough or should i set more.

Ajki · November 30, 2016, 6:38am

I did some testing yesterday and I was able to run 15 concurrent streams.
The only thing that i notice that works worst then with acd_cli is when you are watching a video and you are already watched at 80% and you manually jump to 95% it will stuck at buffering.
( this worked with acdcli mount …at least most of times … need to do more testing to be sure )

Today I added --checkers 40 option and as well tweaked some network settings in /etc/sysctl.conf

Increase system file descriptor limit

fs.file-max = 100000

Discourage Linux from swapping idle processes to disk (default = 60)

vm.swappiness = 10

# Increase Linux autotuning TCP buffer limits
# Set max to 16MB for 1GE and 32M (33554432) or 54M (56623104) for 10GE
# Don't set tcp_mem itself! Let the kernel scale it based on RAM.
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.rmem_default = 16777216
net.core.wmem_default = 16777216
net.core.optmem_max = 40960
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

# Make room for more TIME_WAIT sockets due to more clients,
# and allow them to be reused if we run out of sockets
# Also increase the max packet backlog
net.core.netdev_max_backlog = 50000
net.ipv4.tcp_max_syn_backlog = 30000
net.ipv4.tcp_max_tw_buckets = 2000000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 10

# Disable TCP slow start on idle connections
net.ipv4.tcp_slow_start_after_idle = 0

# If your servers talk UDP, also up these limits
net.ipv4.udp_rmem_min = 8192
net.ipv4.udp_wmem_min = 8192

Guide from: http://www.nateware.com/linux-network-tuning-for-2013.html#.WD5uIOYrL-g
Did not play around with mem/swap yet, currently

      total        used        free      shared  buff/cache   available
Mem: 32004 3703 5910 25 22389 27619
Swap: 16367 0 16367

@ncw how is rclone opening mount threads eg how many can it have running

hyperexpert · November 30, 2016, 8:44am

15 streams? Please do tell. I cannot sustain one solid connection from my Plex to the point that I just gave up on it (Plex)!

I have a VM on Azure with 2 cores and 4GB Ram. Bandwidth is obviously not an issue for me in Azure. When uploading or downloading to the VM with rclone, I get an easy 40MB/s connection. That said, even regular MP4 streams take about 40-50 seconds to start playing, as soon as they do start playing, they buffer about every minute for the first 10 minutes before I get a steady stream. Oh, and forget a second stream. That will just cause both, first and second streams to stop!

Could it be because im running a VM in the cloud? I tried an Amazon instance and had the same, if not worse experience. Will it be better if I get a fast cheap VPS instead? I don’t want to host from home, however, at this point I might if I have to.

I also should mention that I am hosting my files with OneDrive. But like I said, doing uploads or downloads with rclone is very fast.

Ajki · November 30, 2016, 9:57am

Iam running plex ( and plexpy ) on this dedicated server: https://www.hetzner.de/gb/hosting/produkte_rootserver/ex41ssd

Made this test just now: (1 stream - original qualits
Movie info

1 stream original quality, when pressed play it took around 10 seconds to start playing.
Server load ( 8 cpus )

Network stats for duration of the playtime

Amazon rclone upload stats

Copying a movie (1 min stats) from amazon encfs mount (unencrypted ) to local machine eg the same way plex see it

Stats for my library “ls -lahR $acd | grep ^- | wc -l” (unencrypted folder)
Processed 26717 files and it took 376 seconds

Not sure what acd_cli do differently beside having the local database of folder/files but that should only speed up browsing and first access for file/folder. It would be great @ncw could figure out how to solve it.

NOTE: Compared to ACDCLI there is huge problem resuming movies eg skipping to middle or end.

ncw · November 30, 2016, 8:42pm

I have made a test version of rclone with more concurrent buffering in. In my tests it doubles the performance of local copies. I’m not sure exactly what difference it will make for plex - would be interested in any feedback!

http://pub.rclone.org/v1.34-39-g7be5724-fuse-bufferβ/

hyperexpert · November 30, 2016, 10:27pm

Testing… crossing my fingers

Ajki · November 30, 2016, 10:33pm

Let me know as I got my account locked again

Hopefully it will be unlocked by tomorrow, last time I called customer support and they unlocked it.

hyperexpert · November 30, 2016, 11:06pm

This is why I use OneDrive Plus, my library is not huge! Its only ~600GB

Ajki · December 1, 2016, 8:39am

@hyperexpert Its not that bad, they unlocked me again and they dont know why ban is happening, but Iam “abusing” it and yesterday I rent some new vps that was just doing copy from acd to gdrive ( making backup copy of whole library just to be on safe side if perm ban happens ) and it was maxing 200Mbit for 10-12h.

My current setup is 1x download/upload machine ( nzbget/transsmision/sonarr/couchpoato etc… ) and that one is always automatically downloading new stuff and uploading to acd.
1x Plex & Plexpy just for streaming and every 8h uploads the subtitles that were downloaded by plex.

1x VPS for making copy from acd to gdrive.

@ncw I did a quick test now with multiple streams and cant say it better it could be even worse as one of the movie was stopping and buffering a lot. During the streams i also run Lib update on Plex and as well analyze so that would make aditional load

Network stats during the test ( i did not have more then 5, 6 concurrent streams … at work so cant really test it atm)

rclone log file: https://drive.google.com/file/d/0B89x0Pnx4k71U2JkaFJPNjFLamM/view?usp=sharing
A lot of errors

plex log files: https://drive.google.com/file/d/0B89x0Pnx4k71YThqQjQ1OXlKNjQ/view?usp=sharing
some of the errors

p.s. Will try to do more extensive testing tonight.

ladude626 · December 1, 2016, 5:33pm

@Ajki Interesting result. Please keep us update. I’m also waiting for rclone mount to become more stable for use. Right now I’m actually using acd_cli to mount the acd folder and then rclone crypt to decrypt. This is just as stable as acd_cli + encfs.

@ncw I briefly tested the beta fuse buffer version with rclone crypt mount. Seems to be stable for now with 1 direct play. Video start time just bit slower than acd_cli but no buffering so far. I don’t have enough users to test more concurrent streams.

Ajki · December 1, 2016, 6:42pm

@ladude626 so far i found rclone mount almost on pair with acd_cli, couple of tweaks and its there.
atm i removed checkers ( did not saw any difference )
rclone mount --allow-non-empty --allow-other --read-only --max-read-ahead 20000M acd:/ /storage/.acd/ &
( i should mount with --max-read-ahead 20G )

p.s. How big are the videos you are watching, i have problems with those 15GB+ ones i notice it may start buffering, but normal sizes eg 3 to 8GB works normally.

ladude626 · December 1, 2016, 7:08pm

@Ajki I’m playing files to up 16GB. Difference may be that files >9GB is downloaded via a templink instead of the conventional way. I have about 14TB of files on ACD at the moment and my acd_cli cache is only about 20MB, not sure how yours grew to 200MB. And can I ask what VPS you use to transfer that much data? I’ve been starting up an instance at scaleway.com whenever I want to move files with rclone.

calisro · December 1, 2016, 7:10pm

You could play with the templink threshold. I wonder if that would help you. --acd-templink-threshold

Ajki · December 1, 2016, 7:54pm

Great suggestion, will try it out tomorrow, maybe set it to 25GB

Iam mainly uploading from my own server that is on 1GB line, btw I got banned yesterday when i started to use scaleway server to copy from acd to gdrive so now i stopped it there ( back to uploading from my main one)

My current library is just a bit bellow 25TB with 3035 folders and 26717 files.

Btw for streaming purposes with adding more --checkers= you could do more harm eg locks and timeouts , yes Plex would scan library faster but on the other hand it could hinder the actual stream. ( or am I wrong )

hyperexpert · December 1, 2016, 8:13pm

I am on the same boat here. Haven’t noticed any difference with the new beta. My only problem with rclone is the time it takes to initially stream a video, once the video is playing, everything is fine, even skipping back and forth works fast. Its only when the video initially starts playing that takes longer than acd_cli.

Just a though here, is there a way to lets say add a switch to rclone to tell it to cache n% of the beginning of your files? Lets say you have a 3GB movie file, can we have rclone cache the first, 3% or 5% or whatever we tell it so streaming from Plex starts right away with your cached beginning while rclone downloads the rest? Not sure if this is even possible and I know this will require a lot of harddrive space for larger libraries, but then again, people don’t have to turn it on. This will create a hybrid storage of your files and streaming will be almost as fast as having files stored locally.

hyperexpert · December 1, 2016, 8:18pm

@Ajki - oh, and btw, I switched to an SSD instance and HOLY#$@^! My load times are 3-4 times faster now. I went from having to wait 30-40 seconds for any MP4 direct stream to start playing, to 8-10 seconds now. Plus, I can finally do more than 1 stream - 14 to be precise Never thought that would be possible. But still, those files that take 8-10 seconds with rclone, take about 3-6 seconds with acd_cli

Ajki · December 1, 2016, 8:30pm

@hyperexpert well initial play ( for me is always bellow 10 seconds ) will always be slower with rclone as file needs to be accessed and acdcli keeps the exact path in localdb ( that gets corrupted quite often once its above 200MB and especially if you have multiple mounts and changing files )

The bigger problem is that with acdcli jumping forward and backward works much better as with rclone plex gets stucked … thats the only part that needs to be tweaked. ( also need to test how is it working if multiple people are accessing same video … i saw someone mentioning it does not work that good but could be resolved by now )

Someone with knowledge should check what is acd_cli doing differently when it comes to streaming https://github.com/yadayada/acd_cli

p.s. I always had SSD and good to know you have such difference with normal disks.

calisro · December 1, 2016, 8:48pm

I do not believe that is the issue. rclone can also cache the path and does for a time. Check out this post. You can see plex is doing some odd things in analyzing a file bfore playing.