I don’t know what --timeout actually do in terms of rclone mount but it has been a work of wonder!
By setting the --timeout value to 30m, the total cpu and memory usage that is being used during extensive multiple PLEX Scanning, media analysis and media playback has dropped 50%. This is super awesome!
My plex scans feels so much faster while total cpu and memory usage is down by 50%. I scanned over 9000 movies and around 15000 subtitles in just under 2 hours.
This made me curious what exactly does --timeout do that impact plex, rclone cpu and memory usage so much. And what’s the negative part of making the value bigger?
That is suprising! --timeout closes connections (with an error) if they stop transferring data after that time.
It is also used for timing out response headers.
Were you seeing timeout errors in the log before? I would have thought plex would keep a file open for > 30m. Or even > 5m (the default) would it?
I’m not sure why timeout would matter unless you are actually seeing timeout errors in your log. What’s your mount and what did you rclone.log look like?
This would be a timeout:
2018/11/06 15:32:50 ERROR : TV/The Flash (2014)/The.Flash.2014.S05E04.mkv: ReadFileHandle.Read error: low level retry 1/10: read tcp 192.168.1.30:51263->220.127.116.11:443: i/o timeout
I never see any time out error. In fact I rarely see any error in my rclone mount log.
I played with --timeout because @ncw commented in one of the thread that after 5minutes of default value the buffer is being discarded. This is sometimes becomes a problem because my partner watch plex while doing house chores so she pauses a lot and sometimes more than 5 minutes. True, after pausing more than 5 minutes plex sometimes couldn’t be resumed and somehow sometimes rclone will not try to refill the discarded buffer. So I changed the timeout value into much longer and then I realized how less total cpu and memory utilisation during multiple PLEX scans. And playback pause and fast forward is instantly.
I’m guessing because the file were kept open much longer so it saves some time because no need creating new connection? I don’t know, but it works so fast now for plex Scanning.
Here is my mount
ExecStart=/usr/sbin/rclone mount GDrive1: /mnt/Plexdrive
–buffer-size=280M --uid=111 --gid=118 --umask 000
ExecStop=/bin/fusermount -uz /mnt/Plexdrive
If you pause something playing on Plex though, it’ll send a close to the file and the buffer is dropped on a close.
I don’t think a timeout would impact that at all as that’s more related to the connection. That being said, if you can grab a debug log, it would show for sure if the file is being closed or remains open for that long a period of time.
For a scan in my example, there is 0 connections going on if the directory cache is current.
You can check connections with something like this too:
felix@gemini:~$ ps -ef | grep rclone
felix 609 1 0 Nov06 ? 00:22:15 /usr/bin/rclone mount gcrypt: /GD --allow-other --bind 192.168.1.30 --dir-cache-time 72h --drive-chunk-size 32M --log-level INFO --log-file /home/felix/logs/rclone.log --umask 002 --vfs-read-chunk-size 128M --vfs-read-chunk-size-limit off --rc
felix 17168 16389 0 12:00 pts/3 00:00:00 grep rclone
felix@gemini:~$ lsof -p 609 | grep TCP
rclone 609 felix 7u IPv4 6494407 0t0 TCP gemini.animosity.us:45661->lga34s11-in-f10.1e100.net:https (ESTABLISHED)
rclone 609 felix 8u IPv4 17415 0t0 TCP localhost:5572 (LISTEN)
In my case, it only has a single connection going atm.
Well as a matter of fact after changing the timeout value I can pause longer than 5 minutes and then resume playback, fast forward using the old buffer.in android you can see how far you buffered the media and it works. For example I could fast forward until almost the end of marked buffer and it will play instantly. Before changing the timeout value after 5 minutes I could see that rclone is redownloading the buffer again as I can see from netdata tool or from Android player buffer.
So I don’t think the rclone buffer is discarded the moment you hit pause. In using NVIDIA shield TV and kodi and checking the buffer from netdata and visually. The buffer is being reused on plex resume.
Yes,I understand. In my case it’s slightly complicated because I have 3 plex servers Scanning and analysing the same media at almost any moment.so there’s always connection to files as I also set directory cache time to get shorter usually. All my media goes into GDrive using rclone from separate servers so I need that. I’m using different servers each has their own tasks: uploading, organising, plex, deleting.
You’d have to check the rclone log as that shows the open and close of the files.
It also depends on if you are direct streaming/direct playing/transcoding as those all have an impact of how Plex handles it and what plex itself buffers.
Each client is also very different in how it handles playback. I have ATVs / IOS phones and that’s all I can personally test with. ATVs have a habit of closing and opening files as they only request bits of files at a time.
If you can grab a debug log, that would give the information to help out.
Directory cache time shouldn’t matter if files are being uploaded elsewhere as that’s the polling time that would see the new change and it would process the new files regardless of the time.
Are you not seeing that behavior?
You are correct. Polling works for me. But in the early days of rclone it didn’t always work so I have not so strong faith in polling
I add a lot of content every hours when I’m at the time of acquiring new media, like few hundreds of files every 5 hours. So just to be safe I made directory cache time shorter.
Just out of curiosity after reading your last reply. May I ask how large is your gdrive already?
felix@gemini:/opt/plex_autoscan$ rclone about gcrypt:
sorry about hijacking this thread but i get a timeout with your rclone settings (github) and my shield tv when hitting pause and waiting for a minute or two. The video resumes and then it times out and i get back to the plex menue. If i resume it again it works fine but i cannot pause for a longer time. Somehow the cache on the shield tv seems to be empty after pausing the video…
Do you get an error in the logs about something timing out? I don’t have a Shield so I can’t really test. I’ve found that when I pause and come back, the 1 hour timeout worked well for me.
I’m guessing the shield might close out the file and you seem to be getting an error opening it back up? It might be worth it to reproduce it with a debug log and share that and see if we can see what the actual issue is.
I have shield and use timeout 1h. It works on my shield as intended. Pausing, resuming, etc plays instantly. However there was one time when it didn’t work when I started experimenting with /etc/sysctl.conf.
Also, I don’t use animosity setup. Mine is very simple rclone mount settings with timeout and leave most of the other things to default value.
Thanks. I really need to check the log files (not sure why i did not )
Can you provide your complete rclone config? I still get the issues with timeout set to 1h… not sure why
I don’t use unionfs only plain rclone.
This setting works best when you have slow Internet 12Mbps and very weak cpu less than 2000 passmark:
ExecStart=/usr/sbin/rclone mount GD1: /mnt/Plex
–buffer-size=88M --uid=xxx --gid=xxx --umask xxx
So if you have better cpu, more ram and faster Internet you should change some of the numbers above.
With that setting above playback is instantaneous within 6 seconds.