I used cache when it was first released but it didn’t suit my needs at the time so I continued using Plexdrive. Now that it can detect changes on GDrive, I’ve done some testing on it again and it seems to be working fine. I particularly like the write feature.
I would like to move from Plexdrive and UnionFS to only using Rclone with Cache. Here’s my current setup.
I store all the most recent media on the local drive till it’s 95%. I run a script hourly that checks if the disk is full and delete’s everything older than a month to clear up space. I do this because most of my friends and family watch the most recently uploaded items first and performance is much better this way (also less bandwidth).
Rclone:
Decrypt Gdrive Crypt (media)
Mount options: --read-only --allow-other --buffer-size 500M -v
Upload media to Gdrive
So here’s what I’m thinking of for the new setup:
Rclone mount (Gdrive -> Cache -> Crypt)
Go back to using Sonarr instead of Filebot (Sonarr will rename and move directly to the Crypt drive)
Filebot doesn’t handle REPACKS well
Rednoah wants me to pay per release
Set Plex to scan every 2 hours
Questions:
What’s best way to implement something similar to my UnionFS setup with Cache?
To avoid exceeding API thresholds, I setup a script that checks for new files every hour. With Cache, would you say it’s safe to let Plex automatically scan every 2 hours?
Currently I have a speed limit on Rclone uploads set to 8M. If I’m going to use ‘write’ is there a way to limit the upload speed?
Should I use offline uploading instead? I’m worried that normal writes on Cache don’t have retries. However, all my uploads thus far (with normal write) has worked without issue.
I know this is a long post but I’d really appreciate any help I can get. Thanks!
As someone who has done just this, I feel I can probably shed some light on it.
Few things to consider, having --cache-tmp-upload-path enabled pretty much copies your existing unionfs setup.
That is to say, when you write to the mount, say /home/media/mount it’ll show up in there after the move from sonarr is complete, but the actual file will be sitting in your upload cache directory, for example /home/media/.cache/rclone/upload.
This mirrors the functionality of how unionfs will show the local files till they’re uploaded then swap it over to remote. The file will be queued for upload, will be readable the whole time during this process, including during the upload, and when it’s complete and checked for validity - it’ll be deleted locally and rclone will present the remote file to you in /home/media/mount. This means you can add a media file via sonarr, and plex can scan it immediately and start playback while this whole process goes on in the background.
Try plex_autoscan - https://github.com/l3uddz/plex_autoscan/
Currently talking to the owner https://github.com/l3uddz/plex_autoscan/issues/20 about a closely integrated functionality for rclone specifically (as a fallback if API polling fails to detect a change) but even as it is, it’s very handy to have sonarr to hand off to something else to check for the file and scan just that directory rather than a timely scan of the whole library. Fewer API hits if the cache is cold for a start!
Afaik @ncw hasn’t implemented the splitting up of the --bwlimit flag to allow upload and download limits on a mount, it’d be handy when it comes especially with --rc to be able to schedule upload and download speeds throughout the day, with something as simple as a cron script. As it stands now any --bwlimit will be implemented globally.
Using --cache-tmp-upload-path should retry on failure as from what I can tell it’s just doing an rclone copy command behind the scenes on the locally stored queue for upload files, that is to say it’ll retry on failure and shouldn’t delete the local queued upload until the remote has the copy successfully transferred. As to how many times this attempts to upload before giving up, something to ask @remus.bunduc .
If you have any other questions I’ll try to answer, I’m running it on a 2 core VPS with 4GB RAM and 100GB of local space. so my max chunk total size/memory related flags are tweaked accordingly. You may wish to run a huge local cache if you want to minimise remote reads if you have a lot of regular reads of the same files (common example would be many shared users repeatedly viewing the latest week of added TV) - in which case a larger chunk_total_size may suit you. You’ve indicated that’s how you do it now, so I guess nothing stops you from setting chunk_total_size=7500G
Just one correction: --cache-tmp-upload-path= is actually the one that turns on the temporary upload functionality. --cache-writes isn’t even needed as that is a separate way of writing through cache.
cache-writes: will do a direct upload (unreliable) and store it in cache at the same time
cache-tmp-upload-path … : will first copy the file locally, keep it there for how much time it is configured and after that it will move the file to the cloud provider in a reliable way
If you have a lot of postprocessing on files after writing them to cache, I would use the temporary upload one. In fact, I would always use that. I was thinking of just removing the first option completely at some point and have the temp upload method cache the data too.
Will correct, I’ve always defined the upload path at the same time as cache-writes so didn’t realise.
I’d agree on defaulting, to the latter, if not removing the former purely since reliability > some temporary space used.
Now you’ve mentioned it, what happens when both flags are enabled?
Well you’d keep adding stuff to the cache till you hit the max size then it’ll purge the oldest data when adding new data.
The --rc flag allows you to remote control the mount so you can purge individual directories from the cache (structure and or the data chunks too).
Useful when something seems to be missing from the mount but you’re sure it’s on the Google drive, means you can retain the cache data at the same time. Else you’d be forced to send a SIGHUP signal to rclone to kill all cache data and structures. Which if you did have nearly 8TB of locally stored data would be a shame. If you are going down that route I’d consider larger cache chunks to minimise the huge database that would required to index 8TB of 32M chunks.
This isn’t as bandwidth efficient as leaving the data locally and uploading when full though as you’re sending it up the pulling it down at least once.
It might be possible to patch the cache upload routine to upload at a set free space on the drive rather than by time and you’d replicate your existing setup much closer. Would need to talk to @remus.bunduc though. As it stands now it’ll upload with the delay timer variable set.
Third option would be to upload at the timer but not delete the local queued data that successfully uploaded until a free space trigger happens.
Hmm. You’ll eventually download the entire data from the cloud provider or hit the maximum disk space available on the box where rclone is running.
There’s nothing expiring the chunks unless it reaches that value.
Yeah was only talking about emulating their existing setup where they prefer to keep nearly all the 8TB full of the most recently downloaded files and only upload when full.
Though to match it you’d need to give a flag to keep the queued uploads post upload until free space trigger is reached. (Or max size or age)
Not sure if this is more efficient than unionfs, particularly for people who don’t have a separate feeder box and instead have an all-in-one.
In my unionfs setup, when sonarr or radarr move a file to the unionfs mount, it is detected automatically by Plex via inotify and the media appears immediately without having to execute a scan (also without having to use plex_autoscan)
AFAIK a fuse mount can’t integrate with inotify so using the ˋ–cache-tmp-upload-path` instead of unionfs would actually be a step backward.
Very true. I didn’t realise it worked as most guides and my own unionfs mount uses unionfs-fuse so didn’t have the fsnotify benefits. Good points though. I too have a second feeder box so it wouldn’t benefit me.
Sorry to butt in, but could you share your configuration of what happens after an episode is downloaded by Sonarr? Also, your folder structure please, if possible. I am a newbie trying to learn how to setup my own config. Thanks.
My thought process to write locally, store for 24 hours and let rclone upload.
I have a generic mount of “/gmedia” that has a Movies and a TV in there.
At that point, you can just point to the paths on Sonarr/Radarr/Plex and rclone would handle the upload based on your configuration. I wanted to wait 24 hours before uploading as I have space and things tend to get changed in that period of time with a proper release or something like that.
as @remus.bunduc has informed me,--cache-writes and --cache-tmp-upload-path don’t work together, enabling the upload path handles it differently than if you just have --cache-writes on.