One VFS Cache to rule them all?

Good morning,

I have been contemplating how to best set up rclone for my google drive storage use to meet my use case scenario(s). Here is the current situation in order to paint a picture:

I currently have a fairly power hungry setup running with 24 TB in a NetApp DS2248 connected via HBA to a Dell PowerEdge R610 running TrueNAS to provide my local storage needs for 24/7 operations in my home. This hardware was always slated to be temporary for 24/7 storage operations and I'm nearing the time of that ending. I've been (slowly via a 25 mbps upstream link) uploading its contents to google drive with the intent of using rclone to access that stored data and revert the TrueNAS configuration for data science lab work. My downstream connection is 400 mbps - best I can get in my area - no option to get any faster unless I want to pay to have miles of fiber laid down.

My 24/7 operations include a Plex server and a file server. The file server is used more for archival storage of data and documents which may need to be accessed at any time or not accessed for years with no reliable forecast as to when they will need to be next accessed. I occasionally use the file server to edit video and prefer to store all of my source video/media files on the file server for ease of jumping between workstations. I also provide myself remote access to this file share via nextcloud which affords me an easy-to-manage way to share individual files to people in my personal and professional lives. Once the switch is complete I will also use this google drive rclone VFS as a backup solution for lab data which includes archiving VMDKs and other very large files.

My goal is to slim down the power draw on my 24/7 operations compared to the 600 watts it draws now. I have a Cisco Catalyst 2960 PoE switch at the core of my network and I 3D printed a raspberry pi rack. I've been racking raspberry pis into it with PoE hats. I'm not clustering them, just giving each raspberry pi a specific task to perform and building redundancy by establishing more than one pi performing the same work (such as two raspberry pis running DNS).

So back to the subject of this thread: One VFS Cache to rule them all?

Originally I had thought to set up an 8 GB pi4 with a 512 GB microSD (and possibly an external USB SSD) to act as a google drive "gateway" (and nothing else). My thought was to have it mount the google drive storage and bring it in as a VFS and then share that VFS out via SMB and NFS. I'd configure a healthy cache for both read and writes to help compensate for my video editing use case scenario and my poor upload speeds. I configured a 1 week vfs-cache-max-age value because typically if I don't access a file I've saved to storage within the first week, I likely won't access it again for several months or longer. I'm also a very patient person so if it's been a fairly busy week for me and items have fallen out of cache due to lack of space then I'm okay with that and understanding.

In testing this has proven to be unstable at best. Perhaps it was my settings, perhaps it was asking the pi4 to do too much. Perhaps it was another issue. Before I continue further I would like some feedback from experience/insight which I do not have in this matter.

For clarity: During testing the Plex server was run off a VM on the PowerEdge. I experienced smooth direct stream playback starts (5-7 seconds after requesting play) but after 5-10 minutes it would start buffering and I'd notice the rclone gateway pi4 would have an 8+ load average and it would start pulling data from gdrive. This buffering seemed to be almost entirely eliminated if I requested a transcode on the playback. The plex server in this test case had a 300 second transcode back off. I prefer to avoid transcoding so that is not a solution: just a diagnostic finding.

During testing, reading via SMB seemed to be reliable but extremely slow (<~100 mbps average) with wildly unstable speeds (swinging between 1 mbps and 120 mbps) on reads on uncached files. For what it's worth, when initially starting plex playback I've watched the rclone pi4 pull upwards of 400 mbps from gdrive, as expected. I suspect this to be an issue with SMB and I will conduct further testing from an SMB share off of a local file system on the same pi to compare performance characteristics.

I did not test NFS thoroughly enough to provide feedback.

Nextcloud (running on another raspberry pi) had issues with "seeing" files which were definitely listed in the local VFS on the rclone gateway pi and in the NFS mount on the nextcloud pi. I attribute this mostly to Nextcloud internal issues but I'm open to any additional insight as to causality.

At this point I'm contemplating whether I'd be better off having each application run its own VFS locally instead of having a dedicated "gateway". The gateway would be easier for me and reduce overhead on each application "server" but if it's not possible then I'll have to forget the one VFS cache to rule them all.

Here are the settings I used last (ignore any mispellings as I am transcribing this from a screenshot I took and currently do not have direct access to the /etc/systemd/system/rclone.service file) I've made multiple tweaks through the entire process and this is the latest iteration:

mount command:

/usr/bin/rclone mount gcrypt: /mnt/gdrive
--dir-cache-time 168h
--vfs-cache-mode full
--vfs-cache-max-age 168h
--vfs-cache-max-size 500G
--vfs-read-ahead 1G
--vfs-read-chunk-size 100M
--vfs-read-chunk-size-limit 500M
--buffer-size 512M
--log-level INFO


type = drive
scope = drive
token = {"access_token":"XXXXXXXXXXX"}
root_folder_id = XXXXXXXXXX
client_id = XXXXXXXXXXXX
client_secret = XXXXXXXXXX

type = cache
remote = gdrive:/gdrive
plex_url = http://x.x.x.x:32400
plex_username =
plex_password = XXXXXXXXXX
chunk_size = 85M
info_age = 48h
chunk_total_size = 10G

type = crypt
remote = gcache:
filename_encryption = standard
directory_name_encryption = true
password = XXXXXXXX

hello and welcome to the forum,

--fast-list does not work on a mount.
--allow-non-empty is considred a bad idea unless you are 100% sure you need that.

about the cache remote, have you read this?

for plex, not sure that you would need a massive cache for it. just stream what you want to watch.

@asdffdsa I have read the cache status section. I've been considering dropping the cache remote after reading that. Just wanted feedback from someone more experienced with rclone than I.

I'll drop --fast-list. I understand the implications of --allow-non-empty and am merely testing it. I will more than likely drop it for full production.

My reasoning to cache plex files locally for a time is that in my use cases a single item in the Plex library may be viewed 4-6 times over ~48 hours with heavy use of resume/skipping/seeking. This behavior is the typical behavior of my family - someone plays a movie then gets pulled away and leaves it playing. They then return to watching the movie and try to find where they left off by skipping/seeking because they didn't stop playing when they were pulled away. I've tried to "train" them to pause/stop the movie when they walk away but no luck in 10 years. Most of that content has a bitrate of 40-100 mbps. From an economic standpoint it would see me save a ton of bandwidth. From a performance standpoint, such a local cache would likely come closest to emulate my current plex performance characteristics (essentially near RAMDisk speeds where the copper gigabit ethernet connection between the TV and the plex server IS the bottleneck - ZFS's MLA/MFA caching algorithm has really spoiled my family's expectations of Plex's performance.).

I'm interested in any configuration which would enable the VFS cache to pull down the complete file requested to the local cache on an SSD and hold it there for 48h or until the SSD hits about 80% usage.

i am not an expert on the vfs cache, but there are many flags that can be tweaked.

--vfs-cache-max-age duration Max age of objects in the cache. (default 1h0m0s)

and this is the goto guide

as for archiving VMDKs and backups, i would never use a vfs cache, instead i use rclone copy/sync

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.