Good morning,
I have been contemplating how to best set up rclone for my google drive storage use to meet my use case scenario(s). Here is the current situation in order to paint a picture:
I currently have a fairly power hungry setup running with 24 TB in a NetApp DS2248 connected via HBA to a Dell PowerEdge R610 running TrueNAS to provide my local storage needs for 24/7 operations in my home. This hardware was always slated to be temporary for 24/7 storage operations and I'm nearing the time of that ending. I've been (slowly via a 25 mbps upstream link) uploading its contents to google drive with the intent of using rclone to access that stored data and revert the TrueNAS configuration for data science lab work. My downstream connection is 400 mbps - best I can get in my area - no option to get any faster unless I want to pay to have miles of fiber laid down.
My 24/7 operations include a Plex server and a file server. The file server is used more for archival storage of data and documents which may need to be accessed at any time or not accessed for years with no reliable forecast as to when they will need to be next accessed. I occasionally use the file server to edit video and prefer to store all of my source video/media files on the file server for ease of jumping between workstations. I also provide myself remote access to this file share via nextcloud which affords me an easy-to-manage way to share individual files to people in my personal and professional lives. Once the switch is complete I will also use this google drive rclone VFS as a backup solution for lab data which includes archiving VMDKs and other very large files.
My goal is to slim down the power draw on my 24/7 operations compared to the 600 watts it draws now. I have a Cisco Catalyst 2960 PoE switch at the core of my network and I 3D printed a raspberry pi rack. I've been racking raspberry pis into it with PoE hats. I'm not clustering them, just giving each raspberry pi a specific task to perform and building redundancy by establishing more than one pi performing the same work (such as two raspberry pis running DNS).
So back to the subject of this thread: One VFS Cache to rule them all?
Originally I had thought to set up an 8 GB pi4 with a 512 GB microSD (and possibly an external USB SSD) to act as a google drive "gateway" (and nothing else). My thought was to have it mount the google drive storage and bring it in as a VFS and then share that VFS out via SMB and NFS. I'd configure a healthy cache for both read and writes to help compensate for my video editing use case scenario and my poor upload speeds. I configured a 1 week vfs-cache-max-age value because typically if I don't access a file I've saved to storage within the first week, I likely won't access it again for several months or longer. I'm also a very patient person so if it's been a fairly busy week for me and items have fallen out of cache due to lack of space then I'm okay with that and understanding.
In testing this has proven to be unstable at best. Perhaps it was my settings, perhaps it was asking the pi4 to do too much. Perhaps it was another issue. Before I continue further I would like some feedback from experience/insight which I do not have in this matter.
For clarity: During testing the Plex server was run off a VM on the PowerEdge. I experienced smooth direct stream playback starts (5-7 seconds after requesting play) but after 5-10 minutes it would start buffering and I'd notice the rclone gateway pi4 would have an 8+ load average and it would start pulling data from gdrive. This buffering seemed to be almost entirely eliminated if I requested a transcode on the playback. The plex server in this test case had a 300 second transcode back off. I prefer to avoid transcoding so that is not a solution: just a diagnostic finding.
During testing, reading via SMB seemed to be reliable but extremely slow (<~100 mbps average) with wildly unstable speeds (swinging between 1 mbps and 120 mbps) on reads on uncached files. For what it's worth, when initially starting plex playback I've watched the rclone pi4 pull upwards of 400 mbps from gdrive, as expected. I suspect this to be an issue with SMB and I will conduct further testing from an SMB share off of a local file system on the same pi to compare performance characteristics.
I did not test NFS thoroughly enough to provide feedback.
Nextcloud (running on another raspberry pi) had issues with "seeing" files which were definitely listed in the local VFS on the rclone gateway pi and in the NFS mount on the nextcloud pi. I attribute this mostly to Nextcloud internal issues but I'm open to any additional insight as to causality.
At this point I'm contemplating whether I'd be better off having each application run its own VFS locally instead of having a dedicated "gateway". The gateway would be easier for me and reduce overhead on each application "server" but if it's not possible then I'll have to forget the one VFS cache to rule them all.
Here are the settings I used last (ignore any mispellings as I am transcribing this from a screenshot I took and currently do not have direct access to the /etc/systemd/system/rclone.service file) I've made multiple tweaks through the entire process and this is the latest iteration:
mount command:
/usr/bin/rclone mount gcrypt: /mnt/gdrive
--allow-other
--umask=0000
--gid=2000
--uid=2000
--cache-db-purge
--dir-cache-time 168h
--cache-info-age=168h
--vfs-cache-mode full
--vfs-cache-max-age 168h
--vfs-cache-max-size 500G
--vfs-read-ahead 1G
--vfs-read-chunk-size 100M
--vfs-read-chunk-size-limit 500M
--buffer-size 512M
--log-level INFO
--log-file=/var/log/rclone.log
rclone.conf
[gdrive]
type = drive
scope = drive
token = {"access_token":"XXXXXXXXXXX"}
root_folder_id = XXXXXXXXXX
client_id = XXXXXXXXXXXX
client_secret = XXXXXXXXXX[gcache]
type = cache
remote = gdrive:/gdrive
plex_url = http://x.x.x.x:32400
plex_username = account@email.com
plex_password = XXXXXXXXXX
chunk_size = 85M
info_age = 48h
chunk_total_size = 10G
plex_token = XXXXXXXXXXXXXX[gcrypt]
type = crypt
remote = gcache:
filename_encryption = standard
directory_name_encryption = true
password = XXXXXXXX
password2 = XXXXXXXXXXXXXX