Ok I have tried searching and still cannot come up with answer, trying to understand the concept
I have a Windows rclone setup using gdrive and a union to a local drive to assist in sonarr/etc. they all write directly to the rclone mount. My understanding was the union keeps the files local on disk and I run a upload script overnight to push the files to gdrive using rclone move.
My question is:
with vfs-cache-mode full, do I still need to run a upload script, or would it upload things itself?
With vfs-cache-mode full, do I still need union for sonarr/radarr writing, or it would just create its own dir cache and clear it according to max-age?
What version of rclone are you asking about?
What's your union config look like?
You have assumption listed there but missing some details on your setup as those question can change based on what you have setup and what your mount commands are and what version you are running and what OS, etc.
Basically, the stuff in the template helps get a quick answer.
Hello, I apologize for not using template as I thought this question is more general.
With vfs-cache-mode full, are writes buffered to disk and queued for upload later? Or are they uploaded right away? I remember this was a feature that was being worked on but I can't figure out if it's happening.
Because I have a union between gdrive and a local directory, if I write files to the unioned directory (P: in my example) does it get uploaded by rclone vfs automatically or do I need to rely on my nightly upload script?
Rclone version Windows:
rclone v1.53.1
- os/arch: windows/amd64
- go version: go1.15
Mount command mount --cache-dir=c:\rclonecache --dir-cache-time 500h --vfs-cache-max-age 5h --vfs-cache-max-size 10G --buffer-size 64M --vfs-read-ahead 128M --drive-chunk-size 128M --vfs-cache-poll-interval 1m --dir-cache-time 500h --log-level INFO --vfs-read-chunk-size 128M --vfs-read-chunk-size-limit off media_union: P: --config "C:\Users\XXX\.config\rclone\rclone.conf" --vfs-cache-mode=full
rclone config:
[gdrive]
type = drive
client_id = 7REDACTED
client_secret = 6REDACTEDS
scope = drive
token = {"access_token":"yREDACTED","token_type":"Bearer","refresh_token":"1/REDACTED","expiry":"2020-09-28T14:05:09.2388277+08:00"}
root_folder_id = 0REDACTED
[media_union]
type = union
remotes = gdrive: D:\media_union
Back to my question, as I am not sure how this works when used in conjunction with a union:
If I use vfs-cache-mode: full, will writes to the union be uploaded according to this vfs-write-back duration?
Or will it not be written because I am using a union and my local disk is listed last in the mount command?
Second question:
If I do NOT use a union, could I just create a long vfs-write-back duration (for example, 6 hours) to let the mount handle uploading files after a set duration? This would give my scripts time to analyze the media etc before sending to gdrive.
The vfs layer doesn't look at the fact it is wrapping a union - it treats it like any other backend.
That should work too. Though I'm not clear on the advantages of not sending the file to google drive immediately? It will be in the cache before and after sending to google drive.
The reason I ask is I suspect I’m running into some issues with my setup in regards to uploading content. I have my upload script running at 2 am. The script is a simple rclone move.
If I have a download that finishes at 1:50AM, does rclone mount try to upload it again? I added min-age to see if this prevents this issue. I used to have some content inaccessible after being added till hours later. Or I’m just doing it all wrong.
If the file was due to be written to the local directory in the union then rclone will "upload" it to the local directory. You doing the move should have no effect on the VFS layer - the mount won't try to upload it again.