You are combining two that shouldn’t go together. You should either use the cache mode writes or the tmp-upload. Not both.
You should wait for this to get implemented:
If you want a delay in uploading.
True. But without combining the two, the O_TRUNC errors start appearing again. Honestly, it would be much easier if vfs had a reliable upload and could replace the cache altogether.
@ncw I have recently noticed that rclone blocks all operations on cache when a file is being uploaded to the drive from the tmp-upload-path. Is this by design?
@darthShadow would you mind sharing your latest mount please - I keep getting out of memory issues and I’m not sure why.
Thanks
Sure.
/home/darthshadow/rclone/bin/rclone mount GCrypt: /home/darthshadow/GMedia \
--allow-other \
--dir-cache-time=24h \
--cache-db-purge \
--cache-db-path=/dev/shm/rclone/ \
--cache-chunk-path=/home/darthshadow/ssd/ \
--cache-workers=4 \
--cache-tmp-upload-path=/home/darthshadow/rclone/tmp_upload/ \
--cache-tmp-wait-time=6h \
--cache-chunk-size=128M \
--cache-chunk-total-size=75G \
--cache-chunk-clean-interval=300s \
--cache-info-age=36h \
--drive-chunk-size=32M \
--fast-list \
--vfs-read-chunk-size=256M \
--vfs-read-chunk-size-limit=off \
--vfs-cache-mode=writes \
--vfs-cache-max-age=3h \
--vfs-cache-poll-interval=1m \
--cache-dir=/home/darthshadow/rclone/ \
--buffer-size=0 \
--attr-timeout=1s \
--umask 002 \
--rc \
--log-file=/home/darthshadow/rclone/rclone.log \
--stats=1m \
--stats-log-level=NOTICE \
-v
Haven’t had an OOM issue for quite a few months now. I regularly have 5-6 4k direct plays/streams going on and I don’t think the memory usage has ever crossed 20G.
My next experiments will be with the drive-chunk-size and the transfers settings to increase the upload speeds.
thanks - very interesting seeing you combing cache and vfs like that. What are your launch times like compared to just a vfs mount?
I’ve been trying to create a hybrid of 2 mounts, a cache mount for uploading and a vfs mount for playing but I’m going try your version on a test plex server to see if I can get the best of both worlds from one mount.
one more question - what’s the config for your GCrypt - it’s a crypt remote not a cache?
I have never actually tried out just a plain vfs mount. With this config I am typically able to start streams within a couple of seconds (5-10) generally.
Its Drive -> Cache -> Crypt with the Crypt finally being mounted.
1 Like
I’ve been running my mergerfs / vfs mount now for a number of months with zero issues. I do like having a ‘single’ solution, but a great saying is don’t let perfect be the enemy of the good.
All my plays start in 1-2 seconds at most which is the same as local media for the most part. mergerfs handles all the writes / hard links with Sonarr/Radarr so things appear instantly instead of adding a ton of extra IO.
If the delay features make it in for VFS, I might try that.
@darthShadow Thanks - I got my cache working without the errors by following your mount, mainly by adding --vfs-cache-mode=writes
rclone mount --fast-list --allow-other --dir-cache-time=48h --cache-chunk-path=/mnt/user/appdata/other/rclone/cache --cache-chunk-size=128M --cache-chunk-total-size=100G --cache-info-age=72h --cache-db-purge --cache-db-path=/tmp/rclone --cache-workers=4 --buffer-size 0M --bwlimit 9M --cache-tmp-upload-path=/mnt/user/rclone_upload/google_cache --cache-tmp-wait-time=60m --stats=1m --stats-log-level=INFO -v --vfs-read-chunk-size=256M --vfs-read-chunk-size-limit=off --vfs-cache-mode=writes --cache-dir=/mnt/user/rclone_upload/google_cache_dir gdrive_media_cache: /mnt/user/mount_rclone/google_cache &
@ncw it’d be brilliant in the future if the file was written ‘faster’ to the mount as at the moment on my system it appears initially when the file is written to --cache-dir, but then disappears until the ‘2nd write’ to --cache-tmp-upload-path is completed i.e. writes take twice as long to complete.
Under normal operation I wouldn’t notice it as much, but I’m transferring 6TB of files to my new rclone cache that were previously queued to be uploaded by rclone move (i’m now using a vfs mount for playback and a cache mount to upload, so I can remove unionfs from my equation and get the best of both worlds) and it’s very painful…
Duplicating the vfs-cache-mode-writes and the cache-tmp-upload isn’t needed.
You are already automatically uploading with the first mode and re-uploading again basically doubling your outbound traffic, local writes to disk and your daily quota.
I mentioned earlier, you really should not be using both.
It won’t double the daily quota or the outbound traffic. From what I understood earlier, the vfs will upload to the cache backend instead of directly to GDrive. The cache backend, in turn, will upload to GDrive as configured.
This is the exact quote from earlier:
rclone cache alone doesn’t work for me - I get Can't change size without --vfs-cache-mode >= writes errors, which I think is explained here:
https://rclone.org/commands/rclone_mount/#limitations
Without the use of “–vfs-cache-mode” this can only write files sequentially, it can only seek when reading. This means that many applications won’t work with their files on an rclone mount without “–vfs-cache-mode writes” or “–vfs-cache-mode full”. See the File Caching section for more info.
It’s a bit ugly how it goes through the process and resuming stuff/changing things adds quite a bit of complexity to the overall process.
If you turn up the debug logs, you can see the flow of what happens.
It copies to the local vfs cache-dir
It does seem to send that to the cache backend
It does seem to wait for the timer properly to upload
Once it uploads to the cache backend though, it doesn’t remove the file from the vfs cache area.
If you are modifying a file, sure. If you are just copying a file into a directory like a show or movie, you don’t need it.
The problem lies in with Sonarr/Radarr as they write a file with the partial at the end and immediately move it when completed so rclone doesn’t like that.
Your post/question doesn’t seem to have much to do with the post on cache age vs vfs cache age though so it’s a different topic.
It does seem to delete it for me once --vfs-cache-max-age has expired - did you lower this from the default 1h in your test?
I agree, I don’t need it for my scenario but I can’t write to the cache mount without it. I’d love to be able to ditch it and not have the double write.
That seems to be the intended behaviour and it seems logical to me too.
I did lower it, but I had a typo and an extra number at the end so that was my issue with it not removing.
That seems to be fine now.