I have a spinning disk local backend (/mnt/md0) and I want to use the vfs cache so that recently stuff is put in a smaller NVME drive. After enabling the vfs cache, no objects are ever put into it when they're written or read.
The local remote (/mnt/remote2) is merged with two other Google backend remotes using mergerfs
/usr/bin/mergerfs -o category.create=ff,async_read=true,cache.files=partial -o dropcacheonclose=true,minfreespace=0,fsname=mergerfs -o xattr=nosys,statfs=full,statfs_ignore=nc,umask=002,noatime /mnt/local=RW:/mnt/remote=NC:/mnt/remote2=NC:/mnt/remote3=NC /mnt/unionfs
The google backends cache properly, while the local one does not.
rclone v1.62.2
os/version: ubuntu 20.04 (64 bit)
os/kernel: 5.15.0-73-generic (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.20.2
go/linking: static
go/tags: none
Local
rclone copy /tmp remote:tmp)
/usr/bin/rclone mount --user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36 --config=/home/xxx/.config/rclone/rclone.conf --allow-other --async-read=true --dir-cache-time=1000h --buffer-size=256M --poll-interval=15s --rc --rc-no-auth --rc-addr=localhost:5573 --use-mmap --vfs-read-ahead=128M --vfs-read-chunk-size=32M --vfs-read-chunk-size-limit=2G --vfs-cache-max-age=504h --vfs-cache-mode=full --vfs-cache-poll-interval=30s --vfs-cache-max-size=300G --timeout=10m --drive-skip-gdocs --drive-pacer-min-sleep=10ms --drive-pacer-burst=1000 --umask=002 --syslog -v /mnt/md0 /mnt/remote2
No config, using local paths
-vv flag
This is what I see in the logs
rclone[131726]: vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
You'd have to copy something to the mount location for it to populate.
Not super relevant but you have a ton of non needed flags for a local file system as well.
[felix@gemini ~]$ rclone mount /data /home/felix/test --vfs-cache-mode full -vv
2023/06/08 15:02:02 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
2023/06/08 15:02:02 DEBUG : rclone: Version "v1.63.0-beta.7013.d05393b86.fix-6986-times" starting with parameters ["rclone" "mount" "/data" "/home/felix/test" "--vfs-cache-mode" "full" "-vv"]
2023/06/08 15:02:02 DEBUG : Creating backend with remote "/data"
2023/06/08 15:02:02 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2023/06/08 15:02:02 INFO : Local file system at /data: poll-interval is not supported by this remote
2023/06/08 15:02:02 DEBUG : vfs cache: root is "/home/felix/.cache/rclone"
2023/06/08 15:02:02 DEBUG : vfs cache: data root is "/home/felix/.cache/rclone/vfs/local/data"
2023/06/08 15:02:02 DEBUG : vfs cache: metadata root is "/home/felix/.cache/rclone/vfsMeta/local/data"
2023/06/08 15:02:02 DEBUG : Creating backend with remote "/home/felix/.cache/rclone/vfs/local//data"
2023/06/08 15:02:02 DEBUG : fs cache: renaming cache item "/home/felix/.cache/rclone/vfs/local//data" to be canonical "/home/felix/.cache/rclone/vfs/local/data"
2023/06/08 15:02:02 DEBUG : Creating backend with remote "/home/felix/.cache/rclone/vfsMeta/local//data"
2023/06/08 15:02:02 DEBUG : fs cache: renaming cache item "/home/felix/.cache/rclone/vfsMeta/local//data" to be canonical "/home/felix/.cache/rclone/vfsMeta/local/data"
2023/06/08 15:02:02 INFO : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2023/06/08 15:02:02 DEBUG : Local file system at /data: Mounting on "/home/felix/test"
2023/06/08 15:02:02 DEBUG : : Root:
2023/06/08 15:02:02 DEBUG : : >Root: node=/, err=<nil>
2023/06/08 15:02:16 DEBUG : /: Attr:
2023/06/08 15:02:16 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2023/06/08 15:02:19 DEBUG : /: Lookup: name="blah"
2023/06/08 15:02:19 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory
2023/06/08 15:02:19 DEBUG : /: Lookup: name="blah"
2023/06/08 15:02:19 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory
2023/06/08 15:02:19 DEBUG : /: Create: name="blah"
2023/06/08 15:02:19 DEBUG : blah: Open: flags=O_WRONLY|O_CREATE|O_EXCL
2023/06/08 15:02:19 DEBUG : blah: newRWFileHandle:
2023/06/08 15:02:19 DEBUG : blah(0xc0002ba540): openPending:
2023/06/08 15:02:19 DEBUG : blah: vfs cache: truncate to size=0 (not needed as size correct)
2023/06/08 15:02:19 DEBUG : : Added virtual directory entry vAddFile: "blah"
2023/06/08 15:02:19 DEBUG : blah(0xc0002ba540): >openPending: err=<nil>
2023/06/08 15:02:19 DEBUG : blah: >newRWFileHandle: err=<nil>
2023/06/08 15:02:19 DEBUG : : Added virtual directory entry vAddFile: "blah"
2023/06/08 15:02:19 DEBUG : blah: >Open: fd=blah (rw), err=<nil>
2023/06/08 15:02:19 DEBUG : /: >Create: node=blah, handle=&{blah (rw)}, err=<nil>
2023/06/08 15:02:19 DEBUG : blah: Attr:
2023/06/08 15:02:19 DEBUG : blah: >Attr: a=valid=1s ino=0 size=0 mode=-rw-rw-r--, err=<nil>
2023/06/08 15:02:19 DEBUG : &{blah (rw)}: Write: len=192, offset=0
2023/06/08 15:02:19 DEBUG : blah(0xc0002ba540): _writeAt: size=192, off=0
2023/06/08 15:02:19 DEBUG : blah(0xc0002ba540): >_writeAt: n=192, err=<nil>
2023/06/08 15:02:19 DEBUG : &{blah (rw)}: >Write: written=192, err=<nil>
2023/06/08 15:02:19 DEBUG : &{blah (rw)}: Flush:
2023/06/08 15:02:19 DEBUG : blah(0xc0002ba540): RWFileHandle.Flush
2023/06/08 15:02:19 DEBUG : &{blah (rw)}: >Flush: err=<nil>
2023/06/08 15:02:19 DEBUG : &{blah (rw)}: Release:
2023/06/08 15:02:19 DEBUG : blah(0xc0002ba540): RWFileHandle.Release
2023/06/08 15:02:19 DEBUG : blah(0xc0002ba540): close:
2023/06/08 15:02:19 DEBUG : blah: vfs cache: setting modification time to 2023-06-08 15:02:19.277619908 -0400 EDT m=+17.131112003
2023/06/08 15:02:19 INFO : blah: vfs cache: queuing for upload in 5s
2023/06/08 15:02:19 DEBUG : blah(0xc0002ba540): >close: err=<nil>
2023/06/08 15:02:19 DEBUG : &{blah (rw)}: >Release: err=<nil>
2023/06/08 15:02:24 DEBUG : blah: vfs cache: starting upload
2023/06/08 15:02:24 DEBUG : blah: md5 = 2eb05ccb9e16264d184e4503a8445499 OK
and the local cache file
[felix@gemini vfs]$ ls -alR
.:
total 0
drwx------ 3 felix felix 19 Jun 8 15:02 .
drwx------ 4 felix felix 32 Jun 8 15:02 ..
drwx------ 3 felix felix 18 Jun 8 15:02 local
./local:
total 0
drwx------ 3 felix felix 18 Jun 8 15:02 .
drwx------ 3 felix felix 19 Jun 8 15:02 ..
drwx------ 2 felix felix 18 Jun 8 15:02 data
./local/data:
total 4
drwx------ 2 felix felix 18 Jun 8 15:02 .
drwx------ 3 felix felix 18 Jun 8 15:02 ..
-rw------- 1 felix felix 192 Jun 8 15:02 blah
I did try both copying via the mount as well as reading from it and the cache never populated. I reused the config from my other remotes so you may be right, one or more of those flags might be interfering with the cache. I'll remove them and re-test.
You'd want to recreate the issue and share the debug log.
The log you shared is one line and so perhaps you didn't copy anything there.
Looks like that was it. One of the other options appears to be causing the incompatibility. Using this simpler mount command appears to work
rclone mount --config=/home/xxx/.config/rclone/rclone.conf --allow-other --dir-cache-time=1000h --vfs-cache-max-age=504h --vfs-cache-mode=full --vfs-cache-max-size=300G --umask=002 --syslog -v /mnt/md0 /mnt/remote2
EDIT: and to save some trouble for anyone doing this: if you have big files in your backing drives/array then you probably need to add --vfs-fast-fingerprint to your mount, otherwise reads via rclone for uncached content will be much slower than pulling from the backing array due to rclone needing to read the entire file to calculate the fingerprint.
system
July 8, 2023, 8:57pm
6
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.