What is the problem you are having with rclone?
Rclone is working great, but I'm looking for a novel solution to "smart caching" files that have a high likelihood of being used in the future. I have two large HDDs with about 20TB of data between them. I then have a 500GB NVME drive I'm using as a vfs cache drive. I'm trying to minimize how often I need to spin up the large HDDs.
I'm looking for a way that when a file is accessed and triggers a pull into cache, all other files in the same folder are also pulled. I'd imagine a bash script that runs every X min and compares the contents of all the folders in the cache vfs folder compared to the same folder in the mount, and then copies each missing file from the mount to a temporary location and then deletes the file in order to trigger the moving the files into the cache. But maybe there's another way using rclone logs, or something else I'm missing?
If anyone has any suggestions on how to tackle it, I'd greatly appreciate it!
Run the command 'rclone version' and share the full output of the command.
rclone v1.66.0
- os/version: alpine 3.19.1 (64 bit)
- os/kernel: 6.5.0-35-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.22.1
- go/linking: static
- go/tags: none
Which cloud storage system are you using? (eg Google Drive)
Local
The command you were trying to run (eg rclone copy /tmp remote:tmp)
rclone mount "union:" "/mnt/docker/media/rclone/media-internal" --uid 1000 --gid 1000 --umask 002 --default-permissions --allow-non-empty --allow-other --vfs-cache-mode full --vfs-fast-fingerprint --cache-dir "/cache/" --dir-cache-time 9999h --poll-interval=0 --vfs-cache-max-age 9999h --vfs-cache-max-size 500G --vfs-cache-poll-interval 1m --vfs-write-back 9999h --buffer-size 16M --vfs-read-ahead 20G --log-level INFO --stats 15m
Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.
[union]
type = union
upstreams = /mnt/data1 /mnt/data2
action_policy = lfs
create_policy = lfs
min_free_space = 100Gi
A log from the command that you were trying to run with the -vv flag
Log omitted as command run as expected