Refreshing RClone Cache for Specific Paths

I have micro-services architecture in my back-end, one is responsible to upload files to s3, others are processing that files via Redis pub/sub messaging. Instead of downloading files for each server and cache them locally + rotate, decide to use RClone with its caching + rotating/limiting size features.

Imagine case when a file is uploaded into the S3 storage, and after uploading the file is requested from one of the worker servers, when they check a rclone mounted directory there will no be the recent file, according to the rclone doc: changes made directly on the cloud storage by the web interface or a different copy of rclone will only be picked up once the directory cache expires. So currently using kill -SIGHUP $(pidof rclone) or kill -HUP $(pidof rclone) to to flush all directory caches before checking the newly uploaded file from the workers. For now it seems to work, after the signal, will getting updated files including the new file.

But I worry if it refreshes all files/dirs and not only the path I need, what if there are 1000s+ files, etc.
Maybe there is a way to refresh/check remote only for specific path?
Also, there maybe several threads running -SIGHUP, not sure if they interfere and interrupt file reading, etc.

What's best option in my case?

[Service]
Type=notify
ExecStart=/usr/bin/rclone mount \\
    --cache-dir="${CACHE_DIR}" \\
    --vfs-cache-mode full \\
    --vfs-cache-max-age 5000h \\
    --vfs-cache-max-size 100G \\
    --no-modtime \\
    --read-only \\
    --fast-list \\
    --checksum \\
    --allow-other \\
    "${REMOTE_NAME}":"${BUCKET_NAME}" "${MOUNT_DIR}"
ExecStop=/bin/fusermount -uz "${MOUNT_DIR}"
Restart=on-failure
KillMode=none
RestartSec=1600
User=${USER}
Group=$(id -gn "${USER}")
rclone v1.62.2
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 5.15.0-73-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.20.2
- go/linking: static
- go/tags: none

You should read about remote control (rc), specifically vfs/refresh

in your mount command you include:

--rc
--rc-addr 127.0.0.1:1234
--rc-no-auth

then you can run to refresh what you need:

rclone rc --url 127.0.0.1:1234 vfs/refresh  dir=home/junk dir2=data/misc
1 Like

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.