Caching files from a mounted remote location without high CPU usage?

Help and Support
1

What is the problem you are having with rclone?

I'm trying to cache all images from a remote location (mounted google drive) by reading them and redirecting output to /dev/null. However, this causes high CPU usage:

image
image1060×555 187 KB

I used fdfind for this purpose, which is multithreaded:
fd . -e png -e jpg -X cat >/dev/null

To avoid overheating the machine I've switched to a for loop instead (seen later in the above graph):
for i in **/*.png; do echo $i; cat $i >/dev/null; done

Is such high CPU usage normal? Are there better ways to cache/read files from an encrypted, mounted rclone remote?

Run the command 'rclone version' and share the full output of the command.

rclone v1.64.2
- os/version: arch "rolling" (64 bit)
- os/kernel: 6.5.9-arch2-1 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.21.3
- go/linking: dynamic
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

fd . -e png -e jpg -X cat >/dev/null

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[gdrive]
type = drive
client_id = 
client_secret =
scope = drive
token = {"access_token":"","expiry":""}
root_folder_id = 
team_drive = 

[gdrive-encrypted]
type = crypt
remote = gdrive:crypt
password = 
password2 = 

[gdrive-encrypted]
type = crypt
remote = gdrive:crypt
password = 
password2 =

The mount also goes through mergerfs (google drive combined with encrypted aws glacier):

/usr/bin/mergerfs \
  -f \
  -o cache.files=partial \
  -o dropcacheonclose=true \
  /mnt/gdrive-rclone-encrypted:/mnt/aws-glacier-deep-crypt=NC \
  /mnt/gdrive-aws-union-crypt

A log from the command that you were trying to run with the -vv flag

2

hi,

perhaps simplify things a bit, establish baseline by testing without mergerfs and aws?

what is the exact mount command and its debug log?

3

Hi

I've run fd . -e png -e jpg -X cat >/dev/null both on mergerfs mount and on rclone:
image

The two spikes on the sides are from rclone and the one in the middle is from mergerfs. Rclone is smaller although still noticeable.

It's also strange that they are not as big as before - I assume that most of the images were already cached.

First systemd unit for encrypted rclone mount (google drive):

[Unit]
After = network.target

[Service]
Type=notify
RestartSec=5
ExecStart = rclone mount gdrive-encrypted: /mnt/gdrive-rclone-encrypted \
--umask 077 \
--rc --rc-addr :XXXX --rc-user XXXX --rc-pass XXXX \
--cache-dir /home/user/.cache/rclone/gdrive-encrypted \
--vfs-cache-mode full \
--vfs-cache-poll-interval 5m \
--vfs-cache-max-age 8760h \
--vfs-cache-max-size 10G \
--attr-timeout 8700h \
--dir-cache-time 8760h \
--poll-interval 10s \
--multi-thread-streams 0

ExecStop = fusermount -uz /mnt/gdrive-rclone-encrypted
ExecStartPost = rclone rc vfs/refresh recursive=true --fast-list --drive-pacer-min-sleep 10ms _async=true --rc-addr :XXXX --rc-user XXXX --rc-pass XXXX --timeout 0
Restart = on-failure

[Install]
WantedBy = default.target

Then systemd unit for mergerfs (rclone google drive and aws):

[Unit]
After = network.target
Requires = mount-rclone-gdrive-encrypted.service
After = mount-rclone-gdrive-encrypted.service
Requires = mount-aws-glacier-deep-crypt.service
After = mount-aws-glacier-deep-crypt.service

[Service]
Type=simple
ExecStart = /usr/bin/mergerfs \
  -f \
  -o cache.files=partial \
  -o dropcacheonclose=true \
  /mnt/gdrive-rclone-encrypted:/mnt/aws-glacier-deep-crypt=NC \
  /mnt/gdrive-aws-union-crypt
ExecStop = fusermount -uz /mnt/gdrive-aws-union-crypt
Restart = on-failure

[Install]
WantedBy = default.target

aws mount:

[Unit]
After = network.target

[Service]
Type=notify
RestartSec=5
ExecStart = rclone mount aws-glacier-deep-crypt: /mnt/aws-glacier-deep-crypt \
--umask 077 \
--rc --rc-addr :XXXX --rc-user XXXX --rc-pass XXXX \
--cache-dir /home/user/.cache/rclone/aws-glacier-deep-crypt \
--vfs-cache-mode full \
--vfs-cache-poll-interval 5m \
--vfs-cache-max-age 8760h \
--vfs-cache-max-size 10G \
--attr-timeout 8700h \
--dir-cache-time 8760h \
# --poll-interval 10s \
--multi-thread-streams 0 \
--log-file /home/user/.config/rclone/aws-glacier-deep-crypt.log

ExecStop = fusermount -uz /mnt/aws-glacier-deep-crypt
ExecStartPost = rclone rc vfs/refresh recursive=true --fast-list --drive-pacer-min-sleep 10ms _async=true --rc-addr :XXXX --rc-user XXXX --rc-pass XXXX --timeout 0
Restart = on-failure

[Install]
WantedBy = default.target