Uploading many small files causes high CPU usage & high load

Help and Support
1

What is the problem you are having with rclone?

Nextcloud upload of many small files to encrypted GDrive causes high CPU usage and high server load.
I am getting a lot of some kind of cache error (see log below).

What is your rclone version (output from rclone version)

1.52.2

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Ubuntu 20.04 x86-64

Which cloud storage system are you using? (eg Google Drive)

Google Drive (GSuite)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

It's a systemd mount:

/usr/bin/rclone mount \
        --config /home/username/.config/rclone/rclone.conf \
        --vfs-cache-mode writes \
        --vfs-cache-max-age 30m \
	--drive-chunk-size=128M \
        --write-back-cache \
        --allow-other \
	--umask 0027 \

The rclone config contents with secrets removed.

[remotename]
type = crypt
remote = remote:
filename_encryption = standard
directory_name_encryption = true
password = p1
password2 = p2

A log from the command with the -vv flag

Example of journalctl logs for the particular mount:

Nov 08 10:25:10  rclone[8852]: 2020/11/08 10:25:10 ERROR : Nextcloud/data/ilias/files/Sync/My Documents/College/SFSU: cache: double close
Nov 08 10:25:10  rclone[8852]: 2020/11/08 10:25:10 ERROR : Nextcloud/data/ilias/files/Sync/My Documents/College: cache: double close
Nov 08 10:25:10 rclone[8852]: 2020/11/08 10:25:10 ERROR : Nextcloud/data/ilias/files/Sync/My Documents: cache: double close
Nov 08 10:25:10  rclone[8852]: 2020/11/08 10:25:10 ERROR : Nextcloud/data/ilias/files/Sync: cache: double close
Nov 08 10:25:10  rclone[8852]: 2020/11/08 10:25:10 ERROR : Nextcloud/data/ilias/files/Sync/My Documents/College/SFSU/CSC667/AOV/web: cache: double close
Nov 08 10:25:10 rclone[8852]: 2020/11/08 10:25:10 ERROR : Nextcloud/data/ilias/files/Sync/My Documents/College/SFSU/CSC667/AOV: cache: double close
Nov 08 10:25:10  rclone[8852]: 2020/11/08 10:25:10 ERROR : Nextcloud/data/ilias/files/Sync/My Documents/College/SFSU/CSC667: cache: double close
Nov 08 10:25:10  rclone[8852]: 2020/11/08 10:25:10 ERROR : Nextcloud/data/ilias/files/Sync/My Documents/College/SFSU: cache: double close
Nov 08 10:25:10  rclone[8852]: 2020/11/08 10:25:10 ERROR : Nextcloud/data/ilias/files/Sync/My Documents/College: cache: double close
2

migh work better to use rclone copy instead of rclone mount

3

Unfortunately can't do that as this involves a Nextcloud installation that uses the rclone encrypted remote as its data directory, so it needs to be mounted.

4

for copying the files, might try a simpler mount command without the vfs cache.

5

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.