S3 directory cache does not seem to work

Help and Support
1

What is the problem you are having with rclone?

Directory caching that used to work fine with google drive backend don't work with S3 backend (e2). Running an ncdu takes a long time compared to other mounts with google drive backend.

Run the command 'rclone version' and share the full output of the command.

rclone v1.61.1
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 5.15.0-67-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.19.4
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

iDrive e2

Mount service file:

[Unit]
Description=rclone mount media
AssertPathIsDirectory=/rclonmnts/media
Requires=network-online.target
After=network-online.target
Before=docker.service

[Service]
Environment="RCLONE_CONFIG=/rclonfigs/e2.conf"
Environment="RCLONE_TPSLIMIT=5"
Environment="RCLONE_TPSLIMIT_BURST=5"
Environment="RCLONE_VFS_CACHE_MODE=full"
Environment="RCLONE_VFS_CACHE_MAX_SIZE=16G"
Environment="RCLONE_VFS_CACHE_MAX_AGE=1h"
Environment="RCLONE_VFS_CACHE_POLL_INTERVAL=1m"
Environment="RCLONE_CACHE_DIR=/mnt/nvme/rcloncache/media"
Environment="RCLONE_POLL_INTERVAL=0"
Environment="RCLONE_DIR_CACHE_TIME=24h"
Environment="RCLONE_VFS_READ_CHUNK_SIZE=16M"
Environment="RCLONE_VFS_READ_CHUNK_SIZE_LIMIT=2G"
Environment="RCLONE_BUFFER_SIZE=32M"
Environment="RCLONE_VFS_READ_AHEAD=100M"
Type=notify
ExecStart=/usr/bin/rclone \
    -v mount media: /rclonmnts/media \
    --rc --rc-no-auth --rc-addr=":34770" \
    --allow-other \
    --vfs-fast-fingerprint \
    --no-checksum
    
ExecStartPost=/usr/bin/rclone rc --url http://localhost:34770 vfs/refresh recursive=true
ExecStop=/usr/bin/fusermount -u /rclonmnts/media
Restart=on-abort

[Install]
WantedBy=default.target

The rclone config contents with secrets removed.

[e2]
type = s3
provider = IDrive
access_key_id = 00000000000000000000000000000000000000000000
secret_access_key = 00000000000000000000000000000000000000000000
acl = private
endpoint = 0000.par.idrivee2-40.com

[e2bucket1]
type = alias
remote = e2:rcl0000/

[media]
type = crypt
remote = e2bucket1:
filename_encryption = obfuscate
password = 00000000000000000000000000000000000000000000

A log from the command with the -vv flag

2023/03/27 22:50:29 DEBUG : rclone: Version "v1.61.1" starting with parameters ["rclone" "-vv" "--config" "/rclonfigs/e2.conf" "rc" "--url" "http://localhost:34770" "vfs/refresh" "recursive=true"]
{
        "result": {
                "": "OK"
        }
}
2023/03/27 22:50:29 DEBUG : 4 go routines active
2

The cache is in memory and not persistent so if you stop the mount, you lose the cache.

The first time you crawl the mount, it will then be in cache.

3

Thanks. Shouldn't running rclone rc vfs/refresh recursive=true immediately after mounting achieve that? That's how it works for google drive mounts

4

Yes, but without seeing any logs from the mount, it's tough to guess what may or may not be occurring.

5

Note that reading the modification times takes a HEAD request per object - this is likely the cause of the slowdown.

I recommend you use the --use-server-modtime flag or maybe the --no-modtime flag - this will speed things up.

6

Thank you so much! This is it

7

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.