Can we get fast-list on mounts

What is the problem you are having with rclone?

When there’s a mount with hundreds of thousands or millions of deeply nested files and folders, a lot of time is spent when a program needs to scan through the mount. I can increase the directory and attribute cache so that it only occurs on the first scan while subsequent scans are instant. As soon as a remote is mounted, I run a script which essentially does ‘find . ‘ and it takes about an hour to finish.

If I use fast-list to list files, it takes around 5 minutes but of course that can’t be used on a mount.

Would it be possible to get a config option that performs a fast list for the initial amount?

Run the command 'rclone version' and share the full output of the command.

rclone v1.59.1

  • os/version: ubuntu 18.04 (64 bit)
  • os/kernel: 4.9.253-tegra (aarch64)
  • os/type: linux
  • os/arch: arm64
  • go/version: go1.18.5
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive, S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

/usr/bin/rclone mount --config=/home/rclone/.config/rclone/rclone.conf --use-mmap --read-only --allow-other --umask 000 
--no-check-certificate --timeout 10s --contimeout 10s --no-check-dest --no-traverse --no-checksum --fs-cache-expire-duration 180m --fs-cache-expire-interval 180m --tpslimit 8
--attr-timeout 24h --dir-cache-time 24h --poll-interval 12h --buffer-size 64M --drive-chunk-size 64M --vfs-cache-mode minimal --vfs-cache-max-size 4G --vfs-cache-max-age 4h
--log-level ERROR --log-file /var/log/rclone/vision1.log gdvision1: /mnt/remote/vision1

I believe no-check-dest and no traverse don’t work on mounts however it’s been left over from when the systemd service was created last year and therefore included above. I’ll remove it next time there’s a need to change the mount options.

Most folks run a rc command after the mount happens to prime the directory listing:

ExecStartPost=/usr/bin/rclone rc vfs/refresh recursive=true --rc-addr 127.0.0.1:5575 _async=true

My rc runs locally on 5575 and it will run much faster than a find.

I wouldn't touch attribute timeout unless you are certain nothing else is touching a file anywhere else.

Dir cache can be set very high on polling remotes like Google Drive, Dropbox, etc.

# Dropbox is a polling remote so this value can be set very high and any changes are detected via polling.
--dir-cache-time 9999h \
1 Like

My mount is read-only, just like yours. I run this fairly basic command

rclone mount --attr-timeout 5000h --dir-cache-time 5000h --drive-pacer-burst 200 --drive-pacer-min-sleep 10ms --no-checksum --poll-interval 0 --rc --read-only --user-agent ******* -v

followed by

rclone rc vfs/refresh recursive=true --drive-pacer-burst 200 --drive-pacer-min-sleep 10ms --timeout 30m --user-agent *******

Like Ani said, this is much faster. When I make changes to the mount, I use an even more basic command

rclone mount --drive-pacer-burst 200 --drive-pacer-min-sleep 10ms --user-agent ******* -v

This is, of course, slow to list things when there are a lot of files and folders (on Windows, in particular), but I can live with that, since I only use it once a day to move things around.

Thanks @Animosity022 and @VBB.

I couldn't wait so I tried it - I've gone from 40 minute scan per mount to under 1 minute!

The data rarely changes and when it does, someone restarts rclone so having a high attribute and directory cache is fine. Of course, now that I'm aware of rc, I can programmatically refresh the cache when the data sets change.

This isn't a problem for me but just as an observation in case it's helpful to anyone - _async ran 9 seconds faster on average (tested 5x with and 5x without).

# time rclone -vv rc vfs/refresh recursive=true --rc-addr 127.0.0.1:5508
2022/09/01 19:28:37 DEBUG : rclone: Version "v1.59.1" starting with parameters ["rclone" "-vv" "rc" "vfs/refresh" "recursive=true" "--rc-addr" "127.0.0.1:5508"]
{
        "result": {
                "": "OK"
        }
}
2022/09/01 19:29:34 DEBUG : 4 go routines active

real    0m57.046s
user    0m0.068s
sys     0m0.008s
# rclone -vv rc vfs/refresh recursive=true --rc-addr 127.0.0.1:5508 _async=true
2022/09/01 19:31:03 DEBUG : rclone: Version "v1.59.1" starting with parameters ["rclone" "-vv" "rc" "vfs/refresh" "recursive=true" "--rc-addr" "127.0.0.1:5508" "_async=true"]
{
        "jobid": 1
}
2022/09/01 19:31:03 DEBUG : 4 go routines active

# rclone -vv rc --json '{ "jobid":1 }' job/status --rc-addr 127.0.0.1:5508
2022/09/01 19:31:58 DEBUG : rclone: Version "v1.59.1" starting with parameters ["rclone" "-vv" "rc" "--json" "{ \"jobid\":1 }" "job/status" "--rc-addr" "127.0.0.1:5508"]
{
        "duration": 48.78253842,        "endTime": "2022-09-01T19:31:51.803767603+01:00",        "error": "",        "finished": true,        "group": "job/1",        "id": 1,        "output": {                "result": {
                        "": "OK"
                }
        },
        "startTime": "2022-09-01T19:31:03.021229439+01:00",
        "success": true
}
2022/09/01 19:31:58 DEBUG : 4 go routines active

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.