When there’s a mount with hundreds of thousands or millions of deeply nested files and folders, a lot of time is spent when a program needs to scan through the mount. I can increase the directory and attribute cache so that it only occurs on the first scan while subsequent scans are instant. As soon as a remote is mounted, I run a script which essentially does ‘find . ‘ and it takes about an hour to finish.
If I use fast-list to list files, it takes around 5 minutes but of course that can’t be used on a mount.
Would it be possible to get a config option that performs a fast list for the initial amount?
Run the command 'rclone version' and share the full output of the command.
rclone v1.59.1
os/version: ubuntu 18.04 (64 bit)
os/kernel: 4.9.253-tegra (aarch64)
os/type: linux
os/arch: arm64
go/version: go1.18.5
go/linking: static
go/tags: none
Which cloud storage system are you using? (eg Google Drive)
Google Drive, S3
The command you were trying to run (eg rclone copy /tmp remote:tmp)
I believe no-check-dest and no traverse don’t work on mounts however it’s been left over from when the systemd service was created last year and therefore included above. I’ll remove it next time there’s a need to change the mount options.
Like Ani said, this is much faster. When I make changes to the mount, I use an even more basic command
rclone mount --drive-pacer-burst 200 --drive-pacer-min-sleep 10ms --user-agent ******* -v
This is, of course, slow to list things when there are a lot of files and folders (on Windows, in particular), but I can live with that, since I only use it once a day to move things around.
I couldn't wait so I tried it - I've gone from 40 minute scan per mount to under 1 minute!
The data rarely changes and when it does, someone restarts rclone so having a high attribute and directory cache is fine. Of course, now that I'm aware of rc, I can programmatically refresh the cache when the data sets change.
This isn't a problem for me but just as an observation in case it's helpful to anyone - _async ran 9 seconds faster on average (tested 5x with and 5x without).
# time rclone -vv rc vfs/refresh recursive=true --rc-addr 127.0.0.1:5508
2022/09/01 19:28:37 DEBUG : rclone: Version "v1.59.1" starting with parameters ["rclone" "-vv" "rc" "vfs/refresh" "recursive=true" "--rc-addr" "127.0.0.1:5508"]
{
"result": {
"": "OK"
}
}
2022/09/01 19:29:34 DEBUG : 4 go routines active
real 0m57.046s
user 0m0.068s
sys 0m0.008s