I recently switched from GDrive to an SFTP remote. What I did not remember is that GDrive is a polling remote, as such the very large
--dir-cache-time I was using with GDrive was causing new files to not appear in the mount.
--dir-cache-time to 5min which solved that issue but introduced another as a side-effect. When the cache expires it takes so long to refresh it that another software that scans the mount every min completely freezes as it is waiting for I/O.
A strace shows that the rate of stat() is just too low.
For example a tree on the SFTP rclone mount using --max-age without anything else accessing the mount takes:
2573 directories, 4984 files real 18m44.325s user 0m0.191s sys 0m0.396s
In comparison a tree on sshfs with the full files takes:
2579 directories, 42060 files real 15m15.555s user 0m0.619s sys 0m0.847s
not that better, as the problem is caused by FUSE?
Any flags that would help with the SFTP remote or with the dir-cache being populated?
Alternatively is there some other remote I could use instead that would support polling.
I wonder, as the first level of dirs is returned in 1-2 sec, would it help splitting those dirs in 2,3,n connections each listing their respective parts when the cache is to be rebuild.
rclone v1.64.0 - os/version: debian 10.13 (64 bit) - os/kernel: 4.19.0-24-amd64 (x86_64) - os/type: linux - os/arch: amd64 - go/version: go1.21.1 - go/linking: static - go/tags: none
rclone mount --allow-other --read-only --filter - /#*/ --dir-cache-time 2h --poll-interval 25s --buffer-size 16M --cache-dir /opt/rclone --vfs-cache-mode full --vfs-cache-max-age 1440h --vfs-cache-max-size 224G --vfs-cache-poll-interval 1m --vfs-read-chunk-size 128M --vfs-read-chunk-size-limit 256M --transfers 4 --umask 022 --max-age 2022-01-01 sftp_remote:/srv
--poll-interval has no effect on SFTP and some other options are the default values, I just have them there
for ease due to laziness (hope that is not part of the issue ).
Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.
[sftp_remote] type = sftp host = <host> user = <user> port = <port> key_file = ~/.ssh/id_ed25519 shell_type = unix md5sum_command = md5sum sha1sum_command = sha1sum