I need a set of flags that captures changes quickly and yields low latency for an rclone mount

[data-hobbyist]

What is the problem you are having with rclone?

Commands like ls /mnt/path/to/mount/dir take forever (like 8 seconds) to return results on a directory with less than 10 files.

I need to be able to see file changes quickly when they happen so caching is not very useful. I also need to be able to add, delete and list files quickly. Any help with this?

You will see the --vfs-cache-mode off flag below because caching would just mean I see stale data. I want to quickly pick up on file changes.

Run the command 'rclone version' and share the full output of the command.

rclone v1.67.0
- os/version: fedora 40 (64 bit) 
- os/kernel: 6.8.9-300.fc40.x86_64 (x86_64)
- os/type: linux 
- os/arch: amd64
- go/version: go1.22.4
- go/linking: static
- go/tags: none      

Which cloud storage system are you using? (eg Google Drive)

SFTP server at rsync.net.

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount rsyncnet:/path/to/source /mnt/path/to/mount/dir --vfs-cache-mode off --buffer-size 32M --vfs-read-chunk-size 32M --vfs-read-chunk-size-limit off --vfs-cache-max-age 1m --no-modtime --dir-cache-time 2s --poll-interval 2s

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[rsyncnet]
type = sftp
host = XXX
user = XXX
pass = XXX
shell_type = unix
md5sum_command = md5 -r
sha1sum_command = sha1 -r  

[asdffdsa (jojothehumanmonkey)]

welcome to the forum,

--poll-interval is not supported for sftp remotes.

@data-hobbyist

Commands like ls /path/to/file take forever (like 8 seconds) to return results on a directory with less than 10 files

that is because of --dir-cache-time 2s
after two-seconds, the vfs dir cache is considered stale.
so each time you visit that directory, rclone has to re-scan. and sftp is very slow at that.

@data-hobbyist

You will see the --vfs-cache-mode off flag below because caching would just mean I see stale data

that is not correct, it is a bit confusing.
that flag is for the vfs file cache, which has nothing to do with the vfs dir cache.
your issue is with the vfs dir cache.

check out my summary of the two vfs caches

[data-hobbyist]

welcome to the forum

Thanks! Glad to be here.

--poll-interval is not supported for sftp remotes.

Ah, OK my bad. I didn't see that in the mount logs. It seems I need to get used to using the --verbose flag first.

that is because of --dir-cache-time 2s
after two-seconds, the vfs dir cache is considered stale.
so each time you visit that directory, rclone has to re-scan. and sftp is very slow at that.

Maybe I need to change to object-based storage then?

that is not correct, it is a bit confusing.
that flag is for the vfs file cache, which has nothing to do with the vfs dir cache.
your issue is with the vfs dir cache.

Good to know. That helps a lot, thanks!

[asdffdsa (jojothehumanmonkey)]

@data-hobbyist

Maybe I need to change to object-based storage then?

well, most object based providers do not support polling, so will have the same underlying issue.

Which remotes support polling?

Looking at backends source code it is implemented for: OneDrive Google Drive Dropbox Box I also used grep:) In backend directory: grep -R "ChangeNotify" *

when using mount, flags such as --fast-list do not work.
most object storage providers charge for api calls to list files.

there are horror stories in the forum about that.
but if you get to that point, there are way to reduce or eliminate the cost.