VFS Mount to SMB gets rate-limited due to initial stat checks

[sudiptob2 (Sudipto Baral)]

When using the combine backend with SMB in rclone, the initial VFS mount performs a stat check to determine whether each mount point is a directory or a file.

If there are hundreds or thousands of mount points in a combined backend setup, rclone performs a stat check for each mount point during VFS initialization. Each check triggers a NewFs() call, which opens a separate connection to the server. This creates a burst of connections, causing the mount process to get rate-limited

Relevant code section:

	share, dir := f.split("")
	if share == "" || dir == "" {
		return f, nil
	}
	cn, err := f.getConnection(ctx, share)
	if err != nil {
		return nil, err
	}
	stat, err := cn.smbShare.Stat(f.toSambaPath(dir))
	f.putConnection(&cn, err)
	if err != nil {
		// ignore stat error here
		return f, nil
	}
	if !stat.IsDir() {
		f.root, err = path.Dir(root), fs.ErrorIsFile
	}
	fs.Debugf(f, "Using root directory %q", f.root)
	return f, err

Steps to Reproduce

1. Configure SMB and combine backends in rclone config: Assume we have 1000s of mount points.

[smb.server.example]
type = smb
host = <ip-address>
user = <username>
pass = <password>

[smb.server.example-combine]
type = combine
upstreams = share-1=smb.server.example:volumeA/data/share-1 \
            share-2=smb.server.example:volumeA/data/share-2 \
            share-3=smb.server.example:volumeA/data/share-3 \
            share-4=smb.server.example:volumeA/data/share-4 \
            share-5=smb.server.example:volumeA/data/share-5

2. Mount the combined backend:

rclone mount smb.server.example-combine: /mnt/ \
  --vfs-cache-mode writes \
  --links \
  --log-level DEBUG

Potential Solution:

One potential solution I can think of is to bring the directory check under some advanced flag. If the user is certain that all their mount points are directories, they could opt out of the check and avoid hitting rate limits.

Please let me know if there is a better way to solve this problem. Happy to contribute.

[ncw (Nick Craig-Wood)]

The way we've made this efficiency improvement in other backends (eg S3) is if the remote path is "" or ends with a / it must be a directory.

This would avoid the initial check.

Fancy giving that a go?

[sudiptob2 (Sudipto Baral)]

Thank you so much for sharing this neat approach! This method resolves the initial stat check issues on Linux. However, on Windows, the About() method is also called during initialization, to fetch usage stats, which still triggers a connection burst at startup.

I experimented with disabling the About() method, and it seems to work for our use case. Do you have any suggestions on a better way to avoid this About() call at startup? Happy to contribute.

[ncw (Nick Craig-Wood)]

You can use --disable About on the command line. That should work.