Poor list performance with s3 mount

Help and Support
1

I'm trying to diagnose a performance quandry with rclone mount when running on AWS with the S3 storage provider. My bucket and rclone are in the same region, connecting via an S3 endpoint in the VPC. If AWS is to be believed, I have a 40Gbps path between the VM and S3, so I think I've safely ruled out network slowness.

I'm using rclone as a CSI provider using this wrapper https://github.com/wunderio/csi-rclone which sets up the options and hands off to rclone mount. Read and write performance is fine, well within my expectations, but I'm struggling to achieve good performance listing files. I'm looking for some clarification on the following points:

  • Does turning off modtimes not update them, or does it also disable fetching of modification times?
  • Would caching help the directory listing performance? (my hunch is no, but the internals of the Linux VFS are a bit beyond me still)
  • What is the normal performance of listing files I should expect from rclone as compared to the AWS CLI? These operations are nearly instant with the AWS CLI, but can take multiple minutes when going via the VFS mount.

Thanks in advance.

2

hello and welcome to the forum

when you posted, there was template of question, so we can better help you.

you can prime the mount, using vfs/refresh, to have rclone scan and cache the remote folder/file structure on the local computer.
for example, when i need jellyfin to scan for new media on the rclone mount.

  1. get rclone to prime the mount
  2. get jellyfin to scan the mount
3

This is the reason for slow listing likely. With modtimes rclone HEADs each object to read the modtime.

So use --no-modtime or --s3-use-server-modtime

The first doesn't fetch the modtimes, the second causes the S3 backend to use the server last modified instead of rclone's metadata.

It will help with repeated listings but otherwise no.

If you turn off modtimes it should be as quick as the aws tools.

4

Thanks for your quick replies I can add the specifics of this particular setup if you'd like, but I'm really more interested in a theory answer to this question, as I want to understand more why this in particular is slow. I think that ncw's answer largely answers that.

In this particular case, the following information should answer most of your questions, though I am still interested in understanding theory:

Which OS you are using and how many bits (eg Windows 7, 64 bit)

64-bit Linux, from rclone's perspective Alpine 3.9

Which cloud storage system are you using? (eg Google Drive)

AWS S3 with a network adjacent S3 endpoint

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount as specified by the following volume specification for CSI:

id = "sftp"
name = "sftp"
type = "csi"
external_id = "<redacted>"
plugin_id = "rclone0"
access_mode = "multi-node-multi-writer"
attachment_mode = "file-system"

context {
  remote = "s3"
  remotePath = "<redacted>"
  s3-provider = "aws"
  s3-env-auth = true
  s3-region = "us-west-2"

  # Very important, if this isn't set, all writes fail due to md5
  # mismatch since AWS moves the md5sum of the original.
  s3-server-side-encryption = "aws:kms"

  # TODO(maldridge) something is still broken with the above.
  # Fix later and trust TCP check-summing until then.
  ignore-checksum = true
}

The rclone config contents with secrets removed.

rclone is not configured with a config file, rather the context in the above file is converted to flags, and the source and dest paths are inferred.

A log from the command with the -vv flag

This low level log is inaccessible without recompiling the provider binary. I can do this if we really think the log will have valuable information relative to this specific issue, but I think the problem is data-driven as opposed to any kind of bug or deficiency in rclone itself.

5

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.