Rclone mount slow listing on S3 compatible comparing to s3fs

[brlav35 (Bruno Lavoie)]

What is the problem you are having with rclone?

Slow listings of folders when using rclone mount, I have a folder with about 14k files with a total size of 27 GBytes.

What is your rclone version (output from rclone version)

rclone v1.52.3

• os/arch: linux/amd64
• go version: go1.14.7

Which OS you are using and how many bits (eg Windows 7, 64 bit)

CentOS Linux release 7.8.2003 (Core)

Which cloud storage system are you using? (eg Google Drive)

S3 Compatible - Backed by Ceph

The command you were trying to run (eg rclone copy /tmp remote:tmp)

With a freshly mounted bucket, it takes around 25-30 secs the first time and other runs are way faster:

$ time tree mnt-rclone-45446/DATA/ulaval/Rasters/bdtq_20k/tif/ >/dev/null
real    0m27.044s
user    0m0.063s
sys     0m0.276s

$ time tree mnt-rclone-45446/DATA/ulaval/Rasters/bdtq_20k/tif/ >/dev/null
real    0m0.808s                                                                                                                                                                                                                              
user    0m0.054s                                                                                                                                                                                                                              
sys     0m0.197s     

When we do exactly the same with s3fs-fuse, it's constantly returns in around 4 secs:

$ time tree mnt-s3fs-45446/DATA/ulaval/Rasters/bdtq_20k/tif/ >/dev/null
real    0m3.847s
user    0m0.056s
sys     0m0.145s

$ time tree mnt-s3fs-45446/DATA/ulaval/Rasters/bdtq_20k/tif/ >/dev/null
real    0m3.862s
user    0m0.046s
sys     0m0.154s

The rclone config contents with secrets removed.

Default configs in current session:

export AWS_ACCESS_KEY_ID=<REDACTED>
export AWS_SECRET_ACCESS_KEY=<REDACTED>

export RCLONE_S3_PROVIDER="Ceph"
export RCLONE_S3_ENDPOINT="https://s3.mydomain.com"
export RCLONE_S3_ENV_AUTH=true

A log from the command with the -vv flag

No logs yet.

[Animosity022]

You didn't share your mount command nor a debug log.

If you run with debug, you can see what rclone is doing. The first time, it's building the directory list, which is why the second is faster. The s3 client probably does that a bit different as it's specific for S3.