Rclone 25x slower than minio-client listing files

Help and Support
1

What is the problem you are having with rclone?

using rclone lsl --max-depth 1 on a minio s3 bucket is 25 times slower than using mc ls native client.

Run the command 'rclone version' and share the full output of the command.

rclone 1.65.2

  • os/version: nixos 24.05 (Uakari) (64 bit)
  • os/kernel: 6.1.77 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.21.6
  • go/linking: dynamic
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Self host single node/disk Minio on btrfs on lvm on luks on xen on zvol on raidz2 on hdd

quay.io/minio/minio:RELEASE.2024-02-17T01-15-57Z b05958e56139

The command you were trying to run (eg rclone copy /tmp remote:tmp)

time rclone -vvv lsl --max-depth 1 --s3-list-version 2 minio:mybucket/media/pictures/2020

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[minio]
type = s3
provider = Minio
access_key_id = XXX
secret_access_key = XXX
endpoint = http://minio.mytailnet.ts.net:9000

A log from the command that you were trying to run with the -vv flag

2024/02/23 17:19:26 DEBUG : Setting --password-command "/nix/store/akwjk2qrffd3d2jg8bhds9ml4n01pncv-rbw-1.9.0/bin/rbw get --field config rclone" from environment variable RCLONE_PASSWORD_COMMAND="/nix/store/akwjk2qrffd3d2jg8bhds9ml4n01pncv-rbw-1.9.0/bin/rbw get --field config rclone"
2024/02/23 17:19:26 DEBUG : rclone: Version "1.65.2" starting with parameters ["/home/user/.nix-profile/bin/rclone" "-vvv" "lsl" "--max-depth" "1" "--s3-list-version" "2" "minio:mybucket/media/pictures/2020"]
2024/02/23 17:19:26 DEBUG : Creating backend with remote "minio:mybucket/media/pictures/2020"
2024/02/23 17:19:27 DEBUG : Using config file from "/home/user/.config/rclone/rclone.conf"
2024/02/23 17:19:27 DEBUG : minio: detected overridden config - adding "{-7vZW}" suffix to name
2024/02/23 17:19:27 DEBUG : Resolving service "s3" region "us-east-1"
2024/02/23 17:19:27 DEBUG : fs cache: renaming cache item "minio:mybucket/media/pictures/2020" to be canonical "minio{-7vZW}:user/media/pictures/2020"
  1632549 2020-07-01 02:50:53.000000000 mypicture1.jpg
...
rclone -vvv lsl --max-depth 1 --s3-list-version 2   0.21s user 0.16s system 0% cpu 1:07.25 total

the output of the lsl command is file by file. I tried --fast-list and it is the same speed and output.

minio client

mc -v
mc version RELEASE.2024-02-16T11-05-48Z (commit-id=RELEASE.2024-02-16T11-05-48Z)
Runtime: go1.21.6 linux/amd64
time mc ls minio/bucket/media/pictures/2020
[2024-01-26 02:53:07 +07] 1.6MiB STANDARD mypicture.jpg
...
mc ls minio/bucket/media/pictures/2020  0.11s user 0.16s system 9% cpu 2.663 total

the output of mc ls is one stream of output.

187 files in the folder

rclone took 67 seconds
mc took 2.7 seconds

more data:

rclone -vvv lsf --max-depth 1 minio/bucket/media/pictures/2020   0.11s user 0.14s system 9% cpu 2.795 total
rclone -vvv ls --max-depth 1 minio/bucket/media/pictures/2020   0.11s user 0.15s system 11% cpu 2.332 total

these commands cause rclone to output the files in one output stream.

rclone does not like to return the mod time.

EDIT:

Answering my own question...
using lsf --use-server-modtime resolves -- now is there a way to configure a remote with this in the config file?

2

that flag is a global flag, cannot go into the config file.

can use environment variable, something like
export RCLONE_USE_SERVER_MODTIME=true

3

appreciate that. the main reason i found this is because I use the android round-sync app that uses rclone under the covers and defaults to using the lsf option.

It's only configuration is of importing rclone config files and the options within it.

what should 3rd party apps like this do in this case? do you suggest implementing environment variables? I'm not if that would work on Android.

4

sure, that can work well.
on android, i use rclone inside termux. i connect to that using ssh over tailscale.

i have a howto and wiki about termux
https://forum.rclone.org/t/rclone-on-termux-turn-your-android-phone-into-a-media-server/16064

5

I have that installed as well but specifically I am concerned about this Android native app round-sync that allows browsing remotes and using other Android apps to use the storage. It will also serve WebDAV, dlna, http, ftp.

But the rclone default for using modification time on s3 makes it untenable.

6

this is a forum about rclone.

7

I appreciate that. Why is the default on rclone s3 minio remotes to use modification time which makes lsl slow when the minio client uses the server mod time by default and is fast? :thinking:

8

modtime support across many different providers is a core feature of rclone.
rclone makes is very easy to disable when needed.

you did the correct thing, to post at github
Default file listing mode with s3 is very slow

9

After posting that I realized the root problem is with rclone default configuration.

10

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.