Slow listing of bucket with large number of files - swift

[ncw (Nick Craig-Wood)]

I just remembered we have discussed this before in the forum: Ls slow on large directories (500~ files) (Hubic)

Number of checkers makes no difference as it is one per directory

I made this issue about it: swift: speed up directory listings with lots of big files · Issue #1791 · rclone/rclone · GitHub

@dpb

Is there a way to turn off size checking until after the --include --exclude computation would be done? At least in my case, that would solve my problem since I’m only copying a couple named files.

rclone could defer reading the size until it is actually used. That would work too I expect.

@dpb

Is there a way to turn off size checking entirely?

You can try the --ignore-size flag - I'm not sure that will help though.

@dpb

swift list: 8s
ls with checkers: 8 (default), --fast-list: 9m
ls with checkers: 256, --fast-list: 8m
ls with checkers: 1024, --fast-list: 7m45s

I don't think swift list reads the size does it?