In Swift, When I have large numbers of files in a bucket, ls operations are taking ages to complete. I’m wondering if this is a known issue, and if so, is there a way to mitigate it?
swift list returns very quickly (3-4s), but rclone swift: ls will easily run for many minutes before returning. The bucket size I have in front of me is about 11k objects.
Of course, this also affects rclone copy operations with --include filters, which is what I’m really after.
indeed, it is a directory of many large files (compressed log files). They are not all huge, but there are likely thousands of them that are.
Is there a way to turn off size checking until after the --include --exclude computation would be done? At least in my case, that would solve my problem since I’m only copying a couple named files.
Is there a way to turn off size checking entirely?
swift list -l does, and doesn’t appear to impact the run time of the command. But, I see many of what I would expect are the large files are zero-byte, probably signaling that they need another lookup and the swift client doesn’t do that for you.
I just tried and as you suspected, it doesn’t appear to change the speeds. Ah well.