Rclone delete on S3 very slow

The problem here is rclone delete is using recursive listing by default and this essentially means that it does each HEAD request single threaded. You can disable this with --disable ListR.

According to my experiments with a normal directory structure (generated with rclone test makefiles --files 10000 --max-file-size 10b 10000files) the speed of listing with an age filter seems to max out at about --checkers 16 with --disable ListR.

What you are doing

$ time rclone delete --min-age 100d TestS3MinioManual:test --checkers 32

real	0m10.957s
user	0m4.254s
sys	0m0.760s

Now with --disable ListR

$ time rclone delete --min-age 100d TestS3MinioManual:test --checkers 32 --disable ListR

real	0m1.588s
user	0m5.489s
sys	0m0.732s

So I think you should be able to get a significant speedup by adding --disable ListR to your rclone delete.

You can also try this with just plain rclone delete (no --disable ListR)

v1.63.0-beta.6959.b87a9c835.fix-listr-performance on branch fix-listr-performance (uploaded in 15-30 mins)

This does the HEAD requests in parallel so is a lot faster. However it still only fetches one page of things from AWS at once so will top out at 1024 HEADs in parallell.