How to specify the maximum number of concurrent HTTP requests

Help and Support
1

Hello
There are some webservers that only allow 4 concurrent HTTP requests/transfers
However, specifying --transfers=4 doesn't do the trick since it fails with failed to readDir: HTTP Error 429: 429 Too Many Requests
What's the way to specify the maximum number of concurrent requests to the webserver to avoid 429?

2

You'll need to set --transfers and --checkers - together they make the total number of connections needed.

1 Like
3

with --checkers=4 and --transfers=4 I still get Failed to copy: failed to open source object: Open failed: HTTP Error 429: 429 Too Many Requests

4

The total number of connection is --transfers plus --checkers. So try setting them both to 2.

5

both to 2 still fails with a 429.

6

You could try setting them both to 1. You might want to try setting --tpslimit also to slow things down.

I note that the http backend doesn't support low level retries which it really should - that would mean that it retried on the 429 error.

7

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.