Error: Send request failed and retry flags seemingly ignored

Suspected Bug
1

What is the problem you are having with rclone?

At 3 out of 87 remote locations within the past week I began seeing errors when using the rclone copy command. I have been using the same command for years with rclone versions 1.47 and 1.50.2. I have contacted Wasabi technical support but they are not seeing issues on their end and cannot reproduce the problem.

Here is the error message (note: the actual bucket name has a hyphen in the name and the directory name is an integer between 0 - 202.)

2021/02/22 11:51:29 DEBUG : pacer: low level retry 1/10 (error RequestError: send request failed
caused by: Head "https://s3.us-west-1.wasabisys.com/bucket-name/integer": EOF)

I have also tried adding the --retries and --retries-sleep flags but continue receiving the same error:

rclone -vv copy /path/to/my/local/directory wasabi:bucket-name/integer --retries 20 --retries-sleep=5s

It also seems like either I'm using these flags incorrectly or rclone is just ignoring them. In versions 1.54 and 1.55.0-beta.5216.04a8859d2 it doesn't matter what I set these flags to, it always retries 10 times in rapid succession.

What is your rclone version (output from rclone version)

I have reproduced the problem on versions: 1.47, 1.50.2, 1.54 and 1.55.0-beta.5216.04a8859d2.

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Ubuntu versions 16.04.5 and 20.04.2 , both 64 bit.

Which cloud storage system are you using? (eg Google Drive)

Wasabi

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Note: In the following command, I replaced paths with placeholder values due to privacy concerns. The actual bucket name has a hyphen in the name and the directory name is an integer between 0 - 202.

rclone -vv copy /path/to/my/local/directory wasabi:bucket-name/integer

The rclone config contents with secrets removed.

[wasabi]
type = s3
provider = Wasabi
env_auth = false
access_key_id = xxxx
secret_access_key = xxxx
region = us-west-1
endpoint = s3.us-west-1.wasabisys.com
acl = private

A log from the command with the -vv flag

Note: In the following log output, I replaced paths with placeholder values due to privacy concerns. The actual bucket name has a hyphen in the name and the directory name is an integer between 0 - 202.

2021/02/22 11:51:28 DEBUG : rclone: Version "v1.55.0-beta.5216.04a8859d2" starting with parameters ["rclone" "-vv" "copy" "/path/to/my/local/directory" "wasabi:bucket-name/integer"]
2021/02/22 11:51:28 DEBUG : Creating backend with remote "/path/to/my/local/directory"
2021/02/22 11:51:28 DEBUG : Using config file from "/home/username/.config/rclone/rclone.conf"
2021/02/22 11:51:28 DEBUG : Creating backend with remote "wasabi:bucket-name/integer"
2021/02/22 11:51:29 DEBUG : pacer: low level retry 1/10 (error RequestError: send request failed
caused by: Head "https://s3.us-west-1.wasabisys.com/bucket-name/integer": EOF)
2021/02/22 11:51:29 DEBUG : pacer: Rate limited, increasing sleep to 10ms
2021/02/22 11:51:29 DEBUG : pacer: low level retry 2/10 (error RequestError: send request failed
caused by: Head "https://s3.us-west-1.wasabisys.com/bucket-name/integer": EOF)
2021/02/22 11:51:29 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2021/02/22 11:51:29 DEBUG : pacer: low level retry 3/10 (error RequestError: send request failed
caused by: Head "https://s3.us-west-1.wasabisys.com/bucket-name/integer": EOF)
2021/02/22 11:51:29 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2021/02/22 11:51:29 DEBUG : pacer: low level retry 4/10 (error RequestError: send request failed
caused by: Head "https://s3.us-west-1.wasabisys.com/bucket-name/integer": EOF)
2021/02/22 11:51:29 DEBUG : pacer: Rate limited, increasing sleep to 80ms
2021/02/22 11:51:29 DEBUG : pacer: low level retry 5/10 (error RequestError: send request failed
caused by: Head "https://s3.us-west-1.wasabisys.com/bucket-name/integer": EOF)
2021/02/22 11:51:29 DEBUG : pacer: Rate limited, increasing sleep to 160ms
2021/02/22 11:51:29 DEBUG : pacer: low level retry 6/10 (error RequestError: send request failed
caused by: Head "https://s3.us-west-1.wasabisys.com/bucket-name/integer": EOF)
2021/02/22 11:51:29 DEBUG : pacer: Rate limited, increasing sleep to 320ms
2021/02/22 11:51:29 DEBUG : pacer: low level retry 7/10 (error RequestError: send request failed
caused by: Head "https://s3.us-west-1.wasabisys.com/bucket-name/integer": EOF)
2021/02/22 11:51:29 DEBUG : pacer: Rate limited, increasing sleep to 640ms
2021/02/22 11:51:30 DEBUG : pacer: low level retry 8/10 (error RequestError: send request failed
caused by: Head "https://s3.us-west-1.wasabisys.com/bucket-name/integer": EOF)
2021/02/22 11:51:30 DEBUG : pacer: Rate limited, increasing sleep to 1.28s
2021/02/22 11:51:30 DEBUG : pacer: low level retry 9/10 (error RequestError: send request failed
caused by: Head "https://s3.us-west-1.wasabisys.com/bucket-name/integer": EOF)
2021/02/22 11:51:30 DEBUG : pacer: Rate limited, increasing sleep to 2s
2021/02/22 11:51:31 DEBUG : pacer: low level retry 10/10 (error RequestError: send request failed
caused by: Head "https://s3.us-west-1.wasabisys.com/bucket-name/integer": EOF)
2021/02/22 11:51:31 Failed to create file system for "wasabi:bucket-name/integer": RequestError: send request failed
caused by: Head "https://s3.us-west-1.wasabisys.com/bucket-name/integer": EOF
2

hello and welcome to the forum,
i am a long time user of wasabi, never saw that error.

that error is about caused by: Head, pretty sure retries makes no difference, as it just retries the same problem over and over again, with the same results.

rclone, by default, will retry low-level errors 10 times.
https://rclone.org/docs/#low-level-retries-number
which is different from
https://rclone.org/docs/#retries-int

to get more info, perhaps try to copy a single file with these flags.
rclone copy --dump=bodies --retries=1 --low-level-retries=1 /path/to/my/local/directory/filename.txt wasabi:bucket-name/integer

doing a quick test

rclone //?/C:/ex/dummy/01.txt wasabi01:bucketwithdashin-name/202 -vv --retries=1 --low-level-retries=1 
rclone: Version "v1.53.3" starting with parameters ["c:\\data\\rclone\\scripts\\rclone.exe" "copy" "//?/C:/ex/dummy/01.txt" "wasabi01:bucketwithdashin-name/202" "-vv" "--retries=1" "--low-level-retries=1"]
Creating backend with remote "//?/C:/ex/dummy/01.txt"
Using config file from "c:\\data\\rclone\\scripts\\rclone.conf"
fs cache: adding new entry for parent of "//?/C:/ex/dummy/01.txt", "//?/C:/ex/dummy"
Creating backend with remote "wasabi01:bucketwithdashin-name/202"
01.txt: Need to transfer - File not found at Destination
S3 bucket bucketwithdashin-name path 202: Bucket "bucketwithdashin-name" created with ACL "private"
01.txt: MD5 = c4ca4238a0b923820dcc509a6f75849b OK
01.txt: Copied (new)
3

Thank you for the quick reply. After further investigation with Wasabi tech support I determined the endpoint domain s3.us-west-1.wasabisys.com was resolving to a different IP than my other locations. After manually overriding the name resolution to a confirmed Wasabi IP address it worked fine.

I understand now the difference between those retry flags and I'll remember the --dump flag for next time I need to troubleshoot.

1 Like
4

good, that makes sense, as there was nothing wrong with your rclone command and config file

5

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.