Does rclone support retries for check when using --download flag?

Help and Support
1

What is the problem you are having with rclone?

I am using rclone check with --download flag but sometimes I get read: connection reset by peer while it downloads a large object(~2gb). And due to this check command fails with 1 hashes could not be checked.

Run the command 'rclone version' and share the full output of the command.

rclone v1.48.0 
 - os/arch: linux/amd64 
 - go version: go1.12.6

The command you were trying to run (eg rclone copy /tmp remote:tmp)

["./rclone" "-vv" "--retries" "5" "check" "--download" "src_dir" "destination:dest_name" "--multi-thread-streams" "64" "--transfers=64"]

The rclone config contents with secrets removed.

Default config used

A log from the command with the -vv flag

2023/05/29 04:06:19 DEBUG : rclone: Version "v1.48.0" starting with parameters ["./rclone" "-vv" "--retries" "5" "check" "--download" "src_dir" "destination:dest_name" "--multi-thread-streams" "64" "--transfers=64"] 
2023-05-29T04:06:19.750975267Z
 
2023/05/29 04:06:19 NOTICE: Config file "/root/.config/rclone/rclone.conf" not found - using defaults 
2023-05-29T04:06:19.751319806Z
 
2023/05/29 04:06:19 INFO  : S3 bucket dest_name: Waiting for checks to finish 
2023-05-29T04:06:39.883048787Z
 
2023/05/29 04:06:39 ERROR : large: Failed to download: read tcp ip1->ip2: read: connection reset by peer 
2023-05-29T04:06:40.86839994Z
 
 .... Multiple Lines says Object name : OK
2023/05/29 04:06:40 DEBUG : medium: OK 
2023-05-29T04:06:40.892699411Z
 .....
 
2023/05/29 04:06:41 NOTICE: S3 bucket dest_name: 1 differences found 
2023-05-29T04:06:41.22404738Z
 
2023/05/29 04:06:41 NOTICE: S3 bucket dest_name: 1 hashes could not be checked 
2023-05-29T04:06:41.224054905Z
 
2023/05/29 04:06:41 NOTICE: S3 bucket dest_name: 49 matching files 
2023-05-29T04:06:41.224272444Z
 
2023/05/29 04:06:41 Failed to check with 2 errors: last error was: 1 differences found
2

That's a really old version. You'd want to upgrade that.

What storage provider are you using? There isn't a 'default config as that's why we ask for it.

Those are generally network errors but without knowing the providers, it's tough to guess.

Recreate the issue with -vv and share the full debug log.

3

Hey, thank you for the reply.

I understand that this can happen due to network errors and intermittently. I just wanted to check if rclone supports retries in the check command(with --download flag) specifically if there is any error while downloading a single object.

4

Yes, that's why we ask for the debug log as you can see it in that log.

5

I have pasted the logs with -vv flag in the post itself, please take a look.

6

I did take a look as I mentioned, that's not a full debug log so I can't look at what isn't there.

Please update and re-test and repost a log if you'd like to see it.

7

I have multiple logs for this workflow. This is the only logs which I can see :
Only thing which I have extra is this but it contains multiple -> object name : OK lines

 .... Multiple Lines says Object name : OK
2023/05/29 04:06:40 DEBUG : medium: OK 
2023-05-29T04:06:40.892699411Z
 .....

Sorry, if I am missing something. I am running the command with -vv.

8

I wouldn't over complicate it.

Run it one time with -vv and share the full log.

Profit.

9

I just took a quick look at the source code and I think we do not retry failed downloads.

This would be very easy to fix I think though.

I've had a go here if you want to try it

v1.63.0-beta.7055.3b1893071.fix-open on branch fix-open (uploaded in 15-30 mins)

10

I've merged this fix to master now which means it will be in the latest beta in 15-30 minutes and released in v1.63