Rclone continue to try the same file after many low level retries

What is the problem you are having with rclone?

rclone keeps trying same file again after 500 error and 10 low level retries.

I would like it to give up on the file after the 10 retries.

Run the command 'rclone version' and share the full output of the command.

rclone v1.65.0

  • os/version: Microsoft Windows 11 Pro 21H2 (64 bit)
  • os/kernel: 10.0.22000.2538 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.21.4
  • go/linking: static
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Google Photos

The command you were trying to run (eg rclone copy /tmp remote:tmp)

 rclone copy google-photos:media/by-month/2023 s3:google-photos-micamedia/2023 -P -vvv

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[google-photos]
type = google photos
read_only = true
token = XXX

[s3]
type = s3
provider = AWS
access_key_id = XXX
secret_access_key = XXX
region = us-east-1
acl = private
server_side_encryption = aws:kms
sse_kms_key_id = XXX
storage_class = INTELLIGENT_TIERING
bucket_acl = private

[tricia-photos]
type = google photos
read_only = true
token = XXX

A log from the command that you were trying to run with the -vv flag

2023/12/24 13:45:11 DEBUG : 2023-09/original_3db28e23-1b1d-41b2-bd84-d2a3826c578a_PXL_20230907_012011410.jpg: Size:
2023/12/24 13:45:11 DEBUG : 2023-09/original_3db28e23-1b1d-41b2-bd84-d2a3826c578a_PXL_20230907_012011410.jpg: >Size:
2023/12/24 13:45:11 DEBUG : 2023-09/original_3db28e23-1b1d-41b2-bd84-d2a3826c578a_PXL_20230907_012011410.jpg: ModTime:
2023/12/24 13:45:11 DEBUG : 2023-09/original_3db28e23-1b1d-41b2-bd84-d2a3826c578a_PXL_20230907_012011410.jpg: >ModTime:
2023/12/24 13:45:11 DEBUG : 2023-09/original_3db28e23-1b1d-41b2-bd84-d2a3826c578a_PXL_20230907_012011410.jpg: Sizes identical
2023/12/24 13:45:11 DEBUG : 2023-09/original_3db28e23-1b1d-41b2-bd84-d2a3826c578a_PXL_20230907_012011410.jpg: Unchanged skipping
2023/12/24 13:45:11 DEBUG : S3 bucket google-photos-micamedia path 2023: Waiting for transfers to finish
2023/12/24 13:45:11 DEBUG : pacer: low level retry 6/10 (error  (500 500 Internal Server Error))
2023/12/24 13:45:11 DEBUG : pacer: Rate limited, increasing sleep to 2.687658304s
2023/12/24 13:45:13 DEBUG : pacer: low level retry 7/10 (error  (500 500 Internal Server Error))
2023/12/24 13:45:13 DEBUG : pacer: Rate limited, increasing sleep to 4.524450627s
2023/12/24 13:45:16 DEBUG : pacer: low level retry 8/10 (error  (500 500 Internal Server Error))
2023/12/24 13:45:16 DEBUG : pacer: Rate limited, increasing sleep to 8.729856577s
2023/12/24 13:45:20 DEBUG : pacer: low level retry 9/10 (error  (500 500 Internal Server Error))
2023/12/24 13:45:20 DEBUG : pacer: Rate limited, increasing sleep to 16.163298549s
2023/12/24 13:45:29 DEBUG : pacer: low level retry 10/10 (error  (500 500 Internal Server Error))
2023/12/24 13:45:29 DEBUG : pacer: Rate limited, increasing sleep to 16.495918655s
2023/12/24 13:45:29 DEBUG : 2023-06/PXL_20230603_192038000.RESTORED.mp4: >Open:
2023/12/24 13:45:29 DEBUG : 2023-06/PXL_20230603_192038000.RESTORED.mp4: Received error: failed to open source object:  (500 500 Internal Server Error) - low level retry 0/10
2023/12/24 13:45:29 DEBUG : 2023-06/PXL_20230603_192038000.RESTORED.mp4: Size:
2023/12/24 13:45:29 DEBUG : 2023-06/PXL_20230603_192038000.RESTORED.mp4: >Size:
2023/12/24 13:45:29 DEBUG : 2023-06/PXL_20230603_192038000.RESTORED.mp4: Size:
2023/12/24 13:45:29 DEBUG : 2023-06/PXL_20230603_192038000.RESTORED.mp4: >Size:
2023/12/24 13:45:29 DEBUG : 2023-06/PXL_20230603_192038000.RESTORED.mp4: Open:
2023/12/24 13:45:45 DEBUG : pacer: low level retry 1/10 (error  (500 500 Internal Server Error))
2023/12/24 13:45:45 DEBUG : pacer: Rate limited, increasing sleep to 16.097558856s
2023/12/24 13:46:01 DEBUG : pacer: low level retry 2/10 (error  (500 500 Internal Server Error))
2023/12/24 13:46:01 DEBUG : pacer: Rate limited, increasing sleep to 16.088464913s
2023/12/24 13:46:18 DEBUG : pacer: low level retry 3/10 (error  (500 500 Internal Server Error))
2023/12/24 13:46:18 DEBUG : pacer: Rate limited, increasing sleep to 16.825027149s
2023/12/24 13:46:34 DEBUG : pacer: low level retry 4/10 (error  (500 500 Internal Server Error))
2023/12/24 13:46:34 DEBUG : pacer: Rate limited, increasing sleep to 16.364805731s
2023/12/24 13:46:50 DEBUG : pacer: low level retry 5/10 (error  (500 500 Internal Server Error))
2023/12/24 13:46:50 DEBUG : pacer: Rate limited, increasing sleep to 16.67233534s
2023/12/24 13:47:07 DEBUG : pacer: low level retry 6/10 (error  (500 500 Internal Server Error))
2023/12/24 13:47:07 DEBUG : pacer: Rate limited, increasing sleep to 16.188288039s
2023/12/24 13:47:24 DEBUG : pacer: low level retry 7/10 (error  (500 500 Internal Server Error))
2023/12/24 13:47:24 DEBUG : pacer: Rate limited, increasing sleep to 16.377373298s
2023/12/24 13:47:40 DEBUG : pacer: low level retry 8/10 (error  (500 500 Internal Server Error))
2023/12/24 13:47:40 DEBUG : pacer: Rate limited, increasing sleep to 16.39376154s
2023/12/24 13:47:56 DEBUG : pacer: low level retry 9/10 (error  (500 500 Internal Server Error))
2023/12/24 13:47:56 DEBUG : pacer: Rate limited, increasing sleep to 16.646199584s
2023/12/24 13:48:13 DEBUG : pacer: low level retry 10/10 (error  (500 500 Internal Server Error))
2023/12/24 13:48:13 DEBUG : pacer: Rate limited, increasing sleep to 16.294396582s
2023/12/24 13:48:13 DEBUG : 2023-06/PXL_20230603_192038000.RESTORED.mp4: >Open:
2023/12/24 13:48:13 DEBUG : 2023-06/PXL_20230603_192038000.RESTORED.mp4: Received error: failed to open source object:  (500 500 Internal Server Error) - low level retry 1/10
2023/12/24 13:48:13 DEBUG : 2023-06/PXL_20230603_192038000.RESTORED.mp4: Size:
2023/12/24 13:48:13 DEBUG : 2023-06/PXL_20230603_192038000.RESTORED.mp4: >Size:
2023/12/24 13:48:13 DEBUG : 2023-06/PXL_20230603_192038000.RESTORED.mp4: Size:
2023/12/24 13:48:13 DEBUG : 2023-06/PXL_20230603_192038000.RESTORED.mp4: >Size:
2023/12/24 13:48:13 DEBUG : 2023-06/PXL_20230603_192038000.RESTORED.mp4: Open:
2023/12/24 13:48:29 DEBUG : pacer: low level retry 1/10 (error  (500 500 Internal Server Error))
2023/12/24 13:48:29 DEBUG : pacer: Rate limited, increasing sleep to 16.338753906s
2023/12/24 13:48:46 DEBUG : pacer: low level retry 2/10 (error  (500 500 Internal Server Error))
2023/12/24 13:48:46 DEBUG : pacer: Rate limited, increasing sleep to 16.329350968s
2023/12/24 13:49:02 DEBUG : pacer: low level retry 3/10 (error  (500 500 Internal Server Error))
2023/12/24 13:49:02 DEBUG : pacer: Rate limited, increasing sleep to 16.631734679s
2023/12/24 13:49:18 DEBUG : pacer: low level retry 4/10 (error  (500 500 Internal Server Error))
2023/12/24 13:49:18 DEBUG : pacer: Rate limited, increasing sleep to 16.871401123s
2023/12/24 13:49:35 DEBUG : pacer: low level retry 5/10 (error  (500 500 Internal Server Error))
2023/12/24 13:49:35 DEBUG : pacer: Rate limited, increasing sleep to 16.644238991s
2023/12/24 13:49:52 DEBUG : pacer: low level retry 6/10 (error  (500 500 Internal Server Error))
2023/12/24 13:49:52 DEBUG : pacer: Rate limited, increasing sleep to 16.028962956s
2023/12/24 13:50:09 DEBUG : pacer: low level retry 7/10 (error  (500 500 Internal Server Error))
2023/12/24 13:50:09 DEBUG : pacer: Rate limited, increasing sleep to 16.614342836s
2023/12/24 13:50:24 DEBUG : pacer: low level retry 8/10 (error  (500 500 Internal Server Error))
2023/12/24 13:50:24 DEBUG : pacer: Rate limited, increasing sleep to 16.457464728s
2023/12/24 13:50:41 DEBUG : pacer: low level retry 9/10 (error  (500 500 Internal Server Error))
2023/12/24 13:50:41 DEBUG : pacer: Rate limited, increasing sleep to 16.694050353s
2023/12/24 13:50:57 DEBUG : pacer: low level retry 10/10 (error  (500 500 Internal Server Error))
2023/12/24 13:50:57 DEBUG : pacer: Rate limited, increasing sleep to 16.840658169s
2023/12/24 13:50:57 DEBUG : 2023-06/PXL_20230603_192038000.RESTORED.mp4: >Open:
2023/12/24 13:50:57 DEBUG : 2023-06/PXL_20230603_192038000.RESTORED.mp4: Received error: failed to open source object:  (500 500 Internal Server Error) - low level retry 2/10
2023/12/24 13:50:57 DEBUG : 2023-06/PXL_20230603_192038000.RESTORED.mp4: Size:
2023/12/24 13:50:57 DEBUG : 2023-06/PXL_20230603_192038000.RESTORED.mp4: >Size:
2023/12/24 13:50:57 DEBUG : 2023-06/PXL_20230603_192038000.RESTORED.mp4: Size:
2023/12/24 13:50:57 DEBUG : 2023-06/PXL_20230603_192038000.RESTORED.mp4: >Size:
2023/12/24 13:50:57 DEBUG : 2023-06/PXL_20230603_192038000.RESTORED.mp4: Open:
2023/12/24 13:51:14 DEBUG : pacer: low level retry 1/10 (error  (500 500 Internal Server Error))
2023/12/24 13:51:14 DEBUG : pacer: Rate limited, increasing sleep to 16.652695092s
2023/12/24 13:51:31 DEBUG : pacer: low level retry 2/10 (error  (500 500 Internal Server Error))
2023/12/24 13:51:31 DEBUG : pacer: Rate limited, increasing sleep to 16.005812509s
2023/12/24 13:51:48 DEBUG : pacer: low level retry 3/10 (error  (500 500 Internal Server Error))
2023/12/24 13:51:48 DEBUG : pacer: Rate limited, increasing sleep to 16.743634069s
2023/12/24 13:52:04 DEBUG : pacer: low level retry 4/10 (error  (500 500 Internal Server Error))
2023/12/24 13:52:04 DEBUG : pacer: Rate limited, increasing sleep to 16.230532045s
2023/12/24 13:52:20 DEBUG : pacer: low level retry 5/10 (error  (500 500 Internal Server Error))
2023/12/24 13:52:20 DEBUG : pacer: Rate limited, increasing sleep to 16.262239609s
2023/12/24 13:52:37 DEBUG : pacer: low level retry 6/10 (error  (500 500 Internal Server Error))
2023/12/24 13:52:37 DEBUG : pacer: Rate limited, increasing sleep to 16.118456608s
2023/12/24 13:52:53 DEBUG : pacer: low level retry 7/10 (error  (500 500 Internal Server Error))
2023/12/24 13:52:53 DEBUG : pacer: Rate limited, increasing sleep to 16.832421589s
2023/12/24 13:53:09 DEBUG : pacer: low level retry 8/10 (error  (500 500 Internal Server Error))
2023/12/24 13:53:09 DEBUG : pacer: Rate limited, increasing sleep to 16.630781704s
2023/12/24 13:53:11 INFO  : Signal received: interrupt
2023/12/24 13:53:11 INFO  : Google Photos path "media/by-month/2023": Committing uploads - please wait...
2023/12/24 13:53:11 INFO  : Exiting...
1 Like

Rclone retries can be limited with

--retries=N
--low-level-retries=N

Where N is the number of retries. So if you want only one high level retry add --retries=1 to your command. If you want for example 5 high level retries each including 3 low level retries add

--retries=5 --low-level-retries=3

to your command

And if you want to read about (other) flags rclone is well documented

It seems from my logs it's doing 10 low level retries than infinite retries. It keeps doing those 10 low levels over and over again.
I manually stopped it. It would have gone on for hours and hours. Eventual it gives up due to some global time out?

It i am understanding correctly it's not obeying the regular retry. That should default to 3, and it doesn't stop after 3, or ever.

It seems that something changed with Google Photos, rclone does not handle well - there are few more issues reported in the last few days if you look at the forum posts.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.