emboehm
November 1, 2022, 7:06pm
1
upload to BOX fails after a random amount of time with 502 Bad Gateway
rclone --version
rclone v1.60.0
- os/version: centos 7.4.1708 (64 bit)
- os/kernel: 3.10.0-693.11.6.el7.x86_64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.19.2
- go/linking: static
- go/tags: none
Are you on the latest version of rclone? You can validate by checking the version listed here: Rclone downloads
Box
rclone copy /tmp remote:tmp)
/opt/mysql_backup/scripts/rclone --config=/opt/mysql_backup/scripts/rclone.conf copy RB_DASHBOARD_BACKUP/mysql_backup --transfers=16 -vvv --dump-headers box_hnd:RB_DASHBOARD_BACKUP/mysql_backup > /tmp/log 2>&1
[box_hnd]
type = box
token = {removed}
-vv flag
2022/11/01 10:14:04 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/11/01 10:14:04 DEBUG : bcawlan-mj-test-morey/20221031/mysql-database_20221031_022603.tar.gz: Uploading part 36/105 offset 2.188Gi/6.507Gi part size 64Mi
2022/11/01 10:14:05 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/11/01 10:14:05 DEBUG : HTTP REQUEST (req 0xc0009ee900)
2022/11/01 10:14:05 DEBUG : PUT /api/2.0/files/upload_sessions/39F5ECF8D664AD1F689D7615E7BED55C HTTP/1.1
Host: upload.box.com
User-Agent: rclone/v1.60.0
Content-Length: 67108864
Authorization: XXXX
Content-Range: bytes 2348810240-2415919103/6986761358
Content-Type: application/octet-stream
Digest: sha=MkpFa416PAgjTl4Ue7C2wO2QIdI=
Accept-Encoding: gzip
2022/11/01 10:14:05 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/11/01 10:14:17 INFO :
Transferred: 1.859 GiB / 6.507 GiB, 29%, 12.123 MiB/s, ETA 6m32s
Transferred: 0 / 1, 0%
Elapsed time: 20m0.5s
Transferring:
* bcawlan-mj-test-morey/??20221031_022603.tar.gz: 28% /6.507Gi, 12.123Mi/s, 6m32s
2022/11/01 10:15:17 INFO :
Transferred: 1.868 GiB / 6.507 GiB, 29%, 308.211 KiB/s, ETA 4h23m2s
Transferred: 0 / 1, 0%
Elapsed time: 21m0.5s
Transferring:
* bcawlan-mj-test-morey/??20221031_022603.tar.gz: 28% /6.507Gi, 308.207Ki/s, 4h23m2s
2022/11/01 10:15:40 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/11/01 10:15:40 DEBUG : HTTP RESPONSE (req 0xc0005da200)
2022/11/01 10:15:40 DEBUG : HTTP/2.0 502 Bad Gateway
Content-Length: 332
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
Content-Type: text/html; charset=UTF-8
Date: Tue, 01 Nov 2022 17:15:40 GMT
Referrer-Policy: no-referrer
ncw
November 2, 2022, 10:51am
2
emboehm:
502 Bad Gateway
This means that the server beyond the server you were talking to had a problem.
It could be caused by a problem at box. It could also be caused by a local proxy - are you using a proxy?
emboehm
November 2, 2022, 11:38am
3
The connection is from our corporate network so it is possible that there is a proxy or a problem between us and BOX.
I will take it up with our internal support organization and BOX.
Mark_Smith
November 2, 2022, 1:45pm
4
Hi,
We are getting 502 errors as well, did you find to cause?
Thanks
emboehm
November 2, 2022, 2:04pm
5
No, I haven't found the cause yet. It is likely to take some time to get answers from box or our own support organization.
Ole
November 2, 2022, 2:06pm
6
Hi @emboehm and @Mark_Smith ,
Looking in the code, I think rclone should do 10 low level (API request) retries on an 502 before stopping with an error. Unfortunately the snippet posted above is too small to see if this happens.
Can one of you post a link to a full debug (-vv) log (with or without --dump headers)? or confirm that the 10 low level retries are happening with increasing intervals and repeated in 3 attempts (high level retries)?
asdffdsa
November 2, 2022, 2:06pm
7
Ole
November 2, 2022, 2:09pm
8
Exactly, which is the expected behavior of the low level retries to be confirmed in the full debug logs.
emboehm
November 2, 2022, 2:23pm
9
It takes a long time before the 10 low level retries happen. I usually kill it before then.
Ole
November 2, 2022, 2:35pm
10
I would expect low level retries after 10ms, 20ms, 40ms, ... 2s, 2s, 2s.
So something isn't quite as expected - or there are a lot of low level retries and it takes some time until you hit 10 502's in a row.
Perfect, that should be sufficient.
Mark_Smith
November 2, 2022, 2:40pm
11
My command is
rclone move --transfers 16 --checkers 8 /data/BOXSYNC_TEMP/ box-rclone02: --delete-empty-src-dirs --progress --min-age 2m --max-transfer 4G --cutoff-mode=cautious --order-by modtime,ascending --log-file=/home/box-cli/LOGS/BOXSYNC_TEMP.txt --log-level DEBUG --box-upload-cutoff 500Mi --low-level-retries=3
I have tried it down to 4 transfers and 4 checkers and still get the 502 and the whole process sits there not transferring data.
This is showing in my captured log.
2022/11/02 14:25:03 DEBUG : pacer: low level retry 2/10 (error Error "502 Bad Gateway" (502))
emboehm
November 2, 2022, 2:46pm
12
I am seeing multiple times of hitting low level retries but the transfer doesn't abort. Transfer is still running. I haven't killed it yet. Does it take three times of hitting 'low level retry 3/3'?
grep pacer log
2022/11/02 07:27:55 DEBUG : pacer: low level retry 1/3 (error Error "502 Bad Gateway" (502))
2022/11/02 07:27:55 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2022/11/02 07:27:55 DEBUG : pacer: low level retry 1/3 (error Error "502 Bad Gateway" (502))
2022/11/02 07:27:55 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2022/11/02 07:27:55 DEBUG : pacer: low level retry 1/3 (error Error "502 Bad Gateway" (502))
2022/11/02 07:27:55 DEBUG : pacer: Rate limited, increasing sleep to 80ms
2022/11/02 07:30:55 DEBUG : pacer: low level retry 2/3 (error Error "502 Bad Gateway" (502))
2022/11/02 07:30:55 DEBUG : pacer: Rate limited, increasing sleep to 160ms
2022/11/02 07:30:55 DEBUG : pacer: low level retry 2/3 (error Error "502 Bad Gateway" (502))
2022/11/02 07:30:55 DEBUG : pacer: Rate limited, increasing sleep to 320ms
2022/11/02 07:30:55 DEBUG : pacer: low level retry 2/3 (error Error "502 Bad Gateway" (502))
2022/11/02 07:30:55 DEBUG : pacer: Rate limited, increasing sleep to 640ms
2022/11/02 07:33:09 DEBUG : pacer: low level retry 3/3 (error Error "502 Bad Gateway" (502))
2022/11/02 07:33:09 DEBUG : pacer: Rate limited, increasing sleep to 1.28s
2022/11/02 07:33:09 DEBUG : pacer: low level retry 3/3 (error Error "502 Bad Gateway" (502))
2022/11/02 07:33:09 DEBUG : pacer: Rate limited, increasing sleep to 2s
2022/11/02 07:33:16 DEBUG : pacer: low level retry 3/3 (error Error "502 Bad Gateway" (502))
2022/11/02 07:35:10 DEBUG : pacer: low level retry 1/3 (error Error "502 Bad Gateway" (502))
2022/11/02 07:38:34 DEBUG : pacer: low level retry 2/3 (error Error "502 Bad Gateway" (502))
2022/11/02 07:42:52 DEBUG : pacer: low level retry 3/3 (error Error "502 Bad Gateway" (502))
Hello all, our team has ran into this as well.
2022/11/01 15:15:40 DEBUG : Using config file from "/Users/username/.config/rclone/rclone.conf"
2022/11/01 15:15:40 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "copy" "-u" "-vv" "--log-file" "
2022/11/01 07:34:43 DEBUG : Couldn't decode error response: invalid character '<' looking for beginning of value
2022/11/01 07:34:43 DEBUG : pacer: low level retry 2/10 (error Error "502 Bad Gateway" (502))
2022/11/01 07:34:43 DEBUG : pacer: Rate limited, increasing sleep to 320ms
2022/11/01 07:34:43 DEBUG : Couldn't decode error response: invalid character '<' looking for beginning of value
2022/11/01 07:34:43 DEBUG : pacer: low level retry 3/10 (error Error "502 Bad Gateway" (502))
2022/11/01 07:34:43 DEBUG : pacer: Rate limited, increasing sleep to 640ms
2022/11/01 07:35:10 DEBUG : pacer: low level retry 1/10 (error Put "https://upload.box.com/api/2.0/files/upload_sessions/50F0AEFB8620BFECC687ACA81CFE1ACE": http2: timeout awaiting response headers)
2022/11/01 07:35:10 DEBUG : pacer: Rate limited, increasing sleep to 1.28s
2022/11/01 07:35:38 INFO :
Transferred: 519.017M / 661.545 MBytes, 78%, 1.443 MBytes/s, ETA 1m38s
Transferred: 10 / 15, 67%
Elapsed time: 6m8.3s
Transferring:
* XXX:100% /126.601M, 24.862k/s, -
* XXX: 90% /63.947M, 21.902k/s, 4m38s
* XXX: 37% /80.864M, 46.289k/s, 18m36s
* XXX: 0% /84.652M, 0/s,
Looks like we should all get a ticket going with Box support?
Ole
November 2, 2022, 2:52pm
14
Hi @Mark_Smith , @emboehm and @Edgar_Hernandez
Thanks, I would like to see your command too - especially tuning parameters like the number of checkers and transfers. Please add/edit into the post with the log (it makes it easier to keep an overview)
and you rclone version, basically the first line of your debug logs:
2022/11/02 11:57:56 DEBUG : rclone: Version "v1.60.0" starting with parameters ...
1 Like
Mark_Smith
November 2, 2022, 3:02pm
15
My command is
rclone move --transfers 16 --checkers 8 /data/BOXSYNC_TEMP/ box-rclone02: --delete-empty-src-dirs --progress --min-age 2m --max-transfer 4G --cutoff-mode=cautious --order-by modtime,ascending --log-file=/home/box-cli/LOGS/BOXSYNC_TEMP.txt --log-level DEBUG --box-upload-cutoff 500Mi --low-level-retries=3
I have tried it down to 4 transfers and 4 checkers and still get the 502 and the whole process sits there not transferring data.
I've updated my initial comment. We've not tried tunning parameters yet and have been exploring it based on this discussion.
Ole
November 2, 2022, 3:12pm
17
Seems like rclone does the retries correctly
It takes the the third to fail too, "low level retry 3/3" means now trying 3/3. If it fails then the transfer of that file fails, rclone continues trying the other files. If one or more files fail then rclone will do three attempts at the sync/copy.
More info:https://rclone.org/docs/#low-level-retries-number https://rclone.org/docs/#retries-int
Perfect, just what I needed!
Fully agree!
It may help to reduce checkers and transfers (or just use defaults), but not too optimistic - seems like Box is in trouble.
Thanks all for the cooperation!
Mark_Smith
November 2, 2022, 3:15pm
18
I have escalated this thread to Box EMEA Product Support.
Fingers crossed.
2 Likes
emboehm
November 2, 2022, 4:39pm
19
The transfer doesn't seem to be erroring out, even though it looks like it has restarted the transfer five times.
grep -A 1 "Multipart upload session started for 224" *2.log
2022/11/02 07:21:48 DEBUG : bcawlan_20221031.tar.gz.00: Multipart upload session started for 224 parts of size 64Mi
2022/11/02 07:21:48 DEBUG : bcawlan_20221031.tar.gz.00: Uploading part 1/224 offset 0/14Gi part size 64Mi
--
2022/11/02 07:55:35 DEBUG : bcawlan_20221031.tar.gz.00: Multipart upload session started for 224 parts of size 64Mi
2022/11/02 07:55:35 DEBUG : bcawlan_20221031.tar.gz.00: Uploading part 1/224 offset 0/14Gi part size 64Mi
--
2022/11/02 08:16:20 DEBUG : bcawlan_20221031.tar.gz.00: Multipart upload session started for 224 parts of size 64Mi
2022/11/02 08:16:21 DEBUG : bcawlan_20221031.tar.gz.00: Uploading part 1/224 offset 0/14Gi part size 64Mi
--
2022/11/02 08:26:09 DEBUG : bcawlan_20221031.tar.gz.00: Multipart upload session started for 224 parts of size 64Mi
2022/11/02 08:26:09 DEBUG : bcawlan_20221031.tar.gz.00: Uploading part 1/224 offset 0/14Gi part size 64Mi
--
2022/11/02 08:50:21 DEBUG : bcawlan_20221031.tar.gz.00: Multipart upload session started for 224 parts of size 64Mi
2022/11/02 08:50:22 DEBUG : bcawlan_20221031.tar.gz.00: Uploading part 1/224 offset 0/14Gi part size 64Mi
Here are the pacer error messages
grep pacer box_hnd_auto_2.log
2022/11/02 07:27:55 DEBUG : pacer: low level retry 1/3 (error Error "502 Bad Gateway" (502))
2022/11/02 07:27:55 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2022/11/02 07:27:55 DEBUG : pacer: low level retry 1/3 (error Error "502 Bad Gateway" (502))
2022/11/02 07:27:55 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2022/11/02 07:27:55 DEBUG : pacer: low level retry 1/3 (error Error "502 Bad Gateway" (502))
2022/11/02 07:27:55 DEBUG : pacer: Rate limited, increasing sleep to 80ms
2022/11/02 07:30:55 DEBUG : pacer: low level retry 2/3 (error Error "502 Bad Gateway" (502))
2022/11/02 07:30:55 DEBUG : pacer: Rate limited, increasing sleep to 160ms
2022/11/02 07:30:55 DEBUG : pacer: low level retry 2/3 (error Error "502 Bad Gateway" (502))
2022/11/02 07:30:55 DEBUG : pacer: Rate limited, increasing sleep to 320ms
2022/11/02 07:30:55 DEBUG : pacer: low level retry 2/3 (error Error "502 Bad Gateway" (502))
2022/11/02 07:30:55 DEBUG : pacer: Rate limited, increasing sleep to 640ms
2022/11/02 07:33:09 DEBUG : pacer: low level retry 3/3 (error Error "502 Bad Gateway" (502))
2022/11/02 07:33:09 DEBUG : pacer: Rate limited, increasing sleep to 1.28s
2022/11/02 07:33:09 DEBUG : pacer: low level retry 3/3 (error Error "502 Bad Gateway" (502))
2022/11/02 07:33:09 DEBUG : pacer: Rate limited, increasing sleep to 2s
2022/11/02 07:33:16 DEBUG : pacer: low level retry 3/3 (error Error "502 Bad Gateway" (502))
2022/11/02 07:35:10 DEBUG : pacer: low level retry 1/3 (error Error "502 Bad Gateway" (502))
2022/11/02 07:38:34 DEBUG : pacer: low level retry 2/3 (error Error "502 Bad Gateway" (502))
2022/11/02 07:42:52 DEBUG : pacer: low level retry 3/3 (error Error "502 Bad Gateway" (502))
2022/11/02 07:55:25 DEBUG : pacer: low level retry 1/3 (error Error "internal_server_error" (500))
2022/11/02 07:55:31 DEBUG : pacer: Reducing sleep to 1.5s
2022/11/02 07:55:32 DEBUG : pacer: Reducing sleep to 1.125s
2022/11/02 07:55:33 DEBUG : pacer: Reducing sleep to 843.75ms
2022/11/02 07:55:35 DEBUG : pacer: Reducing sleep to 632.8125ms
2022/11/02 07:55:40 DEBUG : pacer: Reducing sleep to 474.609375ms
2022/11/02 07:55:42 DEBUG : pacer: Reducing sleep to 355.957031ms
2022/11/02 07:55:43 DEBUG : pacer: Reducing sleep to 266.967773ms
2022/11/02 07:55:44 DEBUG : pacer: Reducing sleep to 200.225829ms
2022/11/02 07:56:45 DEBUG : pacer: low level retry 1/3 (error Error "502 Bad Gateway" (502))
2022/11/02 07:56:45 DEBUG : pacer: Rate limited, increasing sleep to 400.451658ms
2022/11/02 07:56:45 DEBUG : pacer: low level retry 1/3 (error Error "502 Bad Gateway" (502))
2022/11/02 07:56:45 DEBUG : pacer: Rate limited, increasing sleep to 800.903316ms
2022/11/02 07:56:45 DEBUG : pacer: low level retry 1/3 (error Error "502 Bad Gateway" (502))
2022/11/02 07:56:45 DEBUG : pacer: Rate limited, increasing sleep to 1.601806632s
2022/11/02 07:58:45 DEBUG : pacer: low level retry 2/3 (error Error "502 Bad Gateway" (502))
2022/11/02 07:58:45 DEBUG : pacer: Rate limited, increasing sleep to 2s
2022/11/02 07:58:45 DEBUG : pacer: low level retry 2/3 (error Error "502 Bad Gateway" (502))
2022/11/02 07:58:47 DEBUG : pacer: low level retry 2/3 (error Error "502 Bad Gateway" (502))
2022/11/02 08:00:45 DEBUG : pacer: low level retry 3/3 (error Error "502 Bad Gateway" (502))
2022/11/02 08:00:47 DEBUG : pacer: low level retry 3/3 (error Error "502 Bad Gateway" (502))
2022/11/02 08:00:49 DEBUG : pacer: low level retry 3/3 (error Error "502 Bad Gateway" (502))
2022/11/02 08:02:50 DEBUG : pacer: low level retry 1/3 (error Error "502 Bad Gateway" (502))
2022/11/02 08:13:27 DEBUG : pacer: low level retry 1/3 (error Error "502 Bad Gateway" (502))
2022/11/02 08:15:10 DEBUG : pacer: Reducing sleep to 1.5s
2022/11/02 08:16:18 DEBUG : pacer: Reducing sleep to 1.125s
2022/11/02 08:16:18 DEBUG : pacer: Reducing sleep to 843.75ms
2022/11/02 08:16:19 DEBUG : pacer: Reducing sleep to 632.8125ms
2022/11/02 08:16:20 DEBUG : pacer: Reducing sleep to 474.609375ms
2022/11/02 08:19:36 DEBUG : pacer: low level retry 1/3 (error Error "502 Bad Gateway" (502))
2022/11/02 08:19:36 DEBUG : pacer: Rate limited, increasing sleep to 949.21875ms
2022/11/02 08:19:36 DEBUG : pacer: low level retry 1/3 (error Error "502 Bad Gateway" (502))
2022/11/02 08:19:36 DEBUG : pacer: Rate limited, increasing sleep to 1.8984375s
2022/11/02 08:19:36 DEBUG : pacer: low level retry 1/3 (error Error "502 Bad Gateway" (502))
2022/11/02 08:19:36 DEBUG : pacer: Rate limited, increasing sleep to 2s
2022/11/02 08:21:30 DEBUG : pacer: low level retry 2/3 (error Error "502 Bad Gateway" (502))
2022/11/02 08:21:30 DEBUG : pacer: low level retry 2/3 (error Error "502 Bad Gateway" (502))
2022/11/02 08:21:30 DEBUG : pacer: low level retry 2/3 (error Error "502 Bad Gateway" (502))
2022/11/02 08:22:51 DEBUG : pacer: low level retry 3/3 (error Error "502 Bad Gateway" (502))
2022/11/02 08:22:51 DEBUG : pacer: low level retry 3/3 (error Error "502 Bad Gateway" (502))
2022/11/02 08:22:51 DEBUG : pacer: low level retry 3/3 (error Error "502 Bad Gateway" (502))
2022/11/02 08:22:57 DEBUG : pacer: Reducing sleep to 1.5s
2022/11/02 08:25:58 DEBUG : pacer: low level retry 1/3 (error Error "internal_server_error" (500))
2022/11/02 08:25:58 DEBUG : pacer: Rate limited, increasing sleep to 2s
2022/11/02 08:26:05 DEBUG : pacer: Reducing sleep to 1.5s
2022/11/02 08:26:05 DEBUG : pacer: Reducing sleep to 1.125s
2022/11/02 08:26:07 DEBUG : pacer: Reducing sleep to 843.75ms
2022/11/02 08:26:08 DEBUG : pacer: Reducing sleep to 632.8125ms
2022/11/02 08:26:09 DEBUG : pacer: Reducing sleep to 474.609375ms
2022/11/02 08:27:39 DEBUG : pacer: Reducing sleep to 355.957031ms
2022/11/02 08:27:40 DEBUG : pacer: Reducing sleep to 266.967773ms
2022/11/02 08:28:32 DEBUG : pacer: Reducing sleep to 200.225829ms
2022/11/02 08:28:32 DEBUG : pacer: Reducing sleep to 150.169371ms
2022/11/02 08:28:36 DEBUG : pacer: Reducing sleep to 112.627028ms
2022/11/02 08:28:38 DEBUG : pacer: Reducing sleep to 84.470271ms
2022/11/02 08:28:38 DEBUG : pacer: Reducing sleep to 63.352703ms
2022/11/02 08:28:40 DEBUG : pacer: Reducing sleep to 47.514527ms
2022/11/02 08:28:44 DEBUG : pacer: Reducing sleep to 35.635895ms
2022/11/02 08:28:44 DEBUG : pacer: Reducing sleep to 26.726921ms
2022/11/02 08:28:46 DEBUG : pacer: Reducing sleep to 20.04519ms
2022/11/02 08:28:47 DEBUG : pacer: Reducing sleep to 15.033892ms
2022/11/02 08:32:18 DEBUG : pacer: low level retry 1/3 (error Error "502 Bad Gateway" (502))
2022/11/02 08:32:18 DEBUG : pacer: Rate limited, increasing sleep to 30.067784ms
2022/11/02 08:32:18 DEBUG : pacer: low level retry 1/3 (error Error "502 Bad Gateway" (502))
2022/11/02 08:32:18 DEBUG : pacer: Rate limited, increasing sleep to 60.135568ms
2022/11/02 08:32:18 DEBUG : pacer: low level retry 1/3 (error Error "502 Bad Gateway" (502))
2022/11/02 08:32:18 DEBUG : pacer: Rate limited, increasing sleep to 120.271136ms
2022/11/02 08:37:03 DEBUG : pacer: low level retry 2/3 (error Error "502 Bad Gateway" (502))
2022/11/02 08:37:03 DEBUG : pacer: Rate limited, increasing sleep to 240.542272ms
2022/11/02 08:37:03 DEBUG : pacer: low level retry 2/3 (error Error "502 Bad Gateway" (502))
2022/11/02 08:37:03 DEBUG : pacer: Rate limited, increasing sleep to 481.084544ms
2022/11/02 08:37:03 DEBUG : pacer: low level retry 2/3 (error Error "502 Bad Gateway" (502))
2022/11/02 08:37:03 DEBUG : pacer: Rate limited, increasing sleep to 962.169088ms
2022/11/02 08:38:11 DEBUG : pacer: low level retry 3/3 (error Error "502 Bad Gateway" (502))
2022/11/02 08:38:11 DEBUG : pacer: Rate limited, increasing sleep to 1.924338176s
2022/11/02 08:38:11 DEBUG : pacer: low level retry 3/3 (error Error "502 Bad Gateway" (502))
2022/11/02 08:38:11 DEBUG : pacer: Rate limited, increasing sleep to 2s
2022/11/02 08:38:12 DEBUG : pacer: low level retry 3/3 (error Error "502 Bad Gateway" (502))
2022/11/02 08:39:15 DEBUG : pacer: low level retry 1/3 (error Error "502 Bad Gateway" (502))
2022/11/02 08:42:57 DEBUG : pacer: low level retry 2/3 (error Error "502 Bad Gateway" (502))
2022/11/02 08:44:11 DEBUG : pacer: low level retry 3/3 (error Error "502 Bad Gateway" (502))
2022/11/02 08:50:18 DEBUG : pacer: Reducing sleep to 1.5s
2022/11/02 08:50:18 DEBUG : pacer: Reducing sleep to 1.125s
2022/11/02 08:50:20 DEBUG : pacer: Reducing sleep to 843.75ms
2022/11/02 08:50:21 DEBUG : pacer: Reducing sleep to 632.8125ms
2022/11/02 08:50:27 DEBUG : pacer: Reducing sleep to 474.609375ms
2022/11/02 08:50:28 DEBUG : pacer: Reducing sleep to 355.957031ms
2022/11/02 08:50:30 DEBUG : pacer: Reducing sleep to 266.967773ms
2022/11/02 08:50:31 DEBUG : pacer: Reducing sleep to 200.225829ms
2022/11/02 08:50:34 DEBUG : pacer: Reducing sleep to 150.169371ms
2022/11/02 08:50:35 DEBUG : pacer: Reducing sleep to 112.627028ms
2022/11/02 08:59:25 DEBUG : pacer: Reducing sleep to 84.470271ms
2022/11/02 08:59:26 DEBUG : pacer: Reducing sleep to 63.352703ms
2022/11/02 08:59:26 DEBUG : pacer: Reducing sleep to 47.514527ms
2022/11/02 08:59:52 DEBUG : pacer: Reducing sleep to 35.635895ms
2022/11/02 08:59:52 DEBUG : pacer: Reducing sleep to 26.726921ms
2022/11/02 08:59:57 DEBUG : pacer: Reducing sleep to 20.04519ms
2022/11/02 08:59:58 DEBUG : pacer: Reducing sleep to 15.033892ms
2022/11/02 08:59:59 DEBUG : pacer: Reducing sleep to 11.275419ms
2022/11/02 09:00:15 DEBUG : pacer: Reducing sleep to 10ms
2022/11/02 09:27:15 DEBUG : pacer: low level retry 1/3 (error Error "502 Bad Gateway" (502))
2022/11/02 09:27:15 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2022/11/02 09:28:18 DEBUG : pacer: low level retry 1/3 (error Error "502 Bad Gateway" (502))
2022/11/02 09:28:18 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2022/11/02 09:28:19 DEBUG : pacer: low level retry 1/3 (error Error "502 Bad Gateway" (502))
2022/11/02 09:28:19 DEBUG : pacer: Rate limited, increasing sleep to 80ms
Ole
November 2, 2022, 5:13pm
20
That is entirely up to you, I have the information I was looking for.
Looks like the situation is easing up after 2022/11/02 08:50:18 and then rclone slowly speeds up again seeing only a few 502's in the following 30 minutes, which is acceptable.
So I would keep it running if it does a useful job, unless the pacer sleep times escalated again after 2022/11/02 09:28:19
