Sleep timer for low level retries?

AeroMaxx · February 6, 2019, 12:15am

Is it possible to change the sleep time for the low level retries?

I’m doing some server side copies, and seeing things like this in the logs, note the html response.

2019/02/05 19:12:15 DEBUG : pacer: Rate limited, sleeping for 1.298739376s (1 consecutive low level retries)
2019/02/05 19:12:15 DEBUG : pacer: low level retry 1/20 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2019/02/05 19:13:44 DEBUG : pacer: Rate limited, sleeping for 2.11180996s (2 consecutive low level retries)
2019/02/05 19:13:44 DEBUG : pacer: low level retry 2/20 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2019/02/05 19:15:14 DEBUG : pacer: Rate limited, sleeping for 4.425774727s (3 consecutive low level retries)
2019/02/05 19:15:14 DEBUG : pacer: low level retry 3/20 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2019/02/05 19:16:43 DEBUG : pacer: Rate limited, sleeping for 8.738455059s (4 consecutive low level retries)
2019/02/05 19:16:43 DEBUG : pacer: low level retry 4/20 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2019/02/05 19:18:13 DEBUG : pacer: Rate limited, sleeping for 16.349865094s (5 consecutive low level retries)
2019/02/05 19:18:13 DEBUG : pacer: low level retry 5/20 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2019/02/05 19:19:42 DEBUG : pacer: Rate limited, sleeping for 16.123484406s (6 consecutive low level retries)
2019/02/05 19:19:42 DEBUG : pacer: low level retry 6/20 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2019/02/05 19:21:12 DEBUG : pacer: Rate limited, sleeping for 16.154132132s (7 consecutive low level retries)
2019/02/05 19:21:12 DEBUG : pacer: low level retry 7/20 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2019/02/05 19:22:41 DEBUG : pacer: Rate limited, sleeping for 16.019908605s (8 consecutive low level retries)
2019/02/05 19:22:41 DEBUG : pacer: low level retry 8/20 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2019/02/05 19:24:11 DEBUG : pacer: Rate limited, sleeping for 16.037587134s (9 consecutive low level retries)
2019/02/05 19:24:11 DEBUG : pacer: low level retry 9/20 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=****REMOVED****, userRateLimitExceeded)
2019/02/05 19:25:40 DEBUG : pacer: Rate limited, sleeping for 16.190838396s (10 consecutive low level retries)
2019/02/05 19:25:40 DEBUG : pacer: low level retry 10/20 (error googleapi: Error 503: Backend Error, backendError)
2019/02/05 19:27:10 DEBUG : pacer: Rate limited, sleeping for 16.363107451s (11 consecutive low level retries)
2019/02/05 19:27:10 DEBUG : pacer: low level retry 11/20 (error googleapi: Error 500: Internal Error, internalError)
2019/02/05 19:28:39 DEBUG : pacer: Rate limited, sleeping for 16.742804856s (12 consecutive low level retries)
2019/02/05 19:28:39 DEBUG : pacer: low level retry 12/20 (error googleapi: Error 503: Backend Error, backendError)
2019/02/05 19:30:09 DEBUG : pacer: Rate limited, sleeping for 16.00174843s (13 consecutive low level retries)
2019/02/05 19:30:09 DEBUG : pacer: low level retry 13/20 (error googleapi: Error 503: Backend Error, backendError)
2019/02/05 19:31:38 DEBUG : pacer: Rate limited, sleeping for 16.866998005s (14 consecutive low level retries)
2019/02/05 19:31:38 DEBUG : pacer: low level retry 14/20 (error googleapi: Error 500: Internal Error, internalError)
2019/02/05 19:33:08 DEBUG : pacer: Rate limited, sleeping for 16.477903042s (15 consecutive low level retries)
2019/02/05 19:33:08 DEBUG : pacer: low level retry 15/20 (error googleapi: Error 503: Backend Error, backendError)
2019/02/05 19:34:37 DEBUG : pacer: Rate limited, sleeping for 16.160105015s (16 consecutive low level retries)
2019/02/05 19:34:37 DEBUG : pacer: low level retry 16/20 (error googleapi: Error 503: Backend Error, backendError)
2019/02/05 19:36:07 DEBUG : pacer: Rate limited, sleeping for 16.272542996s (17 consecutive low level retries)
2019/02/05 19:36:07 DEBUG : pacer: low level retry 17/20 (error googleapi: Error 503: Backend Error, backendError)
2019/02/05 19:37:36 DEBUG : pacer: Rate limited, sleeping for 16.670912909s (18 consecutive low level retries)
2019/02/05 19:37:36 DEBUG : pacer: low level retry 18/20 (error googleapi: Error 503: Backend Error, backendError)
2019/02/05 19:39:06 DEBUG : pacer: Rate limited, sleeping for 16.305044035s (19 consecutive low level retries)
2019/02/05 19:39:06 DEBUG : pacer: low level retry 19/20 (error googleapi: Error 503: Backend Error, backendError)
2019/02/05 19:40:35 DEBUG : pacer: Rate limited, sleeping for 16.802306577s (20 consecutive low level retries)
2019/02/05 19:40:35 DEBUG : pacer: low level retry 20/20 (error googleapi: got HTTP response code 502 with body: <!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 502 (Server Error)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>502.</b> <ins>That’s an error.</ins>
  <p>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.  <ins>That’s all we know.</ins>
)

Animosity022 · February 6, 2019, 12:33am

What command are you running to create that?

Are you using your own API key?

AeroMaxx · February 6, 2019, 12:43am

I am using my own API key yes.

I have about 25 bash scripts, 20 of those are doing server side copies from someone elses google drive to mine, and the other 5 are doing upload transfers, 1 isn’t currently being used, the remaining 4 1 uses rclone copy the others I believe from memory are all rclone move.

Animosity022 · February 6, 2019, 12:45am

So as noted from the logs, you have too much going on and need to slow it down.

You can only max 10 transactions per second.

What are those 20 scripts doing command wise? What’s the command you are running in each script?

AeroMaxx · February 6, 2019, 12:54am

Oh I see, I have reduced it to --tpslimit 0.4 will see how I get on with that.

The commands are similar to this, but I am using your lock file upload script.

/usr/bin/rclone copy gmedia_api:'/Folder1' gmedia_api:'/Folder1' --backup-dir gmedia_api:'/Duplicates' --checksum --checkers 1 --drive-chunk-size 256M --drive-export-formats docx,xlsx,pptx,svg,pdf --drive-pacer-min-sleep 3000ms --log-file "/home/localadmin/scripts/rclone/logs/${LOGFILE}" --log-level DEBUG --low-level-retries 20 --progress --retries 20 --retries-sleep 5m --stats 5s --tpslimit 0.4 --transfers 1 2>> "/home/localadmin/scripts/rclone/logs/${LOGFILE}";

The sleep stuff was me trying to fix the userRateLimitExceeded stuff! I guess I could remove all of that now? Also the --drive-chunk-size 256M and --drive-pacer-min-sleep 3000ms was also done to try and reduce the number of API requests.

A bit off topic, but --backup-dir gmedia_api:'/Duplicates' I was trying to use that as some of the source folders have duplicates in them and this is reported in the logs, but that didn’t seem to do what I had hoped it would in copying one file to the original folder, and the others to the duplicates folder, instead it skips all copies rather than copying atleast one.

Animosity022 · February 6, 2019, 1:12am

If you are running 25 scripts at the same time, that’s 25 transfers so you would hit issues I’d imagine as that’s 25 transfers going on.

Since you are using your own key, what are you seeing if you check the quotas per 100 seconds?

That would be a good indicator of if you hitting too many API hits as well although the error message is pretty clear.

AeroMaxx · February 6, 2019, 1:16am

I have been checking that and I couldn’t see any evidence of me going over the 1000 requests per 100 seconds, it got pretty close sometimes like 960 ish, but never over.

Animosity022 · February 6, 2019, 1:34am

1000 is like the max you could hit in a perfect world.

Your hits are way up there so you are really seeing the evidence of that based on the multiple 403s.

AeroMaxx · February 6, 2019, 1:38am

Do you think I should try --tpslimit 0.3 instead then? that should be about 7.5 transactions per 10 seconds, am unsure how the --tpslimit-burst works in terms of increasing the number of transactions per second.

Animosity022 · February 6, 2019, 2:46am

Sounds like a plan. I don’t think there is an exact formula so it will be some trial and error as you are trying to find that sweet suite with no errors and as high as possible throughput.

ncw · February 6, 2019, 10:09am

It isn’t at the moment. The sleep times use google recommended exponential backoff algorithm.

AeroMaxx · February 6, 2019, 5:41pm

@ncw Is there a chance you can increase it so it starts small and stops increasing at 30 seconds instead of 16 seconds, as in the logs I get an error from google saying to wait 30 seconds, this was in the first post, but you’d need to scroll down in the box a little.

Animosity022 · February 6, 2019, 6:01pm

That’s because you have other programs hitting the 403s so it’s progressively worse in that log.

The normal pacing is not 30 seconds as it starts in milliseconds and exponentially gets bigger.

AeroMaxx · February 6, 2019, 6:19pm

I don’t understand what you mean by that? The scripts aren’t doing a server side copy of any folders that are in another script, every folder in each script is unique.

Animosity022 · February 6, 2019, 6:23pm

You can only run 10 transactions per second.

If you have multiple scripts going on that are all hitting rate limits, one script/output is going to be bad as it sees the end of the exponential increase.

The pacer normally goes up in an exponential fashion.

So if you hit an error, the first retry is in milliseconds, the more errors you hit rapidly, the quicker the curve goes until hit 30 seconds.

Your log isn’t the start of a 403 rate limit, it’s the upper curve of it.

AeroMaxx · February 6, 2019, 6:34pm

Oh right I think I understand, at least a bit better than before.

That would explain why I was still seeing internal and backend errors despite setting --tpslimit to 0.1

I’ll try moving all the server side copies into one script, will take longer but I guess I will just have to be patient.

Animosity022 · February 6, 2019, 6:41pm

Think of it like water flowing in a pipe.

If you have an ocean feeding something and your intake is a 5 inch pipe, you can only get so much water through the pipe no matter how hard you try.

AeroMaxx · February 7, 2019, 8:22am

Sure that I can understand, I am obviously doing something Google doesn’t like but unsure what or how I am doing it.

Still getting internal and backend errors, although the user rate limit errors have now gone.

I am using 1 bash script with --tpslimit 0.4 on all of the 20 commands in the file, it does them one by one, not all at once.

2019/02/07 06:26:50 DEBUG : pacer: Rate limited, sleeping for 1.42144409s (1 consecutive low level retries)
2019/02/07 06:26:50 DEBUG : pacer: low level retry 1/20 (error googleapi: Error 500: Internal Error, internalError)
2019/02/07 06:26:52 DEBUG : pacer: Rate limited, sleeping for 2.990292917s (2 consecutive low level retries)
2019/02/07 06:26:52 DEBUG : pacer: low level retry 1/20 (error googleapi: Error 503: Backend Error, backendError)
2019/02/07 06:26:54 DEBUG : pacer: Rate limited, sleeping for 4.784038627s (3 consecutive low level retries)
2019/02/07 06:26:54 DEBUG : pacer: low level retry 1/20 (error googleapi: Error 500: Internal Error, internalError)
2019/02/07 06:28:20 DEBUG : pacer: Rate limited, sleeping for 8.343995166s (4 consecutive low level retries)
2019/02/07 06:28:20 DEBUG : pacer: low level retry 2/20 (error googleapi: Error 500: Internal Error, internalError)
2019/02/07 06:28:22 DEBUG : pacer: Rate limited, sleeping for 16.792274641s (5 consecutive low level retries)
2019/02/07 06:28:22 DEBUG : pacer: low level retry 2/20 (error googleapi: Error 500: Internal Error, internalError)
2019/02/07 06:28:25 DEBUG : pacer: Rate limited, sleeping for 16.570235749s (6 consecutive low level retries)
2019/02/07 06:28:25 DEBUG : pacer: low level retry 2/20 (error googleapi: Error 500: Internal Error, internalError)
2019/02/07 06:29:49 DEBUG : pacer: Rate limited, sleeping for 16.235891187s (7 consecutive low level retries)
2019/02/07 06:29:49 DEBUG : pacer: low level retry 3/20 (error googleapi: Error 503: Backend Error, backendError)
2019/02/07 06:29:54 DEBUG : pacer: Rate limited, sleeping for 16.344003975s (8 consecutive low level retries)
2019/02/07 06:29:54 DEBUG : pacer: low level retry 3/20 (error googleapi: Error 500: Internal Error, internalError)
2019/02/07 06:30:08 DEBUG : pacer: Rate limited, sleeping for 16.709277952s (9 consecutive low level retries)
2019/02/07 06:30:08 DEBUG : pacer: low level retry 3/20 (error googleapi: Error 500: Internal Error, internalError)
2019/02/07 06:31:19 DEBUG : pacer: Rate limited, sleeping for 16.260029147s (10 consecutive low level retries)
2019/02/07 06:31:19 DEBUG : pacer: low level retry 4/20 (error googleapi: Error 503: Backend Error, backendError)
2019/02/07 06:31:32 DEBUG : pacer: Rate limited, sleeping for 16.398892805s (11 consecutive low level retries)
2019/02/07 06:31:32 DEBUG : pacer: low level retry 4/20 (error googleapi: Error 500: Internal Error, internalError)
2019/02/07 06:31:45 DEBUG : pacer: Rate limited, sleeping for 16.955298396s (12 consecutive low level retries)
2019/02/07 06:31:45 DEBUG : pacer: low level retry 4/20 (error googleapi: Error 500: Internal Error, internalError)
2019/02/07 06:32:48 DEBUG : pacer: Rate limited, sleeping for 16.6450497s (13 consecutive low level retries)
2019/02/07 06:32:48 DEBUG : pacer: low level retry 5/20 (error googleapi: Error 500: Internal Error, internalError)
2019/02/07 06:33:01 DEBUG : pacer: Rate limited, sleeping for 16.713783463s (14 consecutive low level retries)
2019/02/07 06:33:01 DEBUG : pacer: low level retry 5/20 (error googleapi: Error 500: Internal Error, internalError)
2019/02/07 06:33:15 DEBUG : pacer: Rate limited, sleeping for 16.626515465s (15 consecutive low level retries)
2019/02/07 06:33:15 DEBUG : pacer: low level retry 5/20 (error googleapi: Error 500: Internal Error, internalError)
2019/02/07 06:34:18 DEBUG : pacer: Rate limited, sleeping for 16.111451241s (16 consecutive low level retries)
2019/02/07 06:34:18 DEBUG : pacer: low level retry 6/20 (error googleapi: Error 503: Backend Error, backendError)
2019/02/07 06:34:31 DEBUG : pacer: Rate limited, sleeping for 16.926988379s (17 consecutive low level retries)
2019/02/07 06:34:31 DEBUG : pacer: low level retry 6/20 (error googleapi: Error 500: Internal Error, internalError)
2019/02/07 06:34:45 DEBUG : pacer: Rate limited, sleeping for 16.299223677s (18 consecutive low level retries)
2019/02/07 06:34:45 DEBUG : pacer: low level retry 6/20 (error googleapi: Error 500: Internal Error, internalError)
2019/02/07 06:35:47 DEBUG : pacer: Rate limited, sleeping for 16.95600742s (19 consecutive low level retries)
2019/02/07 06:35:47 DEBUG : pacer: low level retry 7/20 (error googleapi: Error 500: Internal Error, internalError)
2019/02/07 06:36:01 DEBUG : pacer: Rate limited, sleeping for 16.254507663s (20 consecutive low level retries)
2019/02/07 06:36:01 DEBUG : pacer: low level retry 7/20 (error googleapi: Error 500: Internal Error, internalError)
2019/02/07 06:36:15 DEBUG : pacer: Rate limited, sleeping for 16.791952719s (21 consecutive low level retries)
2019/02/07 06:36:15 DEBUG : pacer: low level retry 7/20 (error googleapi: Error 500: Internal Error, internalError)
2019/02/07 06:37:17 DEBUG : pacer: Rate limited, sleeping for 16.581352358s (22 consecutive low level retries)
2019/02/07 06:37:17 DEBUG : pacer: low level retry 8/20 (error googleapi: Error 500: Internal Error, internalError)
2019/02/07 06:37:31 DEBUG : pacer: Rate limited, sleeping for 16.769347138s (23 consecutive low level retries)
2019/02/07 06:37:31 DEBUG : pacer: low level retry 8/20 (error googleapi: Error 500: Internal Error, internalError)
2019/02/07 06:37:44 DEBUG : pacer: Rate limited, sleeping for 16.238080039s (24 consecutive low level retries)
2019/02/07 06:37:44 DEBUG : pacer: low level retry 8/20 (error googleapi: Error 500: Internal Error, internalError)
2019/02/07 06:38:46 DEBUG : pacer: Rate limited, sleeping for 16.555599983s (25 consecutive low level retries)
2019/02/07 06:38:46 DEBUG : pacer: low level retry 9/20 (error googleapi: Error 503: Backend Error, backendError)
2019/02/07 06:39:00 DEBUG : pacer: Rate limited, sleeping for 16.006084443s (26 consecutive low level retries)
2019/02/07 06:39:00 DEBUG : pacer: low level retry 9/20 (error googleapi: Error 500: Internal Error, internalError)
2019/02/07 06:39:14 DEBUG : pacer: Rate limited, sleeping for 16.846450676s (27 consecutive low level retries)
2019/02/07 06:39:14 DEBUG : pacer: low level retry 9/20 (error googleapi: Error 503: Backend Error, backendError)
2019/02/07 06:40:16 DEBUG : pacer: Rate limited, sleeping for 16.891935705s (28 consecutive low level retries)
2019/02/07 06:40:16 DEBUG : pacer: low level retry 10/20 (error googleapi: Error 500: Internal Error, internalError)
2019/02/07 06:40:30 DEBUG : pacer: Rate limited, sleeping for 16.524878156s (29 consecutive low level retries)
2019/02/07 06:40:30 DEBUG : pacer: low level retry 10/20 (error googleapi: Error 503: Backend Error, backendError)
2019/02/07 06:40:43 DEBUG : pacer: Rate limited, sleeping for 16.663064984s (30 consecutive low level retries)
2019/02/07 06:40:43 DEBUG : pacer: low level retry 10/20 (error googleapi: Error 500: Internal Error, internalError)
2019/02/07 06:41:45 DEBUG : pacer: Rate limited, sleeping for 16.104523373s (31 consecutive low level retries)
2019/02/07 06:41:45 DEBUG : pacer: low level retry 11/20 (error googleapi: Error 500: Internal Error, internalError)
2019/02/07 06:41:59 DEBUG : pacer: Rate limited, sleeping for 16.546927833s (32 consecutive low level retries)
2019/02/07 06:41:59 DEBUG : pacer: low level retry 11/20 (error googleapi: Error 503: Backend Error, backendError)
2019/02/07 06:42:13 DEBUG : pacer: Rate limited, sleeping for 16.350106832s (33 consecutive low level retries)
2019/02/07 06:42:13 DEBUG : pacer: low level retry 11/20 (error googleapi: Error 500: Internal Error, internalError)
2019/02/07 06:43:15 DEBUG : pacer: Rate limited, sleeping for 16.721902292s (34 consecutive low level retries)
2019/02/07 06:43:15 DEBUG : pacer: low level retry 12/20 (error googleapi: Error 503: Backend Error, backendError)
2019/02/07 06:43:29 DEBUG : pacer: Rate limited, sleeping for 16.188704488s (35 consecutive low level retries)
2019/02/07 06:43:29 DEBUG : pacer: low level retry 12/20 (error googleapi: Error 500: Internal Error, internalError)
2019/02/07 06:43:43 DEBUG : pacer: Rate limited, sleeping for 16.982082265s (36 consecutive low level retries)
2019/02/07 06:43:43 DEBUG : pacer: low level retry 12/20 (error googleapi: Error 500: Internal Error, internalError)
2019/02/07 06:44:45 DEBUG : pacer: Rate limited, sleeping for 16.583268485s (37 consecutive low level retries)
2019/02/07 06:44:45 DEBUG : pacer: low level retry 13/20 (error googleapi: Error 500: Internal Error, internalError)
2019/02/07 06:44:58 DEBUG : pacer: Rate limited, sleeping for 16.126214125s (38 consecutive low level retries)
2019/02/07 06:44:58 DEBUG : pacer: low level retry 13/20 (error googleapi: Error 500: Internal Error, internalError)
2019/02/07 06:45:12 DEBUG : pacer: Rate limited, sleeping for 16.184485786s (39 consecutive low level retries)
2019/02/07 06:45:12 DEBUG : pacer: low level retry 13/20 (error googleapi: Error 500: Internal Error, internalError)
2019/02/07 06:46:14 DEBUG : pacer: Rate limited, sleeping for 16.059873705s (40 consecutive low level retries)
2019/02/07 06:46:14 DEBUG : pacer: low level retry 14/20 (error googleapi: Error 500: Internal Error, internalError)
2019/02/07 06:46:14 DEBUG : gmedia_rince_api_1: Loaded invalid token from config file - ignoring
2019/02/07 06:46:14 DEBUG : gmedia_rince_api_1: Saved new token in config file
2019/02/07 06:46:28 DEBUG : pacer: Rate limited, sleeping for 16.878673736s (41 consecutive low level retries)
2019/02/07 06:46:28 DEBUG : pacer: low level retry 14/20 (error googleapi: Error 500: Internal Error, internalError)
2019/02/07 06:46:42 DEBUG : pacer: Rate limited, sleeping for 16.059115538s (42 consecutive low level retries)
2019/02/07 06:46:42 DEBUG : pacer: low level retry 14/20 (error googleapi: Error 503: Backend Error, backendError)
2019/02/07 06:47:46 DEBUG : pacer: Rate limited, sleeping for 16.652200245s (43 consecutive low level retries)
2019/02/07 06:47:46 DEBUG : pacer: low level retry 15/20 (error googleapi: Error 500: Internal Error, internalError)
2019/02/07 06:47:57 DEBUG : pacer: Rate limited, sleeping for 16.655729921s (44 consecutive low level retries)
2019/02/07 06:47:57 DEBUG : pacer: low level retry 15/20 (error googleapi: Error 503: Backend Error, backendError)
2019/02/07 06:48:11 DEBUG : pacer: Rate limited, sleeping for 16.954263069s (45 consecutive low level retries)
2019/02/07 06:48:11 DEBUG : pacer: low level retry 15/20 (error googleapi: Error 500: Internal Error, internalError)
2019/02/07 06:49:15 DEBUG : pacer: Rate limited, sleeping for 16.296065828s (46 consecutive low level retries)
2019/02/07 06:49:15 DEBUG : pacer: low level retry 16/20 (error googleapi: Error 500: Internal Error, internalError)
2019/02/07 06:49:29 DEBUG : pacer: Rate limited, sleeping for 16.242017804s (47 consecutive low level retries)
2019/02/07 06:49:29 DEBUG : pacer: low level retry 16/20 (error googleapi: Error 500: Internal Error, internalError)
2019/02/07 06:49:42 DEBUG : pacer: Rate limited, sleeping for 16.840067109s (48 consecutive low level retries)
2019/02/07 06:49:42 DEBUG : pacer: low level retry 16/20 (error googleapi: Error 500: Internal Error, internalError)
2019/02/07 06:50:45 DEBUG : pacer: Rate limited, sleeping for 16.315115417s (49 consecutive low level retries)
2019/02/07 06:50:45 DEBUG : pacer: low level retry 17/20 (error googleapi: Error 500: Internal Error, internalError)
2019/02/07 06:50:58 DEBUG : pacer: Rate limited, sleeping for 16.682739553s (50 consecutive low level retries)
2019/02/07 06:50:58 DEBUG : pacer: low level retry 17/20 (error googleapi: Error 500: Internal Error, internalError)
2019/02/07 06:51:12 DEBUG : pacer: Rate limited, sleeping for 16.917229449s (51 consecutive low level retries)
2019/02/07 06:51:12 DEBUG : pacer: low level retry 17/20 (error googleapi: Error 500: Internal Error, internalError)
2019/02/07 06:52:14 DEBUG : pacer: Rate limited, sleeping for 16.102270746s (52 consecutive low level retries)
2019/02/07 06:52:14 DEBUG : pacer: low level retry 18/20 (error googleapi: Error 500: Internal Error, internalError)
2019/02/07 06:52:28 DEBUG : pacer: Rate limited, sleeping for 16.823164381s (53 consecutive low level retries)
2019/02/07 06:52:28 DEBUG : pacer: low level retry 18/20 (error googleapi: Error 500: Internal Error, internalError)
2019/02/07 06:52:42 DEBUG : pacer: Rate limited, sleeping for 16.640333122s (54 consecutive low level retries)
2019/02/07 06:52:42 DEBUG : pacer: low level retry 18/20 (error googleapi: Error 500: Internal Error, internalError)
2019/02/07 06:53:44 DEBUG : pacer: Rate limited, sleeping for 16.676188727s (55 consecutive low level retries)
2019/02/07 06:53:44 DEBUG : pacer: low level retry 19/20 (error googleapi: Error 500: Internal Error, internalError)
2019/02/07 06:53:57 DEBUG : pacer: Rate limited, sleeping for 16.525755752s (56 consecutive low level retries)
2019/02/07 06:53:57 DEBUG : pacer: low level retry 19/20 (error googleapi: Error 500: Internal Error, internalError)
2019/02/07 06:54:11 DEBUG : pacer: Rate limited, sleeping for 16.217833334s (57 consecutive low level retries)
2019/02/07 06:54:11 DEBUG : pacer: low level retry 19/20 (error googleapi: Error 500: Internal Error, internalError)
2019/02/07 06:55:13 DEBUG : pacer: Rate limited, sleeping for 16.887251988s (58 consecutive low level retries)
2019/02/07 06:55:13 DEBUG : pacer: low level retry 20/20 (error googleapi: Error 500: Internal Error, internalError)

Animosity022 · February 7, 2019, 11:26am

A 500 error is something different as it’s not a rate limit error.

What is the command you are running and what are the logs before it? Are you doing all server side copies and how much data have you transferred?

AeroMaxx · February 7, 2019, 11:29am

Oh yes I realise that, I am making some progress

Yes all server side copies.

I have no idea, it’s not in the logs, or atleast in the logs that I have.