# Rclone aborting while copying large directory to ACD?

**URL:** <https://forum.rclone.org/t/rclone-aborting-while-copying-large-directory-to-acd/185>\
**Category:** Suspected Bug\
**Created:** [November 3, 2016, 10:16am UTC](https://forum.rclone.org/t/rclone-aborting-while-copying-large-directory-to-acd/185 "2016-11-03T10:16:09Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![durval](https://forum.rclone.org/user_avatar/forum.rclone.org/durval/32/33_2.png) [@durval](https://forum.rclone.org/u/durval)\
**Post date:** [November 3, 2016, 10:16am UTC](https://forum.rclone.org/t/rclone-aborting-while-copying-large-directory-to-acd/185/1 "2016-11-03T10:16:09Z")

</div>

Hello,

I’m using rclone repeatedly here to copy a large local directory tree (actually a backup of my main machine) with a few million, mostly small files to an Amazon Cloud Drive encrypted remote, and it’s aborting after _hours_ of operation with the following:

```
2016/11/03 02:24:19 amazon drive root 'egt6ipr9ofpjnfac0587sqrd8g/dbcd091fet91ga6ortqn0dgimg': Finished reading "4c0aqobr2rnf998n2o0nvljhr4/b7ivnbt6enei8k3673
0qh687vk/pj4spvnq8hb1tkgjrh52ttq110/kmtlepfej8hhmdai6e4l5u2rbs/aioh54p49aips0g2496jvkvnpo/bkhi5bfokdq8kvq0k94a0s0nv4/4obet05vev4tvk8ksavfs333lk/jq1nml5humulhf
9sa244rb6uck/de39t1lmfngbd7phm82vft8mh4/m4qubodpigap8eki0lr0k15cgee8kna6ejbnne467ncb15l3jm80/8g3a37iqpramu6q7ek32qrldj4/5h53hnctf79a8ni4vb20u6b90g/lsn4njrfqht
nvpp839g2mcvl3c/u2d1faie3sfjbrf3h4esv9jlbc/"
2016/11/03 02:24:24 amazon drive root 'egt6ipr9ofpjnfac0587sqrd8g/dbcd091fet91ga6ortqn0dgimg': Error reading : HTTP code 502: "502 Bad Gateway", reponse body:
 <html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>We're sorry!</title>
</head>

<body style="padding: 4em; color: black; background-color: white; font-family: Verdana,Arial,Helvetica,sans-serif;">

<h2>We're sorry!</h2>
<p>
An error occurred when we tried to process your request.
Rest assured, we're already working on the problem and expect to resolve it shortly.
</p>

<h2>D<C3><A9>sol<C3><A9>s!</h2>
<p>
Une erreur s'est produite lorsque nous avons tent<C3><A9> de traiter votre requ<C3><AA>te.
Soyez assur<C3><A9> que nous travaillons d<C3><A9>j<C3><A0> <C3><A0> la r<C3><A9>solution du probl<C3><A8>me que nous pensons trouver tr<C3><A8>s rapidement.
</p>

</body>
</html>

2016/11/03 02:24:24 Attempt 3/3 failed with 0 errors and: HTTP code 502: "502 Bad Gateway", reponse body: <html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>We're sorry!</title>
</head>

<body style="padding: 4em; color: black; background-color: white; font-family: Verdana,Arial,Helvetica,sans-serif;">

<h2>We're sorry!</h2>
<p>
An error occurred when we tried to process your request.
Rest assured, we're already working on the problem and expect to resolve it shortly.
</p>

<h2>D<C3><A9>sol<C3><A9>s!</h2>
<p>
Une erreur s'est produite lorsque nous avons tent<C3><A9> de traiter votre requ<C3><AA>te.
Soyez assur<C3><A9> que nous travaillons d<C3><A9>j<C3><A0> <C3><A0> la r<C3><A9>solution du probl<C3><A8>me que nous pensons trouver tr<C3><A8>s rapidement.
</p>

</body>
</html>

2016/11/03 02:24:24 Failed to copy: HTTP code 502: "502 Bad Gateway", reponse body: <html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>We're sorry!</title>
</head>

<body style="padding: 4em; color: black; background-color: white; font-family: Verdana,Arial,Helvetica,sans-serif;">

<h2>We're sorry!</h2>
<p>
An error occurred when we tried to process your request.
Rest assured, we're already working on the problem and expect to resolve it shortly.
</p>

<h2>D<C3><A9>sol<C3><A9>s!</h2>
<p>
Une erreur s'est produite lorsque nous avons tent<C3><A9> de traiter votre requ<C3><AA>te.
Soyez assur<C3><A9> que nous travaillons d<C3><A9>j<C3><A0> <C3><A0> la r<C3><A9>solution du probl<C3><A8>me que nous pensons trouver tr<C3><A8>s rapidement.
</p>

</body>
</html>

```

I understand that this is happening because Amazon is returning repeated “502 Bad Gateway” errors as logged above.

How can I make rclone _not_ abort in such a scenario. but rather take a delay and try again, repeteadly? I already have about 1M files in the ACD remote directory, and it takes rclone _hours_ just re-reading it to abort when it meets such an error… :-/

Perhaps I should increase one of the “retries” options? In this case, which one?

## Thanks,

Durval.

---

<div class="post-metadata">

**Author:** ![ncw](https://forum.rclone.org/user_avatar/forum.rclone.org/ncw/32/18_2.png) [@ncw](https://forum.rclone.org/u/ncw)\
**Post date:** [November 3, 2016, 6:58pm UTC](https://forum.rclone.org/t/rclone-aborting-while-copying-large-directory-to-acd/185/2 "2016-11-03T18:58:30Z")

</div>

> [@durval](#):
>
> Perhaps I should increase one of the "retries" options? In this case, which one?

502 isn't an error message we normally retry on.... I've added this in

[http://beta.rclone.org/v1.33-99-g452c681/](http://beta.rclone.org/v1.33-99-g452c681/) (will be uploaded in 15-20 mins)

You may not need to change `--low-level-retries` with that but that might help too.

---

<div class="post-metadata">

**Author:** ![Wowfunhappy](https://forum.rclone.org/user_avatar/forum.rclone.org/wowfunhappy/32/39_2.png) [@Wowfunhappy](https://forum.rclone.org/u/Wowfunhappy)\
**Post date:** [November 3, 2016, 7:29pm UTC](https://forum.rclone.org/t/rclone-aborting-while-copying-large-directory-to-acd/185/3 "2016-11-03T19:29:23Z")

</div>

This has happened to me before. It usually ocurrs within the first few minutes of starting rclone, and goes away if I wait a couple of hours before retrying. If I don’t wait, or wait only a couple of minutes, the error persists.

I’ve always assumed it was a problem on Amazon’s end.

---

<div class="post-metadata">

**Author:** ![durval](https://forum.rclone.org/user_avatar/forum.rclone.org/durval/32/33_2.png) [@durval](https://forum.rclone.org/u/durval)\
**Post date:** [November 3, 2016, 7:41pm UTC](https://forum.rclone.org/t/rclone-aborting-while-copying-large-directory-to-acd/185/4 "2016-11-03T19:41:17Z")

</div>

Hello Nick,

> [@ncw](#):
>
> 502 isn't an error message we normally retry on.... I've added this in  
> [http://beta.rclone.org/v1.33-99-g452c681/](http://beta.rclone.org/v1.33-99-g452c681/) (will be uploaded in 15-20 mins)

Thanks for the prompt response! I will be sure to run it the next time the currently running rclone aborts (still not even 50% to finish, so it will almost certainly abort again).

> [@](#):
>
> You may not need to change --low-level-retries with that but that might help too.

Will do. Would "--low-level-retries=10" be adequate, or would you recommend a different value?

## Cheers,

Durval.

---

<div class="post-metadata">

**Author:** ![ncw](https://forum.rclone.org/user_avatar/forum.rclone.org/ncw/32/18_2.png) [@ncw](https://forum.rclone.org/u/ncw)\
**Post date:** [November 3, 2016, 8:54pm UTC](https://forum.rclone.org/t/rclone-aborting-while-copying-large-directory-to-acd/185/5 "2016-11-03T20:54:53Z")

</div>

> [@durval](#):
>
> Will do. Would "--low-level-retries=10" be adequate, or would you recommend a different value?

It reallly depends on how long you get that `502` error. 10 is usually sufficient, but try with the `-v` flag and watch the retries and see what happens.

---

<div class="post-metadata">

**Author:** ![durval](https://forum.rclone.org/user_avatar/forum.rclone.org/durval/32/33_2.png) [@durval](https://forum.rclone.org/u/durval)\
**Post date:** [November 3, 2016, 9:00pm UTC](https://forum.rclone.org/t/rclone-aborting-while-copying-large-directory-to-acd/185/6 "2016-11-03T21:00:35Z")

</div>

Hello Nick,

> [@ncw](#):
>
> It reallly depends on how long you get that 502 error. 10 is usually sufficient, but try with the -v flag and watch the retries and see what happens.

Makes sense. Using 10 for now, will monitor and report my results back here, for those that will come a-googling...

## Cheers,

Durval.

---

<div class="post-metadata">

**Author:** ![ladude626](https://forum.rclone.org/letter_avatar_proxy/v4/letter/l/e99b99/32.png) [@ladude626](https://forum.rclone.org/u/ladude626)\
**Post date:** [November 4, 2016, 1:17am UTC](https://forum.rclone.org/t/rclone-aborting-while-copying-large-directory-to-acd/185/7 "2016-11-04T01:17:37Z")

</div>

Any update with usage of this beta? I was getting a lot of errors uploading large files before too. I consumed 6TB of upload, only to see 3.5TB of files uploaded to ACD due to multiple large file failing.

---

<div class="post-metadata">

**Author:** ![ladude626](https://forum.rclone.org/letter_avatar_proxy/v4/letter/l/e99b99/32.png) [@ladude626](https://forum.rclone.org/u/ladude626)\
**Post date:** [November 4, 2016, 3:26am UTC](https://forum.rclone.org/t/rclone-aborting-while-copying-large-directory-to-acd/185/8 "2016-11-04T03:26:48Z")

</div>

This beta isn’t working so well for me, getting tons of ‘error HTTP code 500: “500 Internal Server Error”, response body: {“message”:“Internal failure”}’ when uploading files \>1GB.

Had to revert back to v1.33-93-g5c96781-acd-range-fixβ/

---

<div class="post-metadata">

**Author:** ![ncw](https://forum.rclone.org/user_avatar/forum.rclone.org/ncw/32/18_2.png) [@ncw](https://forum.rclone.org/u/ncw)\
**Post date:** [November 4, 2016, 9:47am UTC](https://forum.rclone.org/t/rclone-aborting-while-copying-large-directory-to-acd/185/9 "2016-11-04T09:47:52Z")

</div>

> [@ladude626](#):
>
> This beta isn't working so well for me, getting tons of 'error HTTP code 500: "500 Internal Server Error", response body: {"message":"Internal failure"}' when uploading files \>1GB.
> 
> Had to revert back to v1.33-93-g5c96781-acd-range-fixβ/

I think that is probably amazon and rather than the beta. Can you try again with [http://beta.rclone.org/v1.33-99-g452c681/](http://beta.rclone.org/v1.33-99-g452c681/) ?

---

<div class="post-metadata">

**Author:** ![durval](https://forum.rclone.org/user_avatar/forum.rclone.org/durval/32/33_2.png) [@durval](https://forum.rclone.org/u/durval)\
**Post date:** [November 4, 2016, 10:45pm UTC](https://forum.rclone.org/t/rclone-aborting-while-copying-large-directory-to-acd/185/10 "2016-11-04T22:45:45Z")

</div>

Hello @ncw, everyone,

> [@durval](#):
>
> (re: --low-level-retries=10)  
> Using 10 for now, will monitor and report my results back here, for those that will come a-googling...

My latest transfer (with the beta version you recommended above) has been going on non-stop for over 36h now, and has so far transferred 624GB and 285K files, not a single abort.

Reviewing the "-v" log, I saw just three "502 Bad Gateway" errors happening around 20:18, 20:24 and 20:37 UTC today, but all the three errors were in different files, so I didn't get to test your modification yet.

Seems Amazon got its act together for now; I still have a _lot_ of data to upload during the next few weeks, so can you please tell whether this modification has been incorporated on rclone master, or should I keep using the above beta version for now and refrain from updating rclone until I get a large cluster of those errors to be able to test it out?

## Thanks,

Durval.

---

<div class="post-metadata">

**Author:** ![ladude626](https://forum.rclone.org/letter_avatar_proxy/v4/letter/l/e99b99/32.png) [@ladude626](https://forum.rclone.org/u/ladude626)\
**Post date:** [November 5, 2016, 2:03pm UTC](https://forum.rclone.org/t/rclone-aborting-while-copying-large-directory-to-acd/185/11 "2016-11-05T14:03:54Z")

</div>

Ok upload seems to be working better now. Amazon must have fixed something on their end.

---

<div class="post-metadata">

**Author:** ![durval](https://forum.rclone.org/user_avatar/forum.rclone.org/durval/32/33_2.png) [@durval](https://forum.rclone.org/u/durval)\
**Post date:** [November 6, 2016, 11:38am UTC](https://forum.rclone.org/t/rclone-aborting-while-copying-large-directory-to-acd/185/12 "2016-11-06T11:38:16Z")

</div>

Hello,

Answering my own question, to those that may come a-googling:

> [@durval](#):
>
> can you please tell whether this modification has been incorporated on rclone master, or should I keep using the above beta version for now

Apparently the newest "stable" version incorporates it, and so it should become a "common fixture" for the time being:

> [@Rclone release v1.34](https://forum.rclone.org/t/rclone-release-v1-34/197/1):
>
> Add 502 Bad Gateway to list of errors we retry

On a related note: adding the ` --low-level-retries` option as recommended above had the unfortunate side effect of also adding numerous and useless retries to other errors, specially from long-sized dir and file names, see [On-the-fly encryption support · Issue #219 · rclone/rclone · GitHub](https://github.com/ncw/rclone/issues/219#issuecomment-253397112) . This has caused my transfers to stall completely for many _hours_ when rclone stumbled upon a long-size-named directory containing lots of files... :-/ So I'm redoing it but specifying `--exclude="????...????*"` (with 175 '?' characters) in the command line as @ncw recommended here: [Skip long titled folder? --max-name-length · Issue #762 · rclone/rclone · GitHub](https://github.com/ncw/rclone/issues/762#issuecomment-251376745)

EDIT: the limit that worked for me was 160 characters, not 175; see my comment here: [Skip long titled folder? --max-name-length · Issue #762 · rclone/rclone · GitHub](https://github.com/ncw/rclone/issues/762#issuecomment-258724572)

## Cheers,

Durval.

---

<div class="post-metadata">

**Author:** ![system](https://forum.rclone.org/uploads/default/original/2X/3/39ad3235f6ffec76ccd0c81616bd55697b4c300d.png) [@system](https://forum.rclone.org/u/system)\
**Post date:** [March 4, 2020, 12:34am UTC](https://forum.rclone.org/t/rclone-aborting-while-copying-large-directory-to-acd/185/13 "2020-03-04T00:34:22Z")

</div>

Closed on March 4, 2020, 12:34am UTC


