Copy process is hanging - S3

Help and Support
1

Helllo,

I'm trying to copy an object between S3 compliant storages. That object is 10kb and no copying any bits from the source storage.
Object properties: 2022/04/10 19:47:05 10035 object.101

Rclone is not failing and trying to copy for endless time... after 30 mintues I killed the process. It is the same if I try to just copy to the local filesystem. I expect a switch or a workarround that the copy process is hanging for 5 seconds, then it can be marked as failed.

rclone v1.60.0

  • os/version: ubuntu 20.04 (64 bit)
  • os/kernel: 4.4.0-17763-Microsoft (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.19.2
  • go/linking: static
  • go/tags: none

Rclone command and output:

rclone -vv --no-check-certificate --checksum --progress copy --no-traverse S3storage:pro-bucket/object.101 .
2022/11/09 22:49:27 DEBUG : rclone: Version "v1.60.0" starting with parameters ["rclone" "-vv" "--no-check-certificate" "--checksum" "--progress" "copy" "--no-traverse" "S3storage:pro-bucket/object.101" "."]
2022/11/09 22:49:27 DEBUG : Creating backend with remote "S3storage:pro-bucketobject.101"
2022/11/09 22:49:27 DEBUG : Using config file from "/home/ghorvath/.config/rclone/rclone.conf"
2022/11/09 22:49:27 DEBUG : fs cache: adding new entry for parent of "S3storage:pro-bucket/object.101", "S3storage:pro-bucket"
2022/11/09 22:49:27 DEBUG : Creating backend with remote "."
2022/11/09 22:49:27 DEBUG : fs cache: renaming cache item "." to be canonical "/mnt/c/Users/ghorvath"
2022-11-09 22:49:27 DEBUG : object.101: Need to transfer - File not found at Destination
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Transferred:            0 / 1, 0%
Elapsed time:       4m8.0s
Transferring:
 *      object.101: transferring^C

The rclone config:

[S3storage]
type = s3
provider = Other
access_key_id = ...
secret_access_key = ...
endpoint = ...
acl = private
v2_auth = false
disable_http2 = false
no_check_bucket = true

I've alredy tried the flags following flags, but there is no changes:
--timeout 5s
--contimeout 5s

I tried to copy with s3cmd, and I got the folling output:

download: 's3://pro-bucket/object.101' -> './object.101'  [1 of 1]
ERROR: S3 error: 503 (ServiceUnavailable): The server is temporarily unavailable. Please try again later.
2

hi, so the file is very small 10KB - approx. 10,000 bytes?

what is the name of the provider?

but you command is copying from s3 to local current path, correct?
rclone copy S3storage:pro-bucket/object.101 .

3

Hi asdffdsa,

Yes the file is is approx 10kb -> 10035
The provider is other because it is not listed in the provider list: 24 / Any other S3 compatible provider
Yes, I'm copying an object from S3 to local in that case, but I got the same issue if I try to copy to an antoher S3 storage. With that one I want to eliminate the complexity to other issues.

Of course I'm able to copy other objects from the S3 storage to local fs, only this objects has the issue and I need to handle it.

4

sorry, not sure i understand
so you have an issue with only one single file?

what is the name of the provider, where is the s3 hosted from?

5

Yes, I have millions of the file, and I'm copying these file to an another S3 storage but at the first 100k ob object I got this file which is hangig the copy process. I don't know how many issues have in the future, thats the reason why looking for a handling to this issue.

The source is a Cloudian storage and the target is a FlashBlade.

6

--- so the issue also happens with s3cmd and looks like a server issue
The server is temporarily unavailable

--- does the same file fail each time, or random file(s) each time?

--- might add --dump=headers to get a more detailed look.

8

It is only at the same file. If I try to copy an another file, it work fine.

Dump:

rclone -vv --no-check-certificate --checksum --progress --dump headers copy --no-traverse S3storage:pro-blob2/object.101 .
2022/11/09 23:40:04 DEBUG : rclone: Version "v1.60.0" starting with parameters ["rclone" "-vv" "--no-check-certificate" "--checksum" "--progress" "--dump" "headers" "copy" "--no-traverse" "S3storage:pro-blob2/object.101" "."]
2022/11/09 23:40:04 DEBUG : Creating backend with remote "S3storage:pro-blob2/object.101"
2022/11/09 23:40:04 DEBUG : Using config file from "/home/ghorvath/.config/rclone/rclone.conf"
2022/11/09 23:40:04 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2022/11/09 23:40:04 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2022/11/09 23:40:05 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/11/09 23:40:05 DEBUG : HTTP REQUEST (req 0xc000900a00)
2022/11/09 23:40:05 DEBUG : HEAD /pro-blob2/object.101 HTTP/1.1
Host: s3-S3storage.fastcloud.fwb
User-Agent: rclone/v1.60.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20221109T224004Z

2022/11/09 23:40:05 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/11/09 23:40:05 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/11/09 23:40:05 DEBUG : HTTP RESPONSE (req 0xc000900a00)
2022/11/09 23:40:05 DEBUG : HTTP/1.1 200 OK
Content-Length: 10035
Accept-Ranges: bytes
Content-Type: binary/octet-stream
Date: Wed, 09 Nov 2022 22:40:04 GMT
Etag: "80509fa2b370edc141e93430edd2a9eb"
Last-Modified: Sun, 10 Apr 2022 19:47:05 GMT
Server: S3storageS3
Strict-Transport-Security: max-age=31536000
X-Amz-Meta-2: 26911
X-Amz-Meta-3: 26911
X-Amz-Meta-4: ZStd_1
X-Amz-Meta-5: 2474
X-Amz-Meta-Crc-Range: bytes=0-2473
X-Amz-Request-Id: 404ca83a-ff29-1fff-85c2-d8c497780c73

2022/11/09 23:40:05 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/11/09 23:40:05 DEBUG : fs cache: adding new entry for parent of "S3storage:pro-blob2/object.101", "S3storage:pro-blob2"
2022/11/09 23:40:05 DEBUG : Creating backend with remote "."
2022/11/09 23:40:05 DEBUG : fs cache: renaming cache item "." to be canonical "/mnt/c/Users/ghorvath"
2022-11-09 23:40:05 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022-11-09 23:40:05 DEBUG : HTTP REQUEST (req 0xc000a2c200)
2022-11-09 23:40:05 DEBUG : HEAD /pro-blob2/object.101 HTTP/1.1
Host: s3-S3storage.fastcloud.fwb
User-Agent: rclone/v1.60.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20221109T224005Z
2022-11-09 23:40:05 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022-11-09 23:40:05 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022-11-09 23:40:05 DEBUG : HTTP RESPONSE (req 0xc000a2c200)
2022-11-09 23:40:05 DEBUG : HTTP/1.1 200 OK
Content-Length: 10035
Accept-Ranges: bytes
Content-Type: binary/octet-stream
Date: Wed, 09 Nov 2022 22:40:04 GMT
Etag: "80509fa2b370edc141e93430edd2a9eb"
Last-Modified: Sun, 10 Apr 2022 19:47:05 GMT
Server: S3storageS3
Strict-Transport-Security: max-age=31536000
X-Amz-Meta-2: 26911
X-Amz-Meta-3: 26911
X-Amz-Meta-4: ZStd_1
X-Amz-Meta-5: 2474
X-Amz-Meta-Crc-Range: bytes=0-2473
X-Amz-Request-Id: 404ca83c-ff29-1fff-85c2-d8c497780c73
2022-11-09 23:40:05 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022-11-09 23:40:05 DEBUG : object.101: Need to transfer - File not found at Destination
2022-11-09 23:40:05 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022-11-09 23:40:05 DEBUG : HTTP REQUEST (req 0xc000900400)
2022-11-09 23:40:05 DEBUG : GET /pro-blob2/object.101 HTTP/1.1
Host: s3-S3storage.fastcloud.fwb
User-Agent: rclone/v1.60.0
Accept-Encoding: gzip
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20221109T224005Z
2022-11-09 23:40:05 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022-11-09 23:40:05 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022-11-09 23:40:05 DEBUG : HTTP RESPONSE (req 0xc000900400)
2022-11-09 23:40:05 DEBUG : HTTP/1.1 503 Service Unavailable
Content-Length: 273
Accept-Ranges: bytes
Content-Type: application/xml
Date: Wed, 09 Nov 2022 22:40:04 GMT
Etag: "80509fa2b370edc141e93430edd2a9eb"
Last-Modified: Sun, 10 Apr 2022 19:47:05 GMT
Server: S3storageS3
Strict-Transport-Security: max-age=31536000
X-Amz-Meta-2: 26911
X-Amz-Meta-3: 26911
X-Amz-Meta-4: ZStd_1
X-Amz-Meta-5: 2474
X-Amz-Meta-Crc-Range: bytes=0-2473
X-Amz-Request-Id: 404ca83e-ff29-1fff-85c2-d8c497780c73
X-Gmt-Error-Code: ServiceUnavailable
X-Gmt-Message: The server is temporarily unavailable. Please try again later.
2022-11-09 23:40:05 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022-11-09 23:40:06 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022-11-09 23:40:06 DEBUG : HTTP REQUEST (req 0xc000a2c400)
2022-11-09 23:40:06 DEBUG : GET /pro-blob2/object.101 HTTP/1.1
Host: s3-S3storage.fastcloud.fwb
User-Agent: rclone/v1.60.0
Accept-Encoding: gzip
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20221109T224006Z
2022-11-09 23:40:06 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022-11-09 23:40:06 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022-11-09 23:40:06 DEBUG : HTTP RESPONSE (req 0xc000a2c400)
2022-11-09 23:40:06 DEBUG : HTTP/1.1 503 Service Unavailable
Content-Length: 273
Accept-Ranges: bytes
Content-Type: application/xml
Date: Wed, 09 Nov 2022 22:40:05 GMT
Etag: "80509fa2b370edc141e93430edd2a9eb"
Last-Modified: Sun, 10 Apr 2022 19:47:05 GMT
Server: S3storageS3
Strict-Transport-Security: max-age=31536000
X-Amz-Meta-2: 26911
X-Amz-Meta-3: 26911
X-Amz-Meta-4: ZStd_1
X-Amz-Meta-5: 2474
X-Amz-Meta-Crc-Range: bytes=0-2473
X-Amz-Request-Id: 404ca848-ff29-1fff-85c2-d8c497780c73
X-Gmt-Error-Code: ServiceUnavailable
X-Gmt-Message: The server is temporarily unavailable. Please try again later.
2022-11-09 23:40:06 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022-11-09 23:40:08 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022-11-09 23:40:08 DEBUG : HTTP REQUEST (req 0xc000a2c800)
2022-11-09 23:40:08 DEBUG : GET /pro-blob2/object.101 HTTP/1.1
Host: s3-S3storage.fastcloud.fwb
User-Agent: rclone/v1.60.0
Accept-Encoding: gzip
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20221109T224008Z
2022-11-09 23:40:08 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022-11-09 23:40:08 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022-11-09 23:40:08 DEBUG : HTTP RESPONSE (req 0xc000a2c800)
2022-11-09 23:40:08 DEBUG : HTTP/1.1 503 Service Unavailable
Content-Length: 273
Accept-Ranges: bytes
Content-Type: application/xml
Date: Wed, 09 Nov 2022 22:40:07 GMT
Etag: "80509fa2b370edc141e93430edd2a9eb"
Last-Modified: Sun, 10 Apr 2022 19:47:05 GMT
Server: S3storageS3
Strict-Transport-Security: max-age=31536000
X-Amz-Meta-2: 26911
X-Amz-Meta-3: 26911
X-Amz-Meta-4: ZStd_1
X-Amz-Meta-5: 2474
X-Amz-Meta-Crc-Range: bytes=0-2473
X-Amz-Request-Id: 404ca850-ff29-1fff-85c2-d8c497780c73
X-Gmt-Error-Code: ServiceUnavailable
X-Gmt-Message: The server is temporarily unavailable. Please try again later.
2022-11-09 23:40:08 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Transferred:            0 / 1, 0%
Elapsed time:         4.0s
Transferring:
 *      object.101: transferring^C
9

can you post the same log but for a file that does transfer?

fwiw, seen this before recently here in the forum,
where i asked the OP to delete that single file, re-upload it and the problem went away.

10

Yes, this is the log:

rclone -vv --no-check-certificate --checksum --progress --dump headers copy --no-traverse S3storage:pro-blob2/object.102 .
2022/11/09 23:53:02 DEBUG : rclone: Version "v1.60.0" starting with parameters ["rclone" "-vv" "--no-check-certificate" "--checksum" "--progress" "--dump" "headers" "copy" "--no-traverse" "S3storage:pro-blob2/object.102" "."]
2022/11/09 23:53:02 DEBUG : Creating backend with remote "S3storage:pro-blob2/object.102"
2022/11/09 23:53:02 DEBUG : Using config file from "/home/ghorvath/.config/rclone/rclone.conf"
2022/11/09 23:53:02 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2022/11/09 23:53:02 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2022/11/09 23:53:02 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/11/09 23:53:02 DEBUG : HTTP REQUEST (req 0xc0008c8e00)
2022/11/09 23:53:02 DEBUG : HEAD /pro-blob2/object.102 HTTP/1.1
Host: s3-S3storage.fastcloud.fwb
User-Agent: rclone/v1.60.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20221109T225302Z

2022/11/09 23:53:02 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/11/09 23:53:02 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/11/09 23:53:02 DEBUG : HTTP RESPONSE (req 0xc0008c8e00)
2022/11/09 23:53:02 DEBUG : HTTP/1.1 200 OK
Content-Length: 1064061
Accept-Ranges: bytes
Content-Type: binary/octet-stream
Date: Wed, 09 Nov 2022 22:53:02 GMT
Etag: "2405635d68f4f156efebc55b6864eac3"
Last-Modified: Wed, 25 Nov 2020 19:54:16 GMT
Server: S3storageS3
Strict-Transport-Security: max-age=31536000
X-Amz-Meta-2: 1062574
X-Amz-Meta-3: 1062574
X-Amz-Meta-4: QuickLZ_1.5.0
X-Amz-Meta-5: 1478
X-Amz-Meta-Crc-Range: bytes=0-1477
X-Amz-Request-Id: 404cf3e4-ff29-1fff-85c2-d8c497780c73

2022/11/09 23:53:02 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/11/09 23:53:02 DEBUG : fs cache: adding new entry for parent of "S3storage:pro-blob2/object.102", "S3storage:pro-blob2"
2022/11/09 23:53:02 DEBUG : Creating backend with remote "."
2022/11/09 23:53:02 DEBUG : fs cache: renaming cache item "." to be canonical "/mnt/c/Users/ghorvath"
2022-11-09 23:53:02 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022-11-09 23:53:02 DEBUG : HTTP REQUEST (req 0xc00050ee00)
2022-11-09 23:53:02 DEBUG : HEAD /pro-blob2/object.102 HTTP/1.1
Host: s3-S3storage.fastcloud.fwb
User-Agent: rclone/v1.60.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20221109T225302Z
2022-11-09 23:53:02 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022-11-09 23:53:02 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022-11-09 23:53:02 DEBUG : HTTP RESPONSE (req 0xc00050ee00)
2022-11-09 23:53:02 DEBUG : HTTP/1.1 200 OK
Content-Length: 1064061
Accept-Ranges: bytes
Content-Type: binary/octet-stream
Date: Wed, 09 Nov 2022 22:53:02 GMT
Etag: "2405635d68f4f156efebc55b6864eac3"
Last-Modified: Wed, 25 Nov 2020 19:54:16 GMT
Server: S3storageS3
Strict-Transport-Security: max-age=31536000
X-Amz-Meta-2: 1062574
X-Amz-Meta-3: 1062574
X-Amz-Meta-4: QuickLZ_1.5.0
X-Amz-Meta-5: 1478
X-Amz-Meta-Crc-Range: bytes=0-1477
X-Amz-Request-Id: 404cf3e8-ff29-1fff-85c2-d8c497780c73
2022-11-09 23:53:02 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022-11-09 23:53:02 DEBUG : object.102: Need to transfer - File not found at Destination
2022-11-09 23:53:02 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022-11-09 23:53:02 DEBUG : HTTP REQUEST (req 0xc0009be000)
2022-11-09 23:53:02 DEBUG : GET /pro-blob2/object.102 HTTP/1.1
Host: s3-S3storage.fastcloud.fwb
User-Agent: rclone/v1.60.0
Accept-Encoding: gzip
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20221109T225302Z
2022-11-09 23:53:02 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022-11-09 23:53:02 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022-11-09 23:53:02 DEBUG : HTTP RESPONSE (req 0xc0009be000)
2022-11-09 23:53:02 DEBUG : HTTP/1.1 200 OK
Content-Length: 1064061
Accept-Ranges: bytes
Content-Type: binary/octet-stream
Date: Wed, 09 Nov 2022 22:53:02 GMT
Etag: "2405635d68f4f156efebc55b6864eac3"
Last-Modified: Wed, 25 Nov 2020 19:54:16 GMT
Server: S3storageS3
Strict-Transport-Security: max-age=31536000
X-Amz-Meta-2: 1062574
X-Amz-Meta-3: 1062574
X-Amz-Meta-4: QuickLZ_1.5.0
X-Amz-Meta-5: 1478
X-Amz-Meta-Crc-Range: bytes=0-1477
X-Amz-Request-Id: 404cf3ee-ff29-1fff-85c2-d8c497780c73
2022-11-09 23:53:02 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022-11-09 23:53:03 DEBUG : object.102: md5 = 2405635d68f4f156efebc55b6864eac3 OK
2022-11-09 23:53:03 INFO  : object.102: Copied (new)
Transferred:        1.015 MiB / 1.015 MiB, 100%, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         0.3s
2022/11/09 23:53:03 INFO  :
Transferred:        1.015 MiB / 1.015 MiB, 100%, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         0.3s

2022/11/09 23:53:03 DEBUG : 5 go routines active
11

using provider website, can you download /pro-blob2/object.101?

12

I have no acess to that.

13

The server is temporarily unavailable
the server fails the same error with rclone and s3cmd.

as a possible workaround might try
--exclude='/pro-blob2/object.101'

14

Unfortuantely this is not helps me because, I don't know how many objects produces the same, and if I just copy a buch of objects 50-60 million which takes more days, and only 1 objects hangs the copy process.

15

I don't understand why this is hanging the copy process? It should try --low-level-retries times and then continue.

By the look of your log, these retries are being done in the AWS SDK rather than by rclone, so I wonder if there is a bug in there.

If you run it with -vv --dump headers how many times does it try the file?

Try setting --low-level-retries 1 and see what happens.

From looking at your log above, the retries happen pretty quickly so rclone should move on to the next object quite quicky.

16

Hi Nick,

The --low-level-retries worked fine.

rclone -vv --no-check-certificate --checksum --progress --low-level-retries 1 copy --no-traverse S3storage:pro-bucket/object.101 .
2022/11/11 13:38:20 DEBUG : rclone: Version "v1.60.0" starting with parameters ["rclone" "-vv" "--no-check-certificate" "--checksum" "--progress" "--low-level-retries" "1" "copy" "--no-traverse" "S3storage:pro-bucket/object.101" "."]
2022/11/11 13:38:20 DEBUG : Creating backend with remote "S3storage:pro-bucket/object.101"
2022/11/11 13:38:20 DEBUG : Using config file from "/home/ghorvath/.config/rclone/rclone.conf"
2022/11/11 13:38:20 DEBUG : fs cache: adding new entry for parent of "S3storage:pro-bucket/object.101", "S3storage:pro-bucket"
2022/11/11 13:38:20 DEBUG : Creating backend with remote "."
2022/11/11 13:38:20 DEBUG : fs cache: renaming cache item "." to be canonical "/mnt/c/Users/ghorvath"
2022-11-11 13:38:20 DEBUG : object.101: Need to transfer - File not found at Destination
2022-11-11 13:38:21 DEBUG : pacer: low level retry 1/2 (error ServiceUnavailable: The server is temporarily unavailable. Please try again later.
        status code: 503, request id: 409e424c-ff29-1fff-85c2-d8c497780c73, host id: )
2022-11-11 13:38:21 DEBUG : pacer: Rate limited, increasing sleep to 10ms
2022-11-11 13:38:21 DEBUG : pacer: low level retry 2/2 (error ServiceUnavailable: The server is temporarily unavailable. Please try again later.
        status code: 503, request id: 409e424c-ff29-1fff-85c2-d8c497780c73, host id: )
2022-11-11 13:38:21 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2022-11-11 13:38:21 ERROR : object.101: Failed to copy: failed to open source object: ServiceUnavailable: The server is temporarily unavailable. Please try again later.
        status code: 503, request id: 409e424c-ff29-1fff-85c2-d8c497780c73, host id:
2022-11-11 13:38:21 ERROR : Attempt 1/3 failed with 1 errors and: failed to open source object: ServiceUnavailable: The server is temporarily unavailable. Please try again later.
        status code: 503, request id: 409e424c-ff29-1fff-85c2-d8c497780c73, host id:
2022-11-11 13:38:21 DEBUG : pacer: Reducing sleep to 15ms
2022-11-11 13:38:21 DEBUG : object.101: Need to transfer - File not found at Destination
2022-11-11 13:38:22 DEBUG : pacer: low level retry 1/2 (error ServiceUnavailable: The server is temporarily unavailable. Please try again later.
        status code: 503, request id: 409e427e-ff29-1fff-85c2-d8c497780c73, host id: )
2022-11-11 13:38:22 DEBUG : pacer: Rate limited, increasing sleep to 30ms
2022-11-11 13:38:22 DEBUG : pacer: low level retry 2/2 (error ServiceUnavailable: The server is temporarily unavailable. Please try again later.
        status code: 503, request id: 409e427e-ff29-1fff-85c2-d8c497780c73, host id: )
2022-11-11 13:38:22 DEBUG : pacer: Rate limited, increasing sleep to 60ms
2022-11-11 13:38:22 ERROR : object.101: Failed to copy: failed to open source object: ServiceUnavailable: The server is temporarily unavailable. Please try again later.
        status code: 503, request id: 409e427e-ff29-1fff-85c2-d8c497780c73, host id:
2022-11-11 13:38:22 ERROR : Attempt 2/3 failed with 1 errors and: failed to open source object: ServiceUnavailable: The server is temporarily unavailable. Please try again later.
        status code: 503, request id: 409e427e-ff29-1fff-85c2-d8c497780c73, host id:
2022-11-11 13:38:22 DEBUG : pacer: Reducing sleep to 45ms
2022-11-11 13:38:22 DEBUG : object.101: Need to transfer - File not found at Destination
2022-11-11 13:38:23 DEBUG : pacer: low level retry 1/2 (error ServiceUnavailable: The server is temporarily unavailable. Please try again later.
        status code: 503, request id: 409e42a4-ff29-1fff-85c2-d8c497780c73, host id: )
2022-11-11 13:38:23 DEBUG : pacer: Rate limited, increasing sleep to 90ms
2022-11-11 13:38:23 DEBUG : pacer: low level retry 2/2 (error ServiceUnavailable: The server is temporarily unavailable. Please try again later.
        status code: 503, request id: 409e42a4-ff29-1fff-85c2-d8c497780c73, host id: )
2022-11-11 13:38:23 DEBUG : pacer: Rate limited, increasing sleep to 180ms
2022-11-11 13:38:23 ERROR : object.101: Failed to copy: failed to open source object: ServiceUnavailable: The server is temporarily unavailable. Please try again later.
        status code: 503, request id: 409e42a4-ff29-1fff-85c2-d8c497780c73, host id:
2022-11-11 13:38:23 ERROR : Attempt 3/3 failed with 1 errors and: failed to open source object: ServiceUnavailable: The server is temporarily unavailable. Please try again later.
        status code: 503, request id: 409e42a4-ff29-1fff-85c2-d8c497780c73, host id:
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         3.0s
2022/11/11 13:38:23 INFO  :
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         3.0s

2022/11/11 13:38:23 DEBUG : 6 go routines active
2022/11/11 13:38:23 Failed to copy: failed to open source object: ServiceUnavailable: The server is temporarily unavailable. Please try again later.
        status code: 503, request id: 409e42a4-ff29-1fff-85c2-d8c497780c73, host id:

the -vv --dump headers tries during the process is running, Just the wait duration will increased.

Thanks for the help!

1 Like
17

Great.

That will let you do the copy and then afterwards you can use rclone check to make lists of the objects missing.

18

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.