Copy process is hanging - S3

Helllo,

I'm trying to copy an object between S3 compliant storages. That object is 10kb and no copying any bits from the source storage.
Object properties: 2022/04/10 19:47:05 10035 object.101

Rclone is not failing and trying to copy for endless time... after 30 mintues I killed the process. It is the same if I try to just copy to the local filesystem. I expect a switch or a workarround that the copy process is hanging for 5 seconds, then it can be marked as failed.

rclone v1.60.0

  • os/version: ubuntu 20.04 (64 bit)
  • os/kernel: 4.4.0-17763-Microsoft (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.19.2
  • go/linking: static
  • go/tags: none

Rclone command and output:

rclone -vv --no-check-certificate --checksum --progress copy --no-traverse S3storage:pro-bucket/object.101 .
2022/11/09 22:49:27 DEBUG : rclone: Version "v1.60.0" starting with parameters ["rclone" "-vv" "--no-check-certificate" "--checksum" "--progress" "copy" "--no-traverse" "S3storage:pro-bucket/object.101" "."]
2022/11/09 22:49:27 DEBUG : Creating backend with remote "S3storage:pro-bucketobject.101"
2022/11/09 22:49:27 DEBUG : Using config file from "/home/ghorvath/.config/rclone/rclone.conf"
2022/11/09 22:49:27 DEBUG : fs cache: adding new entry for parent of "S3storage:pro-bucket/object.101", "S3storage:pro-bucket"
2022/11/09 22:49:27 DEBUG : Creating backend with remote "."
2022/11/09 22:49:27 DEBUG : fs cache: renaming cache item "." to be canonical "/mnt/c/Users/ghorvath"
2022-11-09 22:49:27 DEBUG : object.101: Need to transfer - File not found at Destination
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Transferred:            0 / 1, 0%
Elapsed time:       4m8.0s
Transferring:
 *      object.101: transferring^C

The rclone config:

[S3storage]
type = s3
provider = Other
access_key_id = ...
secret_access_key = ...
endpoint = ...
acl = private
v2_auth = false
disable_http2 = false
no_check_bucket = true

I've alredy tried the flags following flags, but there is no changes:
--timeout 5s
--contimeout 5s

I tried to copy with s3cmd, and I got the folling output:

download: 's3://pro-bucket/object.101' -> './object.101'  [1 of 1]
ERROR: S3 error: 503 (ServiceUnavailable): The server is temporarily unavailable. Please try again later.

hi, so the file is very small 10KB - approx. 10,000 bytes?

what is the name of the provider?

but you command is copying from s3 to local current path, correct?
rclone copy S3storage:pro-bucket/object.101 .

Hi asdffdsa,

Yes the file is is approx 10kb -> 10035
The provider is other because it is not listed in the provider list: 24 / Any other S3 compatible provider
Yes, I'm copying an object from S3 to local in that case, but I got the same issue if I try to copy to an antoher S3 storage. With that one I want to eliminate the complexity to other issues.

Of course I'm able to copy other objects from the S3 storage to local fs, only this objects has the issue and I need to handle it.

sorry, not sure i understand
so you have an issue with only one single file?

what is the name of the provider, where is the s3 hosted from?

Yes, I have millions of the file, and I'm copying these file to an another S3 storage but at the first 100k ob object I got this file which is hangig the copy process. I don't know how many issues have in the future, thats the reason why looking for a handling to this issue.

The source is a Cloudian storage and the target is a FlashBlade.

--- so the issue also happens with s3cmd and looks like a server issue
The server is temporarily unavailable

--- does the same file fail each time, or random file(s) each time?

--- might add --dump=headers to get a more detailed look.

It is only at the same file. If I try to copy an another file, it work fine.

Dump:

rclone -vv --no-check-certificate --checksum --progress --dump headers copy --no-traverse S3storage:pro-blob2/object.101 .
2022/11/09 23:40:04 DEBUG : rclone: Version "v1.60.0" starting with parameters ["rclone" "-vv" "--no-check-certificate" "--checksum" "--progress" "--dump" "headers" "copy" "--no-traverse" "S3storage:pro-blob2/object.101" "."]
2022/11/09 23:40:04 DEBUG : Creating backend with remote "S3storage:pro-blob2/object.101"
2022/11/09 23:40:04 DEBUG : Using config file from "/home/ghorvath/.config/rclone/rclone.conf"
2022/11/09 23:40:04 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2022/11/09 23:40:04 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2022/11/09 23:40:05 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/11/09 23:40:05 DEBUG : HTTP REQUEST (req 0xc000900a00)
2022/11/09 23:40:05 DEBUG : HEAD /pro-blob2/object.101 HTTP/1.1
Host: s3-S3storage.fastcloud.fwb
User-Agent: rclone/v1.60.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20221109T224004Z

2022/11/09 23:40:05 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/11/09 23:40:05 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/11/09 23:40:05 DEBUG : HTTP RESPONSE (req 0xc000900a00)
2022/11/09 23:40:05 DEBUG : HTTP/1.1 200 OK
Content-Length: 10035
Accept-Ranges: bytes
Content-Type: binary/octet-stream
Date: Wed, 09 Nov 2022 22:40:04 GMT
Etag: "80509fa2b370edc141e93430edd2a9eb"
Last-Modified: Sun, 10 Apr 2022 19:47:05 GMT
Server: S3storageS3
Strict-Transport-Security: max-age=31536000
X-Amz-Meta-2: 26911
X-Amz-Meta-3: 26911
X-Amz-Meta-4: ZStd_1
X-Amz-Meta-5: 2474
X-Amz-Meta-Crc-Range: bytes=0-2473
X-Amz-Request-Id: 404ca83a-ff29-1fff-85c2-d8c497780c73

2022/11/09 23:40:05 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/11/09 23:40:05 DEBUG : fs cache: adding new entry for parent of "S3storage:pro-blob2/object.101", "S3storage:pro-blob2"
2022/11/09 23:40:05 DEBUG : Creating backend with remote "."
2022/11/09 23:40:05 DEBUG : fs cache: renaming cache item "." to be canonical "/mnt/c/Users/ghorvath"
2022-11-09 23:40:05 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022-11-09 23:40:05 DEBUG : HTTP REQUEST (req 0xc000a2c200)
2022-11-09 23:40:05 DEBUG : HEAD /pro-blob2/object.101 HTTP/1.1
Host: s3-S3storage.fastcloud.fwb
User-Agent: rclone/v1.60.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20221109T224005Z
2022-11-09 23:40:05 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022-11-09 23:40:05 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022-11-09 23:40:05 DEBUG : HTTP RESPONSE (req 0xc000a2c200)
2022-11-09 23:40:05 DEBUG : HTTP/1.1 200 OK
Content-Length: 10035
Accept-Ranges: bytes
Content-Type: binary/octet-stream
Date: Wed, 09 Nov 2022 22:40:04 GMT
Etag: "80509fa2b370edc141e93430edd2a9eb"
Last-Modified: Sun, 10 Apr 2022 19:47:05 GMT
Server: S3storageS3
Strict-Transport-Security: max-age=31536000
X-Amz-Meta-2: 26911
X-Amz-Meta-3: 26911
X-Amz-Meta-4: ZStd_1
X-Amz-Meta-5: 2474
X-Amz-Meta-Crc-Range: bytes=0-2473
X-Amz-Request-Id: 404ca83c-ff29-1fff-85c2-d8c497780c73
2022-11-09 23:40:05 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022-11-09 23:40:05 DEBUG : object.101: Need to transfer - File not found at Destination
2022-11-09 23:40:05 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022-11-09 23:40:05 DEBUG : HTTP REQUEST (req 0xc000900400)
2022-11-09 23:40:05 DEBUG : GET /pro-blob2/object.101 HTTP/1.1
Host: s3-S3storage.fastcloud.fwb
User-Agent: rclone/v1.60.0
Accept-Encoding: gzip
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20221109T224005Z
2022-11-09 23:40:05 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022-11-09 23:40:05 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022-11-09 23:40:05 DEBUG : HTTP RESPONSE (req 0xc000900400)
2022-11-09 23:40:05 DEBUG : HTTP/1.1 503 Service Unavailable
Content-Length: 273
Accept-Ranges: bytes
Content-Type: application/xml
Date: Wed, 09 Nov 2022 22:40:04 GMT
Etag: "80509fa2b370edc141e93430edd2a9eb"
Last-Modified: Sun, 10 Apr 2022 19:47:05 GMT
Server: S3storageS3
Strict-Transport-Security: max-age=31536000
X-Amz-Meta-2: 26911
X-Amz-Meta-3: 26911
X-Amz-Meta-4: ZStd_1
X-Amz-Meta-5: 2474
X-Amz-Meta-Crc-Range: bytes=0-2473
X-Amz-Request-Id: 404ca83e-ff29-1fff-85c2-d8c497780c73
X-Gmt-Error-Code: ServiceUnavailable
X-Gmt-Message: The server is temporarily unavailable. Please try again later.
2022-11-09 23:40:05 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022-11-09 23:40:06 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022-11-09 23:40:06 DEBUG : HTTP REQUEST (req 0xc000a2c400)
2022-11-09 23:40:06 DEBUG : GET /pro-blob2/object.101 HTTP/1.1
Host: s3-S3storage.fastcloud.fwb
User-Agent: rclone/v1.60.0
Accept-Encoding: gzip
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20221109T224006Z
2022-11-09 23:40:06 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022-11-09 23:40:06 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022-11-09 23:40:06 DEBUG : HTTP RESPONSE (req 0xc000a2c400)
2022-11-09 23:40:06 DEBUG : HTTP/1.1 503 Service Unavailable
Content-Length: 273
Accept-Ranges: bytes
Content-Type: application/xml
Date: Wed, 09 Nov 2022 22:40:05 GMT
Etag: "80509fa2b370edc141e93430edd2a9eb"
Last-Modified: Sun, 10 Apr 2022 19:47:05 GMT
Server: S3storageS3
Strict-Transport-Security: max-age=31536000
X-Amz-Meta-2: 26911
X-Amz-Meta-3: 26911
X-Amz-Meta-4: ZStd_1
X-Amz-Meta-5: 2474
X-Amz-Meta-Crc-Range: bytes=0-2473
X-Amz-Request-Id: 404ca848-ff29-1fff-85c2-d8c497780c73
X-Gmt-Error-Code: ServiceUnavailable
X-Gmt-Message: The server is temporarily unavailable. Please try again later.
2022-11-09 23:40:06 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022-11-09 23:40:08 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022-11-09 23:40:08 DEBUG : HTTP REQUEST (req 0xc000a2c800)
2022-11-09 23:40:08 DEBUG : GET /pro-blob2/object.101 HTTP/1.1
Host: s3-S3storage.fastcloud.fwb
User-Agent: rclone/v1.60.0
Accept-Encoding: gzip
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20221109T224008Z
2022-11-09 23:40:08 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022-11-09 23:40:08 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022-11-09 23:40:08 DEBUG : HTTP RESPONSE (req 0xc000a2c800)
2022-11-09 23:40:08 DEBUG : HTTP/1.1 503 Service Unavailable
Content-Length: 273
Accept-Ranges: bytes
Content-Type: application/xml
Date: Wed, 09 Nov 2022 22:40:07 GMT
Etag: "80509fa2b370edc141e93430edd2a9eb"
Last-Modified: Sun, 10 Apr 2022 19:47:05 GMT
Server: S3storageS3
Strict-Transport-Security: max-age=31536000
X-Amz-Meta-2: 26911
X-Amz-Meta-3: 26911
X-Amz-Meta-4: ZStd_1
X-Amz-Meta-5: 2474
X-Amz-Meta-Crc-Range: bytes=0-2473
X-Amz-Request-Id: 404ca850-ff29-1fff-85c2-d8c497780c73
X-Gmt-Error-Code: ServiceUnavailable
X-Gmt-Message: The server is temporarily unavailable. Please try again later.
2022-11-09 23:40:08 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Transferred:            0 / 1, 0%
Elapsed time:         4.0s
Transferring:
 *      object.101: transferring^C

can you post the same log but for a file that does transfer?

fwiw, seen this before recently here in the forum,
where i asked the OP to delete that single file, re-upload it and the problem went away.

Yes, this is the log:

rclone -vv --no-check-certificate --checksum --progress --dump headers copy --no-traverse S3storage:pro-blob2/object.102 .
2022/11/09 23:53:02 DEBUG : rclone: Version "v1.60.0" starting with parameters ["rclone" "-vv" "--no-check-certificate" "--checksum" "--progress" "--dump" "headers" "copy" "--no-traverse" "S3storage:pro-blob2/object.102" "."]
2022/11/09 23:53:02 DEBUG : Creating backend with remote "S3storage:pro-blob2/object.102"
2022/11/09 23:53:02 DEBUG : Using config file from "/home/ghorvath/.config/rclone/rclone.conf"
2022/11/09 23:53:02 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2022/11/09 23:53:02 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2022/11/09 23:53:02 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/11/09 23:53:02 DEBUG : HTTP REQUEST (req 0xc0008c8e00)
2022/11/09 23:53:02 DEBUG : HEAD /pro-blob2/object.102 HTTP/1.1
Host: s3-S3storage.fastcloud.fwb
User-Agent: rclone/v1.60.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20221109T225302Z

2022/11/09 23:53:02 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/11/09 23:53:02 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/11/09 23:53:02 DEBUG : HTTP RESPONSE (req 0xc0008c8e00)
2022/11/09 23:53:02 DEBUG : HTTP/1.1 200 OK
Content-Length: 1064061
Accept-Ranges: bytes
Content-Type: binary/octet-stream
Date: Wed, 09 Nov 2022 22:53:02 GMT
Etag: "2405635d68f4f156efebc55b6864eac3"
Last-Modified: Wed, 25 Nov 2020 19:54:16 GMT
Server: S3storageS3
Strict-Transport-Security: max-age=31536000
X-Amz-Meta-2: 1062574
X-Amz-Meta-3: 1062574
X-Amz-Meta-4: QuickLZ_1.5.0
X-Amz-Meta-5: 1478
X-Amz-Meta-Crc-Range: bytes=0-1477
X-Amz-Request-Id: 404cf3e4-ff29-1fff-85c2-d8c497780c73

2022/11/09 23:53:02 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/11/09 23:53:02 DEBUG : fs cache: adding new entry for parent of "S3storage:pro-blob2/object.102", "S3storage:pro-blob2"
2022/11/09 23:53:02 DEBUG : Creating backend with remote "."
2022/11/09 23:53:02 DEBUG : fs cache: renaming cache item "." to be canonical "/mnt/c/Users/ghorvath"
2022-11-09 23:53:02 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022-11-09 23:53:02 DEBUG : HTTP REQUEST (req 0xc00050ee00)
2022-11-09 23:53:02 DEBUG : HEAD /pro-blob2/object.102 HTTP/1.1
Host: s3-S3storage.fastcloud.fwb
User-Agent: rclone/v1.60.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20221109T225302Z
2022-11-09 23:53:02 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022-11-09 23:53:02 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022-11-09 23:53:02 DEBUG : HTTP RESPONSE (req 0xc00050ee00)
2022-11-09 23:53:02 DEBUG : HTTP/1.1 200 OK
Content-Length: 1064061
Accept-Ranges: bytes
Content-Type: binary/octet-stream
Date: Wed, 09 Nov 2022 22:53:02 GMT
Etag: "2405635d68f4f156efebc55b6864eac3"
Last-Modified: Wed, 25 Nov 2020 19:54:16 GMT
Server: S3storageS3
Strict-Transport-Security: max-age=31536000
X-Amz-Meta-2: 1062574
X-Amz-Meta-3: 1062574
X-Amz-Meta-4: QuickLZ_1.5.0
X-Amz-Meta-5: 1478
X-Amz-Meta-Crc-Range: bytes=0-1477
X-Amz-Request-Id: 404cf3e8-ff29-1fff-85c2-d8c497780c73
2022-11-09 23:53:02 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022-11-09 23:53:02 DEBUG : object.102: Need to transfer - File not found at Destination
2022-11-09 23:53:02 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022-11-09 23:53:02 DEBUG : HTTP REQUEST (req 0xc0009be000)
2022-11-09 23:53:02 DEBUG : GET /pro-blob2/object.102 HTTP/1.1
Host: s3-S3storage.fastcloud.fwb
User-Agent: rclone/v1.60.0
Accept-Encoding: gzip
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20221109T225302Z
2022-11-09 23:53:02 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022-11-09 23:53:02 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022-11-09 23:53:02 DEBUG : HTTP RESPONSE (req 0xc0009be000)
2022-11-09 23:53:02 DEBUG : HTTP/1.1 200 OK
Content-Length: 1064061
Accept-Ranges: bytes
Content-Type: binary/octet-stream
Date: Wed, 09 Nov 2022 22:53:02 GMT
Etag: "2405635d68f4f156efebc55b6864eac3"
Last-Modified: Wed, 25 Nov 2020 19:54:16 GMT
Server: S3storageS3
Strict-Transport-Security: max-age=31536000
X-Amz-Meta-2: 1062574
X-Amz-Meta-3: 1062574
X-Amz-Meta-4: QuickLZ_1.5.0
X-Amz-Meta-5: 1478
X-Amz-Meta-Crc-Range: bytes=0-1477
X-Amz-Request-Id: 404cf3ee-ff29-1fff-85c2-d8c497780c73
2022-11-09 23:53:02 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022-11-09 23:53:03 DEBUG : object.102: md5 = 2405635d68f4f156efebc55b6864eac3 OK
2022-11-09 23:53:03 INFO  : object.102: Copied (new)
Transferred:        1.015 MiB / 1.015 MiB, 100%, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         0.3s
2022/11/09 23:53:03 INFO  :
Transferred:        1.015 MiB / 1.015 MiB, 100%, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         0.3s

2022/11/09 23:53:03 DEBUG : 5 go routines active

using provider website, can you download /pro-blob2/object.101?

I have no acess to that.

The server is temporarily unavailable
the server fails the same error with rclone and s3cmd.

as a possible workaround might try
--exclude='/pro-blob2/object.101'

Unfortuantely this is not helps me because, I don't know how many objects produces the same, and if I just copy a buch of objects 50-60 million which takes more days, and only 1 objects hangs the copy process.

I don't understand why this is hanging the copy process? It should try --low-level-retries times and then continue.

By the look of your log, these retries are being done in the AWS SDK rather than by rclone, so I wonder if there is a bug in there.

If you run it with -vv --dump headers how many times does it try the file?

Try setting --low-level-retries 1 and see what happens.

From looking at your log above, the retries happen pretty quickly so rclone should move on to the next object quite quicky.

Hi Nick,

The --low-level-retries worked fine.

rclone -vv --no-check-certificate --checksum --progress --low-level-retries 1 copy --no-traverse S3storage:pro-bucket/object.101 .
2022/11/11 13:38:20 DEBUG : rclone: Version "v1.60.0" starting with parameters ["rclone" "-vv" "--no-check-certificate" "--checksum" "--progress" "--low-level-retries" "1" "copy" "--no-traverse" "S3storage:pro-bucket/object.101" "."]
2022/11/11 13:38:20 DEBUG : Creating backend with remote "S3storage:pro-bucket/object.101"
2022/11/11 13:38:20 DEBUG : Using config file from "/home/ghorvath/.config/rclone/rclone.conf"
2022/11/11 13:38:20 DEBUG : fs cache: adding new entry for parent of "S3storage:pro-bucket/object.101", "S3storage:pro-bucket"
2022/11/11 13:38:20 DEBUG : Creating backend with remote "."
2022/11/11 13:38:20 DEBUG : fs cache: renaming cache item "." to be canonical "/mnt/c/Users/ghorvath"
2022-11-11 13:38:20 DEBUG : object.101: Need to transfer - File not found at Destination
2022-11-11 13:38:21 DEBUG : pacer: low level retry 1/2 (error ServiceUnavailable: The server is temporarily unavailable. Please try again later.
        status code: 503, request id: 409e424c-ff29-1fff-85c2-d8c497780c73, host id: )
2022-11-11 13:38:21 DEBUG : pacer: Rate limited, increasing sleep to 10ms
2022-11-11 13:38:21 DEBUG : pacer: low level retry 2/2 (error ServiceUnavailable: The server is temporarily unavailable. Please try again later.
        status code: 503, request id: 409e424c-ff29-1fff-85c2-d8c497780c73, host id: )
2022-11-11 13:38:21 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2022-11-11 13:38:21 ERROR : object.101: Failed to copy: failed to open source object: ServiceUnavailable: The server is temporarily unavailable. Please try again later.
        status code: 503, request id: 409e424c-ff29-1fff-85c2-d8c497780c73, host id:
2022-11-11 13:38:21 ERROR : Attempt 1/3 failed with 1 errors and: failed to open source object: ServiceUnavailable: The server is temporarily unavailable. Please try again later.
        status code: 503, request id: 409e424c-ff29-1fff-85c2-d8c497780c73, host id:
2022-11-11 13:38:21 DEBUG : pacer: Reducing sleep to 15ms
2022-11-11 13:38:21 DEBUG : object.101: Need to transfer - File not found at Destination
2022-11-11 13:38:22 DEBUG : pacer: low level retry 1/2 (error ServiceUnavailable: The server is temporarily unavailable. Please try again later.
        status code: 503, request id: 409e427e-ff29-1fff-85c2-d8c497780c73, host id: )
2022-11-11 13:38:22 DEBUG : pacer: Rate limited, increasing sleep to 30ms
2022-11-11 13:38:22 DEBUG : pacer: low level retry 2/2 (error ServiceUnavailable: The server is temporarily unavailable. Please try again later.
        status code: 503, request id: 409e427e-ff29-1fff-85c2-d8c497780c73, host id: )
2022-11-11 13:38:22 DEBUG : pacer: Rate limited, increasing sleep to 60ms
2022-11-11 13:38:22 ERROR : object.101: Failed to copy: failed to open source object: ServiceUnavailable: The server is temporarily unavailable. Please try again later.
        status code: 503, request id: 409e427e-ff29-1fff-85c2-d8c497780c73, host id:
2022-11-11 13:38:22 ERROR : Attempt 2/3 failed with 1 errors and: failed to open source object: ServiceUnavailable: The server is temporarily unavailable. Please try again later.
        status code: 503, request id: 409e427e-ff29-1fff-85c2-d8c497780c73, host id:
2022-11-11 13:38:22 DEBUG : pacer: Reducing sleep to 45ms
2022-11-11 13:38:22 DEBUG : object.101: Need to transfer - File not found at Destination
2022-11-11 13:38:23 DEBUG : pacer: low level retry 1/2 (error ServiceUnavailable: The server is temporarily unavailable. Please try again later.
        status code: 503, request id: 409e42a4-ff29-1fff-85c2-d8c497780c73, host id: )
2022-11-11 13:38:23 DEBUG : pacer: Rate limited, increasing sleep to 90ms
2022-11-11 13:38:23 DEBUG : pacer: low level retry 2/2 (error ServiceUnavailable: The server is temporarily unavailable. Please try again later.
        status code: 503, request id: 409e42a4-ff29-1fff-85c2-d8c497780c73, host id: )
2022-11-11 13:38:23 DEBUG : pacer: Rate limited, increasing sleep to 180ms
2022-11-11 13:38:23 ERROR : object.101: Failed to copy: failed to open source object: ServiceUnavailable: The server is temporarily unavailable. Please try again later.
        status code: 503, request id: 409e42a4-ff29-1fff-85c2-d8c497780c73, host id:
2022-11-11 13:38:23 ERROR : Attempt 3/3 failed with 1 errors and: failed to open source object: ServiceUnavailable: The server is temporarily unavailable. Please try again later.
        status code: 503, request id: 409e42a4-ff29-1fff-85c2-d8c497780c73, host id:
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         3.0s
2022/11/11 13:38:23 INFO  :
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         3.0s

2022/11/11 13:38:23 DEBUG : 6 go routines active
2022/11/11 13:38:23 Failed to copy: failed to open source object: ServiceUnavailable: The server is temporarily unavailable. Please try again later.
        status code: 503, request id: 409e42a4-ff29-1fff-85c2-d8c497780c73, host id:

the -vv --dump headers tries during the process is running, Just the wait duration will increased.

Thanks for the help!

1 Like

Great.

That will let you do the copy and then afterwards you can use rclone check to make lists of the objects missing.