Unable to copy files up to 76M

Help and Support
1

What is the problem you are having with rclone?

I am experiencing persistent NoSuchUpload errors when trying to perform multi-threaded uploads using rclone . The uploads consistently fail, citing a status code of 404 with the error NoSuchUpload . This happens despite various adjustments to the concurrency settings and chunk sizes.

Run the command 'rclone version' and share the full output of the command.

rclone v1.66.0

  • os/version: ubuntu 22.04 (64 bit)
  • os/kernel: 5.15.0-25-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.22.1
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using?

I am using S3-compatible storage provided by Contabo.

The command you were trying to run (eg rclone copy /tmp remote:tmp)

/usr/bin/rclone sync -P --update --verbose --transfers 30 "/home/atsys/backup/conjeito.com" "eu2:backup/conjeito.com"

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[eu2]
type = s3
provider = Ceph
access_key_id = XXXX
secret_access_key = XXXX
endpoint = https://eu2.contabostorage.com/bucketlogistica

A log from the command that you were trying to run with the -vv flag

2024/05/10 10:33:54 DEBUG : conjeito.tar: multi-thread copy: abort failed: failed to abort multipart upload "2~qs-lnRAJxIoPNLCCqA7EU-bYwDM8vb1": NoSuchUpload: 
        status code: 404, request id: tx00000b6f4c5af23ec7a70-00663ddbf2-102a64f-default, host id:
2024/05/10 10:33:54 ERROR : conjeito.tar: Failed to copy: multi-thread copy: failed to write chunk: failed to upload chunk 1 with 5242880 bytes: NoSuchUpload: 
        status code: 404, request id: tx000004009de510f9efeb2-00663ddbf2-10264d5-default, host id:
2024/05/10 10:33:54 ERROR : S3 bucket backup path conjeito.com: not deleting files as there were IO errors
2024/05/10 10:33:54 ERROR : S3 bucket backup path conjeito.com: not deleting directories as there were IO errors
2024/05/10 10:33:54 ERROR : Attempt 3/3 failed with 1 errors and: multi-thread copy: failed to write chunk: failed to upload chunk 1 with 5242880 bytes: NoSuchUpload: 
        status code: 404, request id: tx000004009de510f9efeb2-00663ddbf2-10264d5-default, host id:
Transferred:        3.248 MiB / 3.248 MiB, 100%, 831.363 KiB/s, ETA 0s
Errors:                 1 (retrying may help)
Transferred:            3 / 3, 100%
Elapsed time:         4.4s
2024/05/10 10:33:54 INFO  : 
Transferred:   	    3.248 MiB / 3.248 MiB, 100%, 831.363 KiB/s, ETA 0s
Errors:                 1 (retrying may help)
Transferred:            3 / 3, 100%
Elapsed time:         4.4s

2024/05/10 10:33:54 DEBUG : 8 go routines active
2024/05/10 10:33:54 Failed to sync: multi-thread copy: failed to write chunk: failed to upload chunk 1 with 5242880 bytes: NoSuchUpload: 
	status code: 404, request id: tx000004009de510f9efeb2-00663ddbf2-10264d5-default, host id:
2

welcome to the forum,

so there are 76 million files to be copied or the size of the source file larger than 76MiB, 76GiB or what?

not sure what you have tried so far but, as a test, might dumb down rclone and see what happens.
--multi-thread-streams=0 --transfers=1

fwiw, to keep the logs smaller, copy a single file and add--retries=1
and for a deeper look, add --dump=headers

3

Thanks @asdffdsa
76 MiB
I tried with

/usr/bin/rclone copy --verbose --multi-thread-streams=0 --retries=1 --dump=headers --transfers=1 backup-26.04.2024-alebtravel.tar.gz "eu2:backup/alebtravel.com"

2024/05/10 15:34:40 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/05/10 15:34:40 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/05/10 15:34:40 DEBUG : HTTP RESPONSE (req 0xc0005f6240)
2024/05/10 15:34:40 DEBUG : Error: context canceled
2024/05/10 15:34:40 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/05/10 15:34:40 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/05/10 15:34:40 DEBUG : HTTP RESPONSE (req 0xc000aa2900)
2024/05/10 15:34:40 DEBUG : Error: context canceled
2024/05/10 15:34:40 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/05/10 15:34:40 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/05/10 15:34:40 DEBUG : HTTP RESPONSE (req 0xc000b98a20)
2024/05/10 15:34:40 DEBUG : Error: context canceled
2024/05/10 15:34:40 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/05/10 15:34:40 DEBUG : backup-26.04.2024-alebtravel.tar.gz: Cancelling multipart upload
2024/05/10 15:34:40 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/05/10 15:34:40 DEBUG : HTTP REQUEST (req 0xc0004ca5a0)
2024/05/10 15:34:40 DEBUG : DELETE /bucketlogistica/bucketlogistica/backup/alebtravel.com/backup-26.04.2024-alebtravel.tar.gz?uploadId=2~ZuECLnnKltUPz0JU1B-mvxGIiUws9b0 HTTP/1.1
Host: eu2.contabostorage.com
User-Agent: rclone/v1.66.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240510T133440Z
Accept-Encoding: gzip

2024/05/10 15:34:40 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/05/10 15:34:41 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/05/10 15:34:41 DEBUG : HTTP RESPONSE (req 0xc0004ca5a0)
2024/05/10 15:34:41 DEBUG : HTTP/1.1 404 Not Found
Content-Length: 250
Accept-Ranges: bytes
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: *
Content-Type: application/xml
Date: Fri, 10 May 2024 13:34:41 GMT
Ratelimit-Limit: 250
Ratelimit-Remaining: 249
Ratelimit-Reset: 1
Strict-Transport-Security: max-age=16000000; includeSubDomains; preload;
X-Amz-Request-Id: tx000000da59e963206e901-00663e2271-1020179-default
X-Ratelimit-Limit-Second: 250
X-Ratelimit-Remaining-Second: 249

2024/05/10 15:34:41 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/05/10 15:34:41 DEBUG : backup-26.04.2024-alebtravel.tar.gz: Failed to cancel multipart upload: failed to abort multipart upload "2~ZuECLnnKltUPz0JU1B-mvxGIiUws9b0": NoSuchUpload:
status code: 404, request id: tx000000da59e963206e901-00663e2271-1020179-default, host id:
2024/05/10 15:34:41 ERROR : backup-26.04.2024-alebtravel.tar.gz: Failed to copy: failed to upload chunk 1 with 5242880 bytes: NoSuchUpload:
status code: 404, request id: tx000009fdc24cedc7b005a-00663e2270-1021b0e-default, host id:
2024/05/10 15:34:41 ERROR : Attempt 1/1 failed with 1 errors and: failed to upload chunk 1 with 5242880 bytes: NoSuchUpload:
status code: 404, request id: tx000009fdc24cedc7b005a-00663e2270-1021b0e-default, host id:
2024/05/10 15:34:41 INFO :
Transferred: 20 MiB / 20 MiB, 100%, 0 B/s, ETA -
Errors: 1 (retrying may help)
Elapsed time: 7.8s

2024/05/10 15:34:41 DEBUG : 9 go routines active
2024/05/10 15:34:41 Failed to copy: failed to upload chunk 1 with 5242880 bytes: NoSuchUpload:
status code: 404, request id: tx000009fdc24cedc7b005a-00663e2270-1021b0e-default, host id:

4

perhaps that should be endpoint = https://eu2.contabostorage.com

if the problem persists, need to post a full log, not just a snippet, ok?

5

Thank you very much, it has been uploaded successfully :hugs:

6

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.