What is the problem you are having with rclone?
I am trying to perform a copy from Google Cloud Storage to Linode Object storage. The total data volume is 123TB. transfer for 98TB is complete, but post that I am seeing errors like the following:
2023/10/26 16:57:17 ERROR : <redacted-filename>: Failed to copy: multi-thread copy: failed to write chunk: failed to upload chunk 2 with 5242880 bytes: QuotaExceeded:
status code: 403, request id: tx00000ba11001a7bfa6334-00653a9a6c-35b16b-default, host id:
2023/10/26 16:57:17 ERROR : <redacted-filename>: Failed to copy: multi-thread copy: failed to write chunk: failed to upload chunk 4 with 5242880 bytes: QuotaExceeded:
status code: 403, request id: tx00000f57fd9e3e37875dc-00653a9a6c-3523eb-default, host id:
2023/10/26 16:57:18 ERROR : <redacted-filename>: Failed to copy: multi-thread copy: failed to write chunk: failed to upload chunk 2 with 5242880 bytes: QuotaExceeded:
status code: 403, request id: tx00000bb98e921798255cc-00653a9a6d-352427-default, host id:
2023/10/26 16:57:18 ERROR : <redacted-filename>: Failed to copy: multi-thread copy: failed to write chunk: failed to upload chunk 1 with 5242880 bytes: QuotaExceeded:
status code: 403, request id: tx000003b43de1f5c4913b1-00653a9a6d-3523eb-default, host id:
2023/10/26 16:57:18 ERROR : <redacted-filename>: Failed to copy: multi-thread copy: failed to write chunk: failed to upload chunk 4 with 5242880 bytes: QuotaExceeded:
status code: 403, request id: tx00000581efafc53652379-00653a9a6d-35de64-default, host id:
Run the command 'rclone version' and share the full output of the command.
rclone v1.64.2
- os/version: debian 11.7 (64 bit)
- os/kernel: 5.10.0-23-amd64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.21.3
- go/linking: static
- go/tags: none
Which cloud storage system are you using? (eg Google Drive)
Migrating from Google Cloud Storage to Linode Object Storage
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
rclone copy jadoo-eu-west-4:<bucket-name> jadoo-linode-eu-ams:<bucket-name> --transfers 32 --progress
Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.
[jadoo-eu-west-4]
type = google cloud storage
project_number = xxxxxx
service_account_file = xxxx
object_acl = bucketOwnerFullControl
bucket_acl = private
location = europe-west4
storage_class = MULTI_REGIONAL
[jadoo-linode-eu-ams]
type = s3
provider = Other
access_key_id = xxxxx
secret_access_key = xxxxx
endpoint = nl-ams-1.linodeobjects.com
acl = public-read
A log from the command that you were trying to run with the -vv
flag
2023/10/26 15:32:13 DEBUG : <redacted-filename>: open chunk writer: started multipart upload: 2~Pp4xS62PmGZG9SnHmd4LL7SiI9ybAtY
2023/10/26 15:32:13 DEBUG : <redacted-filename>: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 4
2023/10/26 15:32:13 DEBUG : <redacted-filename>: Starting multi-thread copy with 362 chunks of size 5Mi with 4 parallel streams
2023/10/26 15:32:13 DEBUG : <redacted-filename>: multi-thread copy: chunk 4/362 (15728640-20971520) size 5Mi starting
2023/10/26 15:32:13 DEBUG : <redacted-filename>: multi-thread copy: chunk 1/362 (0-5242880) size 5Mi starting
2023/10/26 15:32:13 DEBUG : <redacted-filename>: multi-thread copy: chunk 2/362 (5242880-10485760) size 5Mi starting
2023/10/26 15:32:13 DEBUG : <redacted-filename>: multi-thread copy: chunk 3/362 (10485760-15728640) size 5Mi starting
2023/10/26 15:32:13 DEBUG : <redacted-filename>: multipart upload "2~gQEFyZLyOjwpHFETsMRkX29tUCcwNFn" aborted
2023/10/26 15:32:13 ERROR : <redacted-filename>: Failed to copy: multi-thread copy: failed to write chunk: failed to upload chunk 1 with 5242880 bytes: QuotaExceeded:
status code: 403, request id: tx00000a54d59995d37e7f8-00653a867d-3551a7-default, host id:
-panic: runtime error: index out of range [0] with length 0
goroutine 2565 [running]:
github.com/rclone/rclone/lib/pool.(*RW).readPage(...)
github.com/rclone/rclone/lib/pool/reader_writer.go:91
github.com/rclone/rclone/lib/pool.(*RW).Read(0xc0160dab90, {0xc01bde8000?, 0x41071d?, 0x7effda035108?})
github.com/rclone/rclone/lib/pool/reader_writer.go:127 +0x20d
github.com/aws/aws-sdk-go/aws/request.(*offsetReader).Read(0x4537e9?, {0xc01bde8000?, 0x1c3aec0?, 0xc00170aa01?})
github.com/aws/aws-sdk-go@v1.44.311/aws/request/offset_reader.go:47 +0x11d
io.(*LimitedReader).Read(0xc01e591f08, {0xc01bde8000?, 0xc00170aa80?, 0xc0016950c8?})
io/io.go:480 +0x42
io.copyBuffer({0x2758cc0, 0xc00170aa80}, {0x274a6a0, 0xc01e591f08}, {0x0, 0x0, 0x0})
io/io.go:430 +0x1a6
io.Copy(...)
io/io.go:389
net/http.persistConnWriter.ReadFrom({0x274ca20?}, {0x274a6a0, 0xc01e591f08})
net/http/transport.go:1801 +0x55
bufio.(*Writer).ReadFrom(0xc0031c6040, {0x274a6a0, 0xc01e591f08})
bufio/bufio.go:797 +0x18b
io.copyBuffer({0x274b1e0, 0xc0031c6040}, {0x274a6a0, 0xc01e591f08}, {0x0, 0x0, 0x0})
io/io.go:416 +0x147
io.Copy(...)
io/io.go:389
net/http.(*transferWriter).doBodyCopy(0xc010ffb220, {0x274b1e0?, 0xc0031c6040?}, {0x274a6a0?, 0xc01e591f08?})
net/http/transfer.go:412 +0x48
net/http.(*transferWriter).writeBody(0xc010ffb220, {0x274b1e0, 0xc0031c6040})
net/http/transfer.go:370 +0x3c5
net/http.(*Request).write(0xc001bb0800, {0x274b1e0, 0xc0031c6040}, 0x0, 0xc00196d560, 0xc001b9ff60)
net/http/request.go:738 +0xbad
net/http.(*persistConn).writeLoop(0xc001951440)
net/http/transport.go:2424 +0x18f
created by net/http.(*Transport).dialConn in goroutine 1953
net/http/transport.go:1777 +0x16f1