Hi, download speed around 400 MB download, 140-145 MB upload, using Amazon test server on speedtest dot net
Here is the template answered:
What is the problem you are having with rclone?
Relatively slow speeds observed when trying to use rclone to copy from s3 bucket to another s3 bucket.
Run the command 'rclone version' and share the full output of the command.
rclone v1.66.0
- os/version: darwin 12.7.5 (64 bit)
- os/kernel: 21.6.0 (arm64)
- os/type: darwin
- os/arch: arm64 (ARMv8 compatible)
- go/version: go1.22.1
- go/linking: dynamic
- go/tags: none
Which cloud storage system are you using? (eg Google Drive)
S3
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
rclone copy :s3,access_key_id=redacted,secret_access_key=redacted,region=us-east-2:bucket-name-a/file.example.txt :s3,access_key_id=redacted,secret_access_key=redacted,region=us-east-2:bucket-name-b/file.example.txt -vv -P
Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.
; empty config
Note: I do not have a config file created currently (but maybe having one would help?)
A log from the command that you were trying to run with the -vv
flag
2024/07/15 02:33:18 DEBUG : rclone: Version "v1.66.0" starting with parameters ["rclone" "copy" ":s3,access_key_id=redacted,secret_access_key=redacted,region=us-east-2:bucket-name-a/file.example.txt" ":s3,access_key_id=redacted,secret_access_key=redacted,region=us-east-2:bucket-name-b/file.example.txt" "-P" "-vv"]
2024/07/15 02:33:18 DEBUG : Creating backend with remote ":s3,access_key_id=redacted,secret_access_key=redacted,region=us-east-2:bucket-name-a/file.example.txt"
2024/07/15 02:33:18 NOTICE: Config file "~./config/rclone/rclone.conf" not found - using defaults
2024/07/15 02:33:18 DEBUG : :s3: detected overridden config - adding "{pJafe}" suffix to name
2024/07/15 02:33:18 NOTICE: s3: s3 provider "" not known - please set correctly
2024/07/15 02:33:19 DEBUG : fs cache: adding new entry for parent of ":s3,access_key_id=redacted,secret_access_key=redacted,region=us-east-2:bucket-name-a/file.example.txt", ":s3{pJafe}:bucket-name-a"
2024/07/15 02:33:19 DEBUG : Creating backend with remote ":s3,access_key_id=redacted,secret_access_key=redacted,region=us-east-2:bucket-name-b/file.example.txt"
2024/07/15 02:33:19 DEBUG : :s3: detected overridden config - adding "{GJrII}" suffix to name
2024/07/15 02:33:19 NOTICE: s3: s3 provider "" not known - please set correctly
2024/07/15 02:33:19 DEBUG : fs cache: renaming cache item ":s3,access_key_id=redacted,secret_access_key=redacted,region=us-east-2:bucket-name-b/file.example.txt" to be canonical ":s3{GJrII}:bucket-name-b/upload/file.example.txt"
2024/07/15 02:33:19 DEBUG : file.example.txt: Need to transfer - File not found at Destination
2024/07/15 02:33:19 DEBUG : file.example.txt: size: 593.035Gi, parts: 10000, default: 5Mi, new: 61Mi; default chunk size insufficient, returned new chunk size
2024/07/15 02:33:19 DEBUG : file.example.txt: open chunk writer: started multipart upload: [redacted]
2024/07/15 02:33:19 DEBUG : file.example.txt: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 4
2024/07/15 02:33:19 DEBUG : file.example.txt: Starting multi-thread copy with 9956 chunks of size 61Mi with 4 parallel streams
2024/07/15 02:33:19 DEBUG : file.example.txt: multi-thread copy: chunk 4/9956 (191889408-255852544) size 61Mi starting
2024/07/15 02:33:19 DEBUG : file.example.txt: multi-thread copy: chunk 2/9956 (63963136-127926272) size 61Mi starting
2024/07/15 02:33:19 DEBUG : file.example.txt: multi-thread copy: chunk 1/9956 (0-63963136) size 61Mi starting
2024/07/15 02:33:19 DEBUG : file.example.txt: multi-thread copy: chunk 3/9956 (127926272-191889408) size 61Mi starting
2024/07/15 02:33:53 DEBUG : file.example.txt: multipart upload wrote chunk 4 with 63963136 bytes and etag "..."
2024/07/15 02:33:53 DEBUG : file.example.txt: multi-thread copy: chunk 4/9956 (191889408-255852544) size 61Mi finished
2024/07/15 02:33:53 DEBUG : file.example.txt: multi-thread copy: chunk 5/9956 (255852544-319815680) size 61Mi starting
2024/07/15 02:33:54 DEBUG : file.example.txt: multipart upload wrote chunk 1 with 63963136 bytes and etag "..."
2024/07/15 02:33:54 DEBUG : file.example.txt: multi-thread copy: chunk 1/9956 (0-63963136) size 61Mi finished
2024/07/15 02:33:54 DEBUG : file.example.txt: multi-thread copy: chunk 6/9956 (319815680-383778816) size 61Mi starting
2024/07/15 02:33:55 DEBUG : file.example.txt: multipart upload wrote chunk 2 with 63963136 bytes and etag "..."
2024/07/15 02:33:55 DEBUG : file.example.txt: multi-thread copy: chunk 2/9956 (63963136-127926272) size 61Mi finished
2024/07/15 02:33:55 DEBUG : file.example.txt: multi-thread copy: chunk 7/9956 (383778816-447741952) size 61Mi starting
2024/07/15 02:33:56 DEBUG : file.example.txt: multipart upload wrote chunk 3 with 63963136 bytes and etag "..."
2024/07/15 02:33:56 DEBUG : file.example.txt: multi-thread copy: chunk 3/9956 (127926272-191889408) size 61Mi finished
2024/07/15 02:33:56 DEBUG : file.example.txt: multi-thread copy: chunk 8/9956 (447741952-511705088) size 61Mi starting
2024/07/15 02:34:22 DEBUG : file.example.txt: multipart upload wrote chunk 5 with 63963136 bytes and etag "..."
2024/07/15 02:34:22 DEBUG : file.example.txt: multi-thread copy: chunk 5/9956 (255852544-319815680) size 61Mi finished
2024/07/15 02:34:22 DEBUG : file.example.txt: multi-thread copy: chunk 9/9956 (511705088-575668224) size 61Mi starting
2024/07/15 02:34:25 DEBUG : file.example.txt: multipart upload wrote chunk 7 with 63963136 bytes and etag "..."
2024/07/15 02:34:25 DEBUG : file.example.txt: multi-thread copy: chunk 7/9956 (383778816-447741952) size 61Mi finished
2024/07/15 02:34:25 DEBUG : file.example.txt: multi-thread copy: chunk 10/9956 (575668224-639631360) size 61Mi starting
2024/07/15 02:34:26 DEBUG : file.example.txt: multipart upload wrote chunk 6 with 63963136 bytes and etag "..."
2024/07/15 02:34:26 DEBUG : file.example.txt: multi-thread copy: chunk 6/9956 (319815680-383778816) size 61Mi finished
2024/07/15 02:34:26 DEBUG : file.example.txt: multi-thread copy: chunk 11/9956 (639631360-703594496) size 61Mi starting
2024/07/15 02:34:26 DEBUG : file.example.txt: multipart upload wrote chunk 8 with 63963136 bytes and etag "..."
2024/07/15 02:34:26 DEBUG : file.example.txt: multi-thread copy: chunk 8/9956 (447741952-511705088) size 61Mi finished
2024/07/15 02:34:26 DEBUG : file.example.txt: multi-thread copy: chunk 12/9956 (703594496-767557632) size 61Mi starting
2024/07/15 02:34:39 DEBUG : file.example.txt: multipart upload wrote chunk 9 with 63963136 bytes and etag "..."
2024/07/15 02:34:39 DEBUG : file.example.txt: multi-thread copy: chunk 9/9956 (511705088-575668224) size 61Mi finished
2024/07/15 02:34:39 DEBUG : file.example.txt: multi-thread copy: chunk 13/9956 (767557632-831520768) size 61Mi starting
2024/07/15 02:34:57 DEBUG : file.example.txt: multipart upload wrote chunk 12 with 63963136 bytes and etag "..."
2024/07/15 02:34:57 DEBUG : file.example.txt: multi-thread copy: chunk 12/9956 (703594496-767557632) size 61Mi finished
2024/07/15 02:34:57 DEBUG : file.example.txt: multi-thread copy: chunk 14/9956 (831520768-895483904) size 61Mi starting
2024/07/15 02:35:00 DEBUG : file.example.txt: multipart upload wrote chunk 11 with 63963136 bytes and etag "..."
2024/07/15 02:35:00 DEBUG : file.example.txt: multi-thread copy: chunk 11/9956 (639631360-703594496) size 61Mi finished
2024/07/15 02:35:00 DEBUG : file.example.txt: multi-thread copy: chunk 15/9956 (895483904-959447040) size 61Mi starting
2024/07/15 02:35:00 DEBUG : file.example.txt: multipart upload wrote chunk 10 with 63963136 bytes and etag "..."
2024/07/15 02:35:00 DEBUG : file.example.txt: multi-thread copy: chunk 10/9956 (575668224-639631360) size 61Mi finished
2024/07/15 02:35:00 DEBUG : file.example.txt: multi-thread copy: chunk 16/9956 (959447040-1023410176) size 61Mi starting
2024/07/15 02:35:04 DEBUG : file.example.txt: multipart upload wrote chunk 13 with 63963136 bytes and etag "..."
2024/07/15 02:35:04 DEBUG : file.example.txt: multi-thread copy: chunk 13/9956 (767557632-831520768) size 61Mi finished
2024/07/15 02:35:04 DEBUG : file.example.txt: multi-thread copy: chunk 17/9956 (1023410176-1087373312) size 61Mi starting
Transferred: 906.969 MiB / 593.035 GiB, 0%, 8.149 MiB/s, ETA 20h40m4s
Transferred: 0 / 1, 0%
Elapsed time: 1m58.3s
Transferring:
* file.example.txt: 0% /593.035Gi, 8.147Mi/s, 20h40m23s
This is just a few minutes of the log from starting command but it stays this speed of upload.