What is the problem you are having with rclone?
Files over 1,500 MiB appear to always stop at 192 MiB uploaded. Tested on 1.64. Works fine on cloudflare, so I think it's a b2-specific problem maybe?
Run the command 'rclone version' and share the full output of the command.
rclone v1.64.0
- os/version: arch (64 bit)
- os/kernel: 6.5.6-403.asahi.fc38.aarch64+16k (aarch64)
- os/type: linux
- os/arch: arm64 (ARMv8 compatible)
- go/version: go1.21.1
- go/linking: dynamic
- go/tags: none
Which cloud storage system are you using? (eg Google Drive)
Backblaze b2.
The command you were trying to run (eg rclone copy /tmp remote:tmp)
Create a random file with >1,500 MiB of data. I used this. Then try to transfer it to b2.
rclone copy -Pvv --fast-list file_1 b2:jf-data
The rclone config contents with secrets removed.
[b2]
type = b2
account = redacted
key = redacted
hard_delete = true
[r2]
type = s3
provider = Cloudflare
access_key_id = redacted
secret_access_key = redacted
endpoint = https://redacted.r2.cloudflarestorage.com
acl = private
A log from the command with the -vv flag
2023/10/23 12:20:02 DEBUG : rclone: Version "v1.64.0" starting with parameters ["rclone" "copy" "-Pvv" "--fast-list" "file_1" "b2:jf-data/"]
2023/10/23 12:20:02 DEBUG : Creating backend with remote "file_1"
2023/10/23 12:20:02 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2023/10/23 12:20:02 DEBUG : fs cache: adding new entry for parent of "file_1", "/root"
2023/10/23 12:20:02 DEBUG : Creating backend with remote "b2:jf-data/"
2023/10/23 12:20:03 DEBUG : fs cache: renaming cache item "b2:jf-data/" to be canonical "b2:jf-data"
2023-10-23 12:20:03 DEBUG : Couldn't decode error response: EOF
2023-10-23 12:20:03 DEBUG : file_1: Need to transfer - File not found at Destination
2023-10-23 12:20:04 DEBUG : file_1: multi-thread copy: using backend concurrency of 16 instead of --multi-thread-streams 4
2023-10-23 12:20:04 DEBUG : file_1: Starting multi-thread copy with 17 chunks of size 96Mi with 16 parallel streams
2023-10-23 12:20:04 DEBUG : file_1: multi-thread copy: chunk 16/17 (1509949440-1610612736) size 96Mi starting
2023-10-23 12:20:04 DEBUG : file_1: multi-thread copy: chunk 1/17 (0-100663296) size 96Mi starting
2023-10-23 12:20:04 DEBUG : file_1: multi-thread copy: chunk 9/17 (805306368-905969664) size 96Mi starting
2023-10-23 12:20:04 DEBUG : file_1: multi-thread copy: chunk 14/17 (1308622848-1409286144) size 96Mi starting
2023-10-23 12:20:04 DEBUG : file_1: multi-thread copy: chunk 13/17 (1207959552-1308622848) size 96Mi starting
2023-10-23 12:20:04 DEBUG : file_1: multi-thread copy: chunk 7/17 (603979776-704643072) size 96Mi starting
2023-10-23 12:20:04 DEBUG : file_1: multi-thread copy: chunk 15/17 (1409286144-1509949440) size 96Mi starting
2023-10-23 12:20:04 DEBUG : file_1: multi-thread copy: chunk 10/17 (905969664-1006632960) size 96Mi starting
2023-10-23 12:20:04 DEBUG : file_1: multi-thread copy: chunk 8/17 (704643072-805306368) size 96Mi starting
2023-10-23 12:20:04 DEBUG : file_1: multi-thread copy: chunk 4/17 (301989888-402653184) size 96Mi starting
2023-10-23 12:20:04 DEBUG : file_1: multi-thread copy: chunk 11/17 (1006632960-1107296256) size 96Mi starting
2023-10-23 12:20:04 DEBUG : file_1: multi-thread copy: chunk 12/17 (1107296256-1207959552) size 96Mi starting
2023-10-23 12:20:04 DEBUG : file_1: multi-thread copy: chunk 6/17 (503316480-603979776) size 96Mi starting
2023-10-23 12:20:04 DEBUG : file_1: multi-thread copy: chunk 2/17 (100663296-201326592) size 96Mi starting
2023-10-23 12:20:04 DEBUG : file_1: multi-thread copy: chunk 3/17 (201326592-301989888) size 96Mi starting
2023-10-23 12:20:04 DEBUG : file_1: multi-thread copy: chunk 5/17 (402653184-503316480) size 96Mi starting
Transferred: 0 B / 1.514 GiB, 0%, 0 B/s, ETA -
Transferred: 0 / 1, 0%
Elapsed time: 2.1s
Transferring:
* file_1: 0% /1.514Gi, 0/s, -Transferred: 0 B / 1.514 GiB, 0%, 0 B/s, ETA -
2023-10-23 12:20:05 DEBUG : file_1: Sending chunk 3 length 100663296
2023-10-23 12:20:05 DEBUG : file_1: Sending chunk 6 length 100663296
2023-10-23 12:20:05 DEBUG : file_1: Sending chunk 7 length 100663296
2023-10-23 12:20:05 DEBUG : file_1: Sending chunk 14 length 100663296
2023-10-23 12:20:05 DEBUG : file_1: Sending chunk 12 length 100663296
2023-10-23 12:20:05 DEBUG : file_1: Sending chunk 0 length 100663296
2023-10-23 12:20:05 DEBUG : file_1: Sending chunk 4 length 100663296
2023-10-23 12:20:05 DEBUG : file_1: Sending chunk 11 length 100663296
2023-10-23 12:20:05 DEBUG : file_1: Sending chunk 10 length 100663296
2023-10-23 12:20:05 DEBUG : file_1: Sending chunk 13 length 100663296
2023-10-23 12:20:05 DEBUG : file_1: Sending chunk 15 length 100663296
Transferred: 0 / 1, 0%
Elapsed time: 2.8s
Transferring:
* file_1: 0% /1.514Gi, 0/s, -Transferred: 8.688 MiB / 1.514 GiB, 1%, 0 B/s, ETA -
Transferred: 0 / 1, 0%
Elapsed time: 3.3s
Transferring:
* file_1: 0% /1.514Gi, 5.498Mi/s, 4m40sTransferred: 11.500 MiB / 1.514 GiB, 1%, 3.832 MiB/s, ETA 6m41s
Transferred: 0 / 1, 0%
Elapsed time: 3.8s
Transferring:
* file_1: 0% /1.514Gi, 5.498Mi/s, 4m39sTransferred: 13.094 MiB / 1.514 GiB, 1%, 3.832 MiB/s, ETA 6m41s
Transferred: 0 / 1, 0%
Elapsed time: 4.3s
Transferring:
* file_1: 0% /1.514Gi, 6.543Mi/s, 3m54sTransferred: 17.594 MiB / 1.514 GiB, 1%, 3.522 MiB/s, ETA 7m15s
Transferred: 0 / 1, 0%
Elapsed time: 4.8s
Transferring:
* file_1: 1% /1.514Gi, 6.543Mi/s, 3m54sTransferred: 17.594 MiB / 1.514 GiB, 1%, 3.522 MiB/s, ETA 7m15s
<SNIP... lots of the same progress messages>
Transferred: 0 / 1, 0%
Elapsed time: 43.3s
Transferring:
* file_1: 12% /1.514Gi, 4.334Mi/s, 5m13sTransferred: 191.156 MiB / 1.514 GiB, 12%, 4.185 MiB/s, ETA 5m24s
Transferred: 0 / 1, 0%
Elapsed time: 43.8s
Transferring:
* file_1: 12% /1.514Gi, 4.334Mi/s, 5m13sTransferred: 192 MiB / 1.514 GiB, 12%, 4.185 MiB/s, ETA 5m24s
Transferred: 0 / 1, 0%
Elapsed time: 44.3s
Transferring:
* file_1: 12% /1.514Gi, 4.171Mi/s, 5m25sTransferred: 192 MiB / 1.514 GiB, 12%, 3.977 MiB/s, ETA 5m41s
Transferred: 0 / 1, 0%
Elapsed time: 44.8s
Transferring:
* file_1: 12% /1.514Gi, 4.171Mi/s, 5m25sTransferred: 192 MiB / 1.514 GiB, 12%, 3.977 MiB/s, ETA 5m41s
Transferred: 0 / 1, 0%
Elapsed time: 45.3s
Transferring:
* file_1: 12% /1.514Gi, 3.910Mi/s, 5m47sTransferred: 192 MiB / 1.514 GiB, 12%, 3.728 MiB/s, ETA 6m4s
Transferred: 0 / 1, 0%
Elapsed time: 45.8s
Transferring:
* file_1: 12% /1.514Gi, 3.910Mi/s, 5m47sTransferred: 192 MiB / 1.514 GiB, 12%, 3.728 MiB/s, ETA 6m4s
Transferred: 0 / 1, 0%
Elapsed time: 46.3s
Transferring:
* file_1: 12% /1.514Gi, 3.666Mi/s, 6m10sTransferred: 192 MiB / 1.514 GiB, 12%, 3.495 MiB/s, ETA 6m28s
<SNIP... lots of the same progress messages it'll keep printing this until I kill it>
Transferred: 0 / 1, 0%
Elapsed time: 4m22.8s
Transferring:
* file_1: 12% /1.514Gi, 3/s, 116586h45m2
<SNIP... killed after 4m22s>