Large server-side copy in B2 fails due to 'bad byte range'

Suspected Bug
1

What is the problem you are having with rclone?

When trying to perform a server-side copy of certain files on B2, I encounter a bad byte range: bytes=8589934592-8589934591 error, presumably when Copying chunk 2 length 0. Chunks 0 and 1 add up to the correct filesize, 5,676,749,018 bytes.

I was previously trying to un-nest a file structure server-side, and ran a larger copy command which errored on some, but not all, files larger than 2GB. This is the output of retrying one of those files.

Perhaps some finagling with chunk size settings will produce a workaround.

Run the command 'rclone version' and share the full output of the command.

rclone v1.64.0

  • os/version: darwin 14.0 (64 bit)
  • os/kernel: 23.0.0 (arm64)
  • os/type: darwin
  • os/arch: arm64 (ARMv8 compatible)
  • go/version: go1.21.1
  • go/linking: dynamic
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Backblaze B2

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy vibedoc:vibedoc/vibedoc/proxies/ vibedoc:vibedoc/proxies/ -vv

The rclone config contents with secrets removed.

[vibedoc]
type = b2
account = <removed>
key = <removed?

A log from the command with the -vv flag

2023/10/11 18:16:54 DEBUG : rclone: Version "v1.64.0" starting with parameters ["rclone" "copy" "vibedoc:vibedoc/vibedoc/proxies/VC_0044_Proxy.mov" "vibedoc:vibedoc/proxies/" "-vv"]
2023/10/11 18:16:54 DEBUG : Creating backend with remote "vibedoc:vibedoc/vibedoc/proxies/VC_0044_Proxy.mov"
2023/10/11 18:16:54 DEBUG : Using config file from "/Users/rachelshu/.config/rclone/rclone.conf"
2023/10/11 18:16:55 DEBUG : fs cache: adding new entry for parent of "vibedoc:vibedoc/vibedoc/proxies/VC_0044_Proxy.mov", "vibedoc:vibedoc/vibedoc/proxies"
2023/10/11 18:16:55 DEBUG : Creating backend with remote "vibedoc:vibedoc/proxies/"
2023/10/11 18:16:55 DEBUG : Couldn't decode error response: EOF
2023/10/11 18:16:55 DEBUG : fs cache: renaming cache item "vibedoc:vibedoc/proxies/" to be canonical "vibedoc:vibedoc/proxies"
2023/10/11 18:16:55 DEBUG : Couldn't decode error response: EOF
2023/10/11 18:16:55 DEBUG : VC_0044_Proxy.mov: Need to transfer - File not found at Destination
2023/10/11 18:16:55 DEBUG : VC_0044_Proxy.mov: Starting copy of large file in 2 chunks (id "4_z258dfed870379be387820615_f203876198f33cede_d20231012_m011655_c000_v0001053_t0026_u01697073415594")
2023/10/11 18:16:55 DEBUG : VC_0044_Proxy.mov: Copying chunk 2 length 0
2023/10/11 18:16:55 DEBUG : VC_0044_Proxy.mov: Copying chunk 0 length 4294967296
2023/10/11 18:16:55 DEBUG : VC_0044_Proxy.mov: Copying chunk 1 length 1381781722
2023/10/11 18:16:55 DEBUG : VC_0044_Proxy.mov: Error copying chunk 2 (retry=false): bad byte range: bytes=8589934592-8589934591 (400 bad_request): &api.Error{Status:400, Code:"bad_request", Message:"bad byte range: bytes=8589934592-8589934591"}
2023/10/11 18:16:55 DEBUG : VC_0044_Proxy.mov: Error copying chunk 2: bad byte range: bytes=8589934592-8589934591 (400 bad_request)
2023/10/11 18:16:55 DEBUG : VC_0044_Proxy.mov: Error copying chunk 1 (retry=false): Post "https://api000.backblazeb2.com/b2api/v1/b2_copy_part": context canceled: &url.Error{Op:"Post", URL:"https://api000.backblazeb2.com/b2api/v1/b2_copy_part", Err:(*errors.errorString)(0x105c53610)}
2023/10/11 18:16:55 DEBUG : VC_0044_Proxy.mov: Error copying chunk 1: Post "https://api000.backblazeb2.com/b2api/v1/b2_copy_part": context canceled
2023/10/11 18:16:55 DEBUG : VC_0044_Proxy.mov: Error copying chunk 0 (retry=false): Post "https://api000.backblazeb2.com/b2api/v1/b2_copy_part": context canceled: &url.Error{Op:"Post", URL:"https://api000.backblazeb2.com/b2api/v1/b2_copy_part", Err:(*errors.errorString)(0x105c53610)}
2023/10/11 18:16:55 DEBUG : VC_0044_Proxy.mov: Error copying chunk 0: Post "https://api000.backblazeb2.com/b2api/v1/b2_copy_part": context canceled
2023/10/11 18:16:55 DEBUG : VC_0044_Proxy.mov: Cancelling large file copy
2023/10/11 18:16:56 ERROR : VC_0044_Proxy.mov: Failed to copy: bad byte range: bytes=8589934592-8589934591 (400 bad_request)
2023/10/11 18:16:56 ERROR : Attempt 1/3 failed with 1 errors and: bad byte range: bytes=8589934592-8589934591 (400 bad_request)
2023/10/11 18:16:56 DEBUG : Couldn't decode error response: EOF
2023/10/11 18:16:56 DEBUG : VC_0044_Proxy.mov: Need to transfer - File not found at Destination
2023/10/11 18:16:56 DEBUG : VC_0044_Proxy.mov: Starting copy of large file in 2 chunks (id "4_z258dfed870379be387820615_f20059388d4019fb7_d20231012_m011656_c000_v0001410_t0002_u01697073416137")
2023/10/11 18:16:56 DEBUG : VC_0044_Proxy.mov: Copying chunk 2 length 0
2023/10/11 18:16:56 DEBUG : VC_0044_Proxy.mov: Copying chunk 0 length 4294967296
2023/10/11 18:16:56 DEBUG : VC_0044_Proxy.mov: Copying chunk 1 length 1381781722
2023/10/11 18:16:56 DEBUG : VC_0044_Proxy.mov: Error copying chunk 2 (retry=false): bad byte range: bytes=8589934592-8589934591 (400 bad_request): &api.Error{Status:400, Code:"bad_request", Message:"bad byte range: bytes=8589934592-8589934591"}
2023/10/11 18:16:56 DEBUG : VC_0044_Proxy.mov: Error copying chunk 2: bad byte range: bytes=8589934592-8589934591 (400 bad_request)
2023/10/11 18:16:56 DEBUG : VC_0044_Proxy.mov: Error copying chunk 1 (retry=false): Post "https://api000.backblazeb2.com/b2api/v1/b2_copy_part": context canceled: &url.Error{Op:"Post", URL:"https://api000.backblazeb2.com/b2api/v1/b2_copy_part", Err:(*errors.errorString)(0x105c53610)}
2023/10/11 18:16:56 DEBUG : VC_0044_Proxy.mov: Error copying chunk 0 (retry=false): Post "https://api000.backblazeb2.com/b2api/v1/b2_copy_part": context canceled: &url.Error{Op:"Post", URL:"https://api000.backblazeb2.com/b2api/v1/b2_copy_part", Err:(*errors.errorString)(0x105c53610)}
2023/10/11 18:16:56 DEBUG : VC_0044_Proxy.mov: Error copying chunk 1: Post "https://api000.backblazeb2.com/b2api/v1/b2_copy_part": context canceled
2023/10/11 18:16:56 DEBUG : VC_0044_Proxy.mov: Error copying chunk 0: Post "https://api000.backblazeb2.com/b2api/v1/b2_copy_part": context canceled
2023/10/11 18:16:56 DEBUG : VC_0044_Proxy.mov: Cancelling large file copy
2023/10/11 18:16:56 ERROR : VC_0044_Proxy.mov: Failed to copy: bad byte range: bytes=8589934592-8589934591 (400 bad_request)
2023/10/11 18:16:56 ERROR : Attempt 2/3 failed with 1 errors and: bad byte range: bytes=8589934592-8589934591 (400 bad_request)
2023/10/11 18:16:56 DEBUG : Couldn't decode error response: EOF
2023/10/11 18:16:56 DEBUG : VC_0044_Proxy.mov: Need to transfer - File not found at Destination
2023/10/11 18:16:56 DEBUG : VC_0044_Proxy.mov: Starting copy of large file in 2 chunks (id "4_z258dfed870379be387820615_f2061288290ac6db5_d20231012_m011656_c000_v0001406_t0029_u01697073416689")
2023/10/11 18:16:56 DEBUG : VC_0044_Proxy.mov: Copying chunk 2 length 0
2023/10/11 18:16:56 DEBUG : VC_0044_Proxy.mov: Copying chunk 0 length 4294967296
2023/10/11 18:16:56 DEBUG : VC_0044_Proxy.mov: Copying chunk 1 length 1381781722
2023/10/11 18:16:56 DEBUG : VC_0044_Proxy.mov: Error copying chunk 2 (retry=false): bad byte range: bytes=8589934592-8589934591 (400 bad_request): &api.Error{Status:400, Code:"bad_request", Message:"bad byte range: bytes=8589934592-8589934591"}
2023/10/11 18:16:56 DEBUG : VC_0044_Proxy.mov: Error copying chunk 2: bad byte range: bytes=8589934592-8589934591 (400 bad_request)
2023/10/11 18:16:56 DEBUG : VC_0044_Proxy.mov: Error copying chunk 1 (retry=false): Post "https://api000.backblazeb2.com/b2api/v1/b2_copy_part": context canceled: &url.Error{Op:"Post", URL:"https://api000.backblazeb2.com/b2api/v1/b2_copy_part", Err:(*errors.errorString)(0x105c53610)}
2023/10/11 18:16:56 DEBUG : VC_0044_Proxy.mov: Error copying chunk 1: Post "https://api000.backblazeb2.com/b2api/v1/b2_copy_part": context canceled
2023/10/11 18:16:56 DEBUG : VC_0044_Proxy.mov: Error copying chunk 0 (retry=false): Post "https://api000.backblazeb2.com/b2api/v1/b2_copy_part": context canceled: &url.Error{Op:"Post", URL:"https://api000.backblazeb2.com/b2api/v1/b2_copy_part", Err:(*errors.errorString)(0x105c53610)}
2023/10/11 18:16:56 DEBUG : VC_0044_Proxy.mov: Error copying chunk 0: Post "https://api000.backblazeb2.com/b2api/v1/b2_copy_part": context canceled
2023/10/11 18:16:56 DEBUG : VC_0044_Proxy.mov: Cancelling large file copy
2023/10/11 18:16:57 ERROR : VC_0044_Proxy.mov: Failed to copy: bad byte range: bytes=8589934592-8589934591 (400 bad_request)
2023/10/11 18:16:57 ERROR : Attempt 3/3 failed with 1 errors and: bad byte range: bytes=8589934592-8589934591 (400 bad_request)
2023/10/11 18:16:57 INFO  :
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         2.6s

2023/10/11 18:16:57 DEBUG : 14 go routines active
2023/10/11 18:16:57 Failed to copy: bad byte range: bytes=8589934592-8589934591 (400 bad_request)
2

Pretty sure this is the post you want.

3

I'm not sure that this is helpful to me as I'm not performing an upload but rather a server-side copy, and I don't quite have the understanding as to what I can take away from the other post, most of the flags discussed don't seem to apply?

I just tried creating a new remote config with the same credentials to see if that avoided some existing cache stored locally, but ran into the same error. So I'm not sure that it's a caching error on my local end?

4

playing around with different values for --b2-copy-cutoff does not seem to affect this - the final chunk is always 0 bytes and attempts to address an invalid byte range.

5

Due to the (daft in my opinion) way Range: headers are specified, bytes=8589934592-8589934591 is actually a correct range specifier for a 0 byte chunk. However I also think it is perfectly reasonable for b2 to reject this!

This is a new problem and was caused by the recent refactor of the chunked uploading and copying code. I fixed it and another problem and also added an integration test so it won't happen again.

Please give this a try.

v1.65.0-beta.7429.d8d76ff64.fix-b2-copy on branch fix-b2-copy (uploaded in 15-30 mins)

If that works out OK I'll release it in v1.64.1 which is due very soon.

1 Like
6

Thanks for your quick response and for your hard work maintaining this software! I'm downloading that beta now to take a look and will let you know how it goes.

1 Like
7

The files seem to be being moved successfully, will update when the transfer finishes.

I will note a separate bug here, that the progress indicators don't show any progress still: at each minute's update, it says 0B transferred until the file successfully moves, with no ETA. When it does transfer, the total GB to be transferred goes down and continues to show 0B transferred.

2023/10/12 12:49:36 DEBUG : rclone: Version "v1.65.0-beta.7429.d8d76ff64.fix-b2-copy" starting with parameters ["rclone_beta" "move" "vibedoc:vibedoc/vibedoc/" "vibedoc:vibedoc" "-vv" "--exclude" "/vibedoc/**"]
2023/10/12 12:49:36 DEBUG : Creating backend with remote "vibedoc:vibedoc/vibedoc/"
2023/10/12 12:49:36 DEBUG : Using config file from "/Users/rachelshu/.config/rclone/rclone.conf"
2023/10/12 12:49:36 DEBUG : Couldn't decode error response: EOF
2023/10/12 12:49:36 DEBUG : fs cache: renaming cache item "vibedoc:vibedoc/vibedoc/" to be canonical "vibedoc:vibedoc/vibedoc"
2023/10/12 12:49:36 DEBUG : Creating backend with remote "vibedoc:vibedoc"
2023/10/12 12:49:36 DEBUG : vibedoc: Excluded
2023/10/12 12:49:38 DEBUG : original-media/2023-06-17/FX3_CARD2/PRIVATE/M4ROOT/CLIP/20230617_C1488.MP4: Need to transfer - File not found at Destination
2023/10/12 12:49:38 DEBUG : original-media/2023-06-17/FX3_CARD2/PRIVATE/M4ROOT/CLIP/20230617_C1489.MP4: Need to transfer - File not found at Destination
2023/10/12 12:49:38 DEBUG : original-media/2023-06-16/FX6_CARD3/PRIVATE/XDROOT/Clip/VC_0059.MXF: Need to transfer - File not found at Destination
2023/10/12 12:49:38 DEBUG : original-media/2023-06-16/FX6_CARD2/PRIVATE/XDROOT/Clip/VC_0046.MXF: Need to transfer - File not found at Destination
2023/10/12 12:49:38 DEBUG : original-media/2023-06-16/FX6_CARD2/PRIVATE/XDROOT/Clip/VC_0056.MXF: Need to transfer - File not found at Destination
2023/10/12 12:49:38 DEBUG : original-media/2023-06-16/FX6_CARD2/PRIVATE/XDROOT/Clip/VC_0058.MXF: Need to transfer - File not found at Destination
2023/10/12 12:49:38 DEBUG : original-media/2023-06-17/FX3_CARD2/PRIVATE/M4ROOT/CLIP/20230617_C1488.MP4: Starting copy of large file in 14 chunks (id "4_z258dfed870379be387820615_f212220c17068e133_d20231012_m194938_c000_v0001067_t0059_u01697140178569")
2023/10/12 12:49:38 DEBUG : original-media/2023-06-17/FX3_CARD2/PRIVATE/M4ROOT/CLIP/20230617_C1488.MP4: Copying chunk 3 length 4294967296
2023/10/12 12:49:38 DEBUG : original-media/2023-06-17/FX3_CARD2/PRIVATE/M4ROOT/CLIP/20230617_C1488.MP4: Copying chunk 0 length 4294967296
2023/10/12 12:49:38 DEBUG : original-media/2023-06-17/FX3_CARD2/PRIVATE/M4ROOT/CLIP/20230617_C1488.MP4: Copying chunk 1 length 4294967296
2023/10/12 12:49:38 DEBUG : original-media/2023-06-17/FX3_CARD2/PRIVATE/M4ROOT/CLIP/20230617_C1488.MP4: Copying chunk 2 length 4294967296
2023/10/12 12:49:38 DEBUG : original-media/2023-06-17/FX3_CARD2/PRIVATE/M4ROOT/CLIP/20230617_C1489.MP4: Starting copy of large file in 8 chunks (id "4_z258dfed870379be387820615_f214ef192d8817080_d20231012_m194938_c000_v0001089_t0053_u01697140178599")
2023/10/12 12:49:38 DEBUG : original-media/2023-06-17/FX3_CARD2/PRIVATE/M4ROOT/CLIP/20230617_C1489.MP4: Copying chunk 3 length 4294967296
2023/10/12 12:49:38 DEBUG : original-media/2023-06-17/FX3_CARD2/PRIVATE/M4ROOT/CLIP/20230617_C1489.MP4: Copying chunk 0 length 4294967296
2023/10/12 12:49:38 DEBUG : original-media/2023-06-17/FX3_CARD2/PRIVATE/M4ROOT/CLIP/20230617_C1489.MP4: Copying chunk 1 length 4294967296
2023/10/12 12:49:38 DEBUG : original-media/2023-06-16/FX6_CARD3/PRIVATE/XDROOT/Clip/VC_0059.MXF: Starting copy of large file in 4 chunks (id "4_z258dfed870379be387820615_f2227912099fd9e26_d20231012_m194938_c000_v0001062_t0013_u01697140178653")
2023/10/12 12:49:38 DEBUG : original-media/2023-06-16/FX6_CARD2/PRIVATE/XDROOT/Clip/VC_0046.MXF: Starting copy of large file in 12 chunks (id "4_z258dfed870379be387820615_f214ef192d8817081_d20231012_m194938_c000_v0001073_t0023_u01697140178667")
2023/10/12 12:49:38 DEBUG : original-media/2023-06-17/FX3_CARD2/PRIVATE/M4ROOT/CLIP/20230617_C1489.MP4: Copying chunk 2 length 4294967296
2023/10/12 12:49:38 DEBUG : original-media/2023-06-16/FX6_CARD3/PRIVATE/XDROOT/Clip/VC_0059.MXF: Copying chunk 3 length 1784373808
2023/10/12 12:49:38 DEBUG : original-media/2023-06-16/FX6_CARD3/PRIVATE/XDROOT/Clip/VC_0059.MXF: Copying chunk 0 length 4294967296
2023/10/12 12:49:38 DEBUG : original-media/2023-06-16/FX3_CARD2/PRIVATE/M4ROOT/CLIP/20230616_C1448.MP4: Need to transfer - File not found at Destination
2023/10/12 12:49:38 DEBUG : original-media/2023-06-16/FX3_CARD2/PRIVATE/M4ROOT/CLIP/20230616_C1452.MP4: Need to transfer - File not found at Destination
2023/10/12 12:49:38 DEBUG : original-media/2023-06-16/FX3_CARD2/PRIVATE/M4ROOT/CLIP/20230616_C1453.MP4: Need to transfer - File not found at Destination
2023/10/12 12:49:38 DEBUG : original-media/2023-06-16/FX3_CARD2/PRIVATE/M4ROOT/CLIP/20230616_C1456.MP4: Need to transfer - File not found at Destination
2023/10/12 12:49:38 DEBUG : original-media/2023-06-16/FX6_CARD3/PRIVATE/XDROOT/Clip/VC_0059.MXF: Copying chunk 1 length 4294967296
2023/10/12 12:49:38 DEBUG : original-media/2023-06-16/FX6_CARD3/PRIVATE/XDROOT/Clip/VC_0059.MXF: Copying chunk 2 length 4294967296
2023/10/12 12:49:38 DEBUG : original-media/2023-06-16/FX6_CARD1/PRIVATE/XDROOT/Clip/VC_0026.MXF: Need to transfer - File not found at Destination
2023/10/12 12:49:38 DEBUG : original-media/2023-06-16/FX6_CARD1/PRIVATE/XDROOT/Clip/VC_0043.MXF: Need to transfer - File not found at Destination
2023/10/12 12:49:38 DEBUG : original-media/2023-06-16/FX6_CARD1/PRIVATE/XDROOT/Clip/VC_0044.MXF: Need to transfer - File not found at Destination
2023/10/12 12:49:38 DEBUG : original-media/2023-06-16/FX6_CARD2/PRIVATE/XDROOT/Clip/VC_0046.MXF: Copying chunk 3 length 4294967296
2023/10/12 12:49:38 DEBUG : original-media/2023-06-16/FX6_CARD2/PRIVATE/XDROOT/Clip/VC_0046.MXF: Copying chunk 0 length 4294967296
2023/10/12 12:49:38 DEBUG : original-media/2023-06-16/FX6_CARD2/PRIVATE/XDROOT/Clip/VC_0046.MXF: Copying chunk 1 length 4294967296
2023/10/12 12:49:38 DEBUG : original-media/2023-06-16/FX6_CARD2/PRIVATE/XDROOT/Clip/VC_0046.MXF: Copying chunk 2 length 4294967296
2023/10/12 12:49:38 DEBUG : original-media/2023-06-16/FX3_CARD1/PRIVATE/M4ROOT/CLIP/20230616_C1433.MP4: Need to transfer - File not found at Destination
2023/10/12 12:49:38 DEBUG : original-media/2023-06-16/FX3_CARD1/PRIVATE/M4ROOT/CLIP/20230616_C1443.MP4: Need to transfer - File not found at Destination
2023/10/12 12:49:38 DEBUG : original-media/2023-06-16/FX3_CARD1/PRIVATE/M4ROOT/CLIP/20230616_C1444.MP4: Need to transfer - File not found at Destination
2023/10/12 12:49:39 DEBUG : original-media/2023-06-17/FX6_CARD2/PRIVATE/XDROOT/Clip/VC_0066.MXF: Need to transfer - File not found at Destination
2023/10/12 12:49:39 DEBUG : original-media/2023-06-17/FX6_CARD1/PRIVATE/XDROOT/Clip/VC_0060.MXF: Need to transfer - File not found at Destination
2023/10/12 12:49:39 DEBUG : original-media/2023-06-17/FX6_CARD1/PRIVATE/XDROOT/Clip/VC_0061.MXF: Need to transfer - File not found at Destination
2023/10/12 12:49:39 DEBUG : original-media/2023-06-17/FX6_CARD1/PRIVATE/XDROOT/Clip/VC_0063.MXF: Need to transfer - File not found at Destination
2023/10/12 12:49:39 DEBUG : original-media/2023-06-17/FX6_CARD1/PRIVATE/XDROOT/Clip/VC_0065.MXF: Need to transfer - File not found at Destination
2023/10/12 12:49:39 DEBUG : B2 bucket vibedoc: Waiting for checks to finish
2023/10/12 12:49:39 DEBUG : B2 bucket vibedoc: Waiting for transfers to finish
2023/10/12 12:50:36 INFO  :
Transferred:   	          0 B / 534.742 GiB, 0%, 0 B/s, ETA -
Checks:                 0 / 4, 0%
Transferred:            0 / 21, 0%
Elapsed time:       1m0.2s
Checking:

Transferring:
 * original-media/2023-06…LIP/20230617_C1488.MP4:  0% /52.127Gi, 0/s, -
 * original-media/2023-06…LIP/20230617_C1489.MP4:  0% /28.751Gi, 0/s, -
 * original-media/2023-06…DROOT/Clip/VC_0059.MXF:  0% /13.662Gi, 0/s, -
 * original-media/2023-06…DROOT/Clip/VC_0046.MXF:  0% /44.064Gi, 0/s, -

2023/10/12 12:51:14 DEBUG : original-media/2023-06-16/FX6_CARD3/PRIVATE/XDROOT/Clip/VC_0059.MXF: Done copying chunk 3
2023/10/12 12:51:36 INFO  :
Transferred:   	          0 B / 534.742 GiB, 0%, 0 B/s, ETA -
Checks:                 0 / 4, 0%
Transferred:            0 / 21, 0%
Elapsed time:       2m0.2s
Checking:

Transferring:
 * original-media/2023-06…LIP/20230617_C1488.MP4:  0% /52.127Gi, 0/s, -
 * original-media/2023-06…LIP/20230617_C1489.MP4:  0% /28.751Gi, 0/s, -
 * original-media/2023-06…DROOT/Clip/VC_0059.MXF:  0% /13.662Gi, 0/s, -
 * original-media/2023-06…DROOT/Clip/VC_0046.MXF:  0% /44.064Gi, 0/s, -

2023/10/12 12:52:27 DEBUG : original-media/2023-06-17/FX3_CARD2/PRIVATE/M4ROOT/CLIP/20230617_C1489.MP4: Done copying chunk 1
2023/10/12 12:52:27 DEBUG : original-media/2023-06-17/FX3_CARD2/PRIVATE/M4ROOT/CLIP/20230617_C1489.MP4: Copying chunk 4 length 4294967296
2023/10/12 12:52:31 DEBUG : original-media/2023-06-17/FX3_CARD2/PRIVATE/M4ROOT/CLIP/20230617_C1489.MP4: Done copying chunk 2
2023/10/12 12:52:31 DEBUG : original-media/2023-06-17/FX3_CARD2/PRIVATE/M4ROOT/CLIP/20230617_C1489.MP4: Copying chunk 5 length 4294967296
2023/10/12 12:52:36 INFO  :
Transferred:   	          0 B / 534.742 GiB, 0%, 0 B/s, ETA -
Checks:                 0 / 4, 0%
Transferred:            0 / 21, 0%
Elapsed time:       3m0.2s
Checking:

Transferring:
 * original-media/2023-06…LIP/20230617_C1488.MP4:  0% /52.127Gi, 0/s, -
 * original-media/2023-06…LIP/20230617_C1489.MP4:  0% /28.751Gi, 0/s, -
 * original-media/2023-06…DROOT/Clip/VC_0059.MXF:  0% /13.662Gi, 0/s, -
 * original-media/2023-06…DROOT/Clip/VC_0046.MXF:  0% /44.064Gi, 0/s, -

2023/10/12 12:52:41 DEBUG : original-media/2023-06-16/FX6_CARD2/PRIVATE/XDROOT/Clip/VC_0046.MXF: Done copying chunk 3
2023/10/12 12:52:41 DEBUG : original-media/2023-06-16/FX6_CARD2/PRIVATE/XDROOT/Clip/VC_0046.MXF: Copying chunk 4 length 4294967296
2023/10/12 12:52:46 DEBUG : original-media/2023-06-16/FX6_CARD2/PRIVATE/XDROOT/Clip/VC_0046.MXF: Done copying chunk 1
2023/10/12 12:52:46 DEBUG : original-media/2023-06-16/FX6_CARD2/PRIVATE/XDROOT/Clip/VC_0046.MXF: Copying chunk 5 length 4294967296
2023/10/12 12:52:48 DEBUG : original-media/2023-06-16/FX6_CARD2/PRIVATE/XDROOT/Clip/VC_0046.MXF: Done copying chunk 2
2023/10/12 12:52:48 DEBUG : original-media/2023-06-16/FX6_CARD2/PRIVATE/XDROOT/Clip/VC_0046.MXF: Copying chunk 6 length 4294967296
2023/10/12 12:52:52 DEBUG : original-media/2023-06-16/FX6_CARD3/PRIVATE/XDROOT/Clip/VC_0059.MXF: Done copying chunk 2
2023/10/12 12:52:53 DEBUG : original-media/2023-06-16/FX6_CARD3/PRIVATE/XDROOT/Clip/VC_0059.MXF: Done copying chunk 1
2023/10/12 12:52:53 DEBUG : original-media/2023-06-16/FX6_CARD3/PRIVATE/XDROOT/Clip/VC_0059.MXF: Done copying chunk 0
2023/10/12 12:52:53 DEBUG : original-media/2023-06-16/FX6_CARD3/PRIVATE/XDROOT/Clip/VC_0059.MXF: Finishing large file copy with 4 parts
2023/10/12 12:52:53 DEBUG : original-media/2023-06-16/FX6_CARD3/PRIVATE/XDROOT/Clip/VC_0059.MXF: Src hash empty - aborting Dst hash check
2023/10/12 12:52:53 DEBUG : original-media/2023-06-16/FX6_CARD3/PRIVATE/XDROOT/Clip/VC_0059.MXF: Dst hash empty - aborting Src hash check
2023/10/12 12:52:53 INFO  : original-media/2023-06-16/FX6_CARD3/PRIVATE/XDROOT/Clip/VC_0059.MXF: Copied (server-side copy)
2023/10/12 12:52:54 INFO  : original-media/2023-06-16/FX6_CARD3/PRIVATE/XDROOT/Clip/VC_0059.MXF: Deleted
2023/10/12 12:52:54 DEBUG : original-media/2023-06-16/FX6_CARD2/PRIVATE/XDROOT/Clip/VC_0056.MXF: Starting copy of large file in 7 chunks (id "4_z258dfed870379be387820615_f203876198f62d2ab_d20231012_m195254_c000_v0001066_t0018_u01697140374307")
2023/10/12 12:52:54 DEBUG : original-media/2023-06-16/FX6_CARD2/PRIVATE/XDROOT/Clip/VC_0056.MXF: Copying chunk 3 length 4294967296
2023/10/12 12:52:54 DEBUG : original-media/2023-06-16/FX6_CARD2/PRIVATE/XDROOT/Clip/VC_0056.MXF: Copying chunk 0 length 4294967296
2023/10/12 12:52:54 DEBUG : original-media/2023-06-16/FX6_CARD2/PRIVATE/XDROOT/Clip/VC_0056.MXF: Copying chunk 1 length 4294967296
2023/10/12 12:52:54 DEBUG : original-media/2023-06-16/FX6_CARD2/PRIVATE/XDROOT/Clip/VC_0056.MXF: Copying chunk 2 length 4294967296
2023/10/12 12:53:02 DEBUG : original-media/2023-06-16/FX6_CARD2/PRIVATE/XDROOT/Clip/VC_0046.MXF: Done copying chunk 0
2023/10/12 12:53:02 DEBUG : original-media/2023-06-16/FX6_CARD2/PRIVATE/XDROOT/Clip/VC_0046.MXF: Copying chunk 7 length 4294967296
2023/10/12 12:53:36 INFO  :
Transferred:   	          0 B / 521.080 GiB, 0%, 0 B/s, ETA -
Checks:                 2 / 6, 33%
Deleted:                1 (files), 0 (dirs)
Renamed:                1
Transferred:            1 / 21, 5%
Server Side Copies:     1 @ 13.662 GiB
Elapsed time:       4m0.2s
Checking:

Transferring:
 * original-media/2023-06…LIP/20230617_C1488.MP4:  0% /52.127Gi, 0/s, -
 * original-media/2023-06…LIP/20230617_C1489.MP4:  0% /28.751Gi, 0/s, -
 * original-media/2023-06…DROOT/Clip/VC_0046.MXF:  0% /44.064Gi, 0/s, -
 * original-media/2023-06…DROOT/Clip/VC_0056.MXF:  0% /25.278Gi, 0/s, -
8

It works now! However the transfer is painfully, painfully slow, as opposed to the files that were less than the B2 chunk size limit which moved instantly. 250 GB transferred in 2 hours. It's a little faster if I do one file at a time.

9

Great

Yes this is unfortunate but a known issue. Rclone doesn't have a mechanism for reporting server side copy progress. Not many backends support chunked server side copies (b2 and s3) and it is only those which possibly could give progress.

Great!

That fix is in the latest beta now and will be in v1.64.1

I don't think this will be rclone's fault that it is slow - I suspect that the b2 servers are working quite hard here.

I conjecture it might run quicker if you reduce --b2-copy-cutoff to something smaller than the default of 4G, say 64M or so.

You can then run lots in parallel by increasing --b2-upload-concurrency to 16 or 32 or 64 say.

10

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.