How does rclone handle "multi-thread copy: failed to close object after copy: large files must have at least 2 parts (400 bad_request)" errors in the next sync?

Help and Support
1

What is the problem you are having with rclone?

I'm trying to sync some files to B2. I have seen many errors that read "failed to close object after copy: large files must have at least 2 parts (400 bad_request)". I want to know if rclone will handle these graciously next time I run a sync, because these files exist in my bucket as 0 byte files, and I don't want them to get skipped next time I run the command or anything.

2023/11/08 13:48:35 ERROR : TV Shows/Office, The (US) (2005) [tvdb-73244]/Specials/The Office (US) - S00E16 - Season 4 Blooper Reel Bluray-1080p.mkv: Failed to copy: multi-thread copy: failed to close object after copy: large files must have at least 2 parts (400 bad_request)
2023/11/08 13:52:27 ERROR : TV Shows/Office, The (US) (2005) [tvdb-73244]/Specials/The Office (US) - S00E21 - Season 9 Blooper Reel Bluray-1080p.mkv: Failed to copy: multi-thread copy: failed to close object after copy: large files must have at least 2 parts (400 bad_request)
2023/11/08 13:52:45 ERROR : TV Shows/Office, The (US) (2005) [tvdb-73244]/Specials/The Office (US) - S00E18 - Season 6 Blooper Reel Bluray-1080p.mkv: Failed to copy: multi-thread copy: failed to close object after copy: large files must have at least 2 parts (400 bad_request)
2023/11/08 13:53:11 ERROR : TV Shows/Office, The (US) (2005) [tvdb-73244]/Specials/The Office (US) - S00E20 - Season 8 Blooper Reel Bluray-1080p.mkv: Failed to copy: multi-thread copy: failed to close object after copy: large files must have at least 2 parts (400 bad_request)

Run the command 'rclone version' and share the full output of the command.

rclone v1.64.2

  • os/version: alpine 3.18.4 (64 bit)

  • os/kernel: 4.4.302+ (x86_64)

  • os/type: linux

  • os/arch: amd64

  • go/version: go1.21.3

  • go/linking: static

  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Local hard drive from Synology to B2

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync --log-level DEBUG --log-file "$LOG_FILE" --fast-list --transfers 4 --max-duration 10h --cutoff-mode soft --b2-hard-delete /mnt/Plex b2_com-andresibanez-plex:com-andresibanez-plex/Servers/Kokiri --backup-dir b2_com-andresibanez-plex:com-andresibanez-plex/SafetyNet/Kokiri/$DATE --exclude="@eaDir/**"

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[b2_com-andresibanez-plex]
type = b2
account = XXX
key = XXX
hard_delete = true
upload_cutoff = 4Gi
chunk_size = 500Mi

(I plan on removing the upload_cutoff and chunk_size to see if that prevents me from seeing the errors next time).

A log from the command that you were trying to run with the -vv flag

This is a short excerpt, as the the entire log is a 4.0MB monster

2023/11/08 13:47:21 DEBUG : TV Shows/Office, The (US) (2005) [tvdb-73244]/Specials/The Office (US) - S00E15 - Season 3 Blooper Reel Bluray-1080p.mkv: Done sending chunk 0
2023/11/08 13:47:21 DEBUG : TV Shows/Office, The (US) (2005) [tvdb-73244]/Specials/The Office (US) - S00E15 - Season 3 Blooper Reel Bluray-1080p.mkv: multi-thread copy: chunk 1/1 (0-283300715) size 270.177Mi finished
2023/11/08 13:47:21 DEBUG : TV Shows/Office, The (US) (2005) [tvdb-73244]/Specials/The Office (US) - S00E15 - Season 3 Blooper Reel Bluray-1080p.mkv: Finishing large file upload with 1 parts
2023/11/08 13:47:22 DEBUG : TV Shows/Office, The (US) (2005) [tvdb-73244]/Specials/The Office (US) - S00E15 - Season 3 Blooper Reel Bluray-1080p.mkv: multi-thread copy: cancelling transfer on exit
2023/11/08 13:47:22 DEBUG : TV Shows/Office, The (US) (2005) [tvdb-73244]/Specials/The Office (US) - S00E15 - Season 3 Blooper Reel Bluray-1080p.mkv: Cancelling large file upload
2023/11/08 13:47:22 ERROR : TV Shows/Office, The (US) (2005) [tvdb-73244]/Specials/The Office (US) - S00E15 - Season 3 Blooper Reel Bluray-1080p.mkv: Failed to copy: multi-thread copy: failed to close object after copy: large files must have at least 2 parts (400 bad_request)

Bonus: Grepping only errors:

dekutree@Kokiri:/volume1/docker/rclone/data/logs/Plex/2023/11/08$ cat 00-03-01.log | grep "ERROR : "
2023/11/08 11:11:23 ERROR : TV Shows/Office, The (US) (2005) [tvdb-73244]/Specials/The Office (US) - S00E08 - Threat Level Midnight - The Movie Bluray-1080p.mkv: Failed to copy: multi-thread copy: failed to close object after copy: large files must have at least 2 parts (400 bad_request)
2023/11/08 13:43:57 ERROR : TV Shows/Office, The (US) (2005) [tvdb-73244]/Specials/The Office (US) - S00E13 - The Office Retrospective Bluray-1080p.mkv: Failed to copy: multi-thread copy: failed to close object after copy: large files must have at least 2 parts (400 bad_request)
2023/11/08 13:47:22 ERROR : TV Shows/Office, The (US) (2005) [tvdb-73244]/Specials/The Office (US) - S00E15 - Season 3 Blooper Reel Bluray-1080p.mkv: Failed to copy: multi-thread copy: failed to close object after copy: large files must have at least 2 parts (400 bad_request)
2023/11/08 13:48:35 ERROR : TV Shows/Office, The (US) (2005) [tvdb-73244]/Specials/The Office (US) - S00E16 - Season 4 Blooper Reel Bluray-1080p.mkv: Failed to copy: multi-thread copy: failed to close object after copy: large files must have at least 2 parts (400 bad_request)
2023/11/08 13:52:27 ERROR : TV Shows/Office, The (US) (2005) [tvdb-73244]/Specials/The Office (US) - S00E21 - Season 9 Blooper Reel Bluray-1080p.mkv: Failed to copy: multi-thread copy: failed to close object after copy: large files must have at least 2 parts (400 bad_request)
2023/11/08 13:52:45 ERROR : TV Shows/Office, The (US) (2005) [tvdb-73244]/Specials/The Office (US) - S00E18 - Season 6 Blooper Reel Bluray-1080p.mkv: Failed to copy: multi-thread copy: failed to close object after copy: large files must have at least 2 parts (400 bad_request)
2023/11/08 13:53:11 ERROR : TV Shows/Office, The (US) (2005) [tvdb-73244]/Specials/The Office (US) - S00E20 - Season 8 Blooper Reel Bluray-1080p.mkv: Failed to copy: multi-thread copy: failed to close object after copy: large files must have at least 2 parts (400 bad_request)
2023/11/08 14:49:10 ERROR : B2 bucket com-andresibanez-plex path Servers/Kokiri: not deleting files as there were IO errors
2023/11/08 14:49:10 ERROR : B2 bucket com-andresibanez-plex path Servers/Kokiri: not deleting directories as there were IO errors
2023/11/08 14:49:10 ERROR : B2 bucket com-andresibanez-plex path Servers/Kokiri: max transfer duration reached as set by --max-duration
2023/11/08 14:49:10 ERROR : Cancelling sync due to fatal error: max transfer duration reached as set by --max-duration
2023/11/08 14:49:10 ERROR : Fatal error received - not attempting retries
2

Doesn't transfer files that are identical on source and destination, testing by size and modification time
rclone should notice that the file sizes do not match and re-copy.

the easy and safe way to test:

  • rclone sync --dry-run
  • or pick a single file and rclone copy --dry-run
1 Like
3

not an expert with b2, it has its own quirks.

tho, seems to me, that rclone should not leave zero-bytes files in the dest.

4

Looks like there is a bug here. If you set the chunk size in B2 higher than half of the multi thread cutoff then you'll get this problem.

It is super annoying you can't have multi part uploads to B2 with only 1 part. It fills the B2 backend with special cases.

@AndyIbanezK can you make an issue on GitHub about this? Thanks.

5

@ncw thanks, the issue can be found here:

1 Like