Copying Files Within a B2 Bucket

I managed to replicate this myself now :slight_smile:

I uploaded a 6G file to b2 and I can see the problem.

I think it might be a bug in b2's s3 interface.

I'll investigate further...

Great, thank you so much, really appreciated!

I found a work-around - it looks like using --s3-copy-cutoff 1G will work.

Rclone uses 5G normally which is slightly bigger than b2 wants I think.

Oh nice! I'll check that out, thank you! Probably still worth a bug report for Backblaze maybe...

Yes I'm preparing a bug report as we speak!

Thanks a lot @ncw, and apologies, I should have tested with --s3-copy-cutoff before already, you did mention it previously...
By the way, it also works with --s3-copy-cutoff 4G

B2 have confirmed my bug report and are looking into the resolution.

Hopefully the --s3-copy-cutoff workaround worked for you.

If so then I might just lower the threshold for mulipart copy to 4G say then it will work on B2 and AWS.

Edit reading fail - I see you did try the copy cutoff. I'll nudge the default down in rclone.

Edit2 actually I'll wait until I hear from Backblaze as to what their resolution is.

1 Like

Thank you @ncw, that makes sense.

Hi @ncw, just another thing I've picked up, when using the S3 interface to do a server-side copy of a large file (30GB) and then running the same command again (rclone copy) a few minutes later, it will copy the file again because it sees the modification time as being different. I would have thought the mod time should be the same?

Doing a server side copy is how you update the mod time so if rclone saw the times were different this makes sense.

A quick test indicates that doing a server side copy isn't preserving the modification time.

However it also does this on AWS S3 so this is an rclone bug this time...

I'll post a fix in a little while

Ah I see, I didn't know that a server-side copy updates the mod time :see_no_evil:

It shouldn't, that is a bug!

Here is a potential fix

v1.52.2-275-gec5b9b82-fix-s3-copy-modtime-beta on branch fix-s3-copy-modtime (uploaded in 15-30 mins)

It works on AWS s3 and b2 according to my tests!

Oh, then I misunderstood the first line of your previous comment maybe.
Thanks for the fix, will check it out!

1 Like

Hi @ncw, I was just wondering whether Backblaze ever got back to you on this one?

I think this was an rclone bug and should be fixed in the latest beta. I'm pretty sure I merged the above but not 100% sure. Not at home at the moment so I can't easily check.

Ok thanks @ncw. No hurry! I was under the impression that it was a B2 bug as per your previous message

Sorry we are talking about 2 different thing!

There was a bug in rclone to do with modtimes not being preseved. That's fixed

I don't have an update on the 5GB Vs 5 GiB copy threshold bug though, but backblaze do know about it.

Maybe I should lower the default --s3-copy-cutoff for the 1.53 release?

I've done that now so server side copy should be compatible with the b2 s3 implementation by default.

This is in this beta

v1.53.0-beta.4731.725ae9138 on branch master (uploaded in 15-30 mins)

And will be released in 1.53 which should be this week.

Thank you very much for the update and the fix @ncw, very much appreciated!

1 Like

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.