B2 S3 interface usage

What is the problem you are having with rclone?

When doing an upload of a folder to B2 using the S3 interface where some files in the source dir are changing while the upload is happening, I see multiple versions of some of the files on the B2 web interface once the transfer is done.
See screenshots:

It looks like different versions of the files are present, but the lifecycle settings for the bucket in question have been set to "Keep only the last version"

What is your rclone version (output from rclone version)

rclone v1.52.2-232-gff843516-beta

  • os/arch: linux/amd64
  • go version: go1.14.6

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Ubuntu 20.04, 64 bit

Which cloud storage system are you using? (eg Google Drive)

B2, S3 interface

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy / :s3:bucket-name/folder1 --transfers 14 --fast-list --filter "+ /path/to/backup/folder/**" --filter "- *" -vv --stats 10s --stats-one-line --s3-access-key-id xyz --s3-secret-access-key xyz --s3-region eu-central-003 --s3-endpoint https://s3.eu-central-003.backblazeb2.com --s3-provider Other

The rclone config contents with secrets removed.

See command

A log from the command with the -vv flag

log.log (31.5 KB)

I also see that if I do an rclone purge command on the uploaded folder and then an rclone ls on that same folder, it seems to be gone, but on the B2 web interface it remains present with an asterisk next to the name of the "folder" as well as next to any files under it...

What appears to be happining is that rclone is getting these errors

2020/07/31 14:35:10 ERROR : path/to/backup/folder/userdata/activemq/localhost/KahaDB/db-3.log: Failed to copy: Put "https://s3.eu-central-003.backblazeb2.com/bucket-name/folder1/path/to/backup/folder/userdata/activemq/localhost/KahaDB/db-3.log?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=xyz%2F20200731%2Feu-central-003%2Fs3%2Faws4_request&X-Amz-Date=20200731T123509Z&X-Amz-Expires=900&X-Amz-SignedHeaders=content-md5%3Bcontent-type%3Bhost%3Bx-amz-acl%3Bx-amz-meta-mtime&X-Amz-Signature=bdcf5bb50284d1a293547eb814cb5fe7c25dacf6b5a65949e8d5316e3b51bf52": can't copy - source file is being updated (mod time changed from 2020-07-31 14:35:08.989060021 +0200 SAST to 2020-07-31 14:35:09.885085009 +0200 SAST)

This then causes rclone to abandon the transfer and try it again.

That explains why you get the versions.

If you upload a file, change it and upload it again I expect you'll get the same versions.

There are two things to note here

  1. the existence of extra versions is not something rclone has control over using the s3 protocol. Using the native b2 protocol rclone can control it directly.

  2. The files changing warning can be turned off with --local-no-check-updated which will mean rclone will do its best to transfer the file using the size it read at the start of the transfer.

Thank you @ncw. Yeah, I saw those errors which cause the retries. I just thought that when it retries the upload again and then copies the files again which have changed in the meantime, it would just overwrite the previous file and not create versions.

This is probably a B2 bug then? If the bucket is set to keep only last version, there shouldn't be any versions as far as I'm concerned...

I think it could well be a bug, yes. Though it might be that the extra versions are cleared out at some point - I don't know.

source file is being updated 

on linux, there must be an easy way to take a snapshot and then backup that snapshot.

It depends on which file system you are using. You can with ZFS but not with ext4.

Ubuntu sets things up with LVM by default which you can take snapshots of - I've used that facility before.

I've contacted Backblaze, according to them the lifecycle rules are applied once a day, so the extra versions are deleted then. So it's not a big problem but still somewhat unfortunate that it's different than when using the B2 native API...

I'm glad we've got to the bottom of that! Versions being deleted once per day sounds livable with.

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.