Discrepancy between amount of files in BackBlaze B2 Bucket

What is the problem you are having with rclone?

I tried to use rclone to transfer all of the files from a legacy backblaze b2 bucket to a new one that supports the S3 gateway. The backblaze web UI shows that there currently 626467 files in my bucket.

I initially ran the command rclone copy --progress b2:nerdsinspace-matrix-media-repo-prod b2:nerdsinspace-matrix-media-repo which only copied 542581 files. After doing a bit of research, I found out that additional files may be in my bucket in the form of "previous versions" and learned of the --b2-versions option.

I didn't want to spend another 4 days copying files so I ran a quick dry run test using the command rclone copy --dry-run --b2-versions --progress b2:nerdsinspace-matrix-media-repo-prod b2:nerdsinspace-matrix-media-repo which then only reported that 583882 files would have been copied.

What can explain the discrepancy between the 583882 files rclone reports (including versions) and the 626467 files the backblaze web UI reports?

What is your rclone version (output from rclone version)

I am using the debian package

rclone --version
rclone v1.45
- os/arch: linux/amd64
- go version: go1.11.6

Which OS you are using and how many bits (eg Windows 7, 64 bit)

No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 10 (buster)
Release:        10
Codename:       buster

Which cloud storage system are you using? (eg Google Drive)

BackBlaze B2

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy --dry-run --b2-versions --progress b2:nerdsinspace-matrix-media-repo-prod b2:nerdsinspace-matrix-media-repo

The rclone config contents with secrets removed.

No special config except for my b2 login information

You'd want to grab the current version of rclone and retest as that's a super old version.

I upgraded to rclone v1.54 and rclone copy --dry-run --b2-versions --progress b2:nerdsinspace-matrix-media-repo-prod b2:nerdsinspace-matrix-media-repo still reports that only 583882 files would be moved.

rclone v1.54.0
- os/arch: linux/amd64
- go version: go1.15.7

This could be

  • delete markers - I don't know if backblaze count these as a file, but rclone doesn't
  • directory markers (0 length files to mark the directories) - rclone doesn't count these as files but backblaze will

That would be my guess.

How did the original data get uploaded? Did the original software create directory markers (rclone doesn't).

How many deleted files are there? These will have a dated version but no non-dated version.

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.