Rclone and b2 disagree on size and file numbers

Help and Support
1

What is the problem you are having with rclone?

I am seeing a size/file difference between what rclone crypt on b2 shows vs what b2 says via the website.

Run the command 'rclone version' and share the full output of the command.

rclone v1.71.1

  • os/version: debian 13.1 (64 bit)
  • os/kernel: 6.12.48+deb13-amd64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.25.1
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

b2 (with crypt)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone size --fast-list b2-crypt:
Total objects: 129.306k (129306)
Total size: 474.860 GiB (509876590262 Byte)

yet b2 via the gui says:

Current Files:

179,134

Current Size:

615.3 GB

Within B2 the lifecycle settings are set to:

Keep only the last version of the file

So is this a peculiarity of using the rclone crypt? or is something else going on? Can anyone offer an explanation please as to why there is an extra 140gb ish being used?

Thank you

2

Encryption does not create any extra files (it is 1 to 1 relation) so in your case for sure you are counting two different things. Possibly your B2 destination contains some extra files which are not considered encrypted based on your crypt remote config. Run rclone ls -vv to see debug info. It should give you some answers.

Encrypted files’ sizes are bigger indeed due to some extra info included in every encrypted file. Check docs for details.

This size increase typically is miniscule but can grow (as % value) if you have a lot of small files as explained in docs:

1 byte file will encrypt to

  • 32 bytes header

  • 17 bytes data chunk

49 bytes total

1 MiB (1048576 bytes) file will encrypt to

  • 32 bytes header

  • 16 chunks of 65568 bytes

1049120 bytes total (a 0.05% overhead). This is the overhead for big files.

3

Hi,

Thanks for the reply. rclone ls -vv agreed with what rlcone size is seeing. There is no obvious strange file names or things i wouldn’t expect to see. I’ve also confirmed that crypt and non-crypt versions are similar.

rclone size --fast-list b2: 
Total objects: 129.306k (129306) 
Total size: 474.981 GiB (510006568118 Byte)

vs

rclone size --fast-list b2-crypt: 
Total objects: 129.306k (129306) 
Total size: 474.860 GiB (509876590262 Byte)

It feels more like a b2 rather than a rclone issue - would folk agree?

Thanks

4

Indeed. Though I would not call it an B2 issue. You would have to read B2 docs - it reports something:) And unlikely it is a bug (but can be). Maybe it is not real time? Maybe you have some past versions created before you changed retention policies? Poke around their docs and GUI.

5

Thanks.

For those that find this thread - over 24-36 hours Backblaze ran some kind of automated clean up and the overall space used has dropped massively.

Despite me using ‘Keep only the last version of the file’ via rclone ls : --b2-versions it was clear multiple file versions were being retained. My next step, if it hadn’t have sorted itself out, was to run rclone cleanup.

My problem however is now resolved.

6

I have a vague memory from the b2 docs that they only clean old versions every 24 hours. With b2 you can't disable versions you can only say that you don't want any to be kept which b2 enforces only every 24 hrs.

So I think that explains what you saw.

1 Like
7

Thanks for the replies.