Wrong / different file count / backup size with Backblaze B2?

Help and Support
#1

What is the problem you are having with rclone?

Hi, I'm trying to sync a single folder with Backblaze B2.
I use:
rclone sync /folder b2:bucket --progress --stats-one-line

The result is that rclone sees and syncs 38252 files with a total of 153.361 GB

If you do do

rclone size b2:bucket

You get

38252 files with size 149.892 GB (160945117018 bytes)

Backblaze itself shows 38256 files and 161 GB size

The issue is that Ubuntu, where the files are, shows 39361 files and 161.0 GB

So the backup size is close between Backblaze and Ubuntu, but the files count isn't.

The files are originally from Mac OS, with system files like .DS_Store etc.

Where comes the difference and how to make sure my files are actually all backed up?

What is your rclone version (output from rclone version)

1.51.0

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Ubuntu 20.04, 64 bit

Which cloud storage system are you using? (eg Google Drive)

Backblaze B2

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync /folder b2:bucket --progress --stats-one-line
rclone size b2:bucket

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

I'm not posting debug for sync this since it's 38k of lines...

#2

The difference would be in the debug.

You'd have to look through to find it or post it.

#3

Alright, logging it to a file then. Is there anything in particular I should be looking for?

Edit:
logged the sync debug, all of the rows based on my random skimming are
2020/05/05 12:58:08 DEBUG : Photos/iPhone 5/2013-09-19 06.48.40.png: Unchanged skipping
2020/05/05 12:58:08 DEBUG : Photos/iPhone 5/2013-09-19 06.53.46.png: Size and modification time the same (differ by 0s, within tolerance 1ms)

So no actual issues. Even the .DS_Store files are synced. Maybe something is different in terms of accounting space taken? But that still doesn't explain different files count (maybe folders are not counted in rclone/b2 and counted on Ubuntu?)

#4

It's a bit of a guessing game with the logs.

Generally, I check for ERROR first and it depends on what files aren't synced. I would try to isolate a file that did not work and check for that in the log and see why.

Usually counts are off because rclone doesn't have access to see something which isn't always in the logs.

Each one is a bit unique so it's a matter of looking through the log and going from there.

#5

Thanks. No errors, will try randomly to find files that aren't synced. I wish there would be a more scientific approach to this :slight_smile:

#6

Are you counting any links or anything like that?

#7

You mean like symlinks? They might not sync, reducing file count and backup size?

#8

Yes, they would show up in the logs as well and could be the counts. Hard to tell without actually seeing a log though as things may jump out.

#9

What you want is rclone check which will check the source vs the destination and tell you exactly what is missing and what isn't.

Also try using rclone size on the source so rclone size /path/to/files then at least you are using the same tool on both so we don't have to worry about GiB vs GB etc.

#10

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.