My first backup from local disk to s3 - size differs

Help and Support
1

What is the problem you are having with rclone?

du -skh reports 106G
s3 provider (scaleway) Data usage: 133 GB

  • The number of local files and objects on the server are the same.
  • I am using rclone encrypt (does this make a difference?)

What is your rclone version (output from rclone version)

rclone v1.49.1

  • os/arch: linux/arm64
  • go version: go1.12.9

Which OS you are using and how many bits (eg Windows 7, 64 bit)

aarch64 linux

Which cloud storage system are you using? (eg Google Drive)

scaleway s3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

2

Yes but it is very small - a fraction of a percent.

What does rclone size say when you try it on both the local, the crypted remote and the remote underlying the crypted remote?

One possible difference is you have hardlinked files which rclone will upload twice but will only count once in du.

3

Seems size function outputs same size on both storage's (see below), so is actual size difference reported by Scaleway because of encryption? 133G vs 106G is 25% extra, not a fraction of percent.

local

rclone size 5ido
Total objects: 93248
Total size: 105.095 GBytes (112845332506 Bytes)

remote

rclone size tlc:/5ido
Total objects: 93248
Total size: 105.095 GBytes (112845332506 Bytes)

4

That sounds like too much...

What do you get if you run rclone size on the underlying remote that tlc:/5ido uses?

5

rclone size tlc:
Total objects: 93248
Total size: 105.095 GBytes (112845332506 Bytes)
rclone size tlc:5ido
Total objects: 93248
Total size: 105.095 GBytes (112845332506 Bytes)

Only in the Scaleway website it shows the actual(?) size, which is much more than the data I am backing up.

6

In your config file you have something like this

[tlc]
type = crypt
remote = something:bucket

I'd like you to run

rclone size something:bucket

That should compare directly to what Scaleway is showing.

Do scaleway store old versions? That might explain the difference?

7

it says tlc:5ido, and this is the size I have just posted.

There are no older versions as far as I know, it's my first and only backup.

8

You wrote before

Because these are exactly the same this made me think that you were showing the crypted result, not the result of the underlying storage. I would expect the crypted result to be slightly bigger (0.1%) than the local storage.

Can you paste your config file - removing all secrets?

9

tl is the unencrypted repo, tlc is the encrypted one.

size of tl does infact differ from tlc (by 31,588,496 bytes).

rclone size tl:
Total objects: 93248
Total size: 105.125 GBytes (112876921002 Bytes)

Still does not explain why Scaleway says it's a different size.

[tlc]
type = crypt
remote = tl:xxx
filename_encryption = obfuscate
directory_name_encryption = true

[tl]
type = s3
provider = other
env_auth = false
endpoint = https://s3.nl-ams.scw.cloud
region = nl-ams
acl = private

10

That looks exactly how I expected..

I think this must be some thing to do with the accounting at scaleway, either that or revisions.

Another possibility might be unfinished multipart uploads which should expire eventually.

Can you ask them?