`rclone copy` reporting WAY more data copied to a chunker remote than it actually was

Suspected Bug
1

What is the problem you are having with rclone?

Volume of data (in bytes) reported as "Transferred:" by rclone copy is vastly larger than what as actually copied.

Run the command 'rclone version' and share the full output of the command.

rclone v1.63.1

  • os/version: debian 11.7 (64 bit)
  • os/kernel: 6.1.0-0.deb11.7-cloud-arm64 (aarch64)
  • os/type: linux
  • os/arch: arm64 (ARMv8 compatible)
  • go/version: go1.20.6
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Box, under crypt, under chunker (see below for full config)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone -v copy /tmp/LOCALDIR CHUNKED_ENCRYPTED_REMOTE:REDACTED1/REMOTE_DIR

The rclone config contents with secrets removed.

[BASE_REMOTE]
type = box
token = {"access_token":"REDACTED","token_type":"bearer","refresh_token":"REDACTED","expiry":"REDACTED"}

[ENCRYPTED_BASE_REMOTE]
type = crypt
remote = BASE_REMOTE:
password = REDACTED
filename_encoding = base32768

[CHUNKER_ENCRYPTED_REMOTE]
type = chunker
remote = ebox:
chunk_size = 4G
name_format = *.rcc###
hash_type = sha1all

A log from the command with the -vv flag

It's not easy for me to generate a full -vv log for the command as it's running on a VPS with very little disk space.

Here is what the above rclone copy command prints to its stderr right before when it's finished:

Transferred:        5.467 GiB / 5.467 GiB, 100%, 35.397 MiB/s, ETA 0s
Checks:                29 / 29, 100%
Renamed:               29
Transferred:           29 / 29, 100%
Elapsed time:      2m53.4s

The same rclone copy command when ran replacing the above remote to an crypt-on-top-of-GoogleDrive remote, outputs the correct size when it's done:

Transferred:        2.734 GiB / 2.734 GiB, 100%, 14.984 MiB/s, ETA 0s
Checks:                58 / 58, 100%
Deleted:               29 (files), 5 (dirs)
Renamed:               29
Transferred:           29 / 29, 100%
Elapsed time:      1m38.4s

I strongly suspect the problem to be in the chunker layer.

And here's the output of a du -sh /tmp/LOCALDIR:

2.8G    /tmp/LOCALDIR

Also, here's the output of an rclone size CHUNKED_ENCRYPTED_REMOTE:REDACTED1/REMOTE_DIR right after the copy:

Total objects: 29 (29)
Total size: 2.733 GiB (2934895904 Byte)

This issue is AFAICS perfectly repeatable, has happened to all the directories I've copied to this remote so far.

Please let me know if a -vv log is indeed indispensable as it will not be simple to generate (space is really short at that VPS) and please feel free to contact me if you need more info.

Ditto if I should go ahead and create an issue for this on rclone' s github.

2

It is not a bug but unfortunate design decision to include in total transfer both real and server side operations. Many people were requesting this way of counting. It is impossible to keep everybody happy:)

So in the next rclone version both types of transfers will be split in stats:

you can try it now - link to download in github ticket.

3

Not sure I follow you here. What would be the rationale for including both? Do server side (which I presume are the chunker rename`s) count towards some quota or similar?

Perfect! Glad to hear this is addressed! And many thanks for the prompt and complete response!

4

Here I am in the same camp as you but if you search this forum many think differently:) well. people are never the same:)

1 Like
5

As the proverb says, " different strokes for different folks" -- I guess specially for those that have had strokes :slight_smile:

1 Like
6

There is a beta with the new stats in the issue posted above if you wan to have a try @durval

1 Like
7

Thanks for the great support as usual @ncw! Will try it as soon as I finish migrating my Drive data!

1 Like
8

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.