Progress shows incorrect total size to be transferred

Help and Support
1

I’m not sure if this is a bug or an intended feature, but the total size of files to be transferred seems to be incorrect when I use the --progress flag. The total size starts small and then increases as the sync continues. Is this the expected behavior?

I’m syncing an encrypted restic repository to a wasabi bucket. The total size of the restic repo is 435G. The ETA seems to be affected since it nearly stays the same whether 1G or 79G has transferred.

command:

$ rclone sync -P /mnt/drive/restic wasabi:$BUCKET/restic

output:

Transferred:        1.091G / 49.276 GBytes, 2%, 1.411 MBytes/s, ETA 9h42m41s
Transferred:        2.603G / 50.826 GBytes, 5%, 1.401 MBytes/s, ETA 9h47m28s
Transferred:        9.063G / 57.194 GBytes, 16%, 1.389 MBytes/s, ETA 9h51m27s
Transferred:       66.278G / 114.548 GBytes, 58%, 1.395 MBytes/s, ETA 9h50m42s
Transferred:       79.172G / 127.435 GBytes, 62%, 1.392 MBytes/s, ETA 9h51m37s

version:

rclone v1.45
- os/arch: linux/amd64
- go version: go1.11.2
2

Expected behavior. rclone does many things behind the scenes when you are syncing, including checking existing files in the remote to see if they should be changed, and also rclone queues only a few files at the time for uploading instead of everything on your destination, which is why it may not be accurate whne a sync starts. As a sync progresses and most/all files have been checked and it has queued most files for upload, the reporting will be more accurate.

3

@AndyIbanezK Oh, alright. Thanks for the response. So the ETA is a bit misleading, if not useless, until all the files have been checked/queued? The last transferred output showing 127G was when the repo had already been syncing for 16+ hours.

It’s not a huge deal since I’ll usually be running it in the background. Also, this was the very first sync, and there was nothing on the destination server to check.

Is there anything I can do to have rclone check/queue files faster so I have a more accurate ETA sooner? It looks like it’s going to be another day or two before the ETA is actually useful.

4

Rclone, by default, only gets 10,000 files ahead to save on memory.

You can use this config parameter to allow it to get further ahead.

  --max-backlog int           Maximum number of objects in sync or check backlog. (default 10000)

If that number is bigger than the number of files you have to transfer then you will get accurate transfer stats! Otherwise they will get there eventually.

2 Likes
5

@ncw Oh, neat! Thanks for pointing that out.

Wish I would have noticed that sooner. :face_with_hand_over_mouth: