Rclone sync OneDrive; Scanning files?

Help and Support
1

I am using rclone to sync about 220,000 files from a local HDD to OneDrive.

rclone v1.44-029-g04a0da1f-beta
- os/arch: linux/arm
- go version: go1.11.1

I am confused about the once-per-minute status as shown in the log file; specifically the file counts.

Here are 2 successive status outputs:

Transferred:        1.825G / 48.982 GBytes, 4%, 991.997 kBytes/s, ETA 13h50m46s
Errors:                 0
Checks:               307 / 307, 100%
Transferred:          394 / 10410, 4%


Transferred:        1.883G / 49.026 GBytes, 4%, 992.546 kBytes/s, ETA 13h50m4s
Errors:                 0
Checks:               307 / 307, 100%
Transferred:          404 / 10420, 4%

In the last line Transferred: for file count, I see the left number go up (from 394 to 404), but the right number also goes up. Does that mean that rclone doesn’t have a complete count/list of files to upload before it begins uploading?

Because there are so many files to transfer, this runs over many days. But the ETA always hovers around 13-14 hours, which is incorrect.

Am I correct in assuming rclone scans the files while it is uploading? If so, is there any way to tell rclone to scan everything before it starts transferring, so that the counts (and hopefully the ETA) are correct?

Thanks,

Steven

2

That is correct. Rclone allows itself to get 10,000 files ahead by default, but no more so it doesn’t use too much memory. If you want it to get futher ahead use this flag

  --max-backlog int           Maximum number of objects in sync or check backlog. (default 10000)

If you set it to 1000000 say then rclone would scan your entire disk reasonably quickly and give you an accurate estimate at the cost of using more memory in rclone.

Great question BTW - I should probably put it in the FAQ.

3

Thanks - that option makes rclone gather information on all files, not just 10,000 ahead as you said. It still starts the copy before all files are counted, but within a few minutes it completes the work, and that is fine.

Now the ETA is depressingly accurate:

Transferred:        7.647G / 283.343 GBytes, 3%, 920.633 kBytes/s, ETA 87h13m30s
Errors:                 0
Checks:            160810 / 160810, 100%
Transferred:         1612 / 60408, 3%

One side question: the line for Checks indicates those files that rclone does not have to transfer, correct?

4

Ah well!

Yes those are files that rclone has worked out don’t need to be transferred.