How to get accurate estimate for large transfers

Hi there,

We are doing a large sync to Google Drive, and I am wondering if it is possible to get an accurate estimate of the amount of files left to transfer / time remaining? To try and improve things, I added:

--fast-list

which improved things significantly, both in terms of speed and giving a better initial indication of the amount of data needed to be transferred, and while it works well on most transfers, on the large sync we are doing now (~27TB), the amount of data left to transfer still slowly grows over time, making it very difficult to estimate how much longer the transfer will take.

For example, here is a sample of the output of rclone during the sync:

Transferred:          530.056G / 729.765 GBytes, 73%, 8.532 MBytes/s, ETA 6h39m29s
Errors:                19 (retrying may help)
Checks:           1481621 / 1481621, 100%
Transferred:        24135 / 34147, 71%
Elapsed time:  17h40m19.7s

Then two minutes later:

Transferred:           531.109G / 729.768 GBytes, 73%, 8.535 MBytes/s, ETA 6h37m14s
Errors:                19 (retrying may help)
Checks:           1481621 / 1481621, 100%
Transferred:        24143 / 34155, 71%
Elapsed time:  17h42m2.2s

Then some hours later:

Transferred:             1.152T / 1.279 TBytes, 90%, 8.151 MBytes/s, ETA 4h31m56s
Errors:                20 (retrying may help)
Checks:           1557484 / 1557484, 100%
Transferred:        76375 / 86387, 88%
Elapsed time:  41h9m18.2s

If you look closely, you can see under ‘Transferred:’, the total amount of files to be transferred has gone up. Approximately every minute a few files are added to this total.

Or is this behaviour because this sync is from a filesystem that is currently in use, and rclone is updating to reflect the new files being added to the filesystem?

It is:

  --max-backlog int                              Maximum number of objects in sync or check backlog. (default 10000)

As that is how much is reads ahead for files. You can increase that number at the cost of more memory consumption.

1 Like

Hi Animosity022 - thanks so much for that, I will try that on the next sync.

Much appreciated.

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.