Use the average transfer rate when using --stats-one-line-date

chscott · May 28, 2022, 5:14pm

It appears --stats-one-line-date uses the current transfer rate. That can lead to something like this for brief dips in the transfer rate on a copy that has been running for days and will run for a couple more days.

2022/05/27 15:56:15 INFO  : 2022/05/27 15:56:15 -   688.646 GiB / 1.107 TiB, 61%, 5.386 MiB/s, ETA 23h30m9s (xfr#272368/925168)
2022/05/27 15:56:20 INFO  : 2022/05/27 15:56:20 -   688.665 GiB / 1.107 TiB, 61%, 5.134 MiB/s, ETA 1d39m17s (xfr#272378/925168)
2022/05/27 15:56:25 INFO  : 2022/05/27 15:56:25 -   688.691 GiB / 1.107 TiB, 61%, 5.103 MiB/s, ETA 1d48m6s (xfr#272387/925168)
2022/05/27 15:56:30 INFO  : 2022/05/27 15:56:30 -   688.714 GiB / 1.107 TiB, 61%, 5.267 MiB/s, ETA 1d1m45s (xfr#272399/925168)
2022/05/27 15:56:35 INFO  : 2022/05/27 15:56:35 -   688.734 GiB / 1.107 TiB, 61%, 5.011 MiB/s, ETA 1d1h15m10s (xfr#272404/925168)
2022/05/27 15:56:40 INFO  : 2022/05/27 15:56:40 -   688.762 GiB / 1.107 TiB, 61%, 4.860 MiB/s, ETA 1d2h2m10s (xfr#272410/925168)
2022/05/27 15:56:45 INFO  : 2022/05/27 15:56:45 -   688.789 GiB / 1.107 TiB, 61%, 5.009 MiB/s, ETA 1d1h15m40s (xfr#272419/925168)
2022/05/27 15:56:50 INFO  : 2022/05/27 15:56:50 -   688.806 GiB / 1.107 TiB, 61%, 4.685 MiB/s, ETA 1d3h35s (xfr#272428/925168)
2022/05/27 15:56:55 INFO  : 2022/05/27 15:56:55 -   688.832 GiB / 1.107 TiB, 61%, 4.736 MiB/s, ETA 1d2h42m46s (xfr#272437/925168)
2022/05/27 15:57:00 INFO  : 2022/05/27 15:57:00 -   688.852 GiB / 1.107 TiB, 61%, 4.848 MiB/s, ETA 1d2h5m49s (xfr#272444/925168)
2022/05/27 15:57:05 INFO  : 2022/05/27 15:57:05 -   688.855 GiB / 1.107 TiB, 61%, 3.715 MiB/s, ETA 1d10h3m16s (xfr#272454/925168)
2022/05/27 15:57:10 INFO  : 2022/05/27 15:57:10 -   688.856 GiB / 1.107 TiB, 61%, 2.712 MiB/s, ETA 1d22h38m29s (xfr#272467/925168)
2022/05/27 15:57:15 INFO  : 2022/05/27 15:57:15 -   688.856 GiB / 1.107 TiB, 61%, 1.969 MiB/s, ETA 2d16h15m27s (xfr#272478/925168)
2022/05/27 15:57:20 INFO  : 2022/05/27 15:57:20 -   688.856 GiB / 1.107 TiB, 61%, 1.426 MiB/s, ETA 3d16h42m54s (xfr#272494/925168)
2022/05/27 15:57:25 INFO  : 2022/05/27 15:57:25 -   688.856 GiB / 1.107 TiB, 61%, 1.033 MiB/s, ETA 5d2h26m21s (xfr#272512/925168)
2022/05/27 15:57:30 INFO  : 2022/05/27 15:57:30 -   688.856 GiB / 1.107 TiB, 61%, 771.146 KiB/s, ETA 6d23h59m53s (xfr#272529/925168)
2022/05/27 15:57:35 INFO  : 2022/05/27 15:57:35 -   688.856 GiB / 1.107 TiB, 61%, 565.273 KiB/s, ETA 1w2d13h10m59s (xfr#272545/925168)
2022/05/27 15:57:40 INFO  : 2022/05/27 15:57:40 -   688.856 GiB / 1.107 TiB, 61%, 410.445 KiB/s, ETA 1w6d3h38m8s (xfr#272557/925168)
2022/05/27 15:57:45 INFO  : 2022/05/27 15:57:45 -   688.856 GiB / 1.107 TiB, 61%, 298.798 KiB/s, ETA 2w4d1h34m27s (xfr#272564/925168)
2022/05/27 15:57:50 INFO  : 2022/05/27 15:57:50 -   688.857 GiB / 1.107 TiB, 61%, 282.982 KiB/s, ETA 2w5d1h48m19s (xfr#272573/925168)
2022/05/27 15:57:55 INFO  : 2022/05/27 15:57:55 -   688.857 GiB / 1.107 TiB, 61%, 208.823 KiB/s, ETA 3w4d20h23m5s (xfr#272581/925168)
2022/05/27 15:58:00 INFO  : 2022/05/27 15:58:00 -   688.857 GiB / 1.107 TiB, 61%, 151.860 KiB/s, ETA 5w13h5m18s (xfr#272586/925168)
2022/05/27 15:58:05 INFO  : 2022/05/27 15:58:05 -   688.857 GiB / 1.107 TiB, 61%, 110.301 KiB/s, ETA 6w6d22h30m58s (xfr#272595/925168)
2022/05/27 15:58:10 INFO  : 2022/05/27 15:58:10 -   688.857 GiB / 1.107 TiB, 61%, 80.504 KiB/s, ETA 9w4d1h14m36s (xfr#272603/925168)
2022/05/27 15:58:15 INFO  : 2022/05/27 15:58:15 -   688.857 GiB / 1.107 TiB, 61%, 60.670 KiB/s, ETA 12w4d23h19m12s (xfr#272617/925168)
2022/05/27 15:58:20 INFO  : 2022/05/27 15:58:20 -   688.857 GiB / 1.107 TiB, 61%, 45.117 KiB/s, ETA 17w15h25m43s (xfr#272624/925168)
2022/05/27 15:58:25 INFO  : 2022/05/27 15:58:25 -   688.857 GiB / 1.107 TiB, 61%, 34.435 KiB/s, ETA 22w2d18h9m1s (xfr#272635/925168)
2022/05/27 15:58:30 INFO  : 2022/05/27 15:58:30 -   688.857 GiB / 1.107 TiB, 61%, 29.544 KiB/s, ETA 26w16h53m20s (xfr#272643/925167)
2022/05/27 15:58:35 INFO  : 2022/05/27 15:58:35 -   688.858 GiB / 1.107 TiB, 61%, 21.763 KiB/s, ETA 35w3d51m38s (xfr#272654/925167)
2022/05/27 15:58:40 INFO  : 2022/05/27 15:58:40 -   688.858 GiB / 1.107 TiB, 61%, 16.138 KiB/s, ETA 47w5d11h32m8s (xfr#272669/925167)
2022/05/27 15:58:45 INFO  : 2022/05/27 15:58:45 -   688.858 GiB / 1.107 TiB, 61%, 18.815 KiB/s, ETA 40w6d21h3m24s (xfr#272681/925164)
2022/05/27 15:58:50 INFO  : 2022/05/27 15:58:50 -   688.858 GiB / 1.107 TiB, 61%, 16.001 KiB/s, ETA 48w1d8h3m29s (xfr#272695/925164)
2022/05/27 15:58:55 INFO  : 2022/05/27 15:58:55 -   688.858 GiB / 1.107 TiB, 61%, 12.900 KiB/s, ETA 1y7w4d10h20m47s (xfr#272704/925164)

Note that this is a sampling over less than three minutes and how it leads to absurd ETA estimates. For this use case, I would much rather have the average transfer rate displayed and used to calculate the ETA.

Animosity022 · May 28, 2022, 6:29pm

See:

Transferred rate has changed from average rate to instant rate, and appears twice · Issue #5755 · rclone/rclone (github.com)

ncw · May 30, 2022, 10:06am

Yes, we did change this from the average over the whole transfer to an average over a minute (I think).

People wanted to see that their internet speed was fluctuating.

Your example does look a little strange I agree!

Maybe we should make the time of the average be configurable, or increase during the transfer or something else?

chscott · May 30, 2022, 1:52pm

In my case, I believe it is less about the internet speed fluctuating than the size of files fluctuating. When the copy gets to a tranche of small files, the cumulative size of the files decreases significantly, leading to a moment-in-time data transfer rate that is not reflective of the overall dataset. This is compounded by printing stats every five seconds, I think.

I would be in favor of a couple possible approaches:

A flag to allow going back to using the average of the whole transfer.
A flag to configure the amount of time used to calculate the average.

system · July 29, 2022, 1:53pm

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.