Transfer stats - inaccurate speed

drwtsn32 · October 3, 2024, 4:25pm

What is the problem you are having with rclone?

Transfer stats don't show accurate speed. Value shown is way lower than it should be.

Run the command 'rclone version' and share the full output of the command.

rclone v1.68.1
- os/version: ubuntu 20.04 (64 bit)
- os/kernel: 5.4.0-196-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.23.1
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Azure Blob.

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone sync --verbose --stats-log-level NOTICE --stats=1h --bwlimit "$RC_BW_LIMIT" --retries 1 \
  --track-renames --track-renames-strategy modtime \
  --use-mmap \
  --checkers 40 --transfers 48 --azureblob-upload-concurrency 48 \
  src:/path dst:/path

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[src]
type = smb
host = XXX
user = XXX
pass = XXX
domain = XXX

[dst]
type = azureblob
account = XXX
key = XXX

A log from the command that you were trying to run with the `-vv` flag

Here is a subset of the log. Transfer stats are shown every hour per the command line option I am using. Last three transfer stat outputs are shown.

Note the transfer rate shown is VERY low, ~24KiB/sec. But I'm transferring about 1/2 TiB every hour.

2024/10/03 09:33:04 NOTICE: 
Transferred:       33.638 TiB / 33.644 TiB, 100%, 24.131 KiB/s, ETA 3d8h17m
Checks:          56928469 / 56928469, 100%
Deleted:            18621 (files), 0 (dirs), 1.232 TiB (freed)
Renamed:            18621
Transferred:      5714253 / 5724341, 100%
Server Side Copies: 18621 @ 1.232 TiB
Elapsed time:       3d0.3s

2024/10/03 10:33:04 NOTICE: 
Transferred:       34.066 TiB / 34.065 TiB, 100%, 24.131 KiB/s, ETA -
Checks:          56928475 / 56928475, 100%
Deleted:            18624 (files), 0 (dirs), 1.234 TiB (freed)
Renamed:            18624
Transferred:      5809997 / 5816022, 100%
Server Side Copies: 18624 @ 1.234 TiB
Elapsed time:     3d1h0.3s

2024/10/03 11:33:04 NOTICE: 
Transferred:       34.483 TiB / 34.539 TiB, 100%, 24.131 KiB/s, ETA 4w1d2h
Checks:          56928483 / 56928483, 100%
Deleted:            18628 (files), 0 (dirs), 1.236 TiB (freed)
Renamed:            18628
Transferred:      5909402 / 5919490, 100%
Server Side Copies: 18628 @ 1.236 TiB
Elapsed time:     3d2h0.3s

darthShadow · October 3, 2024, 10:05pm

github.com/rclone/rclone

Speed and ETA are incorrect after a rescan

opened 02:27PM - 28 May 23 UTC

SimplicityGuy

bug stats

#### What is the problem you are having with rclone? During an initial backup, …some files fail and there are errors reported. rclone correctly rescans the source after all files have been uploaded. However, after the rescan the speed and ETA metrics do now update correctly. For instance, this is the estimate after a rescan: `21.510 KiB/s, ETA 1w1d12h`, meanwhile before the rescan the estimate was `1.142 MB/s, ETA 1h20m` #### What is your rclone version (output from `rclone version`) v1.62.2 #### Which OS you are using and how many bits (e.g. Windows 7, 64 bit) Debian 11.7 (64 bit), kernel 5.10.0-20-amd64 #### Which cloud storage system are you using? (e.g. Google Drive) Backblaze B2 #### The command you were trying to run (e.g. `rclone copy /tmp remote:tmp`) `rclone sync --log-file=ams.rclone.log /mnt/md0 backup-b2-crypt:/target/` with these environment variables set: ```bash RCLONE_CHECK_FIRST=1 RCLONE_CHECKSUM=1 RCLONE_LOG_LEVEL=ERROR RCLONE_PROGRESS=1 RCLONE_PROGRESS_TERMINAL_TITLE=1 RCLONE_TRANSFERS=32 ``` #### A log from the command with the `-vv` flag (e.g. output from `rclone -vv copy /tmp remote:tmp`) _did not capture_ 😞 #### How to use GitHub * Please use the 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to show that you are affected by the same issue. * Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue. * Subscribe to receive notifications on status change and new comments.

drwtsn32 · October 3, 2024, 11:56pm

How can I tell if a "rescan" of the source happened?

darthShadow · October 4, 2024, 6:35am

It's not actually related to a rescan. Anytime your process has no transfers for more than a minute, the speed stops updating.

drwtsn32 · October 4, 2024, 4:58pm

Strange. In my case there are definitely transfers in progress at the time the stats are displayed.

Example:

drwtsn32 · October 7, 2024, 10:22pm

New run, it's interesting that the speed shown (for a particular run) is always the same value

darthShadow · October 10, 2024, 6:36am

To clarify the problem further, the value of the speed gets frozen to the value if there ever happens to be a window of 1 minute where there are no transfers active. That value never gets unfrozen even after transfers start again.

system · November 9, 2024, 6:36am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.