Inconsistent progression printing

What is the problem you are having with rclone?

When using options -v -P --stats-one-line --stats 5s during a copy I would expect rclone to print out a new progression line every 5 seconds, instead of that, if no other log INFO/ERROR/etc are printed, each progression status are appended.
It's problematic for me since I am piping the output of the rclone command to another to track progression, it makes it unpractical (especially for shorter stats interval duration).

What is your rclone version (output from rclone version)

rclone v1.56.0

  • os/version: debian 10.10 (64 bit)
  • os/kernel: 4.19.0-14-cloud-amd64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.16.5
  • go/linking: static
  • go/tags: none

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Linux 64bits

Which cloud storage system are you using? (eg Google Drive)

not relevant

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy from: to: -v -P --stats-one-line --stats 5s

The rclone config contents with secrets removed.

not relevant

A log from the command with the -vv flag

...
2021-10-08 08:35:42 INFO : image.png: Copied (new)
445.680Mi / 1.013 GiByte, 43%, 35.924 MiByte/s, ETA 16s607.211Mi / 1.013 GiByte, 59%, 40.692 MiByte/s, ETA 10s839.773Mi / 1.013 GiByte, 81%, 42.119 MiByte/s, ETA 4s1.013Gi / 1.013 GiByte, 100%, 44.023 MiByte/s, ETA 0s
2021/10/08 08:35:55 INFO  :     1.013Gi / 1.013 GiByte, 100%, 41.483 MiByte/s, ETA 0s
...

You've asked for info output with -v but don't want it? Remove the -v

I indeed want the progression to be printed to stdout, and without -v it is not.
With -v, it is printed to stdout but there is no carriage return, which is a problem for me, since i parse all lines to extract progression.

Not sure what you mean:

felix@gemini:~$ rclone copy /etc/hosts GD: -P --stats-one-line
165 / 165 Byte, 100%, 164 Byte/s, ETA 0s

That's my output. What are you expecting? Are you trying to parse stdout or a log file?

if you try with a long time running copy, you will have many progression status appended to the same line.
Try "--stats 1s" with "big" file and you will see what's wrong
I run shell command: "rclone copy ... | other-command-parsing-stdout.sh", and it reads line by line... that's my issue

-P progress is for displaying things in real time.

If you want to parse something, you'd want to use a log file and parse that route. It's just redrawing the window which is what I'd expect in real time output.

Run any command like iotop / top and pipe it into something and it will look awful.

top | less

as an example gives you:

You should change your approach.
-P is intended for humans. -P goes to terminal even if via stdout. Don't parse it.
We specifically intended --stat-one-line and --stat for parsing in console wrappers.
For GUI apps we provide RCd, the REST-based server. It provides plenty of various statistics. Please use IT.
Look at the approach used by latest RCX, the GUI wrapper for Android.
Please RTFM.
Thanks

It's not a issue with "RTFM", it's a issue I dont use GUI apps, I dont want use rest based server. I just want a shell command providing me stats, you should provide an option enabling the progression as an INFO or DEBUG log is -P is not intended for scripting.

For scripted things, you can use json output, parse it out how you want.

      --use-json-log                         Use json log format.

looks good but the problem remains the same : I want to get copy progression in the log.

That's in the log:

rclone copy jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv GD: -v --stats 1s --stats-one-line
2021/10/08 14:13:18 INFO  :       124Ki / 1.402 GiByte, 0%, 0 Byte/s, ETA -
2021/10/08 14:13:19 INFO  :     6.996Mi / 1.402 GiByte, 0%, 2.995 MiByte/s, ETA 7m56s
2021/10/08 14:13:20 INFO  :    15.996Mi / 1.402 GiByte, 1%, 5.997 MiByte/s, ETA 3m56s
2021/10/08 14:13:21 INFO  :    30.996Mi / 1.402 GiByte, 2%, 7.663 MiByte/s, ETA 3m3s
2021/10/08 14:13:22 INFO  :    47.996Mi / 1.402 GiByte, 3%, 11.250 MiByte/s, ETA 2m3s
2021/10/08 14:13:23 INFO  :    63.996Mi / 1.402 GiByte, 4%, 11.599 MiByte/s, ETA 1m58s
2021/10/08 14:13:24 INFO  :    73.996Mi / 1.402 GiByte, 5%, 12.001 MiByte/s, ETA 1m53s
2021/10/08 14:13:25 INFO  :    90.996Mi / 1.402 GiByte, 6%, 12.000 MiByte/s, ETA 1m52s
2021/10/08 14:13:26 INFO  :   111.996Mi / 1.402 GiByte, 8%, 12.499 MiByte/s, ETA 1m45s
2021/10/08 14:13:27 INFO  :   123.996Mi / 1.402 GiByte, 9%, 13.778 MiByte/s, ETA 1m35s
2021/10/08 14:13:28 INFO  :   190.809Mi / 1.402 GiByte, 13%, 14.812 MiByte/s, ETA 1m24s
2021/10/08 14:13:29 INFO  :   277.934Mi / 1.402 GiByte, 19%, 21.417 MiByte/s, ETA 54s
2021/10/08 14:13:30 INFO  :   374.121Mi / 1.402 GiByte, 26%, 27.284 MiByte/s, ETA 38s
2021/10/08 14:13:31 INFO  :   424.996Mi / 1.402 GiByte, 30%, 32.311 MiByte/s, ETA 31s
2021/10/08 14:13:32 INFO  :   464.996Mi / 1.402 GiByte, 32%, 32.791 MiByte/s, ETA 29s
2021/10/08 14:13:33 INFO  :   479.996Mi / 1.402 GiByte, 33%, 31.472 MiByte/s, ETA 30s
2021/10/08 14:13:34 INFO  :   490.996Mi / 1.402 GiByte, 34%, 30.379 MiByte/s, ETA 31s
2021/10/08 14:13:35 INFO  :   510.996Mi / 1.402 GiByte, 36%, 29.480 MiByte/s, ETA 31s
2021/10/08 14:13:36 INFO  :   522.996Mi / 1.402 GiByte, 36%, 28.513 MiByte/s, ETA 31s
2021/10/08 14:13:37 INFO  :   541.996Mi / 1.402 GiByte, 38%, 27.918 MiByte/s, ETA 31s
2021/10/08 14:13:38 INFO  :   553.996Mi / 1.402 GiByte, 39%, 27.048 MiByte/s, ETA 32s
2021/10/08 14:13:39 INFO  :   571.996Mi / 1.402 GiByte, 40%, 26.358 MiByte/s, ETA 32s
2021/10/08 14:13:40 INFO  :   585.996Mi / 1.402 GiByte, 41%, 25.664 MiByte/s, ETA 33s
2021/10/08 14:13:41 INFO  :   605.996Mi / 1.402 GiByte, 42%, 24.857 MiByte/s, ETA 33s
2021/10/08 14:13:42 INFO  :   617.996Mi / 1.402 GiByte, 43%, 24.429 MiByte/s, ETA 33s
2021/10/08 14:13:43 INFO  :   633.996Mi / 1.402 GiByte, 44%, 23.963 MiByte/s, ETA 33s
2021/10/08 14:13:44 INFO  :   651.996Mi / 1.402 GiByte, 45%, 23.279 MiByte/s, ETA 33s
2021/10/08 14:13:45 INFO  :   667.777Mi / 1.402 GiByte, 47%, 23.198 MiByte/s, ETA 33s
2021/10/08 14:13:46 INFO  :   687.996Mi / 1.402 GiByte, 48%, 22.560 MiByte/s, ETA 33s
2021/10/08 14:13:47 INFO  :   701.996Mi / 1.402 GiByte, 49%, 22.026 MiByte/s, ETA 33s
2021/10/08 14:13:48 INFO  :   716.996Mi / 1.402 GiByte, 50%, 22.274 MiByte/s, ETA 32s
2021/10/08 14:13:49 INFO  :   735.996Mi / 1.402 GiByte, 51%, 21.319 MiByte/s, ETA 32s
2021/10/08 14:13:50 INFO  :   749.996Mi / 1.402 GiByte, 52%, 20.987 MiByte/s, ETA 32s
2021/10/08 14:13:51 INFO  :   757.996Mi / 1.402 GiByte, 53%, 20.550 MiByte/s, ETA 32s
2021/10/08 14:13:52 INFO  :   776.996Mi / 1.402 GiByte, 54%, 20.078 MiByte/s, ETA 32s
2021/10/08 14:13:53 INFO  :   792.996Mi / 1.402 GiByte, 55%, 19.886 MiByte/s, ETA 32s
2021/10/08 14:13:54 INFO  :   811.996Mi / 1.402 GiByte, 57%, 19.831 MiByte/s, ETA 31s
2021/10/08 14:13:55 INFO  :   824.996Mi / 1.402 GiByte, 57%, 19.529 MiByte/s, ETA 31s
2021/10/08 14:13:56 INFO  :   840.996Mi / 1.402 GiByte, 59%, 19.182 MiByte/s, ETA 30s
2021/10/08 14:13:57 INFO  :   855.996Mi / 1.402 GiByte, 60%, 19.047 MiByte/s, ETA 30s
2021/10/08 14:13:58 INFO  :   874.996Mi / 1.402 GiByte, 61%, 18.919 MiByte/s, ETA 29s
2021/10/08 14:13:59 INFO  :   884.996Mi / 1.402 GiByte, 62%, 18.487 MiByte/s, ETA 29s
2021/10/08 14:14:00 INFO  :   920.996Mi / 1.402 GiByte, 64%, 18.431 MiByte/s, ETA 27s
2021/10/08 14:14:01 INFO  :   935.996Mi / 1.402 GiByte, 65%, 19.305 MiByte/s, ETA 25s
2021/10/08 14:14:02 INFO  :   951.996Mi / 1.402 GiByte, 66%, 18.973 MiByte/s, ETA 25s
2021/10/08 14:14:03 INFO  :   968.996Mi / 1.402 GiByte, 68%, 19.037 MiByte/s, ETA 24s
2021/10/08 14:14:04 INFO  :   982.996Mi / 1.402 GiByte, 68%, 18.722 MiByte/s, ETA 24s
2021/10/08 14:14:05 INFO  :  1002.996Mi / 1.402 GiByte, 70%, 18.652 MiByte/s, ETA 23s
2021/10/08 14:14:06 INFO  :  1017.996Mi / 1.402 GiByte, 71%, 18.449 MiByte/s, ETA 22s
2021/10/08 14:14:07 INFO  :     1.008Gi / 1.402 GiByte, 72%, 18.297 MiByte/s, ETA 22s
2021/10/08 14:14:08 INFO  :     1.025Gi / 1.402 GiByte, 73%, 18.215 MiByte/s, ETA 21s
2021/10/08 14:14:09 INFO  :     1.042Gi / 1.402 GiByte, 74%, 18.326 MiByte/s, ETA 20s
2021/10/08 14:14:10 INFO  :     1.060Gi / 1.402 GiByte, 76%, 18.243 MiByte/s, ETA 19s
2021/10/08 14:14:11 INFO  :     1.074Gi / 1.402 GiByte, 77%, 18.166 MiByte/s, ETA 18s
2021/10/08 14:14:12 INFO  :     1.100Gi / 1.402 GiByte, 78%, 18.093 MiByte/s, ETA 17s
2021/10/08 14:14:13 INFO  :     1.141Gi / 1.402 GiByte, 81%, 20.188 MiByte/s, ETA 13s
2021/10/08 14:14:14 INFO  :     1.147Gi / 1.402 GiByte, 82%, 19.387 MiByte/s, ETA 13s
2021/10/08 14:14:15 INFO  :     1.158Gi / 1.402 GiByte, 83%, 18.676 MiByte/s, ETA 13s
2021/10/08 14:14:16 INFO  :     1.172Gi / 1.402 GiByte, 84%, 18.508 MiByte/s, ETA 12s
2021/10/08 14:14:17 INFO  :     1.192Gi / 1.402 GiByte, 85%, 18.540 MiByte/s, ETA 11s
2021/10/08 14:14:18 INFO  :     1.210Gi / 1.402 GiByte, 86%, 18.380 MiByte/s, ETA 10s
2021/10/08 14:14:19 INFO  :     1.221Gi / 1.402 GiByte, 87%, 17.919 MiByte/s, ETA 10s
2021/10/08 14:14:20 INFO  :     1.238Gi / 1.402 GiByte, 88%, 17.737 MiByte/s, ETA 9s
2021/10/08 14:14:21 INFO  :     1.248Gi / 1.402 GiByte, 89%, 18.004 MiByte/s, ETA 8s
2021/10/08 14:14:22 INFO  :     1.267Gi / 1.402 GiByte, 90%, 17.503 MiByte/s, ETA 7s
2021/10/08 14:14:23 INFO  :     1.282Gi / 1.402 GiByte, 91%, 17.597 MiByte/s, ETA 6s
2021/10/08 14:14:24 INFO  :     1.295Gi / 1.402 GiByte, 92%, 17.267 MiByte/s, ETA 6s
2021/10/08 14:14:25 INFO  :     1.315Gi / 1.402 GiByte, 94%, 17.230 MiByte/s, ETA 5s
2021/10/08 14:14:26 INFO  :     1.376Gi / 1.402 GiByte, 98%, 19.342 MiByte/s, ETA 1s
2021/10/08 14:14:27 INFO  :     1.391Gi / 1.402 GiByte, 99%, 20.069 MiByte/s, ETA 0s
2021/10/08 14:14:28 INFO  :     1.402Gi / 1.402 GiByte, 100%, 19.753 MiByte/s, ETA 0s
2021/10/08 14:14:29 INFO  :     1.402Gi / 1.402 GiByte, 100%, 18.658 MiByte/s, ETA 0s
2021/10/08 14:14:29 INFO  : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv: Copied (replaced existing)
2021/10/08 14:14:29 INFO  :     1.402Gi / 1.402 GiByte, 100%, 18.658 MiByte/s, ETA 0s

Unsubscribing.

:open_mouth:

indeed, with the right set of options it works as expected :hugs:
Thanks for help

What is the problem here? I am explaining what I want to achieve.

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.