I have a long-running rclone sync command, and I want to be able to view real-time progress in the terminal while also logging progress to a file. Although combining --log-file and --progress flags almost does what I want, I have not been able to find a way to periodically log stats to file in addition to displaying in the terminal.
I want to show progress in the terminal updated every 0.5s and log stats to file every 60s. I tried this set of flags:
However, with these flags the terminal progress interval is set to 60s, while stats are still not logged to file at all. It appears that the --progress flag supersedes normal stats logging.
If there is currently no way to have stats both displayed in the terminal and written to the logs, I'd like to propose this as a new feature. The basic requirements are:
Provide a way to enable both progress and stats logging at the same time
Provide a way to specify separate intervals for progress and stats logging
Preserve backward compatibility with the current behavior
One solution would be to add a new --progress-interval duration flag to specify the progress interval. If both --progress-interval and --stats are given then progress is both displayed in the terminal and written to logs at the respective intervals.
With this hypothetical flag, I could achieve the desired behavior for my use case with:
Progress and stats printing use the same mechanism, that's why you can only have one at a time at the moment.
I think I'd prefer a flag like --log-stats-interval which if set would log stats to the log file at the given interval regardless of the other settings. Maybe at a log level to ensure that the stats were always present.
It also seems like it is getting quite complicated!
This is already what --stats does if --progress is not enabled, which I think would be confusing/redundant.
The reasoning behind my suggestion is as follows. If designing this from scratch I would make --progress and --stats independent flags, each with a duration parameter:
--progress 1s: Print progress only
--stats 5m: Log stats only
--progress 1s --stats 5m: Print progress and log stats
The problem is, this would not be backward compatible with the existing behavior. So my idea is to add a new parameterized flag which can be used instead of --progress to achieve the desired behavior. However I can't really think of a great name for this new flag.
An alternative approach which might be less confusing would be to make --progress a parametrized flag with a no option default value. This has the downside of requiring the user to specify --progress=<duration> instead of --progress <duration>, but otherwise could be used to preserve backward compatibility while supporting the proposed functionality.
If you think this is a reasonable feature I can put together a PR in the next few days.
Hmm, that sounds like a better idea. I hadn't heard of NoOptDefVal before.
Does it? I wasn't sure reading the docs. That makes sense from a parsing point of view though.
Yes....
So what would the combinations mean?
--progress
Show progress at the default rate of 0.5s and don't log stats
--stats 5s
Log stats at the rate of 5s and don't show progress
--progress --stats 5s
Show progress at the rate of 5s don't log stats - this is the case that it would be nice to deprecate.
--progress=5s
Show progress at the rate of 5s and logs stats at the default rate? Is that what we mean by the --progress= that it sets only the progress and not the log stats?
--progress=5s --stats 1m
Show progress at the rate of 5s, stats at the rate of 1m
I wonder if we could deprecate --stats 42s --progress and make a warning message to use --progress=42s instead? Then we could move towards separating the values which does make a lot more sense with the benefit of hindsight!
I note that people use progress they almost always use -P instead of --progress
What you describe is pretty much what I had in mind, except I think it would be more convenient if setting --progress=<duration> without --stats disabled the default stats logging.
However coming back to this after some time, I think this feature request has a pretty marginal benefit so it might not be worth the complexity of implementing it.