Stats processing to give graphical time-series output

tuck1s · November 18, 2022, 6:12pm

Has anyone already come up with a good way to process the summary stats from rclone.log into graphical format? I'm running backup jobs that run for multiple days (slowish ISP upload speed). I'm interested to see the effect of tuning parameters such as --transfers=.

I was thinking to read (or tee) the logfile output through a script to produce a time-series chart.

rclone v1.59.2
- os/version: raspbian 9.13
- os/kernel: 4.19.66+ (armv6l)
- os/type: linux
- os/arch: arm
- go/version: go1.18.6
- go/linking: static
- go/tags: none

Writing to BackBlaze B2.

tuck1s · November 19, 2022, 7:43pm

I made a Python script to pull the date/time and the Transferred: stats out of a human-readable rclone.log. Maybe I should have tried the JSON log output, or the --stats-one-line-date instead, but this seems to work.

TODO: check how rclone treats timestamps with the computer's locale; at the moment it's easy in UK winter time.
Realtime time-series views might be better done via the new GUI web app, or Jupyter Notebook, or Google Data Studio etc.

tuck1s · November 19, 2022, 8:27pm

I just saw some parts of the code have a "remote stats" feed e.g. to Prometheus .. are there docs on that?

asdffdsa · November 19, 2022, 8:54pm

github.com/rclone/rclone

Feature request: export Metrics for Prometheus

opened 05:17PM - 06 Jan 20 UTC

closed 09:58AM - 01 Mar 20 UTC

ccremer

enhancement

#### What is your current rclone version (output from `rclone version`)? Docker image tag 1.50 #### What problem are you are trying to solve? Getting metrics of jobs into a Prometheus scrapable format or pushing into Pushgateway. In a Kubernetes world, where rclone runs as a (Cron)job for backing up application data (e.g. S3 mirroring), the monitoring of jobs and failures is cumbersome. Usually it's done with Prometheus as it's the default monitoring solution. #### How do you think rclone should be changed to solve that? The current method of providing CLI flag `--rc --rc-addr "0.0.0.0:5572"` will enable a webserver that responds with a JSON when accessing `/core/stats`. It would be nice if we can alter the query (e.g. ?format=prometheus) so that it would provide the metrics in OpenMetrics/Prometheus format. That probably should also become a GET request instead of POST. Or another solution altogether. We maintain currently a dedicated project https://github.com/vshn/rclone-exporter that does the JSON-to-prometheus translation, but we figure it's best if it's directly integrated into rclone. For testing purposes one could use this side-project in the meantime.

system · December 19, 2022, 8:54pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.