# Stats processing to give graphical time-series output

**URL:** <https://forum.rclone.org/t/stats-processing-to-give-graphical-time-series-output/34186>\
**Category:** Help and Support\
**Created:** [November 18, 2022, 6:12pm UTC](https://forum.rclone.org/t/stats-processing-to-give-graphical-time-series-output/34186 "2022-11-18T18:12:24Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![tuck1s](https://forum.rclone.org/user_avatar/forum.rclone.org/tuck1s/32/13233_2.png) [@tuck1s](https://forum.rclone.org/u/tuck1s)\
**Post date:** [November 18, 2022, 6:12pm UTC](https://forum.rclone.org/t/stats-processing-to-give-graphical-time-series-output/34186/1 "2022-11-18T18:12:24Z")

</div>

Has anyone already come up with a good way to process the summary stats from `rclone.log` into graphical format? I'm running backup jobs that run for multiple days (slowish ISP upload speed). I'm interested to see the effect of tuning parameters such as `--transfers=`.

I was thinking to read (or `tee`) the logfile output through a script to produce a time-series chart.

```auto
rclone v1.59.2
- os/version: raspbian 9.13
- os/kernel: 4.19.66+ (armv6l)
- os/type: linux
- os/arch: arm
- go/version: go1.18.6
- go/linking: static
- go/tags: none

```

Writing to BackBlaze B2.

---

<div class="post-metadata">

**Author:** ![tuck1s](https://forum.rclone.org/user_avatar/forum.rclone.org/tuck1s/32/13233_2.png) [@tuck1s](https://forum.rclone.org/u/tuck1s)\
**Post date:** [November 19, 2022, 7:43pm UTC](https://forum.rclone.org/t/stats-processing-to-give-graphical-time-series-output/34186/2 "2022-11-19T19:43:42Z")

</div>

I made a [Python script](https://gist.github.com/tuck1s/49099a42e0503cc5137d8cde04852e8a) to pull the date/time and the `Transferred:` stats out of a human-readable rclone.log. Maybe I should have tried the JSON log output, or the `--stats-one-line-date` instead, but this seems to work.

 ![Screenshot 2022-11-19 at 19.39.23](https://forum.rclone.org/uploads/default/original/2X/d/d0a4508f446aaa1f1a0903ac6c177d95d9e15877.png)  
 ![Screenshot 2022-11-19 at 19.40.55](https://forum.rclone.org/uploads/default/original/2X/c/cf6f79d3b5b332f2bbd02d17a7e2e123c8ab6272.jpeg)

TODO: check how rclone treats timestamps with the computer's locale; at the moment it's easy in UK winter time.  
Realtime time-series views might be better done via the new GUI web app, or Jupyter Notebook, or Google Data Studio etc.

---

<div class="post-metadata">

**Author:** ![tuck1s](https://forum.rclone.org/user_avatar/forum.rclone.org/tuck1s/32/13233_2.png) [@tuck1s](https://forum.rclone.org/u/tuck1s)\
**Post date:** [November 19, 2022, 8:27pm UTC](https://forum.rclone.org/t/stats-processing-to-give-graphical-time-series-output/34186/3 "2022-11-19T20:27:32Z")

</div>

I just saw some parts of the code have a "remote stats" feed e.g. to Prometheus .. are there docs on that?

---

<div class="post-metadata">

**Author:** ![asdffdsa](https://forum.rclone.org/letter_avatar_proxy/v4/letter/a/848f3c/32.png) [@asdffdsa](https://forum.rclone.org/u/asdffdsa)\
**Post date:** [November 19, 2022, 8:54pm UTC](https://forum.rclone.org/t/stats-processing-to-give-graphical-time-series-output/34186/4 "2022-11-19T20:54:50Z")

</div>

> <https://github.com/rclone/rclone/issues/3858>
>
> \<!--
> 
> Welcome :-)
> 
> So you've got an idea to improve rclone? We love that! Yo…u'll be glad to hear we've incorporated hundreds of ideas from contributors already.
> 
> Here is a checklist of things to do:
> 
> 1. Please search the old issues first for your idea and +1 or comment on an existing issue if possible.
> 2. Discuss on the forum first: https://forum.rclone.org/
> 3. Make a feature request issue (this is the right place!).
> 4. Be prepared to get involved making the feature :-)
> 
> Looking forward to your great idea!
> 
> The Rclone Developers
> 
> \--\>
> 
> 
> \#### What is your current rclone version (output from \`rclone version\`)?
> 
> Docker image tag 1.50
> 
> \#### What problem are you are trying to solve?
> 
> Getting metrics of jobs into a Prometheus scrapable format or pushing into Pushgateway. In a Kubernetes world, where rclone runs as a (Cron)job for backing up application data (e.g. S3 mirroring), the monitoring of jobs and failures is cumbersome. Usually it's done with Prometheus as it's the default monitoring solution.
> 
> \#### How do you think rclone should be changed to solve that?
> 
> The current method of providing CLI flag \`--rc --rc-addr "0.0.0.0:5572"\` will enable a webserver that responds with a JSON when accessing \`/core/stats\`. It would be nice if we can alter the query (e.g. ?format=prometheus) so that it would provide the metrics in OpenMetrics/Prometheus format. That probably should also become a GET request instead of POST. Or another solution altogether.
> 
> We maintain currently a dedicated project https://github.com/vshn/rclone-exporter that does the JSON-to-prometheus translation, but we figure it's best if it's directly integrated into rclone. For testing purposes one could use this side-project in the meantime.

---

<div class="post-metadata">

**Author:** ![system](https://forum.rclone.org/uploads/default/original/2X/3/39ad3235f6ffec76ccd0c81616bd55697b4c300d.png) [@system](https://forum.rclone.org/u/system)\
**Post date:** [December 19, 2022, 8:54pm UTC](https://forum.rclone.org/t/stats-processing-to-give-graphical-time-series-output/34186/5 "2022-12-19T20:54:59Z")

</div>

Closed on December 19, 2022, 8:54pm UTC

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.
