Prometheus metrics

mmatczuk · February 21, 2020, 1:56pm

I'd like to add more observability to underlying cloud providers' SDKs by adding prometheus metrics.
I'm mostly concerned with monitoring non standard responses, errors and retries.

My initial idea is by adding that functionality to fshttp::Transport

github.com

rclone/rclone/blob/master/fs/fshttp/http.go#L221


	}
	if ci.Cookie {
		client.Jar = cookieJar
	}
	return client
}


// Transport is a our http Transport which wraps an http.Transport
// * Sets the User Agent
// * Does logging
type Transport struct {
	*http.Transport
	dump          fs.DumpFlags
	filterRequest func(req *http.Request)
	userAgent     string
}


// newTransport wraps the http.Transport passed in and logs all
// roundtrips including the body if logBody is set.
func newTransport(ci *fs.ConfigInfo, transport *http.Transport) *Transport {
	return &Transport{

With promhttp package, more or less like in the example (without trace)

The static labels would be

fs name
HTTP host

I'd appreciate feedback regarding this idea and possible improvements to this design draft.

ncw · February 22, 2020, 6:58am

There is an issue about this

it looks like it has stalled - maybe you'd like to take it over?

Adding observability to the rclone Transport is a good way of capturing all the backends so that sounds like a great idea.

Where would you get fs name from? It is possible that the Transport is in use by two backends at once, eg if you are copying from s3 to google drive...

mmatczuk · February 25, 2020, 8:36am

The issue is related but not the same. In the issue they want accounting data as metrics. Here I want what is currently darkness to become light.

We should have /metrics endpoint in the server this is a standard prometheus way of doing things.

FS name can be passed in constructor to fshttp but on the other hand it's not mandatory and not passing it would make the patch smaller.

ncw · February 25, 2020, 10:09am

Agreed, not the same but similar!

OK, happy to take your lead here.

OK, if you think it would be useful enough without then let's go for that.

Dual-O · October 13, 2020, 10:22am

Hello,

is it possible to import rclone mount stats via /metrics in prometheus?
edit: works with mount... thanks

is it possible to see the icoming and outgoing traffic/files seperated?

ncw · October 13, 2020, 1:32pm

I don't think so at the moment.

Dual-O · October 13, 2020, 2:13pm

thanks for the fast reply.
would be nice to have this... can I add a feature request?

ncw · October 13, 2020, 9:07pm

Rclone doesn't measure this at the moment, though with the planned changes to the bandwidth limiters it would fit in there.

By all means make an issue!

random404 · November 11, 2020, 5:50am

Are there any screenshots of what those metrics are? I'm curious

ncw · November 11, 2020, 5:21pm

It is basically the same info that you can get from the core/stats rc call.

zaidmd · May 22, 2021, 4:23am

Hi
I was also going with the same issue for underlying cloud SDK by adding prometheus metrics. After reading and implementing this thread I solved my issue. Also I was curious about the metrics.
Thank you.

ncw · May 22, 2021, 11:03am

@zaimd have you got some pics of the dashboard you could share?

rafaelreisr · October 29, 2021, 1:29pm

The metrics are almost perfect. It would be a killer feature to have Operations reporting there as well (Copied, Deleted, Transfered) file names, error reporting etc.

Logs (stdout and Json) already provide that info. How hard would it be to integrate that to the /metrics endpoint?

ncw · October 30, 2021, 12:43pm

Probably not terribly hard!

The log/console output and prometheus metrics are all based of the stats.

Fancy having a go and sending a PR @rafaelreisr ?