I use rclone to shovel around my backup (to an external HD and to Onedrive), and would love the metrics to be collected by Prometheus. --rc-metrics is doing a great job! However, if I just run my rclone sync command with this option, a) I don't get the latest stats, as Prometheus collects the metrics only every minute or so (thus doesn't receive the stats at the end of the rclone operation but at the latest scrape) and b) the stats are reset every time the operation starts (this is not strictly necessary but would be nice to persist stats between operations).
The best way I could figure out was to run rclone rcd --rc-metrics --rc-htpasswd /etc/rclone.htpasswd and execute the operations as remote calls. However, I am very reluctant to run this service as root because of security considerations, as this exposes my root file system with root rights on a http interface, which has a big attack surface. In particular, prometheus needs to be able to access the http interface to scrape the metrics and thus has full access to the filesystem as root, which is a No-Go (I have seen that you should be able to assign a different password to the /metrics endpoint but still am not comfortable running this daemon as root). The files I need to copy around are only readable by root, so I can't run it as another user...
Can anyone suggest a better way to achieve the desired metrics collection without running an rc daemon as root?
I have a few ideas how this could be achieved, but none of them are ideal:
run the operations locally (not as remote calls) and share/export just the metrics to some long running rclone rcd daemon which is not root and just persists/forwards the metrics
have some service cache the prometheus metrics (I think there are some services for prometheus that do just that, did not dig into that yet though)
Thank you so much for your time, answers and rclone in general, much appreciated !
Run the command 'rclone version' and share the full output of the command.
rclone v1.53.3-DEV
- os/arch: linux/amd64
- go version: go1.18.1
Which cloud storage system are you using? (eg Google Drive)
Onedrive personal and local
The command you were trying to run (eg rclone copy /tmp remote:tmp)
Well possible that it is ancient, it's just what is in the repos for Ubuntu 22 Probably will update to Ubuntu 24 soon anyway, but that won't change anything in the general way things work...
There is no reason to use what Ubuntu provides. Remove it and install the latest version directly from rclone website. Or use better distro. One which keeps things up to date.
You wont find the latest rclone there neither... These type of distros make everything up to date at the point of release. Later only maintaining some key packages like web browser leaving everything less important to rot:) Still better than Ubuntu LTS which IMO has very different purpose than system for workstation.
If you want to use up to date software I would suggest some rolling release distros. Myself I would recommend Suse Tumbleweed - tested rolling release, great documentation and friendly community. Arch is also great but requires solid Linux knowledge to start with.
In addition rclone is single binary without any dependencies so not using your repo provided one does not bring any disadvantages. Simply get rid off repo one and install it from rclone website.
You'd have to run the cloudflared on the server you want and you don't have to expose any ports at all as it's all handled via some setup with Cloudflared.
Thanks for the answer and sorry for the long delay.
I never would expose that service on the internet, not even in my home network. The only reason why I wanted to run rclone rcd is to have a long running service which accumulates the metrics/stats, instead of starting from zero on every call to rclone sync or whatever. And yes, the files I need to copy around need root access.
One idea I came up with that I would be comfortable with, is to run rclone rcd in a docker container, and bind mount only the directories it really needs access to. Like this, it accumulates the stats as long as the machine is running, but I don't have some web API over which my whole root FS is accessible.
What I will finally resort to is to not rely on remote calls at all, but to run rclone sync directly and scrape the metrics by a bash script as long as the job is running, and export the metrics lateron by using the textfile collector of prometheus node exporter. This doesn't accumulate the metrics over multiple calls and might miss some information at the end of the job, but I can live with that.
!
Probably will update to Ubuntu 24 soon anyway, but that won't change anything in the general way things work...