Prometheus /metrics not updated when automatic synching configured

Help and Support
1

Hello,

I want to display the Prometheus-related metrics in a Grafana dashboard. The files are transferred by automatically syncing S3/SeaweedFS based content. The source side is receiving files from time to time by another process. So i am not using a rclone cli command and rely solely on the configuration to sync the files automatically.

This is my rclone.txt file for this:

[csminio]
type = s3
provider = SeaweedFS
env_auth = false
access_key_id = myaccesskey
secret_access_key = mysecretkey
endpoint = http://cs-seaweedfs-s3:8333
location_constraint = us-east-1
acl = private


[dcminio]
type = s3
provider = SeaweedFS
env_auth = false
access_key_id = myaccesskey
secret_access_key = mysecretkey
endpoint = http://dc-seaweedfs-s3:8334
location_constraint = us-east-1
acl = private

I am using the following rclone service settings as part of a docker compose file:

    command:
      - rcd
      - --rc-no-auth
      - --rc-enable-metrics
      - --rc-addr=:5572
      - --rc-web-gui
      - --rc-web-gui-no-open-browser

The /metrics endpoint provides updates for the go-related metrics but all file transfer related details are always zero - while the files are actually being transferred. So, rclone is able to transfer the files and"only" the metrics are not updated.

rclone_bytes_transferred_total 0

Is this a known issue or do i need to change the configuration?

cheers

Juergen

rclone 1.66 (older version same effect)
s3 seaweedfs file repos

[csminio]
type = s3
provider = SeaweedFS
env_auth = false
access_key_id = XXX
secret_access_key = XXX
endpoint = http://cs-seaweedfs-s3:8333
location_constraint = us-east-1
acl = private

[dcminio]
type = s3
provider = SeaweedFS
env_auth = false
access_key_id = XXX
secret_access_key = XXX
endpoint = http://dc-seaweedfs-s3:8334
location_constraint = us-east-1
acl = private
### Double check the config for sensitive info before posting publicly
2

I'm not quite following the flow.

Does the RCD daemon you spawn also do the syncing? How are you doing the 'sync'?

3

Thanks for your question. To clarify, we call a method on a regular basis. Inside basically the following rclone request is build:

        RcloneCoreRequest request = new RcloneCoreRequest();
        request.setCommand("copy");
        request.setArg(Arrays.asList(srcFs,dstFs,"--stats-one-line", "-v","--use-json-log=false"));
        String jsonResponse = rcloneClient.syncDirectory(request);

I see in the logs info like this:

2024/03/20 18:53:13 INFO  : S0001/Metadata/U0001/20240320_000003549.csv: Copied (new)
2024-03-20T18:53:14.369301011Z 2024/03/20 18:53:13 INFO  : S0001/Metadata/U0001/20240320_000003552.csv: Copied (new)
2024-03-20T18:53:14.369301720Z 2024/03/20 18:53:14 INFO  :   326.311 MiB / 326.311 MiB, 100%, 163.144 MiB/s, ETA 0s

So, the files are being transferred. Its just that the metrics are not updated.

Does this help?

4

What does the core/stats rc call report. Are these counting up properly?

Perhaps the Prometheus metrics have become discounted somehow.

5

I see zero values but files jhave been transferred.

curl -X POST 'http://localhost:5572/core/stats'

{
        "bytes": 0,
        "checks": 0,
        "deletedDirs": 0,
        "deletes": 0,
        "elapsedTime": 100.754954337,
        "errors": 0,
        "eta": null,
        "fatalError": false,
        "renames": 0,
        "retryError": false,
        "serverSideCopies": 0,
        "serverSideCopyBytes": 0,
        "serverSideMoveBytes": 0,
        "serverSideMoves": 0,
        "speed": 0,
        "totalBytes": 0,
        "totalChecks": 0,
        "totalTransfers": 0,
        "transferTime": 0,
        "transfers": 0
}
6

Did you set a group when you made the RC call to start the copy?

Are you definitely sending the copy and reading the stats from the same rclone process?

7

The rclone UI does also not show any values. So this seems to be a rclone/seaweedfs related issue. My project stint ended before this has been solved. May still be interesting for folks to get this resolved. What info do you need to help fixing this?

8

If you could make a replication just using command line rclone then I'll be able to fix it.

So something like run an rclone rcd run some rclone rc commands on it then see that the stats don't update.

9

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.