Help Getting Job or Group Stats

What is the problem you are having with rclone?

Can't seem to get stats ( transfer amount, rate etc, ) for a larger file upload.

First I start the upload using async

Then, using the job id, returned by the upload call, I run a script to attempt to gather stats on the transfer.
I will rerun this several times during the transfer and the stats come back empty ( zero )

I believe I am may be missing an important concept or information.
When I try getting stats for all transfers it also returns zero in all the stats fields.

( See code + output below )

Run the command 'rclone version' and share the full output of the command.

rclone v1.61.1

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)


**Power Shell Commands and Output:**
**1.) Call File Upload**
Write-Output("Upload Large File Using core Command copy")
$headers=@{}
$headers.Add("user-agent", "vscode-restclient")
$headers.Add("content-type", "application/json")
$body='{"command": "copy", "arg": ["G:/Data/Work/Development/RCloneResearch/biggerfile.zip",  "GDriveTest:TestFolder"], "_async":"true" }'
Invoke-WebRequest -Uri 'http://127.0.0.1:5572/core/command' -Method POST -Headers $headers -ContentType 'application/json' -Body $body -UseBasicParsing | Select-Object -Expand Content

**Output:**
Upload Large File Using core Command copy
{
	"jobid": 657
}


**2.) Past JOB ID into follow script. Run it to try to get stats on the job.**
Write-Output("==Get List of All Groups==")
$headers=@{}
$headers.Add("user-agent", "vscode-restclient")
$headers.Add("content-type", "application/json")
$body='{ }'
Invoke-WebRequest -Uri 'http://127.0.0.1:5572/core/group-list' -Method POST -Headers $headers -ContentType 'application/json' -Body $body -UseBasicParsing | Select-Object -Expand Content


Write-Output("==Get the Stats for All Groups==")
$headers=@{}
$headers.Add("user-agent", "vscode-restclient")
$headers.Add("content-type", "application/json")
$body='{}'
Invoke-WebRequest -Uri 'http://127.0.0.1:5572/core/stats' -Method POST -Headers $headers -ContentType 'application/json' -Body $body -UseBasicParsing  | Select-Object -Expand Content


Write-Output("==Get the Job Status Using the Job ID ==")
$headers=@{}
$headers.Add("user-agent", "vscode-restclient")
$headers.Add("content-type", "application/json")
$body='{ "jobid": "657" }'
Invoke-WebRequest -Uri 'http://127.0.0.1:5572/job/status' -Method POST -Headers $headers -ContentType 'application/json' -Body $body -UseBasicParsing  | Select-Object -Expand Content

Write-Output("==Get the Core Stats For the Job using the Job Group Name Listed in the Response from the Upload Call==")
$headers=@{}
$headers.Add("user-agent", "vscode-restclient")
$headers.Add("content-type", "application/json")
$body='{ "group": "job/657" }'
Invoke-WebRequest -Uri 'http://127.0.0.1:5572/core/stats' -Method POST -Headers $headers -ContentType 'application/json' -Body $body -UseBasicParsing  | Select-Object -Expand Content

Write-Output("==Get the Transferred Stats for All Groups==")
$headers=@{}
$headers.Add("user-agent", "vscode-restclient")
$headers.Add("content-type", "application/json")
$body='{}'
Invoke-WebRequest -Uri 'http://127.0.0.1:5572/core/transferred' -Method POST -Headers $headers -ContentType 'application/json' -Body $body -UseBasicParsing  | Select-Object -Expand Content

Write-Output("==Get the Transferred Stats for the Upload using the Job Group Name Listed in the Response from the Upload Call==")
$headers=@{}
$headers.Add("user-agent", "vscode-restclient")
$headers.Add("content-type", "application/json")
$body='{"group": "job/657"}'
Invoke-WebRequest -Uri 'http://127.0.0.1:5572/core/transferred' -Method POST -Headers $headers -ContentType 'application/json' -Body $body -UseBasicParsing  | Select-Object -Expand Content


**Output**
==Get List of All Groups==
{
	"groups": [
		"job/27",
		"job/42",
		"job/50",
		"job/117",
		"job/140",
		"job/154",
		"job/646"
	]
}

==Get the Stats for All Groups==
{
	"bytes": 0,
	"checks": 0,
	"deletedDirs": 0,
	"deletes": 0,
	"elapsedTime": 8642.076249,
	"errors": 0,
	"eta": null,
	"fatalError": false,
	"renames": 0,
	"retryError": false,
	"speed": 0,
	"totalBytes": 0,
	"totalChecks": 0,
	"totalTransfers": 0,
	"transferTime": 0,
	"transfers": 0
}

==Get the Job Status Using the Job ID ==
{
	"duration": 0,
	"endTime": "0001-01-01T00:00:00Z",
	"error": "",
	"finished": false,
	"group": "job/657",
	"id": 657,
	"output": null,
	"startTime": "2023-05-18T15:43:25.412056-04:00",
	"success": false
}

==Get the Core Stats For the Job using the Job Group Name Listed in the Response from the Upload Call==
{
	"bytes": 0,
	"checks": 0,
	"deletedDirs": 0,
	"deletes": 0,
	"elapsedTime": 0,
	"errors": 0,
	"eta": null,
	"fatalError": false,
	"renames": 0,
	"retryError": false,
	"speed": 0,
	"totalBytes": 0,
	"totalChecks": 0,
	"totalTransfers": 0,
	"transferTime": 0,
	"transfers": 0
}

==Get the Transferred Stats for All Groups==
{
	"transferred": []
}

==Get the Transferred Stats for the Upload using the Job Group Name Listed in the Response from the Upload Call==
{
	"transferred": []
}

I'm not familiar with powershell but here is now to get the stats with rclone rc

$ rclone rc sync/copy srcFs=/tmp/a dstFs=/tmp/b _async=true
{
	"jobid": 1
}

Then

$ rclone rc core/stats group=job/1
{
	"bytes": 192512,
	"checks": 0,
	"deletedDirs": 0,
	"deletes": 0,
	"elapsedTime": 85.017365441,
	"errors": 0,
	"eta": 1944631,
	"fatalError": false,
	"renames": 0,
	"retryError": false,
	"speed": 552.0580354367565,
	"totalBytes": 1073741824,
	"totalChecks": 0,
	"totalTransfers": 1,
	"transferTime": 0,
	"transferring": [
		{
			"bytes": 192512,
			"eta": 1944260,
			"group": "job/1",
			"name": "1G",
			"percentage": 0,
			"size": 1073741824,
			"speed": 2264.485762144896,
			"speedAvg": 552.1633149681434
		}
	],
	"transfers": 0
}

If you want to see exactly what rclone sent, then add -vv --dump bodies

$ rclone rc core/stats group=job/1 -vv --dump bodies

HTTP request

2023/05/19 10:43:56 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/05/19 10:43:56 DEBUG : HTTP REQUEST (req 0xc000cd2f00)
2023/05/19 10:43:56 DEBUG : POST /core/stats HTTP/1.1
Host: localhost:5572
User-Agent: rclone/v1.63.0-beta.7046.bb4d28ecb.fix-7006-android-metadata
Content-Length: 17
Content-Type: application/json
Accept-Encoding: gzip

{"group":"job/1"}
2023/05/19 10:43:56 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

HTTP response

2023/05/19 10:43:56 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/05/19 10:43:56 DEBUG : HTTP RESPONSE (req 0xc000cd2f00)
2023/05/19 10:43:56 DEBUG : HTTP/1.1 200 OK
Content-Length: 540
Accept-Ranges: bytes
Access-Control-Allow-Headers: authorization, Content-Type
Access-Control-Allow-Origin: http://127.0.0.1:5572/
Access-Control-Request-Method: POST, OPTIONS, GET, HEAD
Content-Type: text/plain; charset=utf-8
Date: Fri, 19 May 2023 09:43:56 GMT
Server: rclone/v1.63.0-beta.7046.bb4d28ecb.fix-7006-android-metadata
X-Rclone-Jobid: 7

{
	"bytes": 356352,
	"checks": 0,
	"deletedDirs": 0,
	"deletes": 0,
	"elapsedTime": 251.012526631,
	"errors": 0,
	"eta": 3386311,
	"fatalError": false,
	"renames": 0,
	"retryError": false,
	"speed": 316.9777253540611,
	"totalBytes": 1073741824,
	"totalChecks": 0,
	"totalTransfers": 1,
	"transferTime": 0,
	"transferring": [
		{
			"bytes": 356352,
			"eta": 3382802,
			"group": "job/1",
			"name": "1G",
			"percentage": 0,
			"size": 1073741824,
			"speed": 1419.6797485271122,
			"speedAvg": 317.3065562676946
		}
	],
	"transfers": 0
}
2023/05/19 10:43:56 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
1 Like

Thank you for the help.
Your answer helped lead me to an answer.

I did some experimenting and found I was able to get stats for "sync/copy" and "operations/copyfile"

But the "core/command" subset doesn't seem to generate stats.

I'm wondering if that is expected behavior?
Maybe stats are not supported for the "core/command" type of commands?

No it doesn't. This essentially shells out to another rclone. Use sync/copy etc in preference.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.