"transferring" details not shown when hit core/stats endpoint

Suspected Bug
1

Hi guys,

With according to documentation [1] "transferring" should be shown when one retrieves stats (via core/stats endpoint). However, we carried out a few tests and none of them were positive.

In "transferring" section should pop up information about transfer details for each transfer (e.g. each transferred file, right?).

Could you verify why it is not showing? Thanks in advance!

[1] https://rclone.org/rc/#core-stats-returns-stats-about-current-transfers

Test I - copyurl

curl -H "Content-Type: application/json" -X POST -d  '{"url": "http://ipv4.download.thinkbroadband.com/100MB.zip", "fs": "","remote": "/dest/1.zip"}' http://localhost:5572/operations/copyurl
curl -H "Content-Type: application/json" -X POST -d  '{"url": "http://ipv4.download.thinkbroadband.com/100MB.zip", "fs": "","remote": "/dest/2.zip"}' http://localhost:5572/operations/copyurl

curl -X POST http://localhost:5572/core/stats
{
    "bytes": 209715200,
    "checks": 0,
    "deletes": 0,
    "elapsedTime": 1908.547180883,
    "errors": 0,
    "fatalError": false,
    "retryError": false,
    "speed": 109882.11457417264,
    "transfers": 2
}

Test II - copy from AWS to local S3

curl --request POST  --url http://1.2.3.4:5572/sync/sync  --header 'cache-control: no-cache'  --header 'content-type: application/json'  --header 'postman-token: <token>'  --data '{    "srcFs":      "AWS:test",    "dstFs":      "LOCAL:foobucket"}'

curl -X POST http://1.2.3.4:5572/core/stats
{
	"bytes": 2965,
	"checks": 1,
	"deletes": 1,
	"elapsedTime": 848.697932742,
	"errors": 0,
	"fatalError": false,
	"retryError": false,
	"speed": 3.493586923701563,
	"transfers": 2
}

Kind regards,
Cezary Zukowski

2

I think what is happening is that the copies are finishing before returning so you aren’t seeing the active stats.

If you want to do the copies asynchronously then set "_async": true in the parameters

$ curl -H "Content-Type: application/json" -X POST -d  '{"_async": true, "url": "http://ipv4.download.thinkbroadband.com/100MB.zip", "fs": "","remote": "/dest/1.zip"}' http://localhost:5572/operations/copyurl
{
	"jobid": 2
}
$ curl -X POST http://localhost:5572/core/stats
{
	"bytes": 222335689,
	"checks": 0,
	"deletes": 0,
	"elapsedTime": 145.922094049,
	"errors": 0,
	"fatalError": false,
	"retryError": false,
	"speed": 1523660.213684575,
	"transferring": [
		{
			"bytes": 12620489,
			"eta": 19,
			"name": "/dest/1.zip",
			"percentage": 12,
			"size": 104857600,
			"speed": 4913045.206002883,
			"speedAvg": 4680788.211296782
		}
	],
	"transfers": 2
}
3

Hello. ncw,
Thank you for your answer . :slight_smile:

Any chance of adding “transferred” file lists in the core/stats results?
In case the list of files grows too big, the information could be saved under specific path and retrieved with another function.

4

Hi ncw! Great news! I also checked with much larger 1GB file and transfer details are there.

Yeah, having “transferred” files in place would be more than required. Otherwise, we might have no opportunity to track transfer details for small files, i.e. core/stats API calls issued after transfer completes, which may be common in terms of small files. Correct me, if I am wrong.

Is there any cache in Rclone that could be employed for that purpose?

5

I’ve had some requests for this for non api work too. I’d probably prefer a flag to turn it on and a different endpoint to return it.

6

If you are using “_async” then you get a jobid back which you can then query the status and completion of the job.

If you aren’t using “_async” then you’ll get a definitive OK/error back when the call completes.

7

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.