Retry logic for cmd based copy vs remote api sync/copy

Hello Experts,

I am using Rclone in remote api mode and using '/sync/copy' to transfer files in async mode.

When I remove internet connection from machine and run "rclone copy" command in command line the retry looks to be fine -
Command : rclone copy awso9dl:o9datalake/targetaws/test1.txt awso9dl:o9datalake/targetaws/test1_copy6.txt --log-level DEBUG --log-file=D:\logs\log4.txt --retries 3 --retries-sleep 10s
I can see 3 retry attempts in logs as it cant reach remote -

But when I test same thing with remote mode with asycn request submitted, whatever the retry count I pass it just takes 120 to 150 seconds and job fails.
I dont see any retry logic related debug logs in log file.

Below is the command with I run remote mode rclone -
rclone.exe rcd --rc-no-auth --rc-job-expire-duration=60m --rc-addr=localhost:5574 --log-level DEBUG --log-file=D:\logs\remotelog10
.txt --retries 10

And below the request sent to rclone for copy -
http://localhost:5574/sync/copy
{
"srcFs": "awso9dl:o9datalake/targetaws",
"dstFs": "awso9dl:o9datalake/targetaws_mh01/a5",
"_filter" : {
"IncludeRule" :
},
"_async" : true
}

Job status gives completed failed response after 120+ seconds
C:\Users\mahendra.hegde>curl -X POST -d '{}' http://localhost:5574/job/status?jobid=1
{
"duration": 128.0721921,
"endTime": "2024-12-05T11:42:56.8807359+05:30",
"error": "RequestError: send request failed\ncaused by: Get "https://o9datalake.s3.us-east-1.amazonaws.com/?delimiter=%2F\u0026encoding-type=url\u0026list-type=2\u0026max-keys=1000\u0026prefix=targetaws%2F\": dial tcp: lookup o9datalake.s3.us-east-1.amazonaws.com: no such host",
"finished": true,
"group": "job/1",
"id": 1,
"output": {},
"startTime": "2024-12-05T11:40:48.8085438+05:30",
"success": false
}

But I dont see anything related to retry in logs.
Logs:

Just want to understand if retry is not happening for remote api mode and just happening for command line mode or am I missing something here?

Rclone version:
rclone v1.63.1

  • os/version: Microsoft Windows 11 Pro 24H2 (64 bit)
  • os/kernel: 10.0.26100.2314 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.20.6
  • go/linking: static
  • go/tags: cmount

hi,

that version is 18 months old?
might want to rclone selfupdate and test again

Hi,
Tested with latest Rclone 'rclone v1.68.2'.
Behavior is same.

I can see 3 (or defined) retries in logs properly but not seeing this behavior with remote mode.

(One difference I am seeing v1.63.1 v/s latest is , async job was giving finished=true with success=false status after almost 120+ seconds in Rclone 1.63.1 but now in latest Rclone its giving the finished=true status in within few milli seconds)

Request:
http://localhost:5574/sync/copy
{
"srcFs": "awso9dl:o9datalake/storehere_v2",
"dstFs": "awso9dl:o9datalake/storehere_MH/three",
"_filter" : {
"IncludeRule" :
},
"_async" : true
}

Status:

curl -X POST -d '{}' http://localhost:5574/job/status?jobid=1
{
"duration": 0.0342843,
"endTime": "2024-12-09T11:04:20.9839977+05:30",
"error": "operation error S3: ListObjectsV2, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "https://o9datalake.s3.us-east-1.amazonaws.com/?delimiter=%2F\u0026encoding-type=url\u0026list-type=2\u0026max-keys=1000\u0026prefix=storehere_v2%2F\": dial tcp: lookup o9datalake.s3.us-east-1.amazonaws.com: no such host",
"finished": true,
"group": "job/1",
"id": 1,
"output": {},
"startTime": "2024-12-09T11:04:20.9497134+05:30",
"success": false
}

Logs :
2024/12/09 11:03:55 DEBUG : rclone: Version "v1.68.2" starting with parameters ["rclone.exe" "rcd" "--rc-no-auth" "--rc-job-expire-duration=60m" "--rc-addr=localhost:5574" "--log-level" "DEBUG" "--log-file=D:\logs\remotelog_10.txt" "--retries" "10"]
2024/12/09 11:03:55 NOTICE: Serving remote control on http://127.0.0.1:5574/
2024/12/09 11:04:20 DEBUG : rc: "sync/copy": with parameters map[_async:true _filter:map[IncludeRule:] dstFs:awso9dl:o9datalake/storehere_MH/three srcFs:awso9dl:o9datalake/storehere_v2]
2024/12/09 11:04:20 DEBUG : rc: "sync/copy": reply map[jobid:1]:
2024/12/09 11:04:20 DEBUG : Creating backend with remote "awso9dl:o9datalake/storehere_v2"
2024/12/09 11:04:20 DEBUG : Using config file from "C:\Users\mahendra.hegde\AppData\Roaming\rclone\rclone.conf"
2024/12/09 11:04:20 DEBUG : Creating backend with remote "awso9dl:o9datalake/storehere_MH/three"
2024/12/09 11:04:20 ERROR : S3 bucket o9datalake path storehere_v2: error reading source root directory: operation error S3: ListObjectsV2, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "https://o9datalake.s3.us-east-1.amazonaws.com/?delimiter=%2F&encoding-type=url&list-type=2&max-keys=1000&prefix=storehere_v2%2F": dial tcp: lookup o9datalake.s3.us-east-1.amazonaws.com: no such host
2024/12/09 11:04:20 DEBUG : S3 bucket o9datalake path storehere_MH/three: Waiting for checks to finish
2024/12/09 11:04:20 DEBUG : S3 bucket o9datalake path storehere_MH/three: Waiting for transfers to finish
2024/12/09 11:04:37 DEBUG : rc: "job/status": with parameters map['{}': jobid:1]
2024/12/09 11:04:37 DEBUG : rc: "job/status": reply map[duration:0.0342843 endTime:2024-12-09T11:04:20.9839977+05:30 error:operation error S3: ListObjectsV2, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "https://o9datalake.s3.us-east-1.amazonaws.com/?delimiter=%2F&encoding-type=url&list-type=2&max-keys=1000&prefix=storehere_v2%2F": dial tcp: lookup o9datalake.s3.us-east-1.amazonaws.com: no such host finished:true group:job/1 id:1 output:map startTime:2024-12-09T11:04:20.9497134+05:30 success:false]:

@asdffdsa @ncw - Any idea about this?
Is it retrying behind the scene without logging or anything I am missing here in remote mode?