What is the problem you are having with rclone?
When an async copy encounters a file not found error, the statistics group does not show any error, nor any information about the submitted job
Run the command 'rclone version' and share the full output of the command.
rclone v1.67.0
- os/version: redhat 8.10 (64 bit)
- os/kernel: 4.18.0-553.5.1.el8_10.x86_64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.22.4
- go/linking: static
- go/tags: none
Which cloud storage system are you using? (eg Google Drive)
S3 compatible, for testing currently IBM COS S3
The command you were trying to run (eg rclone copy /tmp remote:tmp)
rclone --rc-user rclone --rc-pass password --fast-list --checksum -vv rcd
data = {
"srcFs": 'remote:bucket01',
"srcRemote": "test.txt",
"dstFs": 'remote:bucket02',
"dstRemote": "copied.txt",
"_async": True,
"_group": "42"
}
session.post('http://localhost:5572/operations/copyfile', json=data)
Then checking for the single job,
res = session.post('http://localhost:5572/job/status', data={"jobid": rclone_job_id})
which shows the error just fine:
{'duration': 0.114039581,
'endTime': '2024-08-29T11:17:55.036307351+02:00',
'error': 'object not found',
'finished': True,
'group': '42',
'id': 1,
'output': {},
'startTime': '2024-08-29T11:17:54.922267747+02:00',
'success': False}
But when checking the group:
res = session.post('http://localhost:5572/core/stats', data={"group": "42"})
The statistics returned are empty:
{'bytes': 0,
'checks': 0,
'deletedDirs': 0,
'deletes': 0,
'elapsedTime': 2.2232e-05,
'errors': 0,
'eta': None,
'fatalError': False,
'renames': 0,
'retryError': False,
'serverSideCopies': 0,
'serverSideCopyBytes': 0,
'serverSideMoveBytes': 0,
'serverSideMoves': 0,
'speed': 0,
'totalBytes': 0,
'totalChecks': 0,
'totalTransfers': 0,
'transferTime': 0,
'transfers': 0}
Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.
[remote]
type = s3
access_key_id = XXX
endpoint = https://s3.eu-de.cloud-object-storage.appdomain.cloud
provider = IBMCOS
secret_access_key = XXX
Note: I would like to omit the provider config entry in the future and hope it won't make a big difference.
A log from the command that you were trying to run with the -vv flag
2024/08/29 11:14:25 DEBUG : rc: "operations/copyfile": with parameters map[_async:true _group:42 dstFs:remote:bucket02 dstRemote:copied.txt srcFs:remote:bucket01 srcRemote:test.txt]
2024/08/29 11:14:25 DEBUG : rc: "operations/copyfile": reply map[jobid:1]: <nil>
2024/08/29 11:14:25 DEBUG : Creating backend with remote "remote:bucket01"
2024/08/29 11:14:25 DEBUG : Resolving service "s3" region "us-east-1"
2024/08/29 11:14:25 DEBUG : Creating backend with remote "remote:bucket02"
2024/08/29 11:14:25 DEBUG : Resolving service "s3" region "us-east-1"
2024/08/29 11:20:28 DEBUG : rc: "job/status": with parameters map[jobid:1]
2024/08/29 11:20:28 DEBUG : rc: "job/status": reply map[duration:0.713786355 endTime:2024-08-29T11:20:24.422643368+02:00 error:object not found finished:true group:42 id:1 output:map[] startTime:2024-08-29T11:20:23.708856993+02:00 success:false]: <nil>
2024/08/29 11:20:35 DEBUG : rc: "core/stats": with parameters map[group:42]
2024/08/29 11:20:35 DEBUG : rc: "core/stats": reply map[bytes:0 checks:0 deletedDirs:0 deletes:0 elapsedTime:2.2232e-05 errors:0 eta:<nil> fatalError:false renames:0 retryError:false serverSideCopies:0 serverSideCopyBytes:0 serverSideMoveBytes:0 serverSideMoves:0 speed:0 totalBytes:0 totalChecks:0 totalTransfers:0 transferTime:0 transfers:0]: <nil>
Thank you a lot for your help, I hope I gave the correct information.