Rclone copy fails with Invalid Grant

What is the problem you are having with rclone?

rclone copy fails with Invalid Grant

Run the command 'rclone version' and share the full output of the command.

% rclone version
rclone v1.61.1

  • os/version: darwin 13.1 (64 bit)
  • os/kernel: 22.2.0 (x86_64)
  • os/type: darwin
  • os/arch: amd64
  • go/version: go1.19.4
  • go/linking: dynamic
  • go/tags: cmount
Yes

Which cloud storage system are you using? (eg Google Drive)

IBM COS

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone -v -P  copy gp-cos:cpdemo-backup techzone-cos:playground-temp

The rclone config contents with secrets removed.

{
    "gp-cos": {
        "access_key_id": "",
        "acl": "public-read-write",
        "endpoint": "s3.us-east.cloud-object-storage.appdomain.cloud",
        "location_constraint": "us-east-standard",
        "provider": "IBMCOS",
        "region": "us-east",
        "secret_access_key": "",
        "type": "s3"
    },
    "techzone-cos": {
        "access_key_id": "",
        "acl": "public-read-write",
        "endpoint": "s3.us-south.cloud-object-storage.appdomain.cloud",
        "provider": "IBMCOS",
        "region": "us-south",
        "secret_access_key": "",
        "type": "s3"
    }
}

A log from the command with the -vv flag

% rclone -v -P  copy gp-cos:cpdemo-backup techzone-cos:playground-temp -vv
2023/01/05 18:56:29 DEBUG : rclone: Version "v1.61.1" starting with parameters ["rclone" "-v" "-P" "copy" "gp-cos:cpdemo-backup" "techzone-cos:playground-temp" "-vv"]
2023/01/05 18:56:29 DEBUG : Creating backend with remote "gp-cos:cpdemo-backup"
2023/01/05 18:56:29 DEBUG : Using config file from "/Users/pgirish/.config/rclone/rclone.conf"
2023/01/05 18:56:29 DEBUG : Creating backend with remote "techzone-cos:playground-temp"
2023-01-05 18:56:30 DEBUG : S3 bucket playground-temp: Waiting for checks to finish
2023-01-05 18:56:30 DEBUG : S3 bucket playground-temp: Waiting for transfers to finish
2023-01-05 18:56:31 ERROR : pgirish.json: Failed to copy: InvalidGrant: Invalid Grant
	status code: 400, request id: bfbdf4fc-cf94-45a9-b6af-fdffb16458e8, host id:
2023-01-05 18:56:31 ERROR : Attempt 1/3 failed with 1 errors and: InvalidGrant: Invalid Grant
	status code: 400, request id: bfbdf4fc-cf94-45a9-b6af-fdffb16458e8, host id:
2023-01-05 18:56:32 DEBUG : S3 bucket playground-temp: Waiting for checks to finish
2023-01-05 18:56:32 DEBUG : S3 bucket playground-temp: Waiting for transfers to finish
2023-01-05 18:56:33 ERROR : pgirish.json: Failed to copy: InvalidGrant: Invalid Grant
	status code: 400, request id: fb784386-9daa-48eb-ac78-e3d1f1c922ef, host id:
2023-01-05 18:56:33 ERROR : Attempt 2/3 failed with 1 errors and: InvalidGrant: Invalid Grant
	status code: 400, request id: fb784386-9daa-48eb-ac78-e3d1f1c922ef, host id:
2023-01-05 18:56:33 DEBUG : S3 bucket playground-temp: Waiting for checks to finish
2023-01-05 18:56:33 DEBUG : S3 bucket playground-temp: Waiting for transfers to finish
2023-01-05 18:56:34 ERROR : pgirish.json: Failed to copy: InvalidGrant: Invalid Grant
	status code: 400, request id: 2cdc1477-33fc-478d-9b2b-79cf5c578c73, host id:
2023-01-05 18:56:34 ERROR : Attempt 3/3 failed with 1 errors and: InvalidGrant: Invalid Grant
	status code: 400, request id: 2cdc1477-33fc-478d-9b2b-79cf5c578c73, host id:
Transferred:   	   12.738 KiB / 12.738 KiB, 100%, 2.829 KiB/s, ETA 0s
Errors:                 1 (retrying may help)
Elapsed time:         5.4s
2023/01/05 18:56:34 INFO  : 
Transferred:   	   12.738 KiB / 12.738 KiB, 100%, 2.829 KiB/s, ETA 0s
Errors:                 1 (retrying may help)
Elapsed time:         5.4s

2023/01/05 18:56:34 DEBUG : 9 go routines active
2023/01/05 18:56:34 Failed to copy: InvalidGrant: Invalid Grant
	status code: 400, request id: 2cdc1477-33fc-478d-9b2b-79cf5c578c73, host id:

hello and welcome to the forum,

which remote(s) has the issue?

about the location_constraint
--- gp-cos remote uses location_constraint
--- techzone-cos does not use location_constraint

Just to add. 'rclone ls' works on both source and target remotes configured. If I try a command to copy a local file to the target I get the same error

rclone -v -P copy pgirish.json techzone-cos:playground-temp -vv
2023/01/05 19:08:14 DEBUG : rclone: Version "v1.61.1" starting with parameters ["rclone" "-v" "-P" "copy" "pgirish.json" "techzone-cos:playground-temp" "-vv"]
2023/01/05 19:08:14 DEBUG : Creating backend with remote "pgirish.json"
2023/01/05 19:08:14 DEBUG : Using config file from "/Users/pgirish/.config/rclone/rclone.conf"
2023/01/05 19:08:14 DEBUG : fs cache: adding new entry for parent of "pgirish.json", "/Users/pgirish/Downloads"
2023/01/05 19:08:14 DEBUG : Creating backend with remote "techzone-cos:playground-temp"
2023-01-05 19:08:15 DEBUG : pgirish.json: Need to transfer - File not found at Destination
2023-01-05 19:08:16 ERROR : pgirish.json: Failed to copy: InvalidGrant: Invalid Grant
status code: 400, request id: 2fb1e02e-4f50-4261-8bff-53d02fab3e9a, host id:
2023-01-05 19:08:16 ERROR : Attempt 1/3 failed with 1 errors and: InvalidGrant: Invalid Grant
status code: 400, request id: 2fb1e02e-4f50-4261-8bff-53d02fab3e9a, host id:
2023-01-05 19:08:17 DEBUG : pgirish.json: Need to transfer - File not found at Destination
2023-01-05 19:08:17 ERROR : pgirish.json: Failed to copy: InvalidGrant: Invalid Grant
status code: 400, request id: c23babce-4944-4913-97f7-bbb73b47752e, host id:
2023-01-05 19:08:17 ERROR : Attempt 2/3 failed with 1 errors and: InvalidGrant: Invalid Grant
status code: 400, request id: c23babce-4944-4913-97f7-bbb73b47752e, host id:
2023-01-05 19:08:17 DEBUG : pgirish.json: Need to transfer - File not found at Destination
2023-01-05 19:08:18 ERROR : pgirish.json: Failed to copy: InvalidGrant: Invalid Grant
status code: 400, request id: f0abb583-e221-4ca5-92cf-22208dd23c4d, host id:
2023-01-05 19:08:18 ERROR : Attempt 3/3 failed with 1 errors and: InvalidGrant: Invalid Grant
status code: 400, request id: f0abb583-e221-4ca5-92cf-22208dd23c4d, host id:
Transferred: 12.738 KiB / 12.738 KiB, 100%, 4.244 KiB/s, ETA 0s
Errors: 1 (retrying may help)
Elapsed time: 3.9s
2023/01/05 19:08:18 INFO :
Transferred: 12.738 KiB / 12.738 KiB, 100%, 4.244 KiB/s, ETA 0s
Errors: 1 (retrying may help)
Elapsed time: 3.9s

2023/01/05 19:08:18 DEBUG : 7 go routines active
2023/01/05 19:08:18 Failed to copy: InvalidGrant: Invalid Grant
status code: 400, request id: f0abb583-e221-4ca5-92cf-22208dd23c4d, host id:

did you see my post from up above?
does the issue happen with both remotes, or just techzone-cos:?

techzone-cos: - does not have location_constraint

I tried copying a file into both the remotes and I get the 'Invalid Grant' error for both.

i am able to download files from the remote using the 'rclone -v -P copy gp-cos:cpdemo-backup gp-cos -vv' command , but adding or updating a file into the remote seems to be an issue.

with s3, i have never used ACL, but i see from the rclone docs,
for each type of ibm cloud, there is a different set of ACL.

and about public-read-write
Granting this on a bucket is generally not recommended.

as a test, have you tried other ACL such as default

oh yes. that was the issue. I changed the remote config to below

{
    "gp-cos": {
        "access_key_id": "31be62e6f0e24e12bd00b10ca0707bc4",
        "acl": "private",
        "endpoint": "s3.us-east.cloud-object-storage.appdomain.cloud",
        "env_auth": "false",
        "location_constraint": "us-east-standard",
        "provider": "IBMCOS",
        "region": "us-east",
        "secret_access_key": "bbc8d9f40dd25c9653074231f8b18197ba79423033e8d7d4",
        "type": "s3"
    },
    "techzone-cos": {
        "access_key_id": "620bffdd55c14eec91f42af8dd54e5dd",
        "acl": "private",
        "endpoint": "s3.us-south.cloud-object-storage.appdomain.cloud",
        "provider": "IBMCOS",
        "region": "us-south",
        "secret_access_key": "bc67ddf4406676d484a91aeadcfa2920178694061ab9f636",
        "type": "s3"
    }
}

and the copy worked... Infact in the ACL options I see this

   / Owner gets FULL_CONTROL.
 3 | The AllUsers group gets READ and WRITE access.
   | This acl is available on IBM Cloud (Infra), On-Premise IBM COS.
   \ (public-read-write)

This option is not available for IBM Cloud (storage)

Thank you for the responses. I will do some more testing and will close this post..

1 Like

good. let's us know how the testing goes.

All our testing with rclone went fine. We are able to work with our COS instances very well. Thank you for helping out. We are good to close this ticket please.

1 Like

We can hide that for IBMCOS - do you think that would be the right thing to do?

What are the available options? Can you link to docs?

Well, I think what is shown by rclone is fine. rclone is already hiding the option which possibly IBM COS does not support.

1 Like