Rclone ls returns 403 on different VMs

Help and Support
1

Hi,

I am trying to run

rclone ls <profile>:<bucket>

on several different VMs. The storage is IBMCOS. On a VM running version

1.64.1

the command works, but any version greater than that I get a 403.
I have verified on numerous occasions that the config files have identical access and secret keys. Perhaps this is helpful

[sow26user@dh-sow26-prod ~]$ rclone ls -vv *******:*******

2025/03/11 13:20:04 DEBUG : rclone: Version "v1.65.1" starting with parameters ["rclone" "ls" "-vv" "******.*****"]
2025/03/11 13:20:04 DEBUG : Creating backend with remote "*****.*****"
2025/03/11 13:20:04 DEBUG : Using config file from "/home/user/.config/rclone/rclone.conf"
2025/03/11 13:20:04 DEBUG : Resolving service "s3" region "us-east-1"
2025/03/11 13:20:05 DEBUG : 6 go routines active
2025/03/11 13:20:05 Failed to ls: AccessDenied: Access Denied
        status code: 403, request id: b00953dd-d726-4eaf-a474-c110ae6169bd, host id:

Any ideas would be appreciated.

2

welcome to the forum,

first, please do testing on latest rclone and post the output of:

  • rclone version
  • rclone config redacted
  • the rclone command using --dump=headers --retries=1
4

Thanks for getting back to me.
What follows are those three commands on the machine where the command does not work.

[eals@eals0 logs]$ rclone version
rclone v1.68.1
- os/version: redhat 9.5 (64 bit)
- os/kernel: 5.14.0-427.42.1.el9_4.x86_64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.23.1
- go/linking: static
- go/tags: none

[eals@eals0 logs]$ rclone config redacted
[eals-prod-backup]
type = s3
provider = IBMCOS
env_auth = false
access_key_id = XXX
secret_access_key = XXX
endpoint = https://s3.us-south.cloud-object-storage.appdomain.cloud
### Double check the config for sensitive info before posting publicly

[eals@eals0 logs]$ rclone ls --dump=headers --retries=1 eals-prod-backup:eals-prod-backup
2025/03/12 10:26:58 NOTICE: Automatically setting -vv as --dump is enabled
2025/03/12 10:26:58 DEBUG : rclone: Version "v1.68.1" starting with parameters ["rclone" "ls" "--dump=headers" "--retries=1" "eals-prod-backup:eals-prod-backup"]
2025/03/12 10:26:58 DEBUG : Creating backend with remote "eals-prod-backup:eals-prod-backup"
2025/03/12 10:26:58 DEBUG : Using config file from "/home/eals/.config/rclone/rclone.conf"
2025/03/12 10:26:58 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2025/03/12 10:26:58 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2025/03/12 10:26:58 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/03/12 10:26:58 DEBUG : HTTP REQUEST (req 0xc0001f6640)
2025/03/12 10:26:58 DEBUG : GET /eals-prod-backup?delimiter=&max-keys=1000&prefix= HTTP/1.1
Host: s3.us-south.cloud-object-storage.appdomain.cloud
User-Agent: rclone/v1.68.1
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: d7629646-8aef-4fa4-8380-d6785cc9b024
Amz-Sdk-Request: attempt=1; max=10
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20250312T152658Z

2025/03/12 10:26:58 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/03/12 10:26:58 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/03/12 10:26:58 DEBUG : HTTP RESPONSE (req 0xc0001f6640)
2025/03/12 10:26:58 DEBUG : HTTP/1.1 403 Forbidden
Content-Length: 261
Accept-Ranges: bytes
Content-Type: application/xml
Date: Wed, 12 Mar 2025 15:38:56 GMT
Server: Cleversafe
X-Amz-Request-Id: 7fd3db1e-02b4-4dc7-8746-f8a8979ad7e5
X-Clv-Request-Id: 7fd3db1e-02b4-4dc7-8746-f8a8979ad7e5
X-Clv-S3-Version: 2.5

2025/03/12 10:26:58 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/03/12 10:26:58 NOTICE: Time may be set wrong - time from "s3.us-south.cloud-object-storage.appdomain.cloud" is -11m57.081511117s different from this computer
2025/03/12 10:26:58 DEBUG : 6 go routines active
2025/03/12 10:26:58 NOTICE: Failed to ls: operation error S3: ListObjects, https response error StatusCode: 403, RequestID: 7fd3db1e-02b4-4dc7-8746-f8a8979ad7e5, HostID: , api error AccessDenied: Access Denied

Here are those same three commands on a machine with an older version of rclone where it does work:

rclone v1.64.2
- os/version: redhat 9.5 (64 bit)
- os/kernel: 5.14.0-503.21.1.el9_5.x86_64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.21.3
- go/linking: static
- go/tags: none

[als_user@dh-als ~]$ rclone config redacted
[eals-prod-backup]
type = s3
provider = IBMCOS
env_auth = false
access_key_id = XXX
secret_access_key = XXX
endpoint = https://s3.us-south.cloud-object-storage.appdomain.cloud

[als_user@dh-als ~]$ rclone ls --dump=headers --retries=1 eals-prod-backup:eals-prod-backup | more
2025/03/12 12:46:10 NOTICE: Automatically setting -vv as --dump is enabled
2025/03/12 12:46:10 DEBUG : rclone: Version "v1.64.2" starting with parameters ["rclone" "ls" "--dump=headers" "--retries=1" "eals-prod-backup:eals-prod-backup"]
2025/03/12 12:46:10 DEBUG : Creating backend with remote "eals-prod-backup:eals-prod-backup"
2025/03/12 12:46:10 DEBUG : Using config file from "/home/als_user/.config/rclone/rclone.conf"
2025/03/12 12:46:10 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2025/03/12 12:46:10 DEBUG : Resolving service "s3" region "us-east-1"
2025/03/12 12:46:10 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2025/03/12 12:46:10 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/03/12 12:46:10 DEBUG : HTTP REQUEST (req 0xc00059b200)
2025/03/12 12:46:10 DEBUG : GET /eals-prod-backup?delimiter=&max-keys=1000&prefix= HTTP/1.1
Host: s3.us-south.cloud-object-storage.appdomain.cloud
User-Agent: rclone/v1.64.2
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20250312T174610Z
Accept-Encoding: gzip

2025/03/12 12:46:10 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/03/12 12:46:10 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/03/12 12:46:10 DEBUG : HTTP RESPONSE (req 0xc00059b200)
2025/03/12 12:46:10 DEBUG : HTTP/1.1 200 OK
Content-Length: 486746
Accept-Ranges: bytes
Content-Type: application/xml
Date: Wed, 12 Mar 2025 17:47:30 GMT
Ibm-Sse-Kp-Enabled: false
Server: Cleversafe
X-Amz-Request-Id: 72c523cd-7d27-444f-bcc7-bb373e769046
X-Clv-Request-Id: 72c523cd-7d27-444f-bcc7-bb373e769046
X-Clv-S3-Version: 2.5

2025/03/12 12:46:10 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<


I should also mention that I tried this on another VM on which v 1.64.2 was installed (the same as the one immediately above where it does work) and I got a 403.
I initially thought that it worked on an older version, but that is not the case.

Any thoughts?
5

NOTICE: Time may be set wrong - time from "s3.us-south.cloud-object-storage.appdomain.cloud" is -11m57.081511117s different from this computer

6

I see the time difference, but the time on the machine for which I don't have access is the same as the one with access so I don't think that's it.

7

coud be correct, but should be easy to fix and then, have one less issue to deal with?

your issue might be related to S3 backend updated to use AWS SDKv2 as v1 is now unsupported
might try --s3-list-version=1

8

Adding the additional switch you recommended had no effect, but thanks for suggesting. Also, I have other VMs that return 403 that don't show that

Time may be set wrong

message.

9

ok, might experiment with more --dump flags

i tried to setup a free trial account at ibmcos, but they ask for too much private information and a credit card.

10

I did try the

--dump

flag and reported that in an earlier post.

I have reached out to IBM to see if there is something they can tell me.
I don't believe it is rclone which I consider a fantastic tool.

Let's see what they tell me, if anything.