Access denied with rclone and not with awscli

What is the problem you are having with rclone?

I'm probably have configuration issues with rclone (or permissions issues maybe). I'm trying to setup a bucket with multiples folders with their own permissions on Amazon S3. The bucket should be accessible by multiples IAM user but each user should only have access to one folder (prefix). Everything seems to be ok with awscli but I always have access denied when using rclone.

The bucket have been created with this configuration (extract from cloudformation):

      AccessControl: Private

      PublicAccessBlockConfiguration:
        BlockPublicAcls: True
        BlockPublicPolicy: True
        IgnorePublicAcls: True
        RestrictPublicBuckets: True

Here are the policies I have setup for the specific IAM User (extract from cloudforamtion):

    Type: AWS::IAM::User
    Properties:
      Policies:
      - PolicyName: bucket-access
        PolicyDocument:
          Version: '2012-10-17'
          Statement:
            - Effect: Allow
              Action:  
                - s3:PutObject
                - s3:GetObject*
                - s3:HeadObject*
                - s3:ListObject*
              Resource:
              - !Sub arn:aws:s3:::${BucketTransfer}/client/*
      - PolicyName: bucket-ls
        PolicyDocument:
          Version: '2012-10-17'
          Statement:
            - Effect: Allow
              Action:  
                - s3:ListBucket
              Resource:
              - !Sub arn:aws:s3:::${BucketTransfer}
              Condition:
                StringEquals:
                  s3:prefix:
                    - ""
                    - "client/"
                  s3:delimiter:
                    - "/"

Thoses commands are working:

aws s3 ls s3://mybucket/client/
aws s3 ls s3://mybucket/

But with rclone, I have access denied:

rclone ls configname:mybucket/client
rclone ls configname:mybucket

I have a status code 403: Failed to ls: AccessDenied: Access Denied

Run the command 'rclone version' and share the full output of the command.

I'm using version 1.66.0:

rclone v1.66.0
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 6.5.0-26-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.22.1
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Amazon S3

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

My rclone configuration is:

[configname]
type = s3
provider = AWS
env_auth = false
region = eu-west-3
#endpoint = s3.eu-west-3.amazonaws.com
access_key_id = REDACTED
secret_access_key = REDACTED
#location_constraint = eu-west-3
acl = private

A log from the command that you were trying to run with the -vv flag

2024/03/20 09:40:45 Failed to ls: AccessDenied: Access Denied
	status code: 403, request id: REDACTED, host id: REDACTED

Thanks for your help

hi,
hard to know what is going on, without a full debug log?

your bucket policy is not using s3:ListAllMyBuckets, as a result, might need
--s3-no-check-bucket

and for a deeper look, try
--dump=headers --retries=1

I've added the two options without any success. Here is the debug log:

rclone ls --s3-no-check-bucket -vv --dump headers --retries=1 configname:bucketname/client
...
2024/03/21 06:55:52 DEBUG : Using config file from "/home/sylvain/.config/rclone/rclone.conf"
2024/03/21 06:55:52 DEBUG : configname: detected overridden config - adding "{Dn7qA}" suffix to name  # There is only one configname with this name, I don't understand this message
2024/03/21 06:55:52 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2024/03/21 06:55:52 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2024/03/21 06:55:52 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/03/21 06:55:52 DEBUG : HTTP REQUEST (req 0xc0001cd320)
2024/03/21 06:55:52 DEBUG : HEAD /client HTTP/1.1
Host: bucketname.s3.eu-west-3.amazonaws.com
User-Agent: rclone/v1.66.0
Authorization: XXXX
X-Amz-Content-Sha256: REDACTED
X-Amz-Date: 20240321T055552Z

2024/03/21 06:55:52 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/03/21 06:55:53 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/03/21 06:55:53 DEBUG : HTTP RESPONSE (req 0xc0001cd320)
2024/03/21 06:55:53 DEBUG : HTTP/1.1 403 Forbidden
Connection: close
Content-Type: application/xml
Date: Thu, 21 Mar 2024 05:55:52 GMT
Server: AmazonS3
X-Amz-Id-2: REDACTED
X-Amz-Request-Id: REDACTED

2024/03/21 06:55:53 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/03/21 06:55:53 DEBUG : fs cache: renaming cache item "configname:bucketname/client to be canonical "configname{Dn7qA}:bucketname/client"
2024/03/21 06:55:53 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/03/21 06:55:53 DEBUG : HTTP REQUEST (req 0xc0005d06c0)
2024/03/21 06:55:53 DEBUG : GET /?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=client%2F HTTP/1.1
Host: bucketname.s3.eu-west-3.amazonaws.com
User-Agent: rclone/v1.66.0
Authorization: XXXX
X-Amz-Content-Sha256: REDACTED
Accept-Encoding: gzip

2024/03/21 06:55:53 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/03/21 06:55:53 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/03/21 06:55:53 DEBUG : HTTP RESPONSE (req 0xc0005d06c0)
2024/03/21 06:55:53 DEBUG : HTTP/1.1 403 Forbidden
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Thu, 21 Mar 2024 05:55:52 GMT
Server: AmazonS3
X-Amz-Bucket-Region: eu-west-3
X-Amz-Id-2: REDACTED
X-Amz-Request-Id: REDACTED

2024/03/21 06:55:53 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/03/21 06:55:53 DEBUG : 6 go routines active
2024/03/21 06:55:53 Failed to ls: AccessDenied: Access Denied
	status code: 403, request id: REDACTED, host id: REDACTED

I think I've found the problem, I'm fixing it and will make a comment in the post.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.