Acl policy interferes with copyto but not copy

Suspected Bug
1

What is the problem you are having with rclone?

If I grant read permission to another user named B, the owner of the bucket, with full permissions, can no longer use copyto. The owner can use copy successfully. When using copyto, the owner receives a CreateBucket 409 error, BucketAlreadyExists.

Run the command 'rclone version' and share the full output of the command.

rclone v1.68.2
- os/version: rocky 8.9 (64 bit)
- os/kernel: 4.18.0-513.18.1.el8_9.x86_64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.23.3
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

A Ceph installation, named Kopah.

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copyto change-configs.tcsh k_U:U/goo/tryme2.tcsh

however, the following will work with any of the following ACLs which change the behavior of copyto.

rclone copy change-configs.tcsh k_U:U/goo/

copyto fails when there is an ACL like this, where I use U for the owner and B for the user granted read permissions

{
    "Owner": {
        "DisplayName": "U",
        "ID": "U"
    },
    "Grants": [
        {
            "Grantee": {
                "DisplayName": "B",
                "ID": "B",
                "Type": "CanonicalUser"
            },
            "Permission": "READ"
        },
        {
            "Grantee": {
                "DisplayName": "U",
		"ID": "U",
		"Type": "CanonicalUser"
            },
            "Permission": "FULL_CONTROL"
	}
    ]
}

The same is true if the ACL looks like:

{
    "Owner": {
        "DisplayName": "U",
        "ID": "U"
    },
    "Grants": [
        {
            "Grantee": {
                "DisplayName": "B",
                "ID": "B",
                "Type": "CanonicalUser"
            },
            "Permission": "READ"
        }
    ]
}

However, if does work if I remove the reader username B:

{
    "Owner": {
        "DisplayName": "U",
        "ID": "U"
    },
    "Grants": [
        {
            "Grantee": {
                "DisplayName": "U",
                "ID": "U",
                "Type": "CanonicalUser"
            },
            "Permission": "FULL_CONTROL"
        }
    ]
}

The rclone config contents with secrets removed.

[k_U]
type = s3
provider = Ceph
access_key_id = xyz
secret_access_key = xyz
endpoint = https://s3.kopah.my.org

A log from the command with the -vv flag

2024/12/29 17:47:02 DEBUG : rclone: Version "v1.68.2" starting with parameters ["rclone" "copyto" "change-configs.tcsh" "remote:path/to/file.tcsh" "-vv"]
2024/12/29 17:47:02 DEBUG : Creating backend with remote "change-configs.tcsh"
2024/12/29 17:47:02 DEBUG : Using config file from "/path/to/config/rclone.conf"
2024/12/29 17:47:02 DEBUG : fs cache: adding new entry for parent of "change-configs.tcsh", "/path/to/projects"
2024/12/29 17:47:02 DEBUG : Creating backend with remote "remote:path/to/destination/"
2024/12/29 17:47:02 DEBUG : fs cache: renaming cache item "remote:path/to/destination/" to be canonical "remote:path/to/destination"
2024/12/29 17:47:02 DEBUG : change-configs.tcsh: Need to transfer - File not found at Destination
2024/12/29 17:47:02 ERROR : change-configs.tcsh: Failed to copy: failed to prepare upload: operation error S3: CreateBucket, https response error StatusCode: 409, BucketAlreadyExists:
2024/12/29 17:47:02 ERROR : Can't retry any of the errors - not attempting retries
2024/12/29 17:47:02 INFO  :
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (no need to retry)
Elapsed time:         0.0s

2024/12/29 17:47:02 DEBUG : 7 go routines active
2024/12/29 17:47:02 NOTICE: Failed to copyto: failed to prepare upload: operation error S3: CreateBucket, https response error StatusCode: 409, BucketAlreadyExists:
2

You might be right that it is a bug. Not the first one with copyto/moveto. On this forum you can find multiple examples when these two commands simply do not work. IMO it is broken.

Myself I forgot that these commands exist and always use copy/move with filters.

3

welcome to the forum,

try --s3-no-check-bucket

4

I did try --s3-no-check-bucket but it's not available in the version that I'm using (the latest version: v1.68.2), but perhaps --no-traverse could be a similar workaround. Should I post the bug to the github?

5

well, --s3-no-check-bucket was added over four years ago, back at v1.53.0 - 2020-09-02
please post a complete debug log with the exact error.

did you test that?

S3 backend updated to use AWS SDKv2 as v1 is now unsupported.
that was version v1.68.0. so, might want to test an older version, such as v1.67.0

still not sure this is a rclone bug.
for a deeper look, can run the command with `--dump=headers

6

I did not understand what --s3-no-check-bucket meant since I did not see it when I use --help. Perhaps it is a hidden command-line option. I do see the --s3- type syntax now at Amazon S3. If I use no_check_bucket = 1 in the configuration file, it does allow copyto to work as expected.

7

i thought that would work.

good, rclone seems to be working as expected.

8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.