S3 server-side copies with object Tags and no PutObjectTagging gives inconsistent behaviour for single v multipart copies

andrewpatto · November 18, 2024, 5:22am

What is the problem you are having with rclone?

Doing a server side S3 copy - where the objects have tags and the destination does not have s3:PutObjectTagging - has different results for small objects versus large objects.
(as in the single-part code path and the multi-part code path end up with fundamentally different results)

Run the command 'rclone version' and share the full output of the command.

rclone v1.68.2
- os/version: darwin 15.1 (64 bit)
- os/kernel: 24.1.0 (arm64)
- os/type: darwin
- os/arch: arm64 (ARMv8 compatible)
- go/version: go1.23.3
- go/linking: dynamic
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

AWS S3

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

Basically rclone copy AWS:... AWS:... but where the objects have tags
and the destination bucket does not have s3:PutObjectTagging.

More details in the log section.

The rclone config contents with secrets removed.

[AWS]
type = s3
provider = AWS
env_auth = true
region = ap-southeast-2
location_constraint = ap-southeast-2

A log from the command with the `-vv` flag

We have set up two buckets in different accounts. Here are the source objects

+ aws s3 ls --human-readable rclone-bug-source
2024-11-18 14:08:28    2.0 MiB 2mib-notags
2024-11-18 14:08:29    2.0 MiB 2mib-tags
2024-11-18 15:47:23    6.0 GiB 6gib-notags
2024-11-18 15:21:34    6.0 GiB 6gib-tags
+ aws s3api get-object-tagging --bucket rclone-bug-source --key 2mib-tags
{
    "TagSet": [
        {
            "Key": "atag",
            "Value": "avalue"
        }
    ]
}
+ aws s3api get-object-tagging --bucket rclone-bug-source --key 2mib-notags
{
    "TagSet": []
}
+ aws s3api get-object-tagging --bucket rclone-bug-source --key 6gib-tags
{
    "TagSet": [
        {
            "Key": "atag",
            "Value": "avalue6"
        }
    ]
}
+ aws s3api get-object-tagging --bucket rclone-bug-source --key 6gib-notags
{
    "TagSet": []
}

Objects with no tags can successfully be copied irrespective of the destination (s3:PutObjectTagging) - as would be expected.

Object with tags can be succesfully copied where the destination bucket location has s3:PutObjectTagging - and the tags are copied correctly.

A small tagged object copied to a location without put object tagging permissions fails to copy (am happy to accept that this is the correct behaviour!).

+ ./rclone1.68.2 -vv copy --s3-no-check-bucket --s3-disable-checksum --retries=1 --size-only --stats=0 AWS:rclone-bug-source/2mib-tags AWS:test-copy-destination-rclone-bug/no-put-object-tagging/
2024/11/18 16:05:03 DEBUG : rclone: Version "v1.68.2" starting with parameters ["./rclone1.68.2" "-vv" "copy" "--s3-no-check-bucket" "--s3-disable-checksum" "--retries=1" "--size-only" "--stats=0" "AWS:rclone-bug-source/2mib-tags" "AWS:test-copy-destination-rclone-bug/no-put-object-tagging/"]
2024/11/18 16:05:03 DEBUG : Creating backend with remote "AWS:rclone-bug-source/2mib-tags"
2024/11/18 16:05:03 DEBUG : Using config file from "/Users/appat/.config/rclone/rclone.conf"
2024/11/18 16:05:03 DEBUG : AWS: detected overridden config - adding "{YkiGG}" suffix to name
2024/11/18 16:05:03 DEBUG : fs cache: adding new entry for parent of "AWS:rclone-bug-source/2mib-tags", "AWS{YkiGG}:rclone-bug-source"
2024/11/18 16:05:03 DEBUG : Creating backend with remote "AWS:test-copy-destination-rclone-bug/no-put-object-tagging/"
2024/11/18 16:05:03 DEBUG : AWS: detected overridden config - adding "{YkiGG}" suffix to name
2024/11/18 16:05:03 DEBUG : fs cache: renaming cache item "AWS:test-copy-destination-rclone-bug/no-put-object-tagging/" to be canonical "AWS{YkiGG}:test-copy-destination-rclone-bug/no-put-object-tagging"
2024/11/18 16:05:04 DEBUG : 2mib-tags: Need to transfer - File not found at Destination
2024/11/18 16:05:04 ERROR : 2mib-tags: Failed to copy: operation error S3: CopyObject, https response error StatusCode: 403, RequestID: ZZ9JG3ASTMEFDK7K, HostID: lctcUiQWnBUjdk9j3/1J5NqMhn9j9ogIbVNlep+JuYmnEDyS97acVI0kS2RBOSz5M75nwGzkE9Yfup0/n86f7opRy2WPoxyu, api error AccessDenied: Access Denied
2024/11/18 16:05:04 ERROR : Attempt 1/1 failed with 1 errors and: operation error S3: CopyObject, https response error StatusCode: 403, RequestID: ZZ9JG3ASTMEFDK7K, HostID: lctcUiQWnBUjdk9j3/1J5NqMhn9j9ogIbVNlep+JuYmnEDyS97acVI0kS2RBOSz5M75nwGzkE9Yfup0/n86f7opRy2WPoxyu, api error AccessDenied: Access Denied

On the other hand - a large object that forces a multi-part copy - succeeds in the copy, just not the tagging.

+ ./rclone1.68.2 -vv copy --s3-no-check-bucket --s3-disable-checksum --retries=1 --size-only --stats=0 AWS:rclone-bug-source/6gib-tags AWS:test-copy-destination-rclone-bug/no-put-object-tagging/
2024/11/18 16:07:18 DEBUG : rclone: Version "v1.68.2" starting with parameters ["./rclone1.68.2" "-vv" "copy" "--s3-no-check-bucket" "--s3-disable-checksum" "--retries=1" "--size-only" "--stats=0" "AWS:rclone-bug-source/6gib-tags" "AWS:test-copy-destination-rclone-bug/no-put-object-tagging/"]
2024/11/18 16:07:18 DEBUG : Creating backend with remote "AWS:rclone-bug-source/6gib-tags"
2024/11/18 16:07:18 DEBUG : Using config file from "/Users/appat/.config/rclone/rclone.conf"
2024/11/18 16:07:18 DEBUG : AWS: detected overridden config - adding "{YkiGG}" suffix to name
2024/11/18 16:07:18 DEBUG : fs cache: adding new entry for parent of "AWS:rclone-bug-source/6gib-tags", "AWS{YkiGG}:rclone-bug-source"
2024/11/18 16:07:18 DEBUG : Creating backend with remote "AWS:test-copy-destination-rclone-bug/no-put-object-tagging/"
2024/11/18 16:07:18 DEBUG : AWS: detected overridden config - adding "{YkiGG}" suffix to name
2024/11/18 16:07:18 DEBUG : fs cache: renaming cache item "AWS:test-copy-destination-rclone-bug/no-put-object-tagging/" to be canonical "AWS{YkiGG}:test-copy-destination-rclone-bug/no-put-object-tagging"
2024/11/18 16:07:18 DEBUG : 6gib-tags: Need to transfer - File not found at Destination
2024/11/18 16:07:18 DEBUG : 6gib-tags: Starting  multipart copy with 2 parts
2024/11/18 16:09:37 DEBUG : 6gib-tags: Dst hash empty - aborting Src hash check
2024/11/18 16:09:37 DEBUG : 6gib-tags: Src hash empty - aborting Dst hash check
2024/11/18 16:09:37 INFO  : 6gib-tags: Copied (server-side copy)

So the main issue as a bug for us - is the inconsistent behaviour lead us to believe copies were happening based on us seeing large files at the destination - but where in reality none of the small files were coming across (we work genomics so most of our files are large).

We discovered this issue in the 1.67.0 version of rclone in the first instance -so I do not think it is a regression caused by the upgrade of the AWS Go SDK from v1 to v2.

ncw · November 18, 2024, 7:25pm

That does sound like a bug to me.

What do you this the correct behaviour is? To not copy the files or to copy them without tags?

My first thought is that is to do with this line

github.com

rclone/rclone/blob/d8bc542ffcb35fec67a86330e37c5ad23cbaf487/backend/s3/s3.go#L4876


      
          		return nil, err
          	}
          	srcObj, ok := src.(*Object)
          	if !ok {
          		fs.Debugf(src, "Can't copy - not same remote type")
          		return nil, fs.ErrorCantCopy
          	}
          
          	srcBucket, srcPath := srcObj.split()
          	req := s3.CopyObjectInput{
          		MetadataDirective: types.MetadataDirectiveCopy,
          	}
          
          	// Update the metadata if it is in use
          	if ci := fs.GetConfig(ctx); ci.Metadata {
          		ui, err := srcObj.prepareUpload(ctx, src, fs.MetadataAsOpenOptions(ctx), true)
          		if err != nil {
          			return nil, fmt.Errorf("failed to prepare upload: %w", err)
          		}
          		setFrom_s3CopyObjectInput_s3PutObjectInput(&req, ui.req)
          		req.MetadataDirective = types.MetadataDirectiveReplace

Though I'm not sure tagging is metadata in this context or not.

andrewpatto · November 18, 2024, 11:28pm

I think the "proper" (i.e most work) solution would be something that introduces some command line flags that select between the behaviours.

In this instance - for our use cases - we would prefer the object's get copied no matter what. In fact, we would prefer that we can explicitly ask that our tags don't carry through to the destination, even if the permissions were correct.

But I doubt that is universally true.

andrewpatto · November 18, 2024, 11:35pm

S3 Tags are separate from other items that AWS calls metadata.

So for CopyObjectInput there are TaggingDirective and Tagging as separate fields that control this.

I can do some playing around with what combinations work - though I get the feeling that there is no option for "copy if you can". If you ask it to Copy tags (which is the default) - and it fails to do so because of lack of permissions - it fails the entire API call.

TaggingDirective could be set to REPLACE and Tagging left to the default empty set - which would mean that Tags are never copied for these single part copies.

(am still investigating why/how the multi-part copies work in a "copy if you can" mode)

andrewpatto · November 18, 2024, 11:38pm

I also unfortunately have no experience with other S3-like object stores so have no idea about the implementations of Tagging more generally.

system · December 18, 2024, 11:39pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.