Took awhile to set up. No change from the branch, e.g. we error out using the v1.62.0-beta.6743.31cb3beb7.fix-s3-endpoint in the familiar way.
2023/02/28 12:57:13 DEBUG : 2 go routines active
2023/02/28 12:57:13 Failed to copy: SerializationError: failed to unmarshal error message
status code: 400, request id:
caused by: UnmarshalError: failed to unmarshal error message
00000000 3c 3f 78 6d 6c 20 76 65 72 73 69 6f 6e 3d 22 31 |<?xml version="1|
00000010 2e 30 22 20 65 6e 63 6f 64 69 6e 67 3d 22 55 54 |.0" encoding="UT|
00000020 46 2d 38 22 3f 3e 0a 3c 45 72 72 6f 72 3e 3c 43 |F-8"?>.<Error><C|
00000030 6f 64 65 3e 49 6e 76 61 6c 69 64 52 65 71 75 65 |ode>InvalidReque|
00000040 73 74 3c 2f 43 6f 64 65 3e 3c 4d 65 73 73 |st</Code><Mess|
Testing should be simple because the failure occurs when specifying an endpoint and profile, even if we specify a standard public endpoint (just tried that). An exact duplicate test would require a custom endpoint, but the standard one seems just as bad.
I'm very happy to work as directly as you can stand on this, as it's become quite an issue and coding around it is being difficult and expensive! Slack or other screen shares will work fine ... I'm E.T. and free until 5:30P (changed)
To duplicate, something like this failcase:
rclone copy --s3-profile YOURPROFILE --s3-acl "" ~/testbed/bigmovie s3-standard-internet:YOURBUCKET-1 -vv
- The
--s3-aclappears optional assuming the bucket has no ACL - Sourcefile is any old thing
- The provider needs to look something like this:
[s3-standard-internet]
type = s3
provider = AWS
env_auth = true
region = us-east-1
endpoint = https://s3.us-east-1.amazonaws.com
storage_class = STANDARD
The important thing there is to specify an endpoint (I put in the public internet endpoint, which fails as badly as our custom endpoint)
- The profile works like this ... I'm going to get real specific because the term
profileis badly overloaded and can mean either the name of a section in AWS credentials or a section in AWS config. In our case it must refer to AWS config. I don't want to belabor the point, but as you asked for detail, it works like this:
a. In your AWS account, make a role you can assume to test the capability. The role should allow S3 capabilities. If you're not used to IAM, I can walk you thru the role creation. A role in a second AWS account is what we're doing, but for now let's just get it working with any old role.
b. Identify a bucket in your AWS account. For an even better test, make the bucket policy require the new role. If that's too fiddly for now, any old bucket will do at first.
c. Put your AWS credentials in the AWS credentials file, usually located at `~/.aws/credentials. Something like this:
[MyCreds]
aws_access_key_id=AKIAXXXXXXXXXXXX
aws_secret_access_key=YYYYYYYYYYYY
This is not the entry that you'll refer to in the command line!
d. Create an entry in your AWS profile file (usually ~/.aws/config) that names the profile we use on the rclone command ilne, and that ties together the role we'll assume and the credentials you put in the creds file. Like this:
[profile MyProfile]
role_arn = arn:aws:iam::999999999999:role/YOUR-ROLE-FROM-STEP-1
source_profile = MyCreds
duration_seconds = 900
duration_seconds is optional and good for testing timeouts
You can see how Amazon's lack of documentation skills messes people up here. We are literally creating a profile (in the config sense) and pointing it at a "source_profile" (in the credentials sense). This is why engineers can benefit from writing class!
Anyway, flame off. We have a config profile called MyProfile that refers to a credentials entry called MyCreds and says that when invoked we use the MyCreds keys to:
- establish a session with MyCreds
- Assume the role MyRole for the durationSeconds
- (the cool part) Cache those creds, and refresh them when they will expire.
Then just plug it all into a command:
rclone copy --s3-profile MyProfile --s3-acl "" SOMESOURCEFILE s3-standard-internet:YOURBUCKET-1 -vv
and see if it dies.
The easy way to show that it's the presence of endpoint along with an --s3-profile that causes the problem is to try the same copy with a rclone config that has no endpoint specified, e.g.
[s3-standard-none]
type = s3
provider = AWS
env_auth = true
region = us-east-1
storage_class = STANDARD
You should have my work email, or key me with a response and I'll be happy to talk about it, or walk you through.
I did figure out (I think) how to do builds ... easier than I thought, just had to find the docs.
R.
ADDENDUM: Figured out from timestamps you're probably across the pond. Will try to get up early and leave another note in case you want a collaboration to get things going, dogs willing.