Rclone with Amazon S3 access point

Thanks for the feedback :slight_smile:

Maybe a message like this - regardless of the environment state or any auth settings provided. It's simply stating to the user what the behavior of env_auth=False does. To me, it sounds compatible with anonymous access as well.

2023/10/18 14:09:03 DEBUG : Since env_auth=False, the runtime environment is not used for authentication.

This could well be true, in general S3 access-related messages can be cryptic or misleading. Not an rclone problem.

A simple explanation is unfortunately not a short one :smiley:

S3 access points are a relatively new feature.

The problem
With large organisations using many AWS accounts, managing access to S3 buckets cross-account can become a headache. There are bucket policies (with a short 20kB length limit), or IAM users with IAM policies, or IAM roles with IAM policies. This means quite a bit of testing (what does the application support) and eventually spaghetti (a mix of these methods in practice) is needed to provision what should be "simple" access.

Most of the time, IAM users are not allowed because of the risk of having static long-lived credentials. So then it is down to bucket policy or IAM role policy.

If you allow access via the bucket policy, it gets full real quick. Worse - since there can only be one per bucket, it often means one team or repo managing it and that doesn't scale very well. Especially if each consumer needs to do their own fine-grained permissions in the policy...

If you go with IAM role policy, you need to configure this on both the source and target accounts, and then "jump into" the role assuming the application can do this.

A solution?
Access points are basically aliases for a bucket.

A bucket can have many access points defined, for example maybe you create two per consumer - one for "read" and one for "read write".
Then, each access point can have a policy on it that controls that consumer's specific low-level access within the bucket.

The net permissions is still determined by the combination of bucket policy + access point policy + IAM policy.

This still ends up with some spaghetti but it's a bit less than before. Not a complete solution.

One useful thing, you can share an access point to a specific VPC in a consumer account. So you could have a backend VPC with things that are allowed to talk to the bucket, and a frontend VPC which is not allowed.

Another solution?

You now (March 2023) have "cross account access points" which work the same but means the access point can be created in the consumer's account, not co-located with the bucket in its account. The consumer can see and manage their own fine-grained permissions for their application in their own account, and not need to access things in the bucket owner account.

ref Simplify and scale access management to shared datasets with cross-account Amazon S3 Access Points | AWS Storage Blog

TL;DR
It's a way to implement cross-account access to buckets, which scales better in a large organization. Particularly if you have a "central" account with many consumers eg a "data lake" pattern.

I think it will become a more common pattern in future at least in large orgs :slight_smile: