Question on AWS, EKS and IAM

What is the problem you are having with rclone?

Hi, a bit more general question. I would like to understand better how rclone works with IAM roles on EKS cluster.

The docs reads

Or, run rclone in an EKS pod with an IAM role that is associated with a service account (AWS only).

It is not clear for me what "pod with an IAM role" is. Is it just Pod run with a right Service Account?

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  serviceAccountName: IAM-SA-HERE
  ...

?

Provided above is correct, should it just work almost out of the box if config has env_auth = true?

What is your rclone version (output from rclone version)

rclone v1.54.1
- os/arch: linux/amd64
- go version: go1.15.8

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Pod running rclone image in EKS cluster.

Which cloud storage system are you using? (eg Google Drive)

AWS S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone lsd aws-iam:some-test-bucket-name

The rclone config contents with secrets removed.

[aws-iam]
type = s3
provider = AWS
env_auth = true
region = eu-west-2                           # Do I really need this?
location_constraint = eu-west-2              # Do I really need this?

# Do I need anything else?

A log from the command with the -vv flag

Paste  log here

I believe (not an AWS expert) that it will "just work" if you set env_auth = true. That is the intention anyway!

Thanks @ncw, that's positive!

Do you, or someone else on rclone team, would know what are the requirements for the k8s Pod for "just work"? :slight_smile: Like, does it need annotation, specific service account, or secrets to be mounted?

We will be testing it tomorrow and I am trying to prepare as good as possible (without having access to EKS and IAM role...) beforehand.

I'm afraid I don't know the answer to that question. If you can't get it going then please make a new issue on github and I'll tag some people who might know.

When you get it going we should probably improve the docs

You can edit them here (click pencil icon at top right) if you find something out : rclone/s3.md at master · rclone/rclone · GitHub

Thank you @ncw, will let you know how it goes and follow on Github in case of problems / clarifying the docs.

1 Like

The official image is here Docker Hub
Built from this Dockerfile rclone/Dockerfile at v1.54.1 · rclone/rclone · GitHub
Apparently it expects that you bind your config either with volume -v /path/to/rclone.conf:/config/rclone.conf or as a secret, then start the container with whatever rclone CLI arguments.

Thanks @ivandeex,

The rclone config itself I will probably expose as environmental variables, something similar to following example that works with public buckets

apiVersion: v1
kind: Secret
metadata:
  name: seldon-rclone-secret
type: Opaque
stringData:
  RCLONE_CONFIG_S3_TYPE: s3
  RCLONE_CONFIG_S3_PROVIDER: aws

  RCLONE_CONFIG_MYS3_ACCESS_KEY_ID: ""
  RCLONE_CONFIG_MYS3_SECRET_ACCESS_KEY: ""

  RCLONE_CONFIG_S3_ENV_AUTH: "false"
  RCLONE_CONFIG_S3_REGION: eu-west-2

---

apiVersion: v1
kind: Pod
metadata:
  name: rclone-pod
spec:
  containers:
  - name: rclone
    image: rclone/rclone:latest
    command: [ "/bin/sh", "-c", "--", "sleep 3600"]
    envFrom:
    - secretRef:
        name: seldon-rclone-secret

but adjusted to the env_auth = true equivalent aligned with config from the first post.

My question was more along: what else may I need when creating such pod in EKS cluster for it work with IAM roles:

  • annnotation?
  • specific Service Account?
  • something else?

Hi @ncw, unfortunately we run into some problems.
I opened github issue as per your advice Problem copying files from AWS S3 with IAM auth on EKS · Issue #5137 · rclone/rclone · GitHub

1 Like

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.