Hey,
I am trying to get Rclone working with Google's Workload Identity Federation (WIF from now on).
WIF allows one to use external Identity Provider (IdP from now on) and use it to authenticate requests to GCP API. It supports AWS, ADFS and so on, but at most basic level it can be OIDC provider JWKs and this means that any other Kubernetes cluster with JWK issuer can work as IdP. That's what I am testing first.
Essentially, trying to establish trust with GCP that says: "trusts this openid connect (OIDC) token issued to a k8s service account by a given K8s API server
What I am trying to do follows quite closely the example presented here but I am using an actual K8s cluster so do not need expose the issue via ngrok.
I will try to keep the description brief yet complete.
1. Configuring workflow identity federation
I needed to configure WIF on GCP side. For that I obtain the issuer url with
kubectl get --raw /.well-known/openid-configuration | jq .issuer
for completness the response from .well-known/openid-configuration looks like this
{
"issuer": "https://container.googleapis.com/v1/projects/<project id>/locations/<region>/clusters/<cluster name>",
"jwks_uri": "https://<some ip>:443/openid/v1/jwks",
"response_types_supported": [
"id_token"
],
"subject_types_supported": [
"public"
],
"id_token_signing_alg_values_supported": [
"RS256"
]
}
I use the issuer to configure WIF on the GCP side and give it rclone-test-audience as allowed audience. I also set few attributes mapping but this is for now not important.
Note that this is K8s cluster running on GCP as well so normally one should just use Workload Identity (not WIF) but I want to workout more general case.
2. Configuring K8s pod to have K8s SA's creds
Now I configure the K8s service account and make sure that pod (with rclone) has right tokens available
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: rclone-sa
---
apiVersion: v1
kind: Pod
metadata:
name: rclone-wif
spec:
serviceAccountName: rclone-sa
containers:
- name: rclone
image: rclone/rclone:1.63.0
command: ["/bin/sh", "-c", "--", "sleep 3600"]
volumeMounts:
- name: rclone-config-volume
mountPath: /config/rclone
- name: sa-token
mountPath: /var/run/secrets/tokens
volumes:
- name: rclone-config-volume
secret:
secretName: rclone-config
- name: sa-token
projected:
sources:
- serviceAccountToken:
path: sa-token
expirationSeconds: 7200
audience: rclone-test-audience
with this, effectively under the /run/secrets/tokens/sa-token inside the pod one will find a JWT token that after decoding with https://jwt.io/ looks like
{
"aud": [
"rclone-test-audience"
],
"exp": <some timestamp: integer>,
"iat": <some timestamp: integer>,
"iss": "https://container.googleapis.com/v1/projects/<project id>/locations/<region>/clusters/<cluster name>",
"kubernetes.io": {
"namespace": "default",
"pod": {
"name": "rclone-wif",
"uid": "<some uid>"
},
"serviceaccount": {
"name": "rclone-sa",
"uid": "<some uid>"
}
},
"nbf": <some timestamp: integer>,
"sub": "system:serviceaccount:default:rclone-sa"
}
3. Configuring Rclone
Ideally I would want to just point Rclone to JWT token under /run/secrets/tokens/sa-token but that does not seem to be possible. I tried then to create the credentials config
gcloud iam workload-identity-pools create-cred-config \
projects/<project number>/locations/global/workloadIdentityPools/<pool id>/providers/<provider id> \
--service-account=oidc-federated@<project id>.iam.gserviceaccount.com \
--output-file=creds.json \
--credential-source-file=/var/run/secrets/tokens
which creds.json as follow
{
"type": "external_account",
"audience": "//iam.googleapis.com/projects/<project number>/locations/global/workloadIdentityPools/<pool id>/providers/<provider id>",
"subject_token_type": "urn:ietf:params:oauth:token-type:jwt",
"token_url": "https://sts.googleapis.com/v1/token",
"credential_source": {
"file": "/var/run/secrets/tokens"
},
"service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/oidc-federated@<project id>.iam.gserviceaccount.com:generateAccessToken"
}
I then hope to use that creds.json with rclone so I complete my config with
apiVersion: v1
kind: Secret
metadata:
name: rclone-config
type: Opaque
stringData:
rclone.conf: |
[gs]
type = google cloud storage
service_account_file = /config/rclone/creds.json
creds.json: |
{
"type": "external_account",
"audience": "//iam.googleapis.com/projects/<project number>/locations/global/workloadIdentityPools/<pool id>/providers/<provider id>",
"subject_token_type": "urn:ietf:params:oauth:token-type:jwt",
"token_url": "https://sts.googleapis.com/v1/token",
"credential_source": {
"file": "/var/run/secrets/tokens/sa-token"
},
"service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/oidc-federated@<project id>.iam.gserviceaccount.com:generateAccessToken"
}
4. Tests and where I hit issues
Hopeful that this is all I need I exec into the pod and try to access my test bucket
$ rclone ls gs:<test bucket> -vvv
2023/07/09 13:57:54 DEBUG : rclone: Version "v1.63.0" starting with parameters ["rclone" "ls" "gs:<test bucket>" "-vvv"]
2023/07/09 13:57:54 DEBUG : Creating backend with remote "gs:<test bucket>"
2023/07/09 13:57:54 DEBUG : Using config file from "/config/rclone/rclone.conf"
2023/07/09 13:57:54 Failed to create file system for "gs:<test bucket>": failed configuring Google Cloud Storage Service Account: error processing credentials: google: read JWT from JSON credentials: 'type' field is "external_account" (expected "service_account")
The
Failed to create file system for "gs:": failed configuring Google Cloud Storage Service Account: error processing credentials: google: read JWT from JSON credentials: 'type' field is "external_account" (expected "service_account")
error seems to originate from https://github.com/rclone/rclone/blob/473d443874ef2facf16c66342cb70e806c05a1d5/backend/googlecloudstorage/googlecloudstorage.go#L487 which in results call https://github.com/golang/oauth2/blob/62b4eedd7210c3ff2fc694318a8a312b96f01a74/google/google.go#L84-L94
So... either I need to try and use different configuration or rclone needs a new code path to take into account external_account type of credentials. Any help and suggestions will be greatly appreciated!
