Types of access in EKS
- Grant applications and users access to Kubernetes APIs - enable applications or users to authenticate to the Kubernetes API. We can use:
- access entries
- aws-auth ConfigMap
- external OIDC provider
- Grant Kubernetes workloads (processes running in pods) access to AWS using Kubernetes Service Accounts - associate a Kubernetes service account with AWS IAM Roles. In past we had to create IAM User access key and to inject IAM User credentials (key ID and secret) into a pod (see Securing AWS Access from Kubernetes cluster using IRSA & OIDC | by Seifeddine Rajhi | Medium). As using sharing IAM User credentials and using long-term access keys is bad security practice, a new methods for Pod authentication to AWS API emerged so now we can use:
- Pod Identity
- IAM Roles for Service Accounts (IRSA)
Access to Kubernetes APIs
Our cluster has an Kubernetes API endpoint. Kubectl uses this API. We can authenticate to this API using two types of identities:
- An AWS Identity and Access Management (IAM) principal (role or user)
- A user in our own OpenID Connect (OIDC) provider
- Requires authentication to our OIDC provider
- setup: Authenticate users for your cluster from an OpenID Connect identity provider - Amazon EKS
- We can associate one OIDC identity provider to our cluster.
- Kubernetes doesn't provide an OIDC identity provider. We can use an existing public OIDC identity provider, or we can run our own identity provider.
- The issuer URL of the OIDC identity provider must be publicly accessible, so that Amazon EKS can discover the signing keys. Amazon EKS doesn't support OIDC identity providers with self-signed certificates.
- Before we can associate an OIDC identity provider with our cluster, we need the following information from our provider:
- Issuer URL - The URL of the OIDC identity provider that allows the API server to discover public signing keys for verifying tokens. The URL must begin with https:// and should correspond to the iss claim in the provider's OIDC ID tokens. In accordance with the OIDC standard, path components are allowed but query parameters are not. Typically the URL consists of only a host name, like https://server.example.org or https://example.com. This URL should point to the level below .well-known/openid-configuration and must be publicly accessible over the internet.
- Client ID (also known as audience) - The ID for the client application that makes authentication requests to the OIDC identity provider.
We can't disable IAM authentication to our cluster, because it's still required for joining nodes to a cluster. OIDC authentication is optional. Both can be enabled on cluster at the same time.
IAM OIDC identity providers are entities in IAM that describe an external identity provider (IdP) service that supports the OpenID Connect (OIDC) standard, such as Google or Salesforce. You use an IAM OIDC identity provider when you want to establish trust between an OIDC-compatible IdP and your AWS account. This is useful when creating a mobile app or web application that requires access to AWS resources, but you don't want to create custom sign-in code or manage your own user identities.
You can create and manage an IAM OIDC identity provider using the:AWS Management Console, the AWS Command Line Interface, the Tools for Windows PowerShell, or the IAM API.
After you create an IAM OIDC identity provider, you must create one or more IAM roles. A role is an identity in AWS that doesn't have its own credentials (as a user does). But in this context, a role is dynamically assigned to a federated user that is authenticated by your organization's IdP. The role permits your organization's IdP to request temporary security credentials for access to AWS. The policies assigned to the role determine what the federated users are allowed to do in AWS.
Kubernetes workloads access to AWS
A workload is an application running in one or more Kubernetes pods.
A Kubernetes service account provides an identity for processes that run in a Pod.
If your Pod needs access to AWS services, you can map the service account to an IAM identity to grant that access.
Granting IAM permissions to workloads on Amazon Elastic Kubernetes Service clusters
Amazon EKS provides two ways to grant IAM permissions to workloads that run in Amazon EKS clusters:
- IAM roles for service accounts (IRSA)
- Old way
- Allows pods to directly use IAM Roles (no need to inject into pods IAM User access credentials anymore)
- We define the trust relationship between an IAM role and Kubernetes service account (that's a type of account in Kubernetes) in the role's trust policy.
- Each EKS cluster has an OpenID Connect (OIDC) issuer URL associated with it.
- To use/enable IRSA a unique OpenID Connect provider needs to be created for each EKS cluster in IAM.
- EKS Pod Identities
- Modern approach
IAM roles for service accounts (IRSA)
What are Service Accounts?
An AWS EKS cluster service account is a Kubernetes identity assigned to pods, allowing them to authenticate with the (cluster) API server. EKS enhances this by enabling service accounts to assume IAM roles (IRSA - IAM Roles for Service Accounts), providing fine-grained, secure AWS permission access to containers without managing long-lived credentials.
Key Aspects of EKS Service Accounts:
- Identity & Security: Acts as a non-human identity within the cluster for applications to interact with Kubernetes APIs or external AWS services.
- IRSA (IAM Roles for Service Accounts): EKS creates an OIDC provider, allowing Kubernetes service accounts to map directly to AWS IAM roles. Pods use this association to get temporary credentials for services like S3, DynamoDB, etc..
- EKS Pod Identity: A newer, simpler alternative to IRSA that uses an agent to manage credentials directly, bypassing the need for OIDC configuration.
- Management: Defined within Kubernetes manifests (ServiceAccount resource) and linked to IAM policies via OIDC provider setup or EKS Pod Identity Associations.
Why Use EKS Service Accounts?
Instead of assigning IAM roles to worker nodes (which gives all pods the same privileges), service accounts follow the principle of least privilege, giving specific permissions only to the pods that require them.
In brief, how IRSA authentication works?
When an AWS SDK inside a pod needs to access a resource (like an S3 bucket for Loki), it doesn't just send a simple "I am Loki" message. It performs a complex cryptographic handshake called OIDC Federation.
Here is exactly what is sent, what is inside the token, and how STS "connects the dots."
1. What the SDK sends to STS
The SDK calls the AssumeRoleWithWebIdentity API. It sends three primary things:
- RoleArn: The ARN of the IAM Role it wants to assume (e.g., your geeiq-prod-monitoring-k8s-grafana-loki role).
- RoleSessionName: A unique name for the session (usually the pod name).
- WebIdentityToken: This is the most important part—the actual "Identity Card" (JWT) mounted at /var/run/secrets/eks.amazonaws.com/serviceaccount/token.
2. What is inside that Token?
The token is a JSON Web Token (JWT) signed by your EKS cluster's private key. If you were to decode it, you would see a payload like this:
{
"iss": "https://oidc.eks.us-east-2.amazonaws.com/id/570323FD881F43322B5CD6D7693E76DB",
"sub": "system:serviceaccount:grafana-loki:grafana-loki",
"aud": "sts.amazonaws.com",
"exp": 1740000000,
"iat": 1739996400,
"kubernetes.io/namespace": "grafana-loki",
"kubernetes.io/serviceaccount/name": "grafana-loki",
"kubernetes.io/pod/name": "loki-write-0"
}
- iss (Issuer): The URL of your EKS OIDC server. This tells STS exactly who issued the token.
- sub (Subject): The unique identity of the pod (Namespace + ServiceAccount name).
- aud (Audience): This specifies who the token is intended for. This is why your previous IAM fix was so important. If the token says the audience is sts.amazonaws.com, but your IAM policy doesn't require it, the handshake fails.
3. How does STS know which IAM OIDC Provider to talk to?
This is where the "match" we looked at earlier happens:
- Extracting the Issuer: STS looks at the iss field in the incoming token.
- Lookup: STS searches your AWS Account's IAM Identity Providers for a provider that matches that iss URL character-for-character.
- Signature Verification: Once STS finds the matching IAM Provider, it uses the Thumbprint (the SSL fingerprint) to go to the EKS OIDC URL and download the Public Keys.
- Cryptographic Proof: STS uses those public keys to verify that the token was indeed signed by your specific EKS cluster's private key. If the signature is valid, STS knows the token hasn't been tampered with.
- Policy Validation: Finally, STS checks the Trust Relationship on the role you requested. It ensures that the sub and aud in the token match the StringEquals conditions in your role's JSON.
AWS SDK in pods needs to be authenticated against STS so AWS SDK on pods can access AWS resources.
Even though you create the Identity Provider resource in the IAM console, the EKS cluster is the one doing all the actual "work" of identifying the pods.
Think of it like a Passport Office and a Border Agent:
- EKS (The Passport Office): The cluster holds the private key. It issues "passports" (JWT tokens) to pods. EKS is source of truth.
- IAM (The Border Agent): IAM doesn't issue the passports, but it has a record (the Identity Provider resource) of what a "valid" passport from your specific cluster should look like.
1. How EKS becomes the "Issuer"
Every EKS cluster runs a tiny, public-facing web server called the OIDC Discovery Endpoint.
- The URL: This is that https://oidc.eks... URL we discussed.
- The Keys: If you visit that URL (specifically the /.well-known/openid-configuration path), EKS provides a list of public keys.
- The Signing: When a pod starts, EKS creates a token and signs it using its private key. Only your cluster has this private key.
2. Why you create the resource in IAM
IAM is globally shared across your AWS account. It doesn't know your EKS cluster exists until you tell it. By creating the Identity Provider in IAM, you are essentially saying:
"Hey IAM, if someone shows up with a token signed by the keys found at this EKS URL, I want you to consider that a valid identity."
3. The Step-by-Step Identity Exchange
This is exactly how your application pod is supposed to get its e.g. S3 permissions:
- The Token: EKS mounts a projected volume into the App pod containing a signed token (the "passport").
- The Request: App sends this token to AWS STS (Security Token Service) and says, "I want to assume the role mycorp-role"
- The Validation: STS looks at the iss (issuer) field in the token. It sees your EKS URL.
- The Trust Check: STS looks at IAM Identity Providers. It finds the one you created that matches that URL.
- The Signature Check: STS fetches the public key from the EKS URL and verifies that the token was actually signed by your cluster.
- The Condition Check: STS checks the Trust Relationship on the role to ensure the "Subject" matches system:serviceaccount:my-app:my-app.
In 2014, AWS Identity and Access Management added support for federated identities using OpenID Connect (OIDC). This feature allows you to authenticate AWS API calls with supported identity providers and receive a valid OIDC JSON web token (JWT). You can pass this token to the AWS STS AssumeRoleWithWebIdentity API operation and receive IAM temporary role credentials. You can use these credentials to interact with any AWS service, including Amazon S3 and DynamoDB.Each JWT token is signed by a signing key pair. The keys are served on the OIDC provider managed by Amazon EKS and the private key rotates every 7 days. Amazon EKS keeps the public keys until they expire. If you connect external OIDC clients, be aware that you need to refresh the signing keys before the public key expires.Kubernetes has long used service accounts as its own internal identity system. Pods can authenticate with the Kubernetes API server using an auto-mounted token (which was a non-OIDC JWT) that only the Kubernetes API server could validate. These legacy service account tokens don't expire, and rotating the signing key is a difficult process. In Kubernetes version 1.12, support was added for a new ProjectedServiceAccountToken feature. This feature is an OIDC JSON web token that also contains the service account identity and supports a configurable audience.Amazon EKS hosts a public OIDC discovery endpoint for each cluster that contains the signing keys for the ProjectedServiceAccountToken JSON web tokens so external systems, such as IAM, can validate and accept the OIDC tokens that are issued by Kubernetes.
OIDC federation access allows you to assume IAM roles via the Secure Token Service (STS), enabling authentication with an OIDC provider, receiving a JSON Web Token (JWT), which in turn can be used to assume an IAM role. Kubernetes, on the other hand, can issue so-called projected service account tokens, which happen to be valid OIDC JWTs for pods. Our setup equips each pod with a cryptographically-signed token that can be verified by STS against the OIDC provider of your choice to establish the pod’s identity.new credential provider ”sts:AssumeRoleWithWebIdentity”
From Implementing and Understanding IAM Roles for Service Accounts in AWS EKS | by Anil Goyal | Medium:
 |
| IRSA authentication EKS OIDC IdP-signed JWT gets auto-mounted on each pod which uses service account. AWS SDK sends AssumeRoleWithWebIdentity request containing the desired role and JWT. STS uses IAM IdP associated to EKS OIDC IdP in order to verify identity of the pod. |
To use/enable IRSA:
1) Every EKS cluster which supports IRSA has a unique URL - The OIDC Issuer URL.
Example: https://oidc.eks.us-east-2.amazonaws.com/id/EXAMPLEDATA123
- If our cluster supports IAM roles for service accounts, it has an OpenID Connect (OIDC) issuer URL associated with it.
- We can view this URL in the Amazon EKS console, or we can use the following AWS CLI command to retrieve it.
$ aws eks describe-cluster --name cluster_name --query "cluster.identity.oidc.issuer" --output text
The expected output is as follows:
https://oidc.eks.<region-code>.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E
2) A unique OpenID Connect provider (OIDC Provider, Identity Provider) needs to be created in IAM for each EKS cluster. [Create an IAM OIDC provider for your cluster - Amazon EKS]
To use IAM roles for service accounts, an IAM OIDC provider must exist for your cluster's OIDC issuer URL. The Issuer URL generated by EKS cluster must be identical to the Provider URL defined in the IAM Identity Provider resource.
AWS uses this match to verify that the security token presented by pod actually came from a trusted source.
1. The "Exact String" MatchAWS IAM is extremely strict about the URL. If your EKS cluster issuer is:https://oidc.eks.us-east-2.amazonaws.com/id/EXAMPLEDATA123The Identity Provider in IAM must be configured with that exact string. If there is a trailing slash in one but not the other, or if it uses http instead of https, the match fails, and you get the WebIdentityErr you saw in your logs.2. The "Thumbprint" (The Security Guard)Matching the URL is only half the battle. AWS also checks a Thumbprint (a SHA1 hex string).This is the fingerprint of the SSL certificate used by that URL.If the certificate at the URL changes (which happens periodically when AWS updates its services), the thumbprint in your IAM Identity Provider must be updated to match.If our my-app pods are failing with a WebIdentityErr, it's highly likely our cluster was updated/recreated, and the IAM Identity Provider is still holding an old URL or an old Thumbprint.
How to Verify the Match
You can check for a match yourself using the AWS CLI:
Step A: Get the Cluster's URL
aws eks describe-cluster --name <cluster_name> --query "cluster.identity.oidc.issuer" --output text
Step B: List IAM Providers
aws iam list-open-id-connect-providers
Step C: Compare
Pick the ARN from Step B that looks relevant and describe it:
aws iam get-open-id-connect-provider --open-id-connect-provider-arn <ARN_FROM_STEP_B>
Look for the Url field. If it does not match Step A character-for-character, your identity chain is broken.
Any OIDC provider implementation needs to have a public OIDC issuer URL (see Issuer URL in OpenID Connect Discovery should be a working URL? - Stack Overflow). So for each cluster we'll have one implementation of OIDC provider (in IAM).
Your Identity Provider’s Discovery Endpoint contains important configuration information. The OIDC discovery endpoint will always end with /.well-known/openid-configuration as described in theOpenID Provider Configuration Request documentation.You can confirm that the discovery endpoint is correct by entering it in a browser window. If there is a JSON object with metadata about the connection returned, the endpoint is correct.
 |
2) Configure a Kubernetes service account to assume an IAM role
3) Configure Pods to use a Kubernetes service account
4) Use a supported AWS SDK
3) Configure Pods to use a Kubernetes service account
4) Use a supported AWS SDK
Just like we can create OIDC Identity Provider in IAM for representing an external, 3rd party OIDC Provider so we can allow access to AWS for a user authenticated with that 3rd party OIDC Provider, we can also create OIDC Identity Provider in IAM for representing an internal, EKS OIDC Provider which is available for each cluster (each cluster has its own provider). When EKS cluster is created, its OIDC Provider is also created with two pieces of data available:
- OIDC Provider issuer
- has its url which is used for discovery - see the screenshot above
- OIDC Provider server TLS certificate
- This certificate protects the url above (OIDC Provider issuer url) and is used for clients to verify the identity of OIDC Provider server
- TLS certificate is necessary for establishing secure communication with the OIDC provider.
In Terraform, this certificate can be obtained like here:
data "tls_certificate" "example" {
url = aws_eks_cluster.example.identity[0].oidc[0].issuer
}
To create OIDC Identity Provider (IdP) in IAM for this cluster-specific OIDC Provider we can use aws_iam_openid_connect_provider :
resource "aws_iam_openid_connect_provider" "example" {
client_id_list = ["sts.amazonaws.com"]
thumbprint_list = [data.tls_certificate.example.certificates[0].sha1_fingerprint]
url = aws_eks_cluster.example.identity[0].oidc[0].issuer
}
This resource requires few pieces of information:
- url - Describes which OIDC IdP this resource represents.
- The URL of the identity provider. Corresponds to the iss claim.
- thumbprint_list - Describes how will clients communicate with OIDC IdP (servers). HTTP communication goes through TLS secure channel so we need to know the identity of certificates to use.
- A list of server certificate thumbprints for the OpenID Connect (OIDC) identity provider's server certificate(s).
- When we create an OpenID Connect (OIDC) identity provider in IAM, IAM requires the thumbprint for the top intermediate certificate authority (CA) that signed the certificate used by the external identity provider (IdP). The thumbprint is a signature for the CA's certificate that was used to issue the certificate for the OIDC-compatible IdP. When we create an IAM OIDC identity provider, we are trusting identities authenticated by that IdP to have access to our AWS account. By using the CA's certificate thumbprint, we trust any certificate issued by that CA with the same DNS name as the one registered. This eliminates the need to update trusts in each account when we renew the IdP's signing certificate.
- client_id_list - Describes which clients can use this OIDC IdP
- A list of client IDs (also known as audiences). When a mobile or web app registers with an OpenID Connect provider, they establish a value that identifies the application. (This is the value that's sent as the client_id parameter on OAuth requests.)
AWS Security Token Service (STS)
- Web service that enables you to request temporary, limited-privilege credentials for users
- Available as a global service
- All AWS STS requests go to a single endpoint at https://sts.amazonaws.com
- Supports the following actions (requests):
- AssumeRole
- Returns a set of temporary security credentials that you can use to access AWS resources. These temporary credentials consist of an access key ID, a secret access key, and a security token. For example, user can authenticate via company's SSO and on AWS sign-on page can get these credentials that can be copied to ~/.aws/credentials under a profile and then this profile is used when accessing AWS.
- AssumeRoleWithSAML
- AssumeRoleWithWebIdentity
- Issues a role session (temporary session)
- Returns a set of temporary security credentials for users who have been authenticated in a mobile or web application with a web identity provider. Example providers include the OAuth 2.0 providers Login with Amazon and Facebook, or any OpenID Connect-compatible identity provider such as Google or Amazon Cognito federated identities.
- Calling AssumeRoleWithWebIdentity does not require the use of AWS security credentials. Therefore, you can distribute an application (for example, on mobile devices) that requests temporary security credentials without including long-term AWS credentials in the application. You also don't need to deploy server-based proxy services that use long-term AWS credentials. Instead, the identity of the caller is validated by using a token from the web identity provider.
- The temporary security credentials returned by this API consist of an access key ID, a secret access key, and a security token. Applications can use these temporary security credentials to sign calls to AWS service API operations.
- For example, user can authenticate via company's SSO and on AWS sign-on page can get these credentials that can be copied to ~/.aws/credentials under a profile and then this profile is used when accessing AWS.
- By default, the temporary security credentials created by AssumeRoleWithWebIdentity last for one hour. However, you can use the optional DurationSeconds parameter to specify the duration of your session. You can provide a value from 900 seconds (15 minutes) up to the maximum session duration setting for the role. This setting can have a value from 1 hour to 12 hours.
- Required parameters:
- RoleArn - The Amazon Resource Name (ARN) of the role that the caller is assuming.
- RoleSessionName - An identifier for the assumed role session. Typically, you pass the name or identifier that is associated with the user who is using your application. That way, the temporary security credentials that your application will use are associated with that user. This session name is included as part of the ARN and assumed role ID in the AssumedRoleUser response element.
- WebIdentityToken - The OAuth 2.0 access token or OpenID Connect ID token that is provided by the identity provider. Your application must get this token by authenticating the user who is using your application with a web identity provider before the application makes an AssumeRoleWithWebIdentity call. Timestamps in the token must be formatted as either an integer or a long integer. Only tokens with RSA algorithms (RS256) are supported.
- DecodeAuthorizationMessage
- GetAccessKeyInfo
- GetCallerIdentity
- GetFederationToken
- GetSessionToken
Example: Create an IAM role and associate it with a Kubernetes service account
Our custom service account that we have in cluster, my-service-account requires permission to e.g. launch EC2 instances. We need to assign certain IAM role to this service account (IRSA).
We've created OIDC IdP in IAM for OIDC IdP associated with our cluster:
arn:aws:iam::111122223333:oidc-provider/oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE
Prepare the policies for the role that the IdP-authenticated user will assume. As with any role, a role for a service account includes two policies:
- trust policy that specifies who can assume the role
- permissions policy that specifies the AWS actions and resources that the role owner is allowed or denied access to
Trust Policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::111122223333:oidc-provider/oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:sub": "system:serviceaccount:default:my-service-account",
"oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:aud": "sts.amazonaws.com"
}
}
}
]
}
Principal here is OIDC session principal which is a role session principal, see https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_principal.html#sts-session-principals.
This allows anyone who's authenticated with this OIDC IdP and who has my-service-account as the subject (user) and sts.amazonaws.com as the audience (client) in the WebIdentityToken (sent as the parameter of this request) to be able to assume the role that has this policy attached to it.
Our service account authenticates with cluster's OIDC provider (IdP) and from it gets the token. This is Identity Token mentioned in OpenID Connect (OIDC) | My Public Notepad and it contains sub (subject identity, service account in our case) and aud (audience - client - who'll be using this token; STS in our case) claims.
It then sends AssumeRoleWithWebIdentity request with this token and role it requires (e.g. role for creating EC2 instances) to STS. STS (Client) then uses this token against EKS cluster IdP to identify user (service account) and finally to grant it a role.
