Showing posts with label AWS. Show all posts
Showing posts with label AWS. Show all posts

Friday, 26 June 2026

DevOps Interview Questions - k8s ALB alarm




Post-incident review / technical interview questions.

These questions and answers were generated by Claude, upon it analyzed and fixed one CloudWatch alarm which have been flipping. Prompt I used:

I was following your analysis and resolution of this issue in order to acquire your knowledge. I would like to test my knowledge now. Can you compile a list of questions which cover every aspect of the issue and solution here? Don't be shy of creating a really long list of questions. If I am able to answer them all that means I am able to fix the issue on my own next time. Also prepare the answer key.



A. The alarm itself (CloudWatch / ALB fundamentals)

  1. What does the metric TargetResponseTime actually measure, and from whose perspective (client → ALB → target)?

  2. The alarm name was k8s-api-prod-core-7db0ccf2c3-target-response-time. What do the k8s- prefix and the hash portion tell you about how it was created?

  3. The alarm config was: threshold 0.8, GreaterThanThreshold, period 60s, 2 evaluation periods, statistic Average. In plain English, what condition makes it fire?

  4. Why does the alarm use the LoadBalancer dimension only, and not a TargetGroup dimension? What consequence did that have for our investigation?

  5. What is "flapping," and why does this particular threshold/period combination make flapping likely for a bursty workload?

  6. How do you pull an alarm's state-transition history, and what did 9 OK→ALARM cycles in 5 hours tell you?

  7. The alarm fires on Average. Why is that distinction (vs Maximum/p99) absolutely central to both the diagnosis and the fix?

  8. What is the "low-traffic statistical artifact" pattern, where a handful of slow requests inflate the average on a near-idle target — and what evidence did we use to rule it out here?

B. Narrowing from ALB to one service

  1. The ALB fronted six target groups. Name the technique we used to find which one was responsible, and the AWS CLI call behind it.

  2. Why can a single ALB serve six different Kubernetes services? What AWS-LB-Controller concept ties them together (hint: group.name)?

  3. Given the target group name k8s-default-dataservice-88e28b4b6c, how do you map it back to a Kubernetes Service and namespace?

  4. During the burst, data-service showed 5.83s avg / 29.7s max while every other service was <0.5s. Why did that immediately exonerate the shared ALB/ingress as the cause?

C. First look at the workload

  1. What does kubectl top pods show, and why is a single snapshot from it dangerous as evidence?

  2. The first snapshot showed one pod at 997m and three near-idle. What two different explanations are consistent with that, and why can't a snapshot distinguish them?

  3. What's the difference between a CPU request and a CPU limit in Kubernetes?

  4. How did we check whether the pod was being CPU-throttled at its limit, and what file did we read inside the container? What did nr_throttled / throttled_usec tell us?

  5. A deploy to v1.14.3-prod had happened ~15 min earlier. Why was it a red herring, and what evidence dated the flapping as pre-existing?

  6. Distinguish the three probe types (liveness, readiness, startup). What does each one control?

  7. The deployment had a startupProbe and livenessProbe but no readinessProbe. Operationally, what can't the system do without a readiness probe?

D. The load-balancing theory (and why it was wrong)

  1. Explain why round-robin load balancing degrades when request durations are highly variable. Use the "request count vs. total work" framing.

  2. What is the feedback-loop difference between round-robin and least-outstanding-requests (LOR)?

  3. What older, well-known algorithm is LOR equivalent to in nginx/HAProxy terms, and what does AWS's own guidance say about when to use LOR vs round-robin?

  4. "Head-of-line blocking" appeared twice in this incident at two different layers. Name both layers and how each one blocks.

  5. Why is a single kubectl top snapshot insufficient to prove round-robin is causing imbalance, and what data would actually prove or refute it?

  6. We initially called round-robin the "smoking gun," then withdrew it. What specifically made that conclusion wrong?

E. Target mode: instance vs IP (the topology that broke the theory)

  1. What's the difference between an ALB target group in instance mode vs ip mode? How can you tell which one you have from the registered targets and the target-group port?

  2. The target group registered 10 EC2 instances on port 31892. What does that tell you about the request path from ALB to pod?

  3. What is a NodePort service, and what is externalTrafficPolicy: Cluster vs Local?

  4. With instance mode + Cluster policy, describe the full path of a request from client to the backend process, including every hop and who load-balances at each hop.

  5. Given that topology, explain precisely why switching the ALB to least_outstanding_requests would not fix per-pod imbalance.

  6. What two changes together would enable load-aware per-pod routing, and why is that a much bigger change than a one-line annotation?

  7. Why were ALB access logs not useful for confirming per-pod distribution in this setup?

F. Getting the proof (Prometheus / time series)

  1. What is kube-prometheus-stack, and where did it live in the cluster?

  2. Why is kubectl port-forward an acceptable read-only way to query Prometheus, and what does it actually do?

  3. Write (conceptually) the PromQL that gives per-pod CPU usage over time. Why rate(container_cpu_usage_seconds_total[...]) rather than the raw counter?

  4. The time series showed all four pods evenly at 0.5–0.95 cores during the burst. Why did that refute the imbalance theory in one stroke?

  5. Each pod plateaued at ~0.9 cores despite a 1.5-core limit. What does that plateau strongly imply about the process model inside the pod?

G. The real root cause (app server / GIL / async)

  1. What is an application server worker (e.g., Gunicorn), and what's the difference between the master process and a worker process?

  2. What is a Global Interpreter Lock (GIL) or similar single-threaded runtime constraint, and why does it mean one worker process ≈ one core of CPU-bound throughput?

  3. The config had workers = 1 and an asynchronous event-loop worker class specified. Explain what each line does.

  4. What is the difference between a synchronous worker and an asynchronous worker in an application server? When does each shine?

  5. Why is an async (event-loop) worker the wrong model for heavy, synchronous CPU-bound data processing? What does "blocking the event loop" mean concretely?

  6. So with one async worker doing CPU-bound work, what is the per-pod concurrency for heavy requests — and how did that produce the fleet-wide ceiling of ~4 concurrent requests?

  7. How did we confirm the worker count and the CPU quota from inside a running pod (what command, what does cpu.max = 150000 100000 mean, what does worker count = 2 mean)?

  8. Why was the 1.5-core CPU limit effectively unusable given workers = 1?

  9. Tie it together: explain the full causal chain from "traffic burst" to "alarm fires," in one paragraph, using the confirmed root cause.

H. Designing the fix

  1. We considered vertical (more workers/CPU per pod), horizontal (more pods via HPA), and both. What's the trade-off, and why did "both" win?

  2. Why couldn't we just change the worker class to synchronous (or offload to a process pool) as part of this immediate infrastructure fix? What kind of change would that be?

  3. We set workers = 2. Why did the CPU limit have to go up to 2000m at the same time? What would happen if we'd set workers = 2 but left the limit at 1500m?

  4. The pods used ~2.5Gi RSS each at one worker. Why did we expect ~5Gi with two workers, and why wouldn't worker-fork preloading save us here? (What did we learn about when the application memory caches are built?)

  5. There's one import-time load we found (e.g., heavy model loading in a utility file). Why is that one shareable-via-fork but the bulk of the runtime memory is not?

  6. The HPA was autoscaling/v1, target 80%, min 4 / max 7. We measured bursts peaking at ~75% of the 1200m request. Explain mechanically why the HPA never scaled.

  7. targetCPUUtilizationPercentage is a percentage of what? Recompute: at the new 1500m request and a pod using ~0.9 cores, what utilization does the HPA see?

  8. Why did we lower the target to 50% and raise max to 10, rather than just one of those?

  9. What does a topologySpreadConstraints with maxSkew: 1, topologyKey: kubernetes.io/hostname, whenUnsatisfiable: ScheduleAnyway do — and why soft (ScheduleAnyway) rather than hard (DoNotSchedule)?

  10. We deliberately did not add a readiness probe. Explain the failure mode that a naive /health readiness probe would cause in this specific app under heavy load. Why is "no readiness probe" temporarily safer than a bad one here?

  11. What was the container port vs Service target port mismatch situation? Why was alignment low-risk, and why didn't it affect routing?

  12. An orphaned HPA manifest file was deleted. Why was it safe to delete, and how did we confirm it was no longer active?

I. Where the config lives & deploy mechanics

  1. How did we determine the workload was not managed by GitOps tool deployments (e.g., Argo CD), despite the tool being installed? What metadata annotation was the fingerprint?

  2. Which repository and file holds the deployment/HPA, and which separate repository/file holds the ingress? Why do they deploy through different mechanisms?

  3. Describe the deployment pipeline flow end to end. What event triggers it, and what are the key build/test/deploy jobs?

  4. The deploy step does a string substitution (sed 's#$TAG#...#') then kubectl apply. What's the role of the $TAG placeholder, and where does the tag value come from?

  5. Why does merging a PR to the main branch not deploy anything, while pushing a specific environment release tag does?

  6. The pipeline runs on private self-hosted runners. Why does that matter for a private-endpoint cluster?

J. Capacity analysis

  1. What instance types/sizes back the general-purpose compute tier, and how much allocatable CPU/memory does each have?

  2. What is Cluster Autoscaler, and how does it differ from node lifecycle managers like Karpenter? Which one is active in this cluster?

  3. There are two node groups feeding the tier (spot instances and on-demand instances). What are their min/max sizes, and what's the combined node ceiling?

  4. Do the packing math: given ~3920m allocatable CPU and ~400m daemonset overhead, how many pods at 1500m request fit per node? Why is CPU, not memory, the binding constraint?

  5. At HPA max (10 pods), how many nodes are needed, and is that within the ceiling? What's left for other tenants?

  6. Why is horizontal scale-out slow relative to instant traffic bursts? List every contributor to a cold pod's total time-to-serve.

  7. Given that scale-out lag, which part of our fix delivers immediate relief, and which acts as the slower "second line of defense"?

  8. Why did we bump the memory request to 5Gi even though it doesn't change node packing density?

K. Staging-first deploy & verification

  1. Why deploy to the staging environment first when the staging manifest file wasn't even changed by the PR?

  2. Precisely what does the staging deploy validate, and what does it not validate?

  3. Staging runs on the same cluster as production. How is it isolated, and what was the risk we flagged about a tiny 0.5-core staging pod suddenly running workers=2?

  4. List the verification steps we ran on staging, and the pass criteria for each.

  5. Why did we test both GET /health and a heavy POST data processing route, rather than just the health check?

  6. The startup probe warms up by hitting an internal pre-cache endpoint. What does that endpoint do, and why is it the most likely place for a deploy to fail (especially on a CPU-starved staging pod)?

L. Production rollout & confirmation

  1. What version did we tag, and why a patch bump (v1.14.4) rather than a minor/major version shift?

  2. During the production rollout, two new pods went Pending, one with no available node. What happened next, and which log event confirmed the cluster autoscaler reacting?

  3. Why did the rollout take ~6 minutes and stay safe (no dropped traffic) the whole time? What deployment configuration controls the surge/unavailable behavior?

  4. Post-deploy, the new pods showed only ~2.6Gi memory usage, not ~5Gi. Why — and why is that expected rather than a contradiction of our sizing?

  5. After deploy, individual requests still hit ~3s max, but the alarm stayed OK. Why is that consistent with a successful fix? (Connect it back to question 7.)

  6. We monitored for 45 minutes and saw no flapping, yet we kept the incident ticket open. What's the honest gap in that evidence, and what would definitively close it?

  7. The HPA stayed at 4 pods during the entire monitoring window. Why is that a good sign rather than a sign the HPA fix did nothing?

M. Operational / process & gotchas

  1. The cluster API is private-only. What's the practical consequence for local engineers using kubectl, and why does aws sts get-caller-identity succeed while kubectl times out? How do you distinguish an auth problem from a network/VPN problem?

  2. Why did the CI/CD deployment job still work even when our local machine's kubectl couldn't reach the cluster?

  3. The standard code security checks failed on automated image-scanning and vulnerability gates. Diagnose how to isolate the root cause. Were they caused by our configuration change? How do we prove that?

  4. The repository management system showed a BLOCKED merge status, but branch protection rules returned a 404. What's the resolution of that apparent contradiction (e.g., legacy branch protection vs. modern repository rulesets)?

  5. What was the only thing actually gating the code merge, and why were the red security check flags irrelevant to it?

  6. Which actions in this whole deployment flow required explicit manual operator confirmation before execution, and why those specifically?

  7. What's the commit-authorship convention in this engineering environment, and what must never appear in a commit message or PR description?

N. Synthesis & transfer (test of true mastery)

  1. If you were paged for this exact alarm tomorrow with zero prior context, list the first five commands/queries you'd run, in order, and what each one would tell you.

  2. Name three plausible-but-wrong hypotheses for a flapping TargetResponseTime alarm, and the single piece of evidence that kills each.

  3. Suppose the per-pod CPU time series had shown one pod pinned at 100% and three idle (the load imbalance pattern we originally expected). Given the instance-mode topology, what would the real fix have been then — and why is it different from the LOR annotation?

  4. The fix here was vertical + HPA scaling. Under what circumstances would the correct long-term fix instead be an application architecture change, and what would that change look like?

  5. Generalize the core lesson: what specific property of an application workload makes "adding more replicas behind a standard round-robin/random balancer" fail to resolve response time spikes, and what is the class of fixes that does help?

  6. If the morning peak traffic burst still trips the alarm after this infrastructure fix, what are your next two remediation levers (in order), and what data would you collect to choose between them?


Read more »
Posted by at No comments:
Labels: , ,

Thursday, 25 June 2026

AWS EC2: Application Load Balancer

 



An Application Load Balancer (ALB) is a fully managed AWS service that automatically distributes incoming HTTP and HTTPS traffic across multiple backend targets.

It operates at the Application Layer (Layer 7) of the Open Systems Interconnection (OSI) model.

Key Features:

  • Content-Based Routing: Routes traffic based on URL paths (/api vs /images) or hostnames (://example.com).
  • Container Support: Integrates directly with Amazon ECS and EKS using dynamic port mapping.
  • Advanced Protocols: Native support for modern protocols like HTTP/2, gRPC, and WebSockets.
  • Security Integration: Features built-in HTTPS/TLS termination and integrates directly with AWS WAF for web security.

How Components Work Together


  • Listener: Evaluates connection requests from clients using protocols and ports you configure.
  • Rules: Determines how the load balancer routes requests to its registered targets.
  • Target Group: Groups backend resources (like EC2 instances, containers, or IP addresses) that receive the traffic

How ALB health checks keep applications online?


An Application Load Balancer (ALB) keeps your application online by continuously monitoring the health of your backend targets and dynamically redirecting traffic away from failing nodes.

1. Automatic Traffic Redirection


The ALB sends periodic ping requests (health checks) to every registered target. If a target fails to respond correctly, the ALB marks it as unhealthy and immediately stops sending user traffic to it. Traffic is rerouted to the remaining healthy nodes with zero downtime for the user.

2. Auto Scaling Integration


When paired with an Auto Scaling Group (ASG), ALB health checks can trigger the automatic replacement of broken instances.
  • The Problem: An EC2 instance might be running (healthy at the hardware level), but the web server inside it has crashed (unhealthy at the application level).
  • The Solution: The ALB tells the ASG that the instance is failing application health checks. The ASG terminates that specific broken instance and launches a fresh, working one.

3. Graceful Recovery


When an unhealthy instance recovers, or when a new instance is launched, the ALB does not send traffic to it immediately. It enters an initial state and undergoes consecutive successful health checks. Only when it passes the threshold does the ALB safely introduce it back into the traffic rotation.

How to Configure an ALB Health Check


You configure health checks inside the Target Group settings using these parameters:

Parameter                  What it does                                                                          Recommended Setting
========                   =========                                                                           =================
Health Check Path     The URL endpoint the ALB hits 
                                      (e.g., /health or /index.html).                                                    /health
Healthy Threshold      Consecutive successes needed to mark a target as healthy.      3
Unhealthy Threshold  Consecutive failures needed to mark a target as unhealthy.     2
Timeout                        How long the ALB waits for a response before failing.           5 seconds
Interval                         The time between individual health check pings.                    30 seconds
Success Codes              The HTTP status codes that prove the app is working.            200 (or 200-399)



When you configure an ALB, you do not select an Availability Zone (AZ) directly; instead, you must select at least two subnets in different Availability Zones to ensure high availability.

  • How it works: AWS places a load balancer node in each of the specified subnets.
  • The AZ link: Because each subnet belongs to exactly one AZ, this fundamentally binds the ALB's nodes to those corresponding Availability Zones.
  • Custom routing: You can modify the subnets via the Update Availability Zones settings in the EC2 Console at any time.

Public ALB


Binding an ALB to public subnets makes it a public (internet-facing) load balancer. 

When you create an internet-facing ALB, AWS requires you to select public subnets so the ALB nodes can receive a public IP address and route traffic from the internet.

Key Characteristics:

  • Public DNS: The ALB receives a public DNS name that resolves to public IP addresses.
  • Internet Gateway: The selected public subnets must have a route to an Internet Gateway (IGW) in their route tables.
  • Target Routing: Even though the ALB is public, it can still route traffic to EC2 instances living in private subnets
An internet-facing ALB routes traffic directly to the individual backend targets (such as EC2 instances or IP addresses), not to the private subnets themselves.


How Routing Works

  • Target Group Config: You configure the ALB to route traffic to a Target Group.
  • Direct Node Communication: The ALB nodes in the public subnets communicate directly with the private IP addresses of your backend nodes.
  • Cross-Subnet Traffic: AWS handles this routing internally via the VPC router, allowing the public ALB to securely traverse into private subnets.

Configuration Checklist

  • VPC: Both the public subnets (where the ALB lives) and the private subnets (where the nodes live) must be in the same VPC.
  • Security Groups: The private instances must have a security group that allows inbound traffic from the ALB's security group



Private ALB



An internal (private) ALB routes traffic in the exact same way as a public ALB, but it is only accessible within your VPC or connected networks.

It routes traffic directly to individual backend targets, not to subnets.

Key Characteristics

  • Private Subnets: You deploy the ALB nodes into private subnets.
  • Private DNS: The ALB receives a public DNS name, but it resolves exclusively to private IP addresses.
  • No Internet Access: It cannot receive any traffic from the public internet because it lacks a public IP.

Common Use Cases

  • Internal Microservices: Routing traffic from a public-facing web tier to a private backend API tier.
  • Hybrid Networks: Routing traffic coming from an on-premises data centre via AWS Direct Connect or a VPN

Setting Up ALB in AWS Console


AWS Elastic Load Balancing shows basic building blocks of AWS Load Balancer which include listeners and target groups. 

To create Application Load Balancer go to EC2 >> Load balancers >> Create Load balancer >> Select load balancer type (click on Create under Application Load Balancer)



Here we can set:

  • Basic configuration
    • Name
    • Scheme (cannot be changed after the load balancer is created)
      • Internet-facing. An internet-facing load balancer routes requests from clients over the internet to targets. Requires a public subnet. 
      • Internal. An internal load balancer routes requests from clients to targets using private IP addresses.
    • IP address type. Select the type of IP addresses that your subnets use.
      • IPv4. Recommended for internal load balancers.
      • Dualstack. Includes IPv4 and IPv6 addresses.
  • Network mapping. The load balancer routes traffic to targets in the selected subnets, and in accordance with your IP address settings.
    • VPC. Virtual private cloud for your targets. If balancer is internet-facing, only VPCs with an internet gateway are enabled for selection. The selected VPC cannot be changed after the load balancer is created. As VPC is region-specific so is Application Load Balancer.
    • Mappings. Once VPC is selected, its availability zones are listed here and are selectable. Select at least two Availability Zones and one subnet per zone. The load balancer routes traffic to targets in these Availability Zones only. Availability Zones that are not supported by the load balancer or the VPC are not available for selection. We should select all AZs that we listed in the Auto scaling group (if we used it).
  • Security groups. A security group is a set of firewall rules that control the traffic to your load balancer. We can select up to 10 security groups.
    • If our application is listening for HTTP requests on port 80 we should select a security group with:
      • Inbound rule: accept HTTP/TCP traffic on port 80 with source Anywhere-IPv4
      • Outbound rule: allow all traffic for all protocols and port ranges to custom destination 0.0.0.0/0
  • Listeners and routing. A listener is a process that checks for connection requests using the port and protocol you configure. The rules that you define for a listener determine how the load balancer routes requests to its registered targets.
    • Add listener
      • Protocol e.g. HTTP
      • Port e.g. 80. This is a public facing port and it does not need to be the same as the port from the attached target group. E.g. LB can listen on port 80 and forward traffic to target group port 8080.
      • Default action: Forward to (select a target group)
      • Add listener tags
  • Add-on services - optional
    • AWS Global Accelerator
  • Tags - optional


More info on Scheme, from AWS documentation:

When you create a load balancer, you must choose whether to make it an internal load balancer or an internet-facing load balancer.

The nodes of an internet-facing load balancer have public IP addresses.

The nodes of an internal load balancer have only private IP addresses.

Both internet-facing and internal load balancers route requests to your targets using private IP addresses. Therefore, your targets do not need public IP addresses to receive requests from an internal or an internet-facing load balancer.

More info on how ALB routes traffic to multiple Availability Zones (and about what Load Balancer Nodes are):

When you enable an Availability Zone for your load balancer, Elastic Load Balancing creates a load balancer node in the Availability Zone. 

The nodes for your load balancer distribute requests from clients to registered targets. When cross-zone load balancing is enabled, each load balancer node distributes traffic across the registered targets in all enabled Availability Zones. When cross-zone load balancing is disabled, each load balancer node distributes traffic only across the registered targets in its Availability Zone.

Before a client sends a request to your load balancer, it resolves the load balancer's domain name using a Domain Name System (DNS) server. The DNS entry is controlled by Amazon, because your load balancers are in the amazonaws.com domain. The Amazon DNS servers return one or more IP addresses to the client. These are the IP addresses of the load balancer nodes for your load balancer.

As traffic to your application changes over time, Elastic Load Balancing scales your load balancer and updates the DNS entry. The DNS entry also specifies the time-to-live (TTL) of 60 seconds. This helps ensure that the IP addresses can be remapped quickly in response to changing traffic.

The client determines which IP address to use to send requests to the load balancer. The load balancer node that receives the request selects a healthy registered target and sends the request to the target using its private IP address.

With Application Load Balancers, the load balancer node that receives the request uses the following process:

1) Evaluates the listener rules in priority order to determine which rule to apply.

2) Selects a target from the target group for the rule action, using the routing algorithm configured for the target group. The default routing algorithm is round robin. Routing is performed independently for each target group, even when a target is registered with multiple target groups.

For further info: How Elastic Load Balancing works - Elastic Load Balancing

ALB nodes use Elastic Network Interface (Elastic network interfaces - Amazon Elastic Compute Cloud) which has public IP address:

At least one ENI is created and attached to the balancer in each availability zone where the balancer is deployed (except NLB, which should only have one per AZ). Over the life of the balancer, new ENIs will appear and old ones will disappear, as the balancer scales horizontally (number of nodes) and/or vertically (capacity of underlying hardware), all of which is handled transparently by the infrastructure. Even though you can tag them, the tagging will become stale over time.

Source: amazon web services - AWS - Affect Load Balancer's tags to its Network Interfaces (ENI) - Stack Overflow

 

You can determine the IP addresses associated with an internal load balancer or an internet-facing load balancer by resolving the DNS name of the load balancer. These are the IP addresses where the clients should send the requests that are destined for the load balancer. However, Classic Load Balancers and Application Load Balancers use the private IP addresses associated with their elastic network interfaces as the source IP address for requests forwarded to your web servers.

Source: Find the IP address used by a load balancer to forward traffic to web servers

 

Load balancer routes requests to the targets in a target group and performs health checks on the targets. Target group is accepting requests from the load balancer and forwards them to targets. These targets can be e.g. EC2 instances created either manually or through auto scaling group.

How to create a Target Group used by Load Balancer listeners? (This applies for any type of Load Balancer)

EC2 >> Target groups >> Create target group

Step 1: Specify group details

 

Here we can set:

  • Basic configuration. Settings in this section cannot be changed after the target group is created.
    • Target type
      • Instances
        • Supports load balancing to instances within a specific VPC.
        • Facilitates the use of Amazon EC2 Auto Scaling  to manage and scale your EC2 capacity.
      • IP addresses
        • Supports load balancing to VPC and on-premises resources.
        • Facilitates routing to multiple IP addresses and network interfaces on the same instance.
        • Offers flexibility with microservice based architectures, simplifying inter-application communication.
        •  Supports IPv6 targets, enabling end-to-end IPv6 communication, and IPv4-to-IPv6 NAT.
      • Lambda function
        • Facilitates routing to a single Lambda function.
        •  Accessible to Application Load Balancers only.
      • Application Load Balancer
        • Offers the flexibility for a Network Load Balancer to accept and route TCP requests within a specific VPC
        • Facilitates using static IP addresses and PrivateLink with an Application Load Balancer.
    • Target group name
    • Protocol:Port e.g. If our application is accepting HTTP requests on port 8080 this would be HTTP:8080
    • VPC - VPC with the instances that you want to include in the target group.
    • Protocol version
      • HTTP1. Send requests to targets using HTTP/1.1. Supported when the request protocol is HTTP/1.1 or HTTP/2.
      • HTTP2. Send requests to targets using HTTP/2. Supported when the request protocol is HTTP/2 or gRPC, but gRPC-specific features are not available.
      • gRPC. Send requests to targets using gRPC. Supported when the request protocol is gRPC.
  • Health checks. The associated load balancer periodically sends requests, per the settings below, to the registered targets to test their status.
    • Health check protocol
      • HTTP
      • HTTPS
    • Health check path. Use the default path of “/“ to ping the root, or specify a custom path if preferred.
    • Advanced health check settings
      • Port. The port the load balancer uses when performing health checks on targets. The default is the port on which each target receives traffic from the load balancer, but you can specify a different port.
        • Traffic port
        • Override
      • Healthy threshold. The number of consecutive health checks successes required before considering an unhealthy target healthy.
      • Unhealthy threshold. The number of consecutive health check failures required before considering a target unhealthy.
      • Timeout. The amount of time, in seconds, during which no response means a failed health check.
      • Interval. The approximate amount of time between health checks of an individual target
      • Success codes. The HTTP codes to use when checking for a successful response from a target. You can specify multiple values (for example, "200,202") or a range of values (for example, "200-299").
  • Attributes
  • Tags - optional


Step 2: Register targets

This is an optional step to create a target group. However, to ensure that your load balancer routes traffic to this target group you must register your targets.




After load balancer is created it takes several minutes while it's in provisioning state and get into active state. After this, we can use its DNS name in order to see what it's doing.

If we copy its DNS name and paste it to our browser, if we haven't registered any targets in the target group associated with the load balancer, we'll get error 503 - Service Temporary Unavailable.

If we've registered targets and are getting error 504 Gateway time-out, we should check first if security groups (firewalls) for our EC2 instances (inbound rule - source IP range) are set up correctly as this error usually indicates that inbound traffic is not allowed.

AWS Terraform provider offers provisioning all these resources:

 
How is AWS Application Load Balancing usually implemented?
 
Let's say we have our application running on 3 EC2 instances where 2 are in the same region e.g. us-west-2 but in separate availability zones e.g. us-west-2a and us-west-2b. Third EC2 instance is in eu-central-1, in availability zone eu-central-1a.
 
VPC is region-specific but can span multiple availability zones (AZ). 
Subnet is an IP address range within VPC.
VPC can have public and private subnets.
VPC can be divided into multiple subnets but each subnet is AZ-specific.
AZ can have multiple subnets.

So, all EC2 instances belong to the same VPC but, as they are in different AZs, each of them belongs to different subnet.
 
Load balancer must be in the public subnet of VPC as clients communicate with load balancer via internet (public network).
 
Load balancer does not get associated directly with EC2 instances but subnets:

resource "aws_lb" "test" {
    subnets = ["subnet-0001", "subnet-0002"] 
    ...
}

Target group is associated with VPC:
 
resource "aws_alb_target_group" "test" {
    vpc_id   = var.vpc_id
    ...
}
 
 

Difference between ALB and NLB (Network Load Balancer)


An Application Load Balancer (ALB) and a Network Load Balancer (NLB) serve different purposes based on the layer of the network they operate on and the type of traffic they handle.

The core difference is that an ALB understands application-level traffic (Layer 7) like HTTP/HTTPS headers, while an NLB handles low-level network traffic (Layer 4) like TCP/UDP packets at extreme speeds.

Direct Comparison Matrix


Feature          Application Load Balancer (ALB)                     Network Load Balancer (NLB)
======         =========================                      =======================
OSI Layer     Layer 7 (Application)                                            Layer 4 (Transport)
Protocols       HTTP, HTTPS, HTTP/2, gRPC, WebSockets      TCP, UDP, TLS
IP Addresses 
                       Dynamic IPs (Changes automatically; requires a DNS name)  
                                                                                                     Static IPs (Can assign an Elastic IP per AZ)
Routing Features  
                        Advanced (Path, Host, Query parameters, Headers)  
                                                                                                      Basic (Port and IP protocol routing only)
Performance  
                        Optimized for complex web apps (Millions of requests/sec)  
                                                                                                      Optimized for ultra-low latency (Billions of requests/sec)


Key Technical Differences


1. Smart Routing vs. Raw Speed


  • ALB (Smart): Can read the contents of your HTTP requests. It can route traffic bound for ://example.com to an API server cluster, and traffic for ://example.com to a storage cluster.
  • NLB (Fast): Does not look inside the data packet. It simply looks at the target port and forwards the packet instantly. This results in ultra-low latency (measured in milliseconds).

2. IP Addresses and DNS


  • ALB: Scale out dynamically by adding or removing nodes. This causes its underlying IP addresses to change frequently. You must always point your domain name to the ALB's DNS Name, never to a static IP.
  • NLB: Gives you a Static IP address per Availability Zone. You can also assign your own Elastic IP addresses. This is critical if your corporate clients need to whitelist specific, unchanging IPs in their firewalls.

3. Client IP Preservation


  • ALB: Terminates the connection and makes a new one to your backend instances. The backend see the ALB's private IP. To find the real user's IP, your code must read the X-Forwarded-For HTTP header.
  • NLB: Passes the original TCP packet straight through to your backend server. Your backend instances see the original source IP address of the client natively, without needing extra headers.


When to Choose Which?


Choose an ALB if you are building:
  • Standard web applications and microservices.
  • Containerized apps (ECS/EKS) requiring path-based or host-based routing.
  • Applications requiring tight integration with AWS Web Application Firewall (WAF).

Choose an NLB if you are building:
  • Non-HTTP applications (e.g., gaming servers, SFTP, MQTT, database clusters).
  • Architectures requiring fixed, static IP addresses or Elastic IPs.
  • High-frequency financial applications where sub-millisecond network latency is a hard requirement

Which alerts should typically be set for AWS ALB?


To keep your applications highly available, you should set up Amazon CloudWatch alarms for a mix of availability, performance, and target health metrics.

The most critical metrics to monitor for an AWS ALB are grouped by priority below:

1. High Priority (Critical Infrastructure Impact)

UnHealthyHostCount (Per Target Group)
What it means: The number of backend instances failing health checks Target Group Metrics.
Alert Threshold: > 0 (or > 1 for larger clusters).
Why it matters: Signals that your servers are crashing or cannot handle traffic.

HTTPCode_Target_5XX_Count
What it means: The number of 5xx server error codes generated by your backend application ALB Metrics.
Alert Threshold: Depends on baseline traffic, typically > 5 failures within a 1-minute to 5-minute window.
Why it matters: Indicates server crashes, database connection timeouts, or unhandled exceptions in your application code.

HTTPCode_ELB_5XX_Count
What it means: The number of 5xx errors generated directly by the ALB itself (not your servers) ALB Metrics.
Alert Threshold: > 0.
Why it matters: Usually means the ALB cannot find any healthy hosts, or it is experiencing a configuration mismatch (e.g., bad TLS handshake with the target).

2. Medium Priority (Performance & User Experience)

TargetResponseTime
What it means: The time elapsed (in seconds) from when the ALB sent the request to the target until the target started responding ALB Metrics.
Alert Threshold: Use the p95 or p99 statistic. Alert if it exceeds your application’s maximum acceptable latency (e.g., > 2.0 seconds).
Why it matters: Users are experiencing severe application slowdowns, likely due to high CPU/memory usage on your instances.

RejectedConnectionCount
What it means: The load balancer is rejecting connections because it has reached its maximum capacity ALB Metrics.
Alert Threshold: > 0.
Why it matters: Your application is getting sudden traffic spikes and the ALB cannot scale fast enough, or backend targets are failing to keep up.

3. Low Priority (Anomalies & Security)

HTTPCode_Target_4XX_Count
What it means: The number of 4xx client errors (like 404 Not Found or 401 Unauthorized) returned by backend targets ALB Metrics.
Alert Threshold: A significant spike above your standard baseline.
Why it matters: A sudden surge might indicate a broken frontend deployment, a bad API update, or a malicious entity scanning your network for vulnerabilities.

Summary Checklist for CloudWatch Alarms

Metric Name                               Statistic         Recommended       Suggested                      Action
                                                                            Period                     Threshold     
==========                              ======          ===========       ========                     =====
UnHealthyHostCount                    Maximum   1 Minute                         > 0                          Page/On-Call
HTTPCode_ELB_5XX_Count     Sum             1 Minute                         > 0                          Page/On-Call
HTTPCode_Target_5XX_Count  Sum            5 Minutes                > 10 (or > 1% of traffic) Ticket/Slack
TargetResponseTime                     p95              5 Minutes> [Your Limit]                                Ticket/Slack





Resources:


Posted by at No comments:
Labels: , ,

Wednesday, 29 April 2026

Introduction to Amazon Simple Notification Service (SNS)



Amazon Simple Notification Service (SNS) is a fully managed messaging service that enables you to decouple microservices, distributed systems, and serverless applications. Here's how SNS works:

Key Concepts


Topics:

A topic is a logical access point and communication channel. Publishers send messages to a topic, and subscribers receive these messages by subscribing to the topic.

Publishers:

Publishers are entities that send messages to an SNS topic. They could be applications, services, or even other AWS services like Lambda or CloudWatch.

Subscribers:

Subscribers are endpoints that receive messages from an SNS topic. These can include Amazon SQS queues, AWS Lambda functions, HTTP/S endpoints, email addresses, and SMS numbers.

Messages:

Messages are the payload sent by publishers to SNS topics. They can include a variety of data formats, typically JSON.



How It Works


Creating a Topic:

First, you create a topic using the AWS Management Console, AWS CLI, or AWS SDKs. This topic acts as a communication channel.

Subscribing to a Topic:

You then subscribe one or more endpoints to the topic. These endpoints can be other AWS services or external services capable of receiving notifications.

When subscribing, you specify the protocol (such as HTTP, SQS, Lambda, etc.) and the endpoint (like the URL or ARN of the SQS queue).

Publishing a Message:

Publishers send messages to the SNS topic using the Publish API. The message can include a subject, a message body, and optional attributes.

SNS stores multiple copies of the message for redundancy and high availability.

Message Delivery:

SNS distributes the message to all subscribed endpoints.

Each endpoint processes the message according to its protocol. For example, an HTTP endpoint receives a POST request with the message content, and an SQS queue receives the message as a new queue entry.



Use Cases


Fan-out Scenarios:

When a message published to an SNS topic needs to be sent to multiple endpoints, SNS acts as a fan-out service. For example, updating various microservices or notifying multiple systems about an event.

Push Notifications:

SNS can be used to send push notifications to mobile devices through services like Amazon Device Messaging (ADM), Apple Push Notification Service (APNS), and Google Cloud Messaging (GCM).

Application Alerts:

Sending notifications or alerts to administrators or users via SMS, email, or other endpoints.

Event-Driven Computing:

Integrating with AWS Lambda to trigger functions in response to incoming messages, enabling event-driven processing.



Benefits


Scalability:

SNS is designed to handle high-throughput and large volumes of messages, making it suitable for large-scale applications.


Flexibility:

It supports multiple protocols, allowing for versatile messaging architectures.


Reliability:

With built-in redundancy and fault tolerance, SNS ensures reliable message delivery.


Ease of Use:

The service is fully managed, reducing the operational overhead associated with maintaining messaging infrastructure.


Security



Access Control:

Using AWS IAM policies, you can control who can publish or subscribe to topics.


Encryption:

SNS supports server-side encryption (SSE) for messages at rest and SSL/TLS for messages in transit.

Compliance:

SNS is compliant with various industry standards and regulations, ensuring it meets security and compliance requirements.

Amazon SNS is a powerful and flexible tool for building loosely coupled, scalable, and robust messaging solutions in the cloud.



How does Amazon SNS differ from Amazon SQS?



Amazon SNS (Simple Notification Service) and Amazon SQS (Simple Queue Service) are both messaging services provided by AWS, but they serve different purposes and have distinct features. Here's a comparison highlighting their differences:

Purpose and Use Cases


Amazon SNS:
  • Publish/Subscribe Messaging: 
    • SNS is designed for sending notifications or messages to multiple subscribers.
    • It's ideal for broadcasting messages to multiple endpoints, such as sending alerts, notifications, or updates to different systems or users.
  • Use Cases: 
    • Real-time notifications
    • push notifications
    • fan-out scenarios (where a message needs to be sent to multiple recipients simultaneously)
    • application alerts
    • event-driven architectures

Amazon SQS:
  • Message Queuing: 
    • SQS is designed for decoupling and scaling distributed systems. 
    • It allows you to send, store, and receive messages between software components at any volume, without losing messages or requiring other services to be available.
  • Use Cases:
    • Task queues
    • asynchronous processing
    • decoupling microservices
    • job dispatching
    • buffering messages between producer and consumer systems



Messaging Patterns



Amazon SNS:

  • Push-Based: SNS pushes messages to subscribers. Subscribers can be other AWS services (like Lambda, SQS), HTTP/S endpoints, email addresses, SMS numbers, and mobile push notifications.
  • Fan-Out: One message can be sent to multiple subscribers.

Amazon SQS:

  • Pull-Based: Consumers pull messages from the queue. A consumer explicitly retrieves messages from the queue.
  • Point-to-Point: Each message is delivered to and processed by one consumer.


Message Handling


Amazon SNS:

  • Real-Time Delivery: Messages are delivered immediately to all subscribers.
  • No Message Persistence: Messages are not stored after delivery; if a subscriber is unavailable, the message is lost unless it's sent to an SQS queue or some other durable store.

Amazon SQS:

  • Message Persistence: Messages are stored in the queue until they are processed and deleted by a consumer, or until they expire.
  • Delivery Guarantees: Ensures at least once delivery. With FIFO queues, SQS provides exactly-once processing and message ordering.

Scalability and Performance


Amazon SNS:

  • Scalable: Designed to handle massive numbers of messages and deliver them to large numbers of subscribers.
  • Latency: Typically has very low latency for message delivery.

Amazon SQS:

  • Scalable: Automatically scales to handle large volumes of messages. Suitable for high-throughput applications.
  • Latency: Slightly higher latency compared to SNS due to the nature of pull-based consumption.

Features and Capabilities


Amazon SNS:

  • Multiple Protocols: Supports multiple delivery protocols including HTTP/S, email, SMS, SQS, Lambda, and mobile push notifications.
  • Filtering: Allows message filtering, enabling subscribers to receive only the messages that match their filter policies.

Amazon SQS:

  • Visibility Timeout: Temporarily hides a message from other consumers while it is being processed.
  • Dead-Letter Queues (DLQ): Allows you to handle messages that can't be processed successfully.
  • FIFO Queues: Ensures the order of messages and exactly-once processing.
  • Delay Queues: Postpones the delivery of new messages to consumers for a specified amount of time.


Pricing


Amazon SNS:

  • Pricing Model: Based on the number of requests (publishes, deliveries, and notifications) and data transfer.
  • Cost Efficiency: More cost-effective for scenarios requiring a high number of subscribers and real-time notifications.

Amazon SQS:

  • Pricing Model: Based on the number of requests (send, receive, delete) and data transfer.
  • Cost Efficiency: More cost-effective for decoupling microservices and scenarios requiring message persistence and complex message handling.


Integration and Interoperability


Amazon SNS:

  • Integration: Easily integrates with a wide range of AWS services (e.g., Lambda, SQS, HTTP/S endpoints, etc.).
  • Interoperability: Often used in conjunction with SQS for fan-out scenarios where messages need to be processed asynchronously and stored reliably.

Amazon SQS:

  • Integration: Commonly used to decouple systems and provide reliable message delivery. Often used with other AWS services like Lambda, ECS, and EC2.
  • Interoperability: Can be subscribed to SNS topics to receive messages that need persistent storage or further processing.

Summary


In summary, Amazon SNS is a pub/sub messaging service optimized for real-time notifications and broadcasting messages to multiple subscribers, while Amazon SQS is a message queuing service designed for decoupling distributed systems and ensuring reliable message delivery through persistence and processing guarantees. They are often used together to build scalable, resilient, and flexible messaging architectures in AWS.


Push Notification Service - Amazon Simple Notification Service - AWS
Posted by at No comments:
Labels: , , ,

Thursday, 19 March 2026

Amazon EBS CSI Driver



The Amazon EBS CSI Driver is a standard interface that allows Amazon Elastic Kubernetes Service (EKS) clusters to manage the full lifecycle of Amazon EBS volumes as persistent storage for containers. It replaces the older, deprecated "in-tree" Kubernetes storage plugin with a more flexible, decoupled model. 

Key Features

  • Dynamic Provisioning: Automatically creates and attaches EBS volumes when a PersistentVolumeClaim (PVC) is made.
  • Volume Lifecycle Management: Handles the creation, attachment, mounting, and deletion of volumes.
  • Resizing & Snapshots: Supports online volume resizing (for gp3 and others) and taking volume snapshots for data backup.
  • EKS Auto Mode Support: In EKS Auto Mode, routine block storage tasks are automated, and you don't even need to manually install the driver. 

Deployment Methods


You can install and manage the driver through several channels: 
  • EKS Managed Add-on (Recommended): Simplifies installation and updates via the AWS Console, CLI, or Terraform.
  • Helm Chart: Provides highly customizable installation options.
  • Kustomize: Direct deployment using manifests from the official GitHub repository. 

Core Requirements

  • IAM Permissions: The driver requires an IAM role with the AmazonEBSCSIDriverPolicy to interact with EBS resources.
  • Service Accounts: Typically uses IAM Roles for Service Accounts (IRSA) to securely provide AWS credentials to the driver pods.
  • Compatibility: Supports Linux and Windows worker nodes, as well as ARM64 architectures.

Driver Components


The driver is typically deployed into the kube-system namespace and consists of two main parts: 
  • Controller Deployment: Runs as a set of replicas (ebs-csi-controller) to communicate with the AWS EC2 API and manage volume operations.
  • Node DaemonSet: Runs on every worker node (ebs-csi-node) to handle the actual mounting and unmounting of volumes to pods on that specific host. 

In the Amazon EBS CSI driver architecture, the ebs-csi-controller and ebs-csi-node are the two primary components that work together to manage the lifecycle of EBS volumes in a Kubernetes cluster.

Core Feature Differences


ebs-csi-controller

  • Deployment Type: 
    • Deployment (typically 2 replicas for HA)
  • Main Function: 
    • Control Plane operations: Creating, deleting, attaching, and detaching volumes
  • AWS Interaction: 
    • Calls the AWS EC2 API to manage EBS resources
  • IAM Permissions: 
    • Requires an IAM role with permissions like ec2:CreateVolume and ec2:AttachVolume

ebs-csi-node

  • Deployment Type: 
    • DaemonSet (runs on every worker node)
  • Main Function: 
    • Node-level operations: Mounting and unmounting volumes to the local filesystem
  • AWS Interaction: 
    • Interacts with the local OS (privileged system calls) to handle block devices
  • IAM Permissions: 
    • Generally requires fewer/no AWS API permissions, as it mostly performs local mount actions

How They Work Together
  • Provisioning & Attachment: When you create a PersistentVolumeClaim (PVC), the ebs-csi-controller watches the request and calls the AWS API to create the EBS volume and attach it to the correct EC2 instance.
  • Mounting: Once the volume is physically attached to the EC2 instance, the ebs-csi-node pod running on that specific node detects the new block device and mounts it into the container’s path so your application can use it. 

Key Considerations
  • Security: For better security, you can schedule the ebs-csi-controller on hardened management nodes, while the ebs-csi-node must run everywhere your workloads need storage.
  • Fargate: You can run the controller on Fargate nodes, but the ebs-csi-node (as a DaemonSet) only runs on EC2 instances.
  • Troubleshooting: If a volume fails to "attach," check the controller logs; if it fails to "mount" or "format," check the node logs.

Pods for both the ebs-csi-controller and ebs-csi-node typically share the same value for the app.kubernetes.io/name label. 

In standard deployments (such as via the official Helm chart or EKS add-on), both components use this label to identify that they belong to the same overarching application: the Amazon EBS CSI Driver.

Label Comparisons


While they share the same application name, they use the app.kubernetes.io/component label to distinguish between their specific roles.

Label Key                           ebs-csi-controller Pods     ebs-csi-node Pods
------------                                  ------------------------------    ----------------------
app.kubernetes.io/name           aws-ebs-csi-driver       aws-ebs-csi-driver
app.kubernetes.io/instance   aws-ebs-csi-driver       aws-ebs-csi-driver
app.kubernetes.io/component   csi-driver (or controller)     csi-driver (or node)
app (Legacy label)                   ebs-csi-controller               ebs-csi-node


How to Verify in Your Cluster


You can check these labels yourself using kubectl. This is useful if you are writing Prometheus rules or network policies that need to target the entire driver or just one part of it. 

To see labels for all EBS CSI pods:

kubectl get pods -n kube-system -l app.kubernetes.io/name=aws-ebs-csi-driver --show-labels

To target only the controller:

kubectl get pods -n kube-system -l app=ebs-csi-controller


---

Posted by at No comments:
Labels: , , , , ,

Friday, 6 February 2026

Amazon EKS Autoscaling with Karpenter



Kubernetes autoscaling is a function that scales resources in and out depending on the current workload. AWS supports two autoscaling implementations:
  • Cluster Autoscaler
  • Karpenter 
    • Karpenter
    • flexible, high-performance Kubernetes cluster autoscaler and node provisioner
    • helps improve application availability and cluster efficiency
    • launches right-sized compute resources (for example, Amazon EC2 instances) in response to changing application load in under a minute
    • can provision just-in-time compute resources that precisely meet the requirements of our workload
    • automatically provisions new compute resources based on the specific requirements of cluster workloads. These include compute, storage, acceleration, and scheduling requirements. 
    • creates Kubernetes nodes directly from EC2 instances
    • improves the efficiency and cost of running workloads on the cluster
    • open-source


Pod Scheduler


  • Kubernetes cluster component responsible for determining which node Pods get assigned to
  • default Pod scheduler for Kubernetes is kube-scheduler
    • logs the reasons Pods can't be scheduled

Unschedulable Pods



A Pod is unschedulable when it's been put into Kubernetes' scheduling queue, but can't be deployed to a node. This can be for a number of reasons, including:
  • The cluster not having enough CPU or RAM available to meet the Pod's requirements.
  • Pod affinity or anti-affinity rules preventing it from being deployed to available nodes.
  • Nodes being cordoned due to updates or restarts.
  • The Pod requiring a persistent volume that's unavailable, or bound to an unavailable node.

How to detect unschedulable Pods?

Pods waiting to be scheduled are held in the "Pending" status, but if the Pod can't be scheduled, it will remain in this state. However, Pods that are being deployed normally are also marked as "Pending." The difference comes down to how long a Pod remains in "Pending." 

How to  fix unschedulable Pods? 
There is no single solution for unschedulable Pods as they have many different causes. However, there are a few things we can try depending on the cause. 
  • Enable cluster autoscaling
    • If we're using a managed Kubernetes service like Amazon EKS or Google Kubernetes Engine (GKE), we can very easily take advantage of autoscaling to increase and decrease cluster capacity on-demand. With autoscaling enabled, Kubernetes' Cluster Autoscaler will trigger our provider to add nodes when needed. As long as we've configured our cluster node pool and it hasn't reached its max node limit, our provider will automatically provision a new node and add it to the pool, making it available to the cluster and to our Pods.
  • Increase our node capacity
  • Check our Pod requests
  • Check our affinity and anti-affinity rules 

 

In this article we'll show how to enable cluster autoscaling with Karpenter.


How does the regular Kubernetes Autoscaler work in AWS?


When we create a regular Kubernetes cluster in AWS, each node group is managed by the AWS Auto-scaling group [Auto Scaling groups - Amazon EC2 Auto Scaling]. Cluster native autoscaler adjusts the desired size based on the load in the cluster to fit all unscheduled pods.

HorizontalPodAutoscaler (HPA) [Horizontal Pod Autoscaling | Kubernetes] is built into Kubernetes and it uses metrics like CPU usage, memory usage or custom metrics we can write to decide when to spin up or down additional pods in the node of the cluster. If our app is receiving more traffic, HPA will kick in and provision additional pods. 

VerticalPodAutoscaler (VPA) can also be installed in cluster where it manages the resource (like CPU and memory) allocation to pods that are already running.

What about when there's not enough capacity to schedule any more pods in the (existing) node(s)? That's when we'll need an additional node. So we have a pod that needs to be scheduled but we don't know where to put it. We could call AWS API, spin up an additional EC2 node, get added it to our cluster or if we're using managed groups we can use Managed Node Group API, bump up the desired size but easier approach is to use cluster auto-scaler. There is a mature open-source solution called Cluster Auto-Scaler (CAS).

CAS was built to handle hundreds of different combinations of nodes types, zones, purchase options available in AWS. CAS works directly with managed node groups or self-managed managed nodes and auto-scaling groups which are AWS constructs to help us manage nodes. 


What are the issues with the regular Kubernetes Autoscaler?


Let's say CAS is installed on node, in cluster and manages one managed node group (MNG). It's filling up and we have an additional pod that needs to be provisioned so CAS tells MNG to bump up the number of nodes so it spins up another one so pod can now be scheduled. But this is not ideal. We have a single pod in a node, we don't need such a big node. 

This can be solved by creating a different MNG with a smaller instance type and now CAS recognizes that instance and provisions pod on a more appropriately-sized node.

Unfortunately, we might end up with many MNGs, based on requirements which might be a challenge to manage especially when looking best practices in terms of cost efficiency and high availability. 


How does Karpenter work?


Karpenter works differently, It doesn't use MNG or ASGs and manages each node directly. Let's say we have different pods, of different sizes. Let's say that HPA says that we need more of the smaller pods. Karpenter will intelligently pick the right instance type for that workload. If we need to spin up a larger pod it will again pick the right instance type. 

Karpenter picks exactly the right type of node for our workload. 

If we're using spot instances and spot capacity is not available, Karpenter does retries more quickly. Karpenter offers, faster, dynamic, more intelligent compute, using best practices without operational overhead of managing nodes ourselves. 

How to control how Karpenter operates?

There are many dimensions here. We can set constraints on Karpenter to limit the instances type, we can set up taints to isolate workloads to specific types of nodes. Different teams can have isolated access to different pods, one team can access billing pods, another GPU-based instances. 

Workload Consolidation feature: Pods are consolidated into fewer nodes.. let's say we have 3 nodes, two at 70% and one at 20% utilization. Karpenter detects this and will move pods from underutilized node to those two and shut down this now empty node (instances are terminated). This leads to lower costs.

Karpenter is making it easier to use spot and graviton instances which can also lead to lower costs. 

A feature to keep our nodes up to date. ttlSecondsUntilExpired parameter tells Karpenter to terminate nodes after a set amount of time. These nodes will automatically be replaced with new nodes, running the latest AMIs.

Karpenter:
1) lower costs
2) higher application availability 
3) lower operation overhead


Karpenter needs permissions to create EC2 instances in AWS. 

If we use a self-hosted (on bare metal boxes or EC2 instances), self-managed (we have full control over all aspects of Kubernetes) Kubernetes cluster, for example by using kOps (see also Is k8s Kops preferable than eks? : r/kubernetes), we can add additional IAM policies to the existing IAM role attached to Kubernetes nodes. 

If using EKS, the best way to grant access to internal service is with IAM roles for service accounts (IRSA).


Karpenter's Kubernetes Custom Resources


NodePool


NodePool is the primary Custom Resource (CR) in Karpenter that defines scheduling constraints, how nodes are provisioned and managed (node management policies). It is the successor to the older Provisioner API and acts as the brain that tells Karpenter which nodes to create and how to handle them over time. It acts as the "brain" for scheduling decisions by evaluating the requirements of pending pods and matching them to infrastructure constraints.

Core Role of NodePool
  • Scheduling Authority: It defines the constraints (instance types, zones, architectures) that determine which nodes can be created.
  • Successor to Provisioner: It replaced the older Provisioner API to provide a more scalable and configuration-based approach.
  • Management Hub: It handles node lifecycle settings, including disruption policies (consolidation and expiration) and aggregate resource limits (CPU/Memory).

Core Functions

A NodePool manages three primary aspects of our cluster's compute capacity: 
  • Scheduling Constraints: Restricts which nodes can be provisioned using requirements for instance types, zones, architectures (e.g., x86 vs. ARM), and capacity types (Spot vs. On-Demand).
  • Disruption Policies: Governs how Karpenter optimizes the cluster by defining when nodes should be expired or consolidated to save costs.
  • Resource Limits: Sets a cap on the total CPU and memory that the NodePool can provision, preventing runaway costs. 

Key Components of a NodePool

The specification is divided into several functional areas:
  • template: Defines the configuration for the nodes that will be created.
  • requirements: Uses well-known Kubernetes labels (e.g., karpenter.sh/capacity-type) to select hardware.
  • nodeClassRef: Points to an EC2NodeClass for cloud-provider-specific settings like subnets and security groups.
  • disruption: Replaces older TTL settings with a unified policy for consolidationPolicy (e.g., WhenUnderutilized) and expireAfter.
  • limits: Defines the maximum aggregate resources (e.g., cpu: 1000) allowed for this pool. 

Example v1 Configuration

This example demonstrates a production-ready NodePool that prioritises Spot instances but allows for On-
Demand fallback. 

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general-purpose
spec:
  template:
    spec:
      requirements:
        - key: "karpenter.sh/capacity-type"
          operator: In
          values: ["spot", "on-demand"]
        - key: "karpenter.k8s.aws/instance-category"
          operator: In
          values: ["c", "m", "r"]
        - key: "kubernetes.io/arch"
          operator: In
          values: ["amd64", "arm64"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
  disruption:
    consolidationPolicy: WhenUnderutilized
    expireAfter: 720h # 30 days
  limits:
    cpu: "500"
    memory: 1000Gi


Comparison with Other Objects

While the NodePool is the central configuration object, it works in a hierarchy with two other key resources: 
  • NodePool
    • Purpose: The Logic: Defines what nodes should look like and how they should behave.
  • EC2NodeClass
    • Purpose: The Infrastructure: Defines where and with what AWS-specific settings (subnets, AMIs, security groups) nodes launch.
  • NodeClaim
    • Purpose: The Instance: Represents an individual node currently being managed or provisioned by Karpenter.

Every NodePool must reference at least one EC2NodeClass to successfully provision capacity on AWS.

Useful Commands:

To see all node pools:

% kubectl get nodepools                   
NAME                NODECLASS           NODES   READY   AGE
clickhouse          clickhouse          0       True    140d
clickhouse-backup   clickhouse-backup   0       True    140d 

Cluster user needs to have permission to list resource "nodepools" in API group "karpenter.sh" at the cluster scope.

To debug a specific node pool:

kubectl describe nodepool <nodepool-name>

Cluster user needs to have permission to get resource "nodepools" in API group "karpenter.sh" at the cluster scope.

% kubectl describe nodepool clickhouse
Name:         clickhouse
Namespace:    
Labels:       <none>
Annotations:  karpenter.sh/nodepool-hash: 12671849087427876759
              karpenter.sh/nodepool-hash-version: v3
API Version:  karpenter.sh/v1
Kind:         NodePool
Metadata:
  Creation Timestamp:  2025-10-22T15:02:58Z
  Generation:          2
  Resource Version:    1073678
  UID:                 f7869dd3-ac24-4600-98a6-059073645769
Spec:
  Disruption:
    Budgets:
      Nodes:               10%
    Consolidate After:     0s
    Consolidation Policy:  WhenEmptyOrUnderutilized
  Template:
    Metadata:
      Labels:
        Karpenter - Node - Pool:  clickhouse
    Spec:
      Expire After:  720h
      Node Class Ref:
        Group:  karpenter.k8s.aws
        Kind:   EC2NodeClass
        Name:   clickhouse
      Requirements:
        Key:       node.kubernetes.io/instance-type
        Operator:  In
        Values:
          r8g.xlarge
          r8g.2xlarge
          r8g.4xlarge
          r8g.8xlarge
        Key:       karpenter.sh/capacity-type
        Operator:  In
        Values:
          on-demand
          spot
Status:
  Conditions:
    Last Transition Time:  2025-10-22T15:02:59Z
    Message:               
    Observed Generation:   2
    Reason:                ValidationSucceeded
    Status:                True
    Type:                  ValidationSucceeded
    Last Transition Time:  2025-10-22T15:03:07Z
    Message:               
    Observed Generation:   2
    Reason:                NodeClassReady
    Status:                True
    Type:                  NodeClassReady
    Last Transition Time:  2025-10-23T17:24:01Z
    Message:               
    Observed Generation:   2
    Reason:                Ready
    Status:                True
    Type:                  Ready
  Resources:
    Cpu:                  0
    Ephemeral - Storage:  0
    Memory:               0
    Nodes:                0
    Pods:                 0
Events:                   <none>



EC2NodeClass


EC2NodeClass is a Custom Resource (CR) used to define AWS-specific infrastructure configurations for the nodes Karpenter provisions. 

While a NodePool handles high-level scheduling constraints (like instance types or taints), the EC2NodeClass dictates the underlying Amazon EC2 settings. 

Key Responsibilities


The EC2NodeClass abstracts cloud provider-specific details, including: 
  • Networking: Selects subnets using subnetSelectorTerms.
  • Security: Identifies security groups via securityGroupSelectorTerms.
  • Identity: Assigns the IAM role or instance profile for the nodes.
  • Storage: Configures blockDeviceMappings for EBS volumes.
  • Images: Specifies the Amazon Machine Image (AMI) family (e.g., AL2, Bottlerocket) or selects specific AMIs.
  • Customisation: Includes userData for custom bootstrap scripts. 

Relationship with NodePools


A NodePool must reference an EC2NodeClass using the nodeClassRef field. Multiple NodePools can point to the same EC2NodeClass if they share the same infrastructure requirements (e.g., same VPC and IAM role).

Example Configuration


A basic EC2NodeClass manifest typically looks like this: 

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: default
spec:
  amiFamily: AL2
  role: "KarpenterNodeRole-my-cluster"
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: my-cluster
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: my-cluster

Useful Commands:


To see all EC2NodeClasses

kubectl get ec2nodeclasses 

Cluster user needs to have permission to list resource "ec2nodeclasses" in API group "karpenter.k8s.aws" at the cluster scope.

Example:

% kubectl get ec2nodeclasses        
NAME                READY   AGE
clickhouse          True    140d
clickhouse-backup   True    140d

To debug a specific node that isn't coming online:

kubectl describe ec2nodeclasses <ec2nodeclass-name>

Cluster user needs to have permission to get resource "ec2nodeclasses" in API group "karpenter.k8s.aws" at the cluster scope.

Example:

% kubectl describe ec2nodeclass clickhouse
Name:         clickhouse
Namespace:    
Labels:       <none>
Annotations:  karpenter.k8s.aws/ec2nodeclass-hash: 358699366951558737
              karpenter.k8s.aws/ec2nodeclass-hash-version: v4
API Version:  karpenter.k8s.aws/v1
Kind:         EC2NodeClass
Metadata:
  Creation Timestamp:  2025-10-22T15:02:58Z
  Finalizers:
    karpenter.k8s.aws/termination
  Generation:        1
  Resource Version:  73323969
  UID:               25c663e7-cc29-47b2-8a97-937fb5f39825
Spec:
  Ami Family:  AL2023
  Ami Selector Terms:
    Alias:              al2023@latest
  Detailed Monitoring:  true
  Metadata Options:
    Http Endpoint:                enabled
    httpProtocolIPv6:             disabled
    Http Put Response Hop Limit:  1
    Http Tokens:                  required
  Role:                           KarpenterNodeRole-mycorp-prod-clickhouse-k8s
  Security Group Selector Terms:
    Tags:
      karpenter.sh/discovery/mycorp-prod-clickhouse-k8s:  true
  Subnet Selector Terms:
    Tags:
      karpenter.sh/discovery:  true
      private_subnet:          true
  Tags:
    Name:                                              mycorp-prod-clickhouse-k8s-karpenter-clickhouse
    karpenter.sh/discovery/mycorp-prod-clickhouse-k8s:  true
Status:
  Amis:
    Id:    ami-06ab427136b8ffa61
    Name:  amazon-eks-node-al2023-x86_64-nvidia-1.33-v20260304
    Requirements:
      Key:       kubernetes.io/arch
      Operator:  In
      Values:
        amd64
      Key:       karpenter.k8s.aws/instance-gpu-count
      Operator:  Exists
    Id:          ami-08f492a005f7b8703
    Name:        amazon-eks-node-al2023-x86_64-neuron-1.33-v20260304
    Requirements:
      Key:       kubernetes.io/arch
      Operator:  In
      Values:
        amd64
      Key:       karpenter.k8s.aws/instance-accelerator-count
      Operator:  Exists
    Id:          ami-0023c4931d42779e6
    Name:        amazon-eks-node-al2023-x86_64-standard-1.33-v20260304
    Requirements:
      Key:       kubernetes.io/arch
      Operator:  In
      Values:
        amd64
      Key:       karpenter.k8s.aws/instance-gpu-count
      Operator:  DoesNotExist
      Key:       karpenter.k8s.aws/instance-accelerator-count
      Operator:  DoesNotExist
    Id:          ami-061bed77c8a6d03cd
    Name:        amazon-eks-node-al2023-arm64-standard-1.33-v20260304
    Requirements:
      Key:       kubernetes.io/arch
      Operator:  In
      Values:
        arm64
      Key:       karpenter.k8s.aws/instance-gpu-count
      Operator:  DoesNotExist
      Key:       karpenter.k8s.aws/instance-accelerator-count
      Operator:  DoesNotExist
  Conditions:
    Last Transition Time:  2025-10-22T15:02:59Z
    Message:               
    Observed Generation:   1
    Reason:                AMIsReady
    Status:                True
    Type:                  AMIsReady
    Last Transition Time:  2025-10-22T15:02:59Z
    Message:               
    Observed Generation:   1
    Reason:                SubnetsReady
    Status:                True
    Type:                  SubnetsReady
    Last Transition Time:  2025-10-22T15:02:59Z
    Message:               
    Observed Generation:   1
    Reason:                SecurityGroupsReady
    Status:                True
    Type:                  SecurityGroupsReady
    Last Transition Time:  2025-10-22T15:02:59Z
    Message:               
    Observed Generation:   1
    Reason:                InstanceProfileReady
    Status:                True
    Type:                  InstanceProfileReady
    Last Transition Time:  2025-10-22T15:03:07Z
    Message:               
    Observed Generation:   1
    Reason:                ValidationSucceeded
    Status:                True
    Type:                  ValidationSucceeded
    Last Transition Time:  2025-10-22T15:03:07Z
    Message:               
    Observed Generation:   1
    Reason:                Ready
    Status:                True
    Type:                  Ready
  Instance Profile:        mycorp-prod-clickhouse-k8s_15693974848685646064
  Security Groups:
    Id:    sg-09f3cd41bcef827c0
    Name:  mycorp-prod-clickhouse-k8s-node-20251020164545608400000006
  Subnets:
    Id:       subnet-04xxxxxxxxxx5d30b
    Zone:     us-east-1b
    Zone ID:  use1-az2
    Id:       subnet-00xxxxxxxxxx08cef
    Zone:     us-east-1c
    Zone ID:  use1-az3
    Id:       subnet-02xxxxxxxxxxx8711
    Zone:     us-east-1a
    Zone ID:  use1-az1
Events:       <none>


NodeClaim


In Karpenter, a NodeClaim is the Custom Resource (CR) that represents a single, specific instance of compute capacity. 

While a NodePool is the template and a NodeClass is the blueprint, the NodeClaim is the actual request sent to the cloud provider to launch a specific node. 

Key Characteristics


  • 1:1 Relationship: Each NodeClaim typically corresponds to exactly one EC2 instance and its associated Kubernetes Node.
  • Immutable: Once created, a NodeClaim cannot be changed. If the requirements for a node change (e.g., due to "drift"), Karpenter deletes the existing NodeClaim and creates a new one.
  • Lifecycle Management: It tracks the instance from its initial "launch" request through "registration" with the cluster until it is fully "initialized" and ready to run pods. 

Why We Should Monitor NodeClaims


NodeClaims are the best place to look when debugging provisioning failures. We can use them to identify why a node failed to join the cluster: 
  • Status Conditions: A NodeClaim status will show if an instance failed to launch (e.g., "LaunchFailed" due to AWS service limits) or if it's stuck during initialization.
  • Provider Information: It stores the specific providerID (e.g., EC2 instance ID) and the nodeName once the instance successfully joins the cluster.
  • Drift Detection: Karpenter adds a Drifted status to a NodeClaim if the underlying instance no longer matches the current NodePool or EC2NodeClass configuration. 

How it Fits the Hierarchy:

  • NodePoolDefines 
    • Function: scheduling rules and resource limits.
    • Analogy: The Foreman
  • EC2NodeClass
    • Function: Defines AWS-specific infrastructure (subnets, AMIs).
    • Analogy: The Blueprint
  • NodeClaim
    • Function: Represents the actual instance being provisioned.
    • Analogy: The Building Material

Useful Commands:


To see all active node claims and their status:

kubectl get nodeclaims 

Cluster user needs to have permission to list resource "nodeclaims" in API group "karpenter.sh" at the cluster scope.

To debug a specific node that isn't coming online:

kubectl describe nodeclaim <nodeclaim-name>

Cluster user needs to have permission to get resource "nodeclaims" in API group "karpenter.sh" at the cluster scope.

Example:

% kubectl get nodeclaims         
No resources found

How to Debug Misfunctioning Karpenter


Based on the output we've provided above, our NodePool and EC2NodeClass are both in a Ready state and have correctly discovered subnets, security groups, and AMIs. However, we have 0 nodes and 0 NodeClaims. The most common reason for this behavior—where the infrastructure is ready but Karpenter isn't scaling—is a scheduling mismatch between our pending Pods and the NodePool's requirements. Here is a breakdown of why Karpenter might be sitting idle:

1. Missing Architecture Requirement

Our clickhouse NodePool specifies r8g instance types. These are AWS Graviton (ARM64) instances. In our NodePool spec, we haven't explicitly defined the architecture requirement. While Karpenter is smart, if our Pods do not have a nodeSelector or toleration for kubernetes.io/arch: arm64, or if the Pods are explicitly asking for amd64, Karpenter will not use this NodePool to satisfy them.Check our Pods:

kubectl get pods -A | grep -i pending
kubectl describe pod <pending-pod-name>

Look for NodeSelectors or Affinity rules in the Pod spec. If the Pod doesn't allow arm64, Karpenter won't spin up an r8g node.

2. Missing Taints/Tolerations

ClickHouse workloads often use specialized nodes. If our NodePool (or the nodes it creates) has a Taint that we haven't shown in the snippet, or if our Pods have a Toleration for a taint that doesn't exist on this NodePool, Karpenter will ignore it.

Check if our pending Pods have specific nodeSelector labels that aren't present in the NodePool template.metadata.labels. Our NodePool only has one label: Karpenter - Node - Pool: clickhouse.

3. The "Karpenter Controller" Logs

If the logic seems correct but nothing is happening, the answer is always in the controller logs. Karpenter will explicitly tell us why it is passing over a Pod.Run this to see the scheduling decisions:

kubectl logs -n karpenter -l app.kubernetes.io/name=karpenter | grep -i "scheduling"

Look for messages like:no reachable nodeclassesno possible pod inventoryunschedulable, ... did not match requirements

4. Service Linked Role / Permissions

Since our EC2NodeClass is Ready, our basic AWS tags are likely fine. However, double-check that the KarpenterNodeRole-xxxxx-prod-clickhouse-k8s actually exists in IAM and has the AmazonEKSWorkerNodePolicy and AmazonEC2ContainerRegistryReadOnly attached. If the role is missing or misconfigured, the EC2 instance might start but fail to join the cluster, causing Karpenter to terminate it immediately.

Summary Checklist

Potential Issue Fix/Action 
Arch Mismatch => Add kubernetes.io/arch with arm64 to NodePool requirements or Pod nodeSelector.

Pending Pods => Ensure there are actually Pods in Pending state. Karpenter only scales in response to unschedulable pods.

Instance Availability => r8g instances are relatively new. Ensure they are available in us-east-2 for the capacity type (Spot/On-Demand) we requested.


How to install Karpenter in the cluster?


Namespace


Current Karpenter best practices recommend deploying it in the kube-system namespace rather than its own dedicated namespace. 

While many early adopters used a separate karpenter namespace, the project shifted toward kube-system starting with version v0.33.0. 

Why kube-system is preferred:
  • API Priority & Fairness: By default, Kubernetes grants higher priority to requests coming from the kube-system namespace. This ensures the Karpenter controller can still communicate with the API server to provision nodes even during periods of heavy cluster congestion.
  • Critical Component Status: Placing Karpenter in kube-system denotes it as a critical cluster component, aligning it with other essential services like kube-proxy or the VPC CNI.
  • Reduced Complexity: Using a standard system namespace avoids the need to manually configure custom FlowSchemas or PriorityLevelConfigurations that would otherwise be required to give a custom namespace the same level of reliability. 

When to use a separate namespace:
  • Legacy Installations: If we installed Karpenter before v0.33.0, it likely lives in a karpenter namespace. Migrating is recommended but requires updating our IAM Roles for Service Accounts (IRSA) trust policy to reflect the new namespace.
  • Fargate Isolation: If we run Karpenter on AWS Fargate, we must create a Fargate profile for the specific namespace where Karpenter is deployed.

Labels


While most standard Karpenter installations use the label app.kubernetes.io/name=karpenter for the controller pods, it is not guaranteed for every pod in every environment. 

Why it might differ:
  • Helm Chart Customisation: If we (or our platform team) overrode the podLabels or nameOverride values during the Karpenter Helm installation, this label will be different.
  • Version Variance: Very old versions of Karpenter sometimes used different labelling conventions (e.g., just app=karpenter), though modern versions follow Kubernetes recommended labels.
  • Webhook Pods: In some configurations, Karpenter may run separate pods for webhooks that might carry slightly different descriptive labels depending on the deployment strategy.


How to configure Karpenter?


We can configure specific Karpenter NodePools or Provisioners.

How to set up weighted NodePools for multi-tenant isolation?



In Karpenter, Weighted NodePools allow us to control which NodePool is selected when a pod's requirements match multiple pools. This is a powerful tool for multi-tenant isolation, enabling us to prioritize specific hardware or cost models for certain teams while providing a fallback mechanism. 

How Weighting Works

  • Precedence: Higher weight values indicate higher priority.
  • Default: If no weight is specified, it defaults to 0.
  • Selection: If a pending pod matches the requirements of multiple NodePools, Karpenter will always select the one with the highest weight first. 

Multi-Tenant Strategy: Isolation & Priority

For multi-tenant environments, we can use weights to enforce distinct tiers of service or cost:
  • Reserved/Savings Plan Tier (Highest Weight): 
    • Create a NodePool that specifically includes instance types covered by our Savings Plans or Reserved Instances. By giving this pool a high weight (e.g., 100), Karpenter will prioritize using this pre-paid capacity before launching new nodes.
  • Spot Instance Tier (Medium Weight): 
    • A general-purpose pool for non-critical workloads or "Team A" can be set with a medium weight (e.g., 50) and restricted to spot capacity.
  • On-Demand Fallback (Lowest Weight): 
    • A "catch-all" NodePool with a low weight (e.g., 10) that allows on-demand instances. This ensures that if Spot capacity is unavailable or Savings Plans are exhausted, workloads still have a place to land. 

Implementation Example

Below is an example of two overlapping NodePools where the "Premium" pool is prioritized for any workload that could run on it.

# NodePool 1: High Priority (e.g., Reserved Capacity)
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: premium-reserved
spec:
  weight: 100  # Higher weight = Higher priority
  template:
    spec:
      requirements:
        - key: "node.kubernetes.io/instance-type"
          operator: In
          values: ["m5.large", "m5.xlarge"] # Specific reserved types
      nodeClassRef:
        name: default
---
# NodePool 2: Standard Priority (e.g., Spot)
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: standard-spot
spec:
  weight: 50
  template:
    spec:
      requirements:
        - key: "karpenter.sh/capacity-type"
          operator: In
          values: ["spot"]
      nodeClassRef:
        name: default


Best Practices for Isolation

  • Mutual Exclusivity: While weights handle overlaps, the official Karpenter guidance suggests making NodePools mutually exclusive whenever possible (using taints/tolerations or unique labels) to simplify debugging.
  • Resource Limits: Always set spec.limits on tenant-specific pools to prevent one team from consuming the entire cluster's budget.
  • Billing Attribution: Use the spec.template.metadata.labels field in each NodePool to add "Team" or "Project" tags. These labels propagate to the EC2 instances, making it easy to track costs per tenant

How to implement Taints and Tolerations alongside weights for stricter tenant "hard" isolation?


While weights allow Karpenter to prefer one NodePool over another, Taints and Tolerations are required for hard isolation. They ensure that nodes provisioned for one tenant "repel" pods from all other tenants. 

The Isolation Strategy

To achieve strict tenant separation, we combine three elements:
  • Taints: Applied to the NodePool to prevent unauthorized pods from scheduling on its nodes.
  • Tolerations: Applied to the tenant's pods so they can "bypass" the taint.
  • Node Affinity: Applied to the tenant's pods to "attract" them specifically to their dedicated nodes. 

1. Dedicated Tenant NodePool 

In the NodePool spec, add a taint. Any node Karpenter creates from this pool will automatically carry this "keep out" sign. 

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: tenant-a-pool
spec:
  weight: 50
  template:
    spec:
      taints:
        - key: "tenant"
          value: "team-a"
          effect: "NoSchedule" # Only pods with matching toleration can land here
      labels:
        tenant: "team-a" # Used for affinity
      nodeClassRef:
        name: default

2. Tenant Pod Configuration

For Team A's workloads to run, their pods must explicitly tolerate the taint and prefer (or require) the tenant label. 

apiVersion: v1
kind: Pod
metadata:
  name: team-a-app
spec:
  containers:
    - name: app
      image: nginx
  tolerations:
    - key: "tenant"
      operator: "Equal"
      value: "team-a"
      effect: "NoSchedule"
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: "tenant"
                operator: In
                values: ["team-a"]

Why use both?

  • Taint + Toleration alone stops other pods from accidentally using Team A's nodes, but it doesn't stop Team A's pods from accidentally landing on "General" nodes.
  • Node Affinity ensures Team A's pods only go to their dedicated nodes.
  • Weights (e.g., weight: 100) can still be used within a tenant's pool to prioritize Spot vs. On-Demand specifically for that tenant. 

Best Practices

  • Mutually Exclusive Pools: It is recommended to design NodePools so they do not overlap. If a pod matches multiple pools, Karpenter uses the one with the highest weight.
  • NoExecute for Critical Changes: Use the NoExecute effect if we need to evict existing pods immediately when a node becomes inappropriate for them.
  • Limit Resources: Always set spec.limits on each tenant pool to prevent a single team's auto-scaling from exhausting the entire AWS account's resources.

How to ensure our cluster has at least 3 nodes spread across 3 different Availability Zones (AZs)?


This is important if we want to implement highly available architecture. We want nodes to be spread across multiple data centres and with them, pod which belong to our application. 

We can define a NodePool that forces a spread across zones using topology:

apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: default
spec:
  template:
    spec:
      requirements:
        # Force the nodes to be spread across these zones
        - key: "topology.kubernetes.io/zone"
          operator: In
          values: ["us-east-1a", "us-east-1b", "us-east-1c"]
        - key: "karpenter.sh/capacity-type"
          operator: In
          values: ["on-demand"]
  # Ensure the autoscaler keeps a minimum of 3 nodes
  limits:
    cpu: 1000


BONUS: Forcing Pods to use all 3 Zones


Even if we have 3 nodes in 3 zones, Kubernetes might try to put all our pods on just one of those nodes to be "efficient." 

To prevent this, we use Topology Spread Constraints. This is the modern, more powerful version of Anti-Affinity. It ensures our pods are distributed evenly across the zones we just created.

spec:
  topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: "topology.kubernetes.io/zone"
    whenUnsatisfiable: DoNotSchedule # Or ScheduleAnyway
    labelSelector:
      matchLabels:
        app: my-app

maxSkew: 1: This means the difference in the number of pods between any two zones can't be more than 1. (e.g., 1-1-1 is fine, 2-1-0 is not).


How to check if Karpenter is deployed and operational in the cluster?

To verify that Karpenter is correctly configured and operational in our EKS cluster, we should follow validation steps described below.

1. Check Controller Health


a) Check Pod Status


Ensure the Karpenter controller pods are running without errors in the dedicated namespace (usually kube-system or karpenter).

We know that its pods should be installed in kube-system namespace and that they should have label app.kubernetes.io/name=karpenter so we can filter pods by these two criterias:

% kubectl get pods -n kube-system -l app.kubernetes.io/name=karpenter
NAME                         READY   STATUS    RESTARTS   AGE
karpenter-598976645b-96dps   1/1     Running   0          11h
karpenter-598976645b-nxm24   1/1     Running   0          12h

b) Inspect Logs


To watch for successful discovery of our cluster endpoint and region use:

% kubectl logs -f -n kube-system -l app.kubernetes.io/name=karpenter -c controller

-f = follow (command does not return)
- l = logs from objects with specified label
-c = only logs from specified container


To verify successful discovery of our EKS cluster endpoint and region, we should look for specific initialisation and informer messages in the Karpenter controller logs. 

Key Success Indicators

When Karpenter starts, it must connect to the AWS EKS API to "describe" the cluster. Look for these signs in the output of kubectl logs -n kube-system -l app.kubernetes.io/name=karpenter -c controller: 

  • "Starting informers...": This indicates Karpenter has successfully authenticated with the Kubernetes API server and is beginning to watch for unschedulable pods.
  • Absence of "DescribeCluster" Errors: If discovery is working, we will not see errors like failed to detect the cluster CIDR or AccessDeniedException: ... eks:DescribeCluster.
  • Region and Cluster Verification: In newer versions, Karpenter logs its configuration during startup. Look for a log entry mentioning the cluster name and AWS region we provided in our Helm values. 
Example log:

{"level":"DEBUG","time":"2026-03-11T01:10:28.203Z","logger":"controller","caller":"operator/operator.go:132","message":"discovered karpenter version","commit":"1c39126","version":"1.3.2"}

{"level":"DEBUG","time":"2026-03-11T01:10:28.461Z","logger":"controller","caller":"operator/operator.go:124","message":"discovered region","commit":"1c39126","region":"us-east-1"}

{"level":"DEBUG","time":"2026-03-11T01:10:28.749Z","logger":"controller","caller":"operator/operator.go:129","message":"discovered region","commit":"1c39126","region":"us-east-1"}

{"level":"DEBUG","time":"2026-03-11T01:10:28.909Z","logger":"controller","caller":"operator/operator.go:135","message":"discovered cluster endpoint","commit":"1c39126","cluster-endpoint":"https://CA0xxxxxxx5FDD.yxx.us-east-1.eks.amazonaws.com"}

{"level":"DEBUG","time":"2026-03-11T01:10:28.914Z","logger":"controller","caller":"operator/operator.go:143","message":"discovered kube dns","commit":"1c39126","kube-dns-ip":"172.20.0.10"}

{"level":"INFO","time":"2026-03-11T01:10:28.948Z","logger":"controller.controller-runtime.metrics","caller":"server/server.go:208","message":"Starting metrics server","commit":"1c39126"}

{"level":"INFO","time":"2026-03-11T01:10:28.948Z","logger":"controller","caller":"manager/runnable_group.go:226","message":"starting server","commit":"1c39126","name":"health probe","addr":"[::]:8081"}

{"level":"INFO","time":"2026-03-11T01:10:28.950Z","logger":"controller.controller-runtime.metrics","caller":"server/server.go:247","message":"Serving metrics server","commit":"1c39126","bindAddress":":8080","secure":true}

{"level":"INFO","time":"2026-03-11T01:10:29.052Z","logger":"controller","caller":"leaderelection/leaderelection.go:215","message":"attempting to acquire leader lease kube-system/karpenter-leader-election...","commit":"1c39126"}

{"level":"DEBUG","time":"2026-03-11T06:00:19.215Z","logger":"controller","caller":"provisioning/provisioner.go:128","message":"computing scheduling decision for provisionable pod(s)","commit":"1c39126","controller":"provisioner","namespace":"","name":"","reconcileID":"921af0a4-f057-4041-bff5-d1861d9f72d1","pending-pods":1,"deleting-pods":0}

{"level":"DEBUG","time":"2026-03-11T06:00:21.223Z","logger":"controller","caller":"provisioning/provisioner.go:128","message":"computing scheduling decision for provisionable pod(s)","commit":"1c39126","controller":"provisioner","namespace":"","name":"","reconcileID":"9f0c7833-8e01-4661-8728-890f0001a634","pending-pods":1,"deleting-pods":0}

{"level":"INFO","time":"2026-03-11T06:00:29.230Z","logger":"controller","caller":"lifecycle/controller.go:148","message":"initialized nodeclaim","commit":"1c39126","controller":"nodeclaim.lifecycle","controllerGroup":"karpenter.sh","controllerKind":"NodeClaim","NodeClaim":{"name":"xxxx-ms587"},"namespace":"","name":"xxxxx","reconcileID":"35624d4f-833a-4939-9785-24df4c975e0e","provider-id":"aws:///us-east-1c/i-0123456df20484e26","Node":
{"name":"ip-10-1-46-231.us-east-1.compute.internal"},"allocatable":{"cpu":"3920m","ephemeral-storage":"192128045146","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"15147932Ki","pods":"58"}}

{"level":"DEBUG","time":"2026-03-11T06:00:29.741Z","logger":"controller","caller":"disruption/controller.go:99","message":"marking consolidatable","commit":"1c39126","controller":"nodeclaim.disruption","controllerGroup":"karpenter.sh","controllerKind":"NodeClaim","NodeClaim":{"name":"xxxx-ms587"},"namespace":"","name":"xxxx-ms587","reconcileID":"8c5c3d20-36eb-4a78-b0e8-792532db530d","Node":{"name":"ip-10-2-45-230.us-east-1.compute.internal"}}

{"level":"INFO","time":"2026-03-11T06:01:46.399Z","logger":"controller","caller":"disruption/controller.go:193","message":"disrupting node(s)","commit":"1c39126","controller":"disruption","namespace":"","name":"","reconcileID":"acc96c52-0cda-475f-b8a9-1251e7a98dc1","command-id":"3fa9d95e-8f45-48a9-b524-94786e1ac91a","reason":"empty","decision":"delete","disrupted-node-count":1,"replacement-node-count":0,"pod-count":0,"disrupted-nodes":[{"Node":{"name":"ip-10-2-45-230.us-east-1.compute.internal"},"NodeClaim":{"name":"xxxx-ms587"},"capacity-type":"on-demand","instance-type":"m5.xlarge"}],"replacement-nodes":[]}


Common Error Patterns to Watch For

If discovery fails, the logs will explicitly mention connectivity or permission issues:
  • DNS/Endpoint Issues: Look for i/o timeout or lookup sts.<region>.amazonaws.com. This often means Karpenter can't reach the AWS STS endpoint to get credentials.
  • IAM Permission Issues: Messages stating is not authorized to perform: eks:DescribeCluster mean the controller's IAM role (IRSA) is missing the necessary permissions to discover the cluster details.
  • Controller Crash/Restart: If the logs show repeated restarts right after "Starting informers", it often points to a mismatch between the provided clusterName and the actual cluster. 

Tip: Enable Debug Logging 

If we don't see enough detail, we can increase the log verbosity. Update our Helm deployment with --set logLevel=debug or change the LOG_LEVEL environment variable in the deployment to debug


2. Verify CRD Configurations 


Karpenter requires specific Custom Resource Definitions (CRDs) to know how to provision nodes. 

(1) List NodePools: Run kubectl get nodepools to ensure our provisioning logic is active.



(2) List EC2NodeClasses: Run kubectl get ec2nodeclasses to confirm AWS-specific settings (like subnets and security groups) are defined. 


3. Perform a Scaling Test ("Inflate" Test) 


The standard way to test Karpenter is by deploying a "dummy" workload that exceeds current cluster capacity. 

(1) Deploy a test app: Apply a deployment (often called inflate) with high CPU/Memory requests.

(2) Scale it up: Run: 

% kubectl scale deployment inflate --replicas=5

(3) Watch for new nodes: Monitor:

% kubectl get nodes -w

If configured correctly, Karpenter will detect the pending pods and provision a new EC2 instance within about a minute. 

During the inflate scaling test, how to know that a new node was provisioned by karpenter and not cluster autoscaler?


During an inflate scaling test, we can distinguish between nodes provisioned by Karpenter and those from Cluster Autoscaler (CAS) by checking for specific labels, console status, and controller logs. 

1. Check for Specific Kubernetes Labels 

Karpenter automatically injects unique labels into every node it creates. CAS nodes usually belong to an Auto Scaling Group (ASG) and do not have these specific Karpenter markers. 

Run this command to see the labels on our nodes:

kubectl get nodes --show-labels 

Look for these Karpenter-exclusive labels:
  • karpenter.sh/nodepool: The name of the NodePool that provisioned the node.
  • karpenter.sh/capacity-type: Set to spot or on-demand.
  • karpenter.k8s.aws/instance-category: (e.g., c, m, r). 

Nodes provisioned by Cluster Autoscaler (CAS) don't have a unique "CAS" label. Instead, they carry labels that identify them as members of an Auto Scaling Group (ASG) or an EKS Managed Node Group (MNG).

If we are looking at a node and trying to confirm if it came from CAS, look for these specific markers:

1. Managed Node Group Labels (Most Common)

If we use EKS Managed Node Groups with CAS, the nodes will always have:
  • eks.amazonaws.com/nodegroup: The name of the MNG.
  • eks.amazonaws.com/nodegroup-image: The AMI ID used.
  • eks.amazonaws.com/capacityType: Usually ON_DEMAND or SPOT
  • eks.amazonaws.com/sourceLaunchTemplateId: 
  • eks.amazonaws.com/sourceLaunchTemplateVersion: 

2. Auto Scaling Group Labels

Since CAS works by increasing the "Desired Capacity" of an ASG, the underlying EC2 instance is tagged by AWS. Kubernetes reflects these as:
  • alpha.eksctl.io/nodegroup-name: (If using eksctl)
  • node.kubernetes.io/instance-type: (Standard, but CAS uses this to match ASG definitions)

3. The "Missing" Labels

The easiest way to identify a CAS node during a Karpenter test is by what it doesn't have. A CAS node will NEVER have:
  • ❌ karpenter.sh/nodepool
  • ❌ karpenter.sh/provisioner-name (deprecated)
  • ❌ karpenter.k8s.aws/instance-category

Quick Check Command

Run this to see which nodes belong to Karpenter vs. CAS/MNG:

kubectl get nodes -L karpenter.sh/nodepool,://eks.amazonaws.com

If the nodepool column is populated, it's Karpenter.
If the nodegroup column is populated, it's CAS/MNG.


2. Identify via "Self-Managed" Status in EKS Console 

In the AWS EKS Console under the Compute tab: 
  • Karpenter Nodes: Appear as "Self-managed" because Karpenter bypasses Auto Scaling Groups to launch instances directly via the EC2 Fleet API.
  • Cluster Autoscaler Nodes: Appear as part of a "Managed Node Group" or are tied to a specific ASG. 

3. Check for the NodeClaim Object 

Karpenter creates a NodeClaim for every node it provisions. Cluster Autoscaler does not use this resource. Run

kubectl get nodeclaims 

...during the test. If we see new entries appearing that correspond to our inflate pods, Karpenter is doing the work. 

4. Monitor Controller Logs

We can watch Karpenter’s real-time decision-making process by tailing its logs. It will explicitly state when it discovers unschedulable pods and which instance type it is launching. 

kubectl logs -n kube-system -l app.kubernetes.io/name=karpenter -f

CAS logs, by contrast, will show it interacting with ASGs and increasing the "desired capacity" of a group


How to disable Cluster Autoscaler temporarily to ensure Karpenter is the only one responding to our tests?


To ensure Karpenter is the only controller responding to our scaling tests, we can temporarily disable the Cluster Autoscaler (CAS) by scaling its deployment to zero replicas. 

1. Identify the CAS Deployment

The Cluster Autoscaler typically runs in the kube-system namespace. Verify its name first: 

kubectl get deployments -n kube-system | grep cluster-autoscaler

2. Scale to Zero

Run the following command to stop the CAS from running. This will terminate the pod responsible for monitoring the cluster and scaling our Auto Scaling Groups (ASGs): 

kubectl scale deployment cluster-autoscaler -n kube-system --replicas=0


3. Verify the Shutdown

Ensure no CAS pods are running to prevent them from interfering with our inflate test:

kubectl get pods -n kube-system -l app.kubernetes.io/name=aws-cluster-autoscaler

4. (Optional) Remove ASG Tags

If we want a more permanent "hard" disable without deleting the deployment, we can remove the specific AWS tags from our Auto Scaling Groups that the CAS uses for auto-discovery:
  • k8s.io/cluster-autoscaler/enabled
  • k8s.io/cluster-autoscaler/<cluster-name> 

Without these tags, the CAS will ignore those node groups even if the deployment is scaled back up. 

To Re-enable

Once our tests are complete, we can restore the Cluster Autoscaler by scaling it back to its original replica count:

kubectl scale deployment cluster-autoscaler -n kube-system --replicas=1


To confirm which instance types Karpenter chose during our inflate test, we can watch the controller logs in real-time. Karpenter will log exactly how it batches our pods and which instances it requests from AWS.

1. Tail Karpenter Logs

Run the following command while our inflate pods are in a Pending state:

kubectl logs -f -n kube-system -l app.kubernetes.io/name=karpenter

Note: Some installations use the karpenter namespace instead of kube-system. 

2. What to Look For

Karpenter logs its decisions in JSON or text format. Look for INFO messages containing found provisionable pod(s) or created nodeclaim.

A typical log entry looks like this:

2024-03-12T10:00:00.000Z INFO controller.provisioner created nodeclaim {"commit": "...", "nodeclaim": "default-abc12", "nodepool": "general-purpose", "requests": {"cpu":"4","memory":"8Gi"}, "instance-types": "m5.xlarge, m6i.xlarge, c5.2xlarge..."}

  • requests: Shows the total CPU/Memory requested by our inflate pods.
  • instance-types: Lists the candidates Karpenter passed to the EC2 Fleet API. Karpenter usually sends a diversified list (up to 60 types) to ensure high availability and best pricing. 

3. Identify the Winning Instance

Once the node is live, we can confirm the final selected type by checking the NodeClaim status:

kubectl get nodeclaims -o custom-columns=NAME:.metadata.name,TYPE:.status.instanceType,ZONE:.status.zone

This command explicitly shows the specific instance type (e.g., m5.large) that AWS actually provisioned for that claim. 

4. Enable Debug Logging (Optional) 

If we don't see enough detail, we can increase the log verbosity to debug. This will reveal exactly how Karpenter "binpacks" our pods onto different instance options. 

kubectl patch deployment karpenter -n kube-system --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/env/0/value", "value": "debug"}]'

Tip: After our test, remember to set the LOG_LEVEL back to info to avoid excessive log storage costs.

4. Monitor NodeClaims


Karpenter uses NodeClaims to manage the lifecycle of the nodes it creates. 
  • Check Status: Run kubectl get nodeclaims to see if requests for new nodes are being fulfilled.
  • Debug Failures: If a node fails to join, use kubectl describe nodeclaim <name> to check for errors like LaunchFailed or issues with IAM permissions. 

5. Verify Tagging


Karpenter discovers infrastructure via tags. Ensure our Subnets and Security Groups have the following tag: 
Key: karpenter.sh/discovery
Value: <our-cluster-name>


How to know if node was provisioned by Karpenter?


Karpenter applies labels on nodes it provisions so let's check labels:

% kubectl get nodes --show-labels

If labels like karpenter.sh/nodepool or karpenter.sh/provisioner-name exist, Karpenter launched the node.


When setting up EKS Managed Node Group, we set desired, min and max number of nodes for ASG. Does that enable Cluster Autoscaler automatically? How does ASG play with Karpenter?


Setting the desired, min, and max size on an EKS Managed Node Group only configures the underlying AWS Auto Scaling Group (ASG).
  • What AWS does: If a node crashes, the ASG will see that the "current" count is less than the "min" (or "desired") and spin up a new node to replace it.
  • What AWS does NOT do: It will not look at your pending Kubernetes pods and say, "Oh, we need more space, let's increase the count from 3 to 4."
To get that "intelligent" scaling based on pod demand, we must install a separate controller.

Is Cluster Autoscaler (CAS) enabled by default?

No. Kubernetes Cluster Autoscaler is not enabled by default on EKS.

If we want to use it, we must:
  • Deploy the Cluster Autoscaler as a Pod in our cluster (usually via Helm).
  • Give that Pod an IAM Role (IRSA) that has permission to update your ASG's desired_capacity.
  • Add specific tags to our Node Group so the Autoscaler knows which ASG to "manage."

Do we need to disable CAS to use Karpenter?


Yes, absolutely. We should not run Cluster Autoscaler and Karpenter simultaneously on the same nodes.
  • The Conflict: CAS tries to scale nodes by changing the "desired capacity" of an ASG. Karpenter works differently—it bypasses ASGs entirely and talks directly to the EC2 Fleet API to launch specific instances.
  • The Result of Running Both: They will fight over the cluster. CAS might try to shrink a group while Karpenter is trying to add capacity, leading to "flapping" nodes and unpredictable costs.

If we switch to Karpenter:
  • Uninstall/Scale down the Cluster Autoscaler deployment.
  • Set our Node Group sizes to fixed values (or migrate to "headless" node groups where Karpenter manages the entire lifecycle).
  • Karpenter is the "New Way": Most AWS users are moving toward Karpenter because it is faster (seconds vs minutes) and more efficient at picking the right instance sizes.

Summary Comparison


Feature   ASG (Default)             Cluster Autoscaler (CAS)              Karpenter
---------   ------------------             -------------------------------              ------------
Logic      "Keep X nodes alive"  "Add nodes if Pods are Pending"  "Provision exactly what Pods need"
Speed      Slow (Health-based)   Medium (Polling ASG)                  Fast (Direct EC2 API)
Setup      Built-in to EKS            Manual Install + IAM                    Manual Install + IAM
Best for   Fixed capacity             Traditional scaling                         Cost-optimization & high speed


Updating Karpenter Version


Karpetner version is compatible only to certain k8s versions. If we plan to upgrade k8s version in the cluster where Karpenter operates, we need first to check if current Karpenter version is compatible with the new k8s version. If it's not, we first need to upgrade Karpenter.

Karpenter upgrade boils down to updating Karpenter contoller (which runs in its pods which are deployed on static nodes - provisioned manually or controlled via Cluster Autoscaler). If Karpenter is deployed via Helm chart, we might only need to change the version there. 


Explain why karpenter wants to do rolling update of nodes in its nodepool after karpenter controller version is bumped up? What runs on those nodes that needs update? Some karpenter agents?                                                                                 

There's no Karpenter agent on the nodes. The replacement isn't about updating something running on them. Karpenter has no per-node component. Karpenter runs only as a central controller Deployment (e.g. 2 replicas on the static nodes). Unlike aws-node/kube-proxy/ebs-csi-node (which are DaemonSets on every node), Karpenter puts nothing on the worker nodes it provisions. Node lifecycle — interruption handling, health, termination — is all done centrally by the controller via the EC2 APIs and the interruption SQS queue, not by an on-node daemon. So when the controller version bumps, there is literally no Karpenter binary on the nodes that needs upgrading.

So why does it want to replace them? → Drift + immutable nodes


Karpenter's model is immutable infrastructure: it never reconfigures a running node in place. If a node's actual configuration no longer matches the desired configuration, Karpenter's remedy is always the same — provision a new node, drain the old one. This disruption mode is called Drift (the others are Consolidation, Expiration, Interruption).

How it detects drift: for every node it owns (each NodeClaim), Karpenter computes a hash of the desired config derived from the NodePool + EC2NodeClass specs, and stores it as annotations on the node/NodeClaim:
  karpenter.sh/nodepool-hash            + karpenter.sh/nodepool-hash-version
  karpenter.k8s.aws/ec2nodeclass-hash   + karpenter.k8s.aws/ec2nodeclass-hash-version
  
On each reconcile it recomputes the hash and compares. Mismatch → node marked Drifted → replace.

Why the v1.12 controller bump specifically triggers it


Two ways a controller upgrade can change the computed hash even when you didn't touch your specs:

1. The hashing logic itself changes. v1.12 "adds support for drift on CA-bundle" — it changed what goes into the hash (now includes the node's CA bundle). So the moment the new controller reconciles, every existing node's stored hash (computed by 1.3.2) differs from the
  freshly computed one → all flagged drifted. That's exactly the release note you saw.

2. New default fields/behaviour in the spec that old nodes were created without.
  
Important nuance: Karpenter tries not to churn on every upgrade. The *-hash-version annotations exist precisely so that when only the hashing scheme version bumps (but your effective config is unchanged), the controller silently re-stamps the stored hash instead of drifting. Many controller upgrades cause zero node replacement for this reason. v1.12 is a deliberate exception because the inputs to the hash changed, not just the version — so it re-drifts everything once.

  What actually lives on those nodes (i.e. what would need updating)


  - kubelet / container runtime / OS → come from the AMI. These only change when the node is replaced with a newer AMI — that's the 1.34 upgrade concern, not the Karpenter bump.
  - DaemonSets (VPC CNI, kube-proxy, EBS CSI node, alloy/node-exporter) → owned by their own addons/controllers, not Karpenter.
  - Karpenter → nothing.

So for the Karpenter controller bump, the node replacement carries no payload — it's pure bookkeeping: re-creating the node so its stored config hash matches the new controller's view.

  Why this validates our plan

Because the drift is just bookkeeping (the nodes are functionally fine on 1.3.2's config), we don't have to act on it immediately. Setting the NodePool disruption budget to nodes: "0" lets the controller upgrade land while the drift sits pending and harmless — then we satisfy that drift once, together with the actual 1.34 AMI change in xxx, instead of churning the ES nodes twice for no functional gain.



Updating Kubernetes version on nodes managed by Karpenter



References:



Posted by at No comments:
Labels: , , , , , , ,
Subscribe to: Posts (Atom)