Friday, 11 September 2026

Kubernetes Resource Sizing Cheat Sheet

Friday, 7 August 2026

Splitting One Domain Across GitHub Pages and Blogger: A Complete DNS Setup Guide

If you own a single domain but want to run two separate things on it — say, a static landing page on GitHub Pages at the root domain, and a blog on Blogger at a subdomain — you'll quickly discover that the actual mechanics of DNS, custom domain verification, and HTTPS certificate provisioning are scattered across three different systems that don't talk to each other and don't always explain their errors clearly.

This post documents the full setup, end to end: what records you need, what each provider actually requires, and the specific errors you're likely to hit along the way (all of which I hit myself while setting this up).

The goal, concretely:

example.com (the apex/root domain) → a static site hosted on GitHub Pages
blog.example.com (a subdomain) → a Blogger blog, with its original xxxx.blogspot.com address redirecting to it
www.example.com → optionally also works, redirecting to the apex

I'll use bojankomazec.com as the running example, since that's the domain I set this up on.

1. The Big Picture: Three Systems, One Domain

Before touching any settings, it helps to understand the three layers involved and what each one owns:

Layer	What it controls	Where you configure it
DNS registrar (e.g. GoDaddy, Namecheap, Cloudflare)	Which records exist for your domain — A, CNAME, MX, etc. This is the actual "phone book" that tells the internet where to send traffic.	Your registrar's DNS management panel
GitHub Pages	Which domain(s) it will accept and serve your static site for, and whether it issues an HTTPS certificate for them	Repository → Settings → Pages
Blogger	Which domain(s) it will accept and serve your blog for, and whether it issues an HTTPS certificate for them	Blogger → Settings → Publishing

The critical thing to understand: DNS records alone are not enough. Both GitHub Pages and Blogger require you to also tell them, inside their own settings, which custom domain you're using. If DNS points to them but their own settings don't know about your domain, you'll get connection errors — not because DNS is wrong, but because the receiving service doesn't recognize the incoming hostname and refuses it.

This is the single most common source of confusion in this kind of setup: DNS can be 100% correct and the site can still fail to load, because DNS and application-level domain configuration are two separate steps that both have to be done.

2. DNS Records You'll Need

Here's the full set, using bojankomazec.com as the example domain, blog as the subdomain, and bojankomazec as the GitHub username (so the Pages repo is bojankomazec.github.io).

Apex domain → GitHub Pages

DNS doesn't allow a CNAME record at the apex/root of a domain (technically, because the apex must also hold other record types like SOA and often MX, and a hostname can't mix CNAME with anything else). So GitHub Pages publishes a fixed set of IP addresses, and you point the apex at those directly with A records:

Type: A    Host: @    Value: 185.199.108.153
Type: A    Host: @    Value: 185.199.109.153
Type: A    Host: @    Value: 185.199.110.153
Type: A    Host: @    Value: 185.199.111.153

(@ is the conventional way most DNS panels represent "the apex/root domain itself.") If your provider supports IPv6 and you want dual-stack support, GitHub also publishes AAAA records:

Type: AAAA    Host: @    Value: 2606:50c0:8000::153
Type: AAAA    Host: @    Value: 2606:50c0:8001::153
Type: AAAA    Host: @    Value: 2606:50c0:8002::153
Type: AAAA    Host: @    Value: 2606:50c0:8003::153

Important: check whether your registrar auto-created a placeholder A record for @ when the domain was registered (many do, often pointing to a parking page). If so, delete it — a leftover default record will conflict with the four you're adding.

www subdomain → GitHub Pages (optional)

If you also want www.example.com to work (redirecting to the apex once GitHub verifies both), add:

Type: CNAME    Host: www    Value: bojankomazec.github.io.

Note this points to your GitHub Pages hostname (<username>.github.io), not to a URL, and not to the repo's github.com page. A CNAME value is always a hostname, never a full URL with https:// or a path.

Blog subdomain → Blogger

Type: CNAME    Host: blog    Value: ghs.google.com.

This is the one that trips people up: it's tempting to point this directly at your xxxx.blogspot.com address, since that's "where the blog lives." Don't do that. Blogger's custom-domain routing works through a shared Google endpoint (ghs.google.com) that inspects the incoming hostname and routes it to the correct blog based on what's configured in your Blogger account. Pointing the CNAME straight at your blogspot address sends the request to blogspot-specific infrastructure that doesn't know how to handle a request arriving under a different hostname — the connection gets abruptly closed rather than serving anything (ERR_CONNECTION_CLOSED in Chrome).

Blogger domain verification CNAME

Separately from the routing CNAME above, Blogger needs to confirm you actually own the subdomain before it'll accept it. When you enter your custom domain into Blogger's settings, it will generate a unique verification CNAME — something like:

Type: CNAME    Host: <random-string>    Value: gv-<random-string>.dv.googlehosted.com.

This value is generated per-domain and per-account — you can't predict or reuse it, you have to copy it exactly from Blogger's settings screen after entering your custom domain. Add it as its own DNS record alongside the routing CNAME.

Gotcha: if you ever change which subdomain Blogger is using (e.g. you originally set it up as www.example.com and later switch to blog.example.com), the old verification CNAME becomes stale and should be deleted — Blogger will issue a new one specific to the new subdomain when you re-save.

Full example record set

A       @        185.199.108.153
A       @        185.199.109.153
A       @        185.199.110.153
A       @        185.199.111.153
CNAME   www      bojankomazec.github.io.
CNAME   blog     ghs.google.com.
CNAME   <verify> gv-xxxxxxxxxxxx.dv.googlehosted.com.

3. The GitHub Pages Side

In your <username>.github.io repository, go to Settings → Pages.
Under Custom domain, enter your apex domain (e.g. bojankomazec.com — no www, no https://).
Save. GitHub will write a file literally named CNAME (no extension) into the root of your repository, containing just the domain name. This file is how GitHub Pages knows which domain to serve your content for — DNS alone doesn't tell it this.
GitHub will attempt to verify your DNS. If your A records are already correct, this usually succeeds quickly. If you see a DNS valid for primary message alongside a separate www.example.com is improperly configured warning — that's normal. The www check is a secondary, automatic check GitHub runs to see if it can also offer www support and a unified certificate; it doesn't block your primary domain from working. If your primary domain error clears and only the www one remains, your site is live.
Once verified, tick Enforce HTTPS (this option is often greyed out until GitHub finishes issuing a certificate — see the HTTPS section below).

4. The Blogger Side

In Blogger, go to Settings → Publishing.
Under Custom domain, enter your blog subdomain (e.g. blog.bojankomazec.com).
Save. Blogger will likely reject the save the first time and show you the verification CNAME described above. Copy it exactly.
Add both the routing CNAME (ghs.google.com) and the verification CNAME to your DNS.
Wait for propagation (see below), then go back and save again in Blogger. It should now accept the domain.
Check the Redirect domain setting on the same page carefully — see the dedicated warning about this below, it's a common trap.
Look for an HTTPS availability toggle further down the page. It usually reads "Status: Unknown" for a while after you first set up the domain — this is Blogger's certificate issuance still in progress, not an error (see HTTPS section).

The "Redirect domain" trap

Blogger's Publishing settings include a "Redirect domain" option, something like "Redirect example.com to blog.example.com." This setting, if enabled, makes Blogger redirect your apex domain to your blog. If your apex domain is meant to serve something else entirely (like a GitHub Pages landing page, as in this setup), this option will silently hijack it — visitors to example.com will get redirected to your blog instead of seeing your landing page, even though your DNS and GitHub Pages configuration are both completely correct.

This setting can also change target without obviously telling you: if you initially set up Blogger's custom domain as www.example.com and later switch it to blog.example.com, the "Redirect domain" checkbox may still be checked, just silently re-pointed at whatever your current custom domain is. If you don't want your apex domain touched by Blogger at all, make sure this is switched off, and re-check it any time you change Blogger's custom domain setting.

5. HTTPS and Certificates: Why It Takes a While

Both GitHub Pages and Blogger use automated certificate issuance (GitHub uses Let's Encrypt; Google's infrastructure handles Blogger's) that kicks in after your domain passes DNS verification — not before, and not instantly.

What this means in practice:

Your site can be fully reachable over plain http:// while https:// still fails, sometimes with a confusing certificate error, for anywhere from a few minutes to around 24–48 hours after you first set up or change a custom domain.
The specific error you'll likely see in Chrome during this window is something like:

  Your connection is not private
  net::ERR_CERT_COMMON_NAME_INVALID
  This server could not prove that it is www.example.com;
  its security certificate is from *.github.io

This happens because the connection is reaching GitHub's infrastructure, but GitHub hasn't yet issued a certificate specifically covering your hostname, so it falls back to serving its generic wildcard certificate — which your browser correctly flags as invalid for your domain.

Don't repeatedly remove and re-add the custom domain while waiting — each change can reset the certificate issuance queue and make the wait longer, not shorter.
Once issuance completes, https:// starts working with no further action needed on your part.

A red herring worth knowing about: antivirus HTTPS scanning

If you inspect your site's certificate in Chrome and see something unexpected — like the certificate authority being your antivirus software (Avast, Kaspersky, and others do this) rather than Let's Encrypt or Google — this is not a sign of a compromised or misconfigured site. Many antivirus suites include an "HTTPS scanning" or "Web Shield" feature that intercepts encrypted connections locally, decrypting and re-encrypting them with a certificate the antivirus generated and installed into your system's trust store, in order to scan traffic for threats.

This substitution happens for every HTTPS site you visit, not just yours, so it's not diagnostic of anything wrong with your setup. To see the actual certificate your host issued, either temporarily disable the antivirus's HTTPS scanning feature, or check from an external tool that doesn't go through your local machine at all — such as SSL Labs' SSL Test, which queries your server directly.

6. Troubleshooting Toolkit

A handful of tools make this whole process far less guess-and-check:

dig <domain> <record-type> (Mac/Linux, or Windows via WSL) — queries DNS directly and shows you exactly what's published, bypassing any local caching. E.g. dig bojankomazec.com A or dig blog.bojankomazec.com CNAME.
nslookup <domain> — similar, built into Windows natively.
dnschecker.org — checks DNS propagation across many resolvers worldwide simultaneously, which is invaluable for figuring out whether a record change has actually rolled out everywhere yet, or is still mid-propagation.
SSL Labs SSL Test — inspects a site's actual certificate chain independent of your local browser/antivirus interference.

General debugging approach when something doesn't load:

Confirm the DNS record itself is correct and propagated (dnschecker.org).
Confirm the receiving service (GitHub Pages / Blogger) has the domain configured in its own settings, not just DNS.
Try plain http:// before https:// — this isolates whether the problem is routing/configuration (affects both) or just certificate issuance (only affects https).
If a certificate error shows an unexpected issuer (like an antivirus name instead of Let's Encrypt/Google), suspect local HTTPS interception before suspecting your DNS setup.

7. Common Errors and What They Actually Mean

Error	What it usually means
`InvalidCNAMEError` (GitHub)	The named subdomain's CNAME doesn't point where GitHub expects, or DNS hasn't propagated to the resolver GitHub's checker used yet. Verify with dnschecker.org before assuming misconfiguration.
`NotServedByPagesError` (GitHub)	DNS looks fine, but the domain isn't yet resolving to GitHub Pages from GitHub's own check — often transient/propagation-related.
`ERR_CERT_COMMON_NAME_INVALID`	The connection reached the right server, but no certificate has been issued for this specific hostname yet — a provisioning delay, not a misconfiguration.
`ERR_CONNECTION_CLOSED` (on a Blogger subdomain)	Usually means the CNAME points somewhere that doesn't recognize the incoming hostname — most often, pointing directly at a `.blogspot.com` address instead of `ghs.google.com`.
A domain unexpectedly redirecting somewhere else entirely	Check Blogger's "Redirect domain" setting — it may be silently redirecting your apex domain to your blog.
Certificate issuer shown as an antivirus company	Local HTTPS scanning/interception by your antivirus software — check via an external tool instead.

8. Summary Checklist

Four A records for the apex domain, pointing at GitHub Pages' published IPs
(Optional) CNAME for www pointing at <username>.github.io
CNAME for your blog subdomain pointing at ghs.google.com (not directly at your .blogspot.com address)
Blogger's unique verification CNAME, copied exactly from its settings screen
Custom domain entered in GitHub Pages settings (creates the repo's CNAME file)
Custom domain entered in Blogger's Publishing settings
Blogger's "Redirect domain" setting switched off, unless you deliberately want your apex redirected to your blog
Patience for HTTPS certificate issuance (minutes to ~48 hours) on both platforms
Verify using http:// first, then https://, and cross-check unexpected certificate issuers against local antivirus software before assuming a server-side problem

Once all of this settles, you end up with a clean split: your apex domain serving whatever static site or landing page you want full control over, and a subdomain serving your blog — both fully on their own custom domains, both secured with HTTPS, and both surviving independently of where the underlying content is actually hosted.

Wednesday, 5 August 2026

kube-state-metrics

kube-state-metrics (KSM) is an official Kubernetes add-on agent that listens to the Kubernetes API server and generates Prometheus-format metrics about the state of Kubernetes objects (such as Pods, Deployments, Nodes, StatefulSets, and CronJobs).

Unlike agent metrics collectors like cAdvisor or node_exporter, KSM does not measure resource usage (like CPU usage, memory consumption, or disk I/O). Instead, it translates the raw state of objects in the Kubernetes API into structured metrics.

Core Concept: Usage vs. State

cAdvisor / Node Exporter (Usage): "How much CPU is Pod X consuming right now?"

kube-state-metrics (State): "How many replicas are ready in Deployment Y?", "When did CronJob Z last succeed?", "What phase is Pod A in?"

How kube-state-metrics Exposes Metrics

┌─────────────────────────┐

│ Kubernetes API Server │

└────────────┬────────────┘

│ Watch / List (Informer Cache)

▼

┌─────────────────────────┐

│ kube-state-metrics │ ─── (Generates OpenMetrics in-memory)

└────────────┬────────────┘

│ HTTP GET /metrics (Port 8080)

▼

┌─────────────────────────┐

│ Prometheus Server │

└─────────────────────────┘

1. Consuming the API via Kubernetes Informers

KSM connects directly to the Kubernetes API server using standard Client-GO Informers.

Instead of constantly polling the API server with heavy requests, KSM maintains an in-memory cache updated in real-time via long-polling watch streams.
When a resource changes (e.g., a Deployment scales down or a Pod enters CrashLoopBackOff), the local cache updates instantly.

2. Generating Prometheus OpenMetrics In-Memory

KSM parses the fields of the Kubernetes object specs and status subresources (such as .status.phase, .spec.replicas, .status.conditions) and converts them directly into Prometheus gauge/counter metrics.

3. Serving via HTTP Endpoint

KSM hosts a lightweight HTTP server (default port :8080 at path /metrics).

It does not store metrics over time, nor does it push metrics out.

When a Prometheus server scrapes KSM's /metrics endpoint, KSM reads its in-memory snapshot, formats the raw text payload, and streams it back.

Key Technical Properties

Property

Details

Port & Path

Port 8080 (/metrics for cluster state), Port 8081 (/metrics for KSM internal performance telemetry)

Data Format

Prometheus text format / OpenMetrics standard

Metric Types

Predominantly Gauge metrics representing instantaneous state (1 or 0 state representations)

Scaling

Supports Horizontal Sharding and resource filtering (--resources, --namespaces) for large clusters

Example Metric Output

A scrape request to kube-state-metrics for a Deployment produces plain-text metrics like this:

# HELP kube_deployment_spec_replicas The desired number of pods declared in the deployment spec.

# TYPE kube_deployment_spec_replicas gauge

kube_deployment_spec_replicas{namespace="default",deployment="api-server"} 3

# HELP kube_deployment_status_replicas_available The number of available pods created by the deployment.

# TYPE kube_deployment_status_replicas_available gauge

kube_deployment_status_replicas_available{namespace="default",deployment="api-server"} 2

Do we need kube-state-metrics if we have Prometheus deployed in the cluster?

Yes, you still need kube-state-metrics (KSM) even if Prometheus is deployed in your cluster.

Prometheus is the engine that collects, stores, and queries time-series data, while kube-state-metrics is the agent that generates metrics about the state of Kubernetes API objects.

Prometheus does not natively inspect Kubernetes API objects on its own—it relies on exporters like KSM to expose that data.

What Prometheus collects out-of-the-box vs. with KSM?

Metric Source

Responsible Agent

Examples of Metrics Provided

Node / OS Metrics

node_exporter

Node CPU utilization, RAM usage, disk space, network traffic.

Container Usage

cAdvisor (built into kubelet)

Container CPU throttling, memory usage (RSS), network bytes per container.

Control Plane

kube-apiserver, etcd, coredns

API latency, request counts, etcd commit durations.

Kubernetes State

kube-state-metrics

Deployment replica counts, CronJob last success times, Pod restart counts, pending PVCs, ingress statuses, TLS certificate secret expiration.

What happens without kube-state-metrics?

If you run Prometheus without KSM, you lose visibility into the high-level health and configuration of your cluster resources. You will not be able to:

Alert on Deployment Health: You won't know if spec.replicas (desired) doesn't match status.available_replicas.
Track Job / CronJob Success: Metrics like kube_cronjob_status_last_successful_time or kube_job_status_failed won't exist.
Monitor Pod Lifecycle States: You won't have metrics for Pods stuck in Pending, CrashLoopBackOff status codes, or ImagePullBackOff.
Track Resource Requests vs. Limits: You won't be able to compare requested CPU/memory (kube_pod_container_resource_requests) against node capacity to measure cluster overcommit.

How They Work Together

┌─────────────────────────────────────────────────────────┐

│ Kubernetes API │

└────────────────────────────┬────────────────────────────┘

│ Watch API Objects

▼

┌───────────────────────┐

│ kube-state-metrics │

└───────────┬───────────┘

│ Exposes /metrics

▼

┌───────────────────────┐

│ Prometheus Server │ ◄── (Scrapes KSM)

└───────────────────────┘

KSM listens to the API server and generates metrics representing resource states.
Prometheus scrapes KSM's /metrics endpoint along with cAdvisor, node-exporter, and your application endpoints.
Prometheus evaluates Alertmanager rules and stores the metrics for Grafana dashboards.

Standard Stack Setup

In standard Kubernetes monitoring deployments (such as the kube-prometheus-stack Helm chart or Prometheus Operator), kube-state-metrics is included by default as a core component alongside Prometheus and Alertmanager.

How to install kube-state-metrics?

kube-state-metrics (KSM) is typically deployed using one of four common approaches:

1. As Part of the kube-prometheus-stack (Most Common)

If you are setting up cluster monitoring with Prometheus and Grafana, kube-state-metrics is usually installed automatically as a bundled sub-chart.

Using the kube-prometheus-stack Helm chart:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

helm repo update

helm install prometheus-stack prometheus-community/kube-prometheus-stack

kube-state-metrics runs out of the box with ServiceMonitors already configured.

2. Standalone Helm Chart

If you already have a Prometheus instance or another monitoring agent (like Datadog, New Relic, or Grafana Alloy) and just need KSM running, you can deploy the official standalone Helm chart:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

helm repo update

helm install kube-state-metrics prometheus-community/kube-state-metrics --namespace kube-system

3. Native Kustomize / Standard Manifests (kubectl apply)

For gitops workflows or lightweight clusters where Helm isn't used, you can apply the official manifests maintained directly in the upstream Git repository using Kustomize:

git clone https://github.com/kubernetes/kube-state-metrics.git

cd kube-state-metrics

kubectl apply -k examples/standard

Note: This creates the necessary ServiceAccount, ClusterRole, ClusterRoleBinding, Deployment, and Service in the kube-system namespace.

4. Cloud Managed Kubernetes Add-ons

On managed platforms (like GKE, EKS, or AKS), kube-state-metrics is often enabled as a one-click add-on or managed component integrated into cloud-native observability services (e.g., Google Cloud Observability, AWS CloudWatch Container Insights, or Azure Monitor).

What Gets Created in the Cluster?

Regardless of how you deploy it, KSM will create:

ServiceAccount & ClusterRole/Binding: Grants read-only access (list, watch, get) to cluster API resources.
Deployment: Runs the kube-state-metrics container.
Service: Exposes the HTTP endpoint (usually on port 8080 at /metrics).

How to inspect which default metrics KSM exposes or how to disable unused ones?

Understanding which metrics kube-state-metrics (KSM) emits and how to prune unwanted metrics is key to controlling Prometheus ingestion volume and storage overhead.

Part 1: Inspecting Exposed Metrics

You can inspect the metrics exposed by KSM using three different methods:

1. Official Documentation Reference

The KSM GitHub repository contains generated documentation for every supported Kubernetes API group.

Standard Docs: kube-state-metrics/docs contains dedicated files for each resource (e.g., pod-metrics.md, cronjob-metrics.md).
Resource Status: The docs categorize metrics as STABLE (default), EXPERIMENTAL (alpha/beta features), or DEPRECATED.

2. Direct In-Cluster Scraping (curl / port-forward)

To see the exact payload KSM generates in your active cluster, port-forward to the KSM service and fetch /metrics:

# Port-forward the KSM service locally

kubectl port-forward svc/kube-state-metrics -n monitoring 8080:8080

# In a separate terminal, curl the metrics endpoint

curl http://localhost:8080/metrics

Filter for specific resource types using grep:

# View all Pod-related metrics exposed by KSM

curl -s http://localhost:8080/metrics | grep "^kube_pod_"

# View unique metric family names

curl -s http://localhost:8080/metrics | grep -v "^#" | cut -d'{' -f1 | sort -u

3. Prometheus Metric Explorer

In Prometheus, run the following query in the Expression Browser to list all distinct metric names harvested from the KSM job:

count by (__name__) ({job="kube-state-metrics"})

Part 2: Disabling or Filtering Unused Metrics

There are two primary ways to reduce metric volume: at the source (KSM Flags) or at the scraper (Prometheus Relabeling).

Method 1: Filtering at Source via KSM Flags (Recommended)

You can instruct KSM to only collect or expose specific resources or metric families using command-line arguments. This reduces both KSM memory utilization and network output.

--resources: Restricts KSM to watch specific Kubernetes objects only (e.g., ignore configmaps or secrets).

--metric-allowlist: Explicitly lists metric families to expose (all others are dropped).

--metric-denylist: Drops specific metric families while keeping the rest.

Setting via Helm (values.yaml):

# Helm configuration for prometheus-community/kube-state-metrics

resources:

# Watch only specific K8s API resources

resources:

- pods

- deployments

- statefulsets

- cronjobs

- nodes

# Disable specific metric families across enabled resources

extraArgs:

- --metric-denylist=kube_configmap_info,kube_secret_info,kube_pod_labels

Setting via Argo CD Application Manifest:YAMLapiVersion: argoproj.io/v1alpha1

kind: Application

metadata:

name: kube-state-metrics

namespace: argocd

spec:

# ... (repo configuration omitted)

source:

helm:

extraArgs:

- --resources=pods,deployments,cronjobs,nodes

- --metric-denylist=kube_pod_created,kube_pod_completion_time

Method 2: Dropping Metrics at Prometheus Ingestion

If you do not manage KSM CLI flags directly, configure Prometheus or Grafana Agent/Alloy to drop metrics during the scrape phase using metric_relabel_configs:

Prometheus CustomResource (Prometheus / ServiceMonitor):

apiVersion: monitoring.coreos.com/v1

kind: ServiceMonitor

metadata:

name: kube-state-metrics

namespace: monitoring

spec:

endpoints:

- port: http-metrics

metricRelabelings:

# Drop entire metric families matching a regex pattern

- action: drop

sourceLabels: [__name__]

regex: "(kube_configmap_.*|kube_secret_.*|kube_lease_.*)"

# Drop high-cardinality container state metrics if unused

- action: drop

sourceLabels: [__name__]

regex: "kube_pod_container_state_started"

kube_job_status_failed Source Filtering vs. Scraper DroppingFeatureKSM CLI Flags (--resources, --metric-denylist)Prometheus metric_relabel_configsKSM Memory UsageLower (KSM does not store or process ignored objects in memory)Unchanged (KSM processes objects regardless)Network TrafficLower (HTTP payload size is smaller)Unchanged (KSM transmits full payload; Prometheus discards post-fetch)Prometheus DB StorageLowerLowerConfig PlacementKSM Deployment manifestPrometheus scrape configuration

Prometheus Metrics Exposed by kube-state-metrics

kube_job_status_succeeded

kube_job_status_succeeded is a Prometheus metric exposed by kube-state-metrics that tracks the status of Kubernetes Jobs.

It returns a gauge value representing whether a Job execution succeeded, partitioned by label selectors.

Metric Breakdown

Metric Name: kube_job_status_succeeded
Type: Gauge
Value:

1: The Job has completed successfully.
0: The Job is in progress, failed, or has not succeeded.

Key Labels

Label Description

==== =========

job_name The name of the Kubernetes Job resource

namespace The namespace where the Job resides

condition Condition status (typically true when checking successful completion)

Common PromQL Queries

1. List All Currently Succeeded Jobs

kube_job_status_succeeded{condition="true"} == 1

2. Detect Jobs That Failed or Did Not Succeed

kube_job_status_succeeded{condition="true"} == 0

3. Alerting Rule: Job Failure

Alert when a Job has completed its execution attempt without succeeding:

- alert: KubernetesJobFailed

expr: kube_job_status_failed{condition="true"} == 1

for: 5m

labels:

severity: warning

annotations:

summary: "Kubernetes Job {{ $labels.namespace }}/{{ $labels.job_name }} failed"

4. Track Job Completion Rate Over Time

Calculate the rate of successful Job completions across a cluster:

sum(increase(kube_job_status_succeeded{condition="true"}[1h]))

kube_job_status_failed

kube_job_status_failed is a Prometheus metric exposed by kube-state-metrics that tracks whether a Kubernetes Job has failed to execute successfully.

It returns a gauge value representing the failure status of a Job, partitioned by label selectors.

Metric Breakdown

Metric Name: kube_job_status_failed
Type: Gauge
Value:

1: The Job reached its failure condition (e.g., exceeded backoffLimit or failed execution).
0: The Job has not failed (it is currently running, pending, or succeeded).

Key Labels

Label Description

==== =========

job_name The name of the Kubernetes Job resource

namespace The namespace where the Job resides

condition Condition status (typically true when evaluating an active failure state)

reason The reason for failure if populated (e.g., BackoffLimitExceeded, DeadlineExceeded)

Common PromQL Queries

1. List All Currently Failed Jobs

kube_job_status_failed{condition="true"} == 1

2. Count Failed Jobs by Namespace

sum by (namespace) (kube_job_status_failed{condition="true"} == 1)

3. Prometheus Alerting Rule for Job Failures

Alert when a Job has reached a failed condition and remained failed for 5 minutes:

- alert: KubernetesJobFailed

expr: kube_job_status_failed{condition="true"} == 1

for: 5m

labels:

severity: warning

annotations:

summary: "Kubernetes Job {{ $labels.namespace }}/{{ $labels.job_name }} failed"

description: "Job {{ $labels.job_name }} in namespace {{ $labels.namespace }} failed to complete."

4. Filter CronJobs / Generated Jobs by Prefix

If Jobs are generated dynamically by CronJobs, group or filter using regular expressions:

kube_job_status_failed{condition="true", job_name=~"nightly-backup-.*"} == 1

kube_cronjob_status_last_successful_time

kube_cronjob_status_last_successful_time is a gauge metric exposed by kube-state-metrics.

It records the Unix timestamp (in seconds) of when a Kubernetes CronJob last completed execution successfully.

Metric Details

Attribute:Value

Exporter: kube-state-metrics
Metric Type:Gauge
Value: Unix timestamp (seconds) or omitted if the CronJob has never completed
Labels: cronjob, namespace

Common PromQL Use Cases

1. Time Since Last Successful Run

Calculates how many seconds have elapsed since the CronJob last succeeded:

time() - kube_cronjob_status_last_successful_time

2. Alert on CronJob Failure or Missed Schedule

Alerts if the time since the last successful execution exceeds a target threshold (e.g., 1 day / 86,400 seconds):

(time() - kube_cronjob_status_last_successful_time) > 86400

3. Alerting when Schedule Run Failed

Compares the last scheduled time (kube_cronjob_status_last_schedule_time) against the last successful time. If last_schedule_time is greater than last_successful_time, the most recent run failed or is taking abnormally long:

kube_cronjob_status_last_schedule_time - kube_cronjob_status_last_successful_time > 0

4. Filter out Suspended CronJobs

Combine with kube_cronjob_spec_suspend to suppress false alerts on intentionally paused jobs:

(

(time() - kube_cronjob_status_last_successful_time) > 86400

) and on (cronjob, namespace) (

kube_cronjob_spec_suspend == 0

)

Tuesday, 4 August 2026

Monitoring term: Dead-man's switch

It's monitoring inverted: instead of alerting when you observe something bad, you alert when you stop observing something good.

Example: ping on a successful run. If run is unsuccessful, monitoring catches missing ping and triggers alert.

The name comes from industrial safety — the lever on a train's throttle or a chainsaw that has to be actively held down. If the operator dies or lets go, the machine stops. Safety is the default state; it takes continuous positive action to keep running.

Normal alerting is presence-based. Something goes wrong, it emits a signal, you alert on the signal: error rate spikes, latency crosses a threshold, a pod enters CrashLoopBackOff. It works well when failures are noisy.

A dead-man's switch is absence-based. The healthy system periodically says "still fine." You alert when that message doesn't arrive on time.

Kubernetes Debugging Scenario: Node.JS CronJob dies with a V8 JavaScript heap OOM

Problem Scenario

A Node.js batch job running as a Kubernetes CronJob aborts with FATAL ERROR: Ineffective mark-compacts near heap limit at ~4 GB. No NODE_OPTIONS, no resources block. Each scheduled run leaves several failed pods behind.

Knowledge required to fix the problem (Q&A)

Detailed Q&A

1. Node.js / V8 memory model

Q: What does --max-old-space-size actually control, and what does it not control?

It caps V8's old space — the long-lived generation of the JS heap. It does not cap new space (--max-semi-space-size), code space, large object space, or external/off-heap memory such as Buffer and ArrayBuffer allocations, native addon memory, thread-pool stacks, or glibc malloc arenas. So a process with a 6 GB old-space ceiling can easily have an RSS well above 6 GB.

V8 is Google's open source high-performance JavaScript and WebAssembly engine, written in C++. It is used in Chrome and in Node.js, among others.

--max-old-space-size sets the maximum memory limit (in megabytes) allocated to the Old Generation heap space inside V8, the JavaScript engine powering Node.js.

When V8 allocates memory for your application, it divides the JavaScript heap into distinct regions based on object lifecycle. This flag configures the largest region where long-lived objects reside.

What It Measures & Controls

--max-old-space-size explicitly caps memory allocated for:

Old Generation JavaScript Objects: Objects, arrays, functions, closures, and strings that have survived initial garbage collection cycles in the Young Generation space and were promoted to the Old Generation.
Old Pointer Space & Old Data Space: Regions holding objects that contain pointers to other objects and raw data (like numbers or unboxed scalars).

What It Does NOT Control

A common misconception is that --max-old-space-size caps the entire Resident Set Size (RSS) or system memory footprint of your Node.js process. It does not limit:
Node.js Buffers (ArrayBuffers): Since Node.js v8.0+, binary Buffer allocations use off-heap C++ memory backing stores (ArrayBuffer). While the JavaScript wrapper object lives on the V8 heap, the underlying raw bytes do not count toward the old space limit.
Native C++ Allocations: Memory used by native C++ add-ons, libuv threads, or external libraries compiled into Node.
Other V8 Heap Spaces:

New Space (Nursery/Young Generation): Where new allocations land (--max-semi-space-size).
Code Space: JIT-compiled bytecode and machine code.
Map/Cell Spaces: V8 internal hidden classes and metadata.

Call Stack Memory: Memory used by execution contexts and local variables on the stack.

Because of off-heap memory, a Node.js process with --max-old-space-size=2048 (2 GB) can easily consume 3 GB or more of total system RAM (RSS).

What Happens When the Limit Is Reached

Aggressive Garbage Collection: As old space usage approaches the limit, V8 triggers blocking, high-overhead Mark-Sweep-Compact garbage collection cycles to reclaim dead objects.
Process Crash: If V8 cannot free enough memory to fit the next allocation below the configured threshold, Node.js crashes with a fatal error:

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

Default Values & Usage

Default Behavior: In modern Node.js versions, V8 dynamically sets the limit based on total available system RAM—typically around 2 GB to 4 GB on 64-bit systems if unspecified.

Command Line Flag:

node --max-old-space-size=4096 app.js

Environment Variable:

export NODE_OPTIONS="--max-old-space-size=4096"

Q: Why did the process die at ~4064 MB when nobody configured a heap limit?

V8 picks a default heap ceiling from the memory it believes is available, and on 64-bit builds that lands at roughly 4 GB. The Mark-Compact 4064.3 MB line in the log is the giveaway that it hit that default ceiling rather than any limit you set.

Q: Why doesn't Node just size its heap to the container's memory limit?

Historically V8 read host RAM, not the cgroup limit, so a Node process in a 512 Mi container would happily set a multi-gigabyte heap and get OOM-killed. Newer Node versions do consult cgroup constraints, but the behaviour varies by version — which is why the defensive answer is always to set the flag explicitly rather than rely on auto-detection.

Q: The workload starts with npm run start. Does setting NODE_OPTIONS in the container env actually reach the Node process?

Yes — NODE_OPTIONS is an environment variable, so it's inherited by every child process npm spawns. Two caveats worth naming: it also applies to the npm wrapper process itself (harmless, just an extra reservation), and not every V8 flag is permitted inside NODE_OPTIONS. The alternative is passing the flag in the npm script itself, which is more surgical but easier to lose.

2. Diagnosis: which kind of OOM is this?

Q: How do you tell a V8 heap OOM from a kernel OOMKill from a kubelet eviction?

Signal	V8 heap OOM	OOMKilled	Evicted
Log line	`FATAL ERROR: Ineffective mark-compacts near heap limit`	none from the app — killed mid-flight	none from the app
Signal / exit	`SIGABRT`, exit 134	`SIGKILL`, exit 137	pod deleted
Pod status	`Error`	`OOMKilled` in `lastState.terminated.reason`	`Failed`, reason `Evicted`
Fix direction	raise heap ceiling or reduce allocation	raise container limit	set requests so you aren't the first target

The ticket's evidence — the mark-compact message plus signal SIGABRT — puts it firmly in column one. That matters because raising the container limit alone would have changed nothing: V8 would still have aborted at 4 GB.

Q: How would you size the flag rather than guessing?

Instrument before you tune. --trace-gc shows the heap trajectory over the run; process.memoryUsage() sampled periodically distinguishes heapUsed from external; --heapsnapshot-near-heap-limit=1 writes a snapshot right before the abort that you can open in Chrome DevTools to find the retaining structure. That tells you whether the working set is genuinely ~6 GB or whether one unbounded array is the whole problem.

Q: Is raising the heap the right fix at all?

Usually it's a mitigation, not a fix. A benchmark job whose memory scales with input size will hit any ceiling you pick — the durable fix is streaming, batching, or paginating so peak memory is bounded by chunk size rather than dataset size. Raising the flag is defensible as a stopgap; the honest version says so in the ticket and files the follow-up. Note that in this case the real numbers came out at 10 GB heap with a 5-hour runtime, which is a fairly loud hint that the algorithm is the underlying issue.

3. Kubernetes resource management

Q: What's the difference between a memory request and a memory limit?

The request is what the scheduler reserves — it decides which node the pod fits on and is the baseline the kubelet uses when deciding who to evict. The limit is enforced at runtime by the cgroup; exceed it and the kernel OOM-kills the container. Memory, unlike CPU, is incompressible: there's no throttling, only killing.

Q: What QoS class does a pod with no resources block get, and why does that matter here?

BestEffort — the first thing evicted under node memory pressure, and it contributes nothing to the scheduler's accounting so the node can be oversubscribed into pressure in the first place. Setting requests equal to limits gives Guaranteed; requests below limits gives Burstable.

Q: How do you choose the relationship between the heap flag and the container limit?

Limit strictly above heap ceiling, with headroom for everything --max-old-space-size doesn't cover — off-heap buffers, native memory, the npm and node process overhead, plus GC working room. The ticket proposed 6 GB heap under a 7 Gi limit; what actually shipped was 10 GB heap under a 12 Gi limit. Too tight and you convert a clean SIGABRT into a much harder-to-debug OOMKill.

Q: What's the risk of setting limits.memory well above requests.memory?

You're overcommitting the node. It schedules against the request but can consume up to the limit, so several such pods on one node can drive it into memory pressure and trigger evictions of unrelated workloads. Matching them costs you scheduling flexibility but makes the blast radius predictable.

Q: You set requests.memory: 8Gi and the pod never starts. What's your first check?

Whether any node has 8 Gi of allocatable memory free — allocatable is capacity minus kube-reserved, system-reserved, and eviction thresholds. The pod sits Pending with an Insufficient memory scheduling event. Big-request batch jobs are a classic case for a dedicated or autoscaling node pool.

4. CronJobs and Jobs

Q: What produced seven Error pods from one scheduled run?

backoffLimit retries on failure, and a deterministic OOM fails identically every time — so the Job burned through its retries producing one dead pod each. The fix applied was a podFailurePolicy that fails the Job on the application's exit code instead of retrying, plus ttlSecondsAfterFinished so finished Jobs get garbage-collected rather than accumulating.

Q: What does podFailurePolicy require to work?

restartPolicy: Never on the pod template, and rules matching on either container exit codes (onExitCodes) or pod conditions (onPodConditions, e.g. DisruptionTarget). Actions are FailJob, Ignore, Count, and FailIndex. The point is distinguishing retryable infrastructure failures from deterministic application failures — retrying a heap OOM six times is pure waste.

Q: Which CronJob fields govern history and overlap?

successfulJobsHistoryLimit / failedJobsHistoryLimit for retained Job objects, ttlSecondsAfterFinished on the Job for automatic cleanup, concurrencyPolicy (Allow / Forbid / Replace) for overlapping runs, startingDeadlineSeconds for missed schedules, and activeDeadlineSeconds as a wall-clock kill switch. For a job that runs five hours, concurrencyPolicy: Forbid deserves a hard look.

Q: The schedule is 30 10 */14 * *. Does that run every 14 days?

No — and this is the trap. Step values in day-of-month are evaluated within each month, so it fires on the 1st, 15th, and 29th, then resets. The gap between the 29th and the following 1st is two or three days, not fourteen. Genuine "every N days" needs an external scheduler or a daily run that no-ops based on a stored timestamp.

5. Container memory accounting

Q: When you read a container's memory usage, what are you actually seeing?

Under cgroup v2 the kubelet reports working set derived from memory.current minus inactive file cache; memory.max is the hard limit. Crucially memory.current includes page cache, so a process doing heavy file I/O can look alarming without any anonymous-memory problem. RSS is anonymous plus mapped pages for the process specifically, and glibc often doesn't return freed memory to the OS — so RSS is sticky and lags real usage downward.

Q: Why is container_memory_rss a poor alerting signal for some workloads?

Because it only captures what lives in RSS. For a JVM or Node process the heap is anonymous memory and RSS tracks it reasonably; for something like Percona MongoDB, where WiredTiger's cache sits in the OS page cache rather than RSS, the metric is structurally blind to the thing you care about — you want cache fill percentage instead. Matching the metric to the workload's memory architecture is the actual skill.

6. Verification

Q: How do you prove the fix worked?

Trigger a manual run (kubectl create job --from=cronjob/experience-benchmarks) and confirm the Job reaches Complete with no SIGABRT and no Error pods. Then compare peak usage against the limit — completing at 95% of the ceiling is luck, not a fix. Verification model: live CronJob spec matches main, last three runs all Complete, runtimes recorded, zero Error pods.

Q: How do you confirm what's actually running in prod matches what's in the repo?

Diff the live object against the manifest — kubectl get cronjob experience-benchmarks -o yaml against deploy/prod.yml. Drift between a merged PR and the running cluster is exactly the kind of gap that lets a "fixed" ticket keep failing, and it's the check that would have surfaced the tickets overlap before any code was written.

Q: What should you have checked before writing a single line for this ticket?

Whether the problem still existed. The ticket sat in Backlog for four days, a ticket shipped a superset of the fix during that window, and the work that followed would have lowered the heap from 10 GB to 6 GB — reintroducing the OOM. Reading main and the live spec before implementing is the cheapest step in the whole process and the one that was skipped.

Brief Q&A

V8 / Node

Q: What does --max-old-space-size cap? Only V8's old space. Not new space, code space, or off-heap memory (Buffer, ArrayBuffer, native addons). RSS can exceed it substantially.

Q: Why die at ~4 GB with no flag set? That's V8's default ceiling on 64-bit. Node has historically sized it from host RAM, not the cgroup limit — so always set it explicitly.

Q: Does NODE_OPTIONS reach a process started via npm run start? Yes, it's inherited by child processes. It also applies to the npm wrapper itself.

Diagnosis

Q: Distinguish the three OOM flavours. V8 heap OOM → mark-compact message, SIGABRT, exit 134, pod Error. Kernel kill → no app log, SIGKILL, exit 137, OOMKilled. Eviction → pod Failed, reason Evicted. Only the first is fixed by the heap flag.

Q: How do you size the flag instead of guessing? --trace-gc for the trajectory, process.memoryUsage() for heap vs. external, --heapsnapshot-near-heap-limit=1 for a snapshot at the abort.

Q: Is raising the heap the real fix? Usually a stopgap. If memory scales with input size, any ceiling eventually fails — stream or batch so peak is bounded by chunk size.

Kubernetes resources

Q: Request vs. limit? Request drives scheduling and eviction ranking; limit is cgroup-enforced. Memory is incompressible — no throttling, only killing.

Q: No resources block means what QoS? BestEffort — first evicted under node pressure, and invisible to scheduler accounting. Requests == limits gives Guaranteed.

Q: How do heap ceiling and container limit relate? Limit strictly above the heap, with headroom for off-heap and process overhead. Too tight converts a clean SIGABRT into a harder-to-debug OOMKill.

Q: Request set high and the pod won't schedule? Check node allocatable (capacity minus reserved and eviction thresholds). Expect Pending with Insufficient memory.

Jobs / CronJobs

Q: Why several failed pods per run? backoffLimit retries, and a deterministic OOM fails identically each time. Use podFailurePolicy with onExitCodes (requires restartPolicy: Never) to fail fast, plus ttlSecondsAfterFinished for cleanup.

Q: Does 30 10 */14 * * run every 14 days? No. Day-of-month steps reset monthly → the 1st, 15th, and 29th. True "every N days" needs external scheduling.

Verification

Q: How do you prove it's fixed? Trigger a manual run from the CronJob, confirm Complete with no failed pods, and compare peak usage to the limit — finishing at 95% of the ceiling is luck.

Q: What do you check before writing any code? That the problem still exists. Diff the live object against the repo manifest; a stale ticket can lead you to lower limits that a since-merged fix raised.