The 4 Core Prometheus Metric Types
Prometheus defines four primary metric types. Choosing the right type depends on how the value behaves over time.
┌───────────────────────────┐
│ Prometheus Metric Types │
└─────────────┬─────────────┘
│
┌───────────────────┬──────────────┴───────────────┬────────────────────┐
▼ ▼ ▼ ▼
┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Gauge │ │ Counter │ │ Histogram │ │ Summary │
├──────────────┤ ├──────────────┤ ├──────────────┤ ├──────────────┤
│ Value goes │ │ Value ONLY │ │ Groups values│ │ Calculates │
│ UP and DOWN │ │ increases │ │ into buckets │ │ quantiles │
│ (Snapshot) │ │ (Cumulative) │ │ (Client-side)│ │ (Client-side)│
└──────────────┘ └──────────────┘ └──────────────┘ └──────────────┘
1. Gauge
- Behavior: Can increase, decrease, or stay the same.
- Use Case: Current state, temperatures, memory usage, concurrent connections, replica counts.
- PromQL Functions: avg_over_time(), max_over_time(), direct evaluation.
In Prometheus (and OpenMetrics standards), a Gauge is a metric that represents a single numerical value that can arbitrarily go up and down.
It acts like a digital display or a vehicle's speedometer—it gives you a point-in-time snapshot of a current state.
Understanding the Gauge Type
Because a Gauge can fluctuate freely, it is used to measure snapshot values, current state levels, and instantaneous statuses.
- Example Real-World Analogy: A car's speedometer (can go from 0 to 70 mph, back to 30 mph), ambient temperature, or fuel tank level.
- Kubernetes/KSM Examples:
- kube_cronjob_status_last_successful_time: Holds a Unix timestamp (moves forward or stays fixed).
- kube_deployment_status_replicas_available: Number of ready pods (e.g., 3 → 5 → 2).
- node_memory_MemAvailable_bytes: Available system memory in bytes.
Key Feature: Instantaneous Operations
Because gauges fluctuate, you do not run rate or increase functions like rate() or increase() on them in PromQL. Instead, you query their current value directly or perform operations like running averages (avg_over_time()), min/max checks, or scalar math (e.g., time() - gauge).
2. Counter
- Behavior: A cumulative metric that only increases (or resets to 0 upon process restart). It never decreases naturally.
- Use Case: Counting total occurrences of events (e.g., total HTTP requests served, total pipeline errors, network bytes transmitted).
- PromQL Functions: rate(), irate(), increase().
Why use a Counter instead of a Gauge? A counter allows PromQL's rate() function to accurately calculate "per-second rates" while automatically handling process restarts (resets).
3. Histogram
- Behavior: Samples observations (usually things like request durations or payload sizes) and counts them in configurable buckets. It also provides a sum of all observed values.
- Exposed Data:
- <basename>_bucket{le="<upper_bound>"}: Counter of observations with value <= upper bound.
- <basename>_count: Total number of observations (Counter).
- <basename>_sum: Sum of all observed values (Counter).
- Use Case: Measuring latencies, response times, or request sizes where you want to calculate percentiles (e.g., p95, p99) on the server side using PromQL (histogram_quantile()).
4. Summary
- Behavior: Similar to a histogram, but calculates configurable quantiles (e.g., 0.50, 0.90, 0.99) directly on the client application side over a sliding time window.
- Exposed Data:
- <basename>{quantile="0.95"}: The 95th percentile value.
- <basename>_count: Total observations.
- <basename>_sum: Sum of observations.
- Use Case: When client-side percentile calculation is required and you don't need to aggregate quantiles across multiple instances in Prometheus.
Summary Comparison Matrix
Metric Type
- Can Decrease?
- Typical PromQL Functions
- Best Used For
Gauge
- Yes
- Direct value, avg_over_time()
- Current state, memory usage, counts of current objects
Counter
- No (only on restart)
- rate(), increase()
- Total count of events over time, request volume
Histogram
- No (bucket counts increase)
- histogram_quantile(), rate()
- Request latencies, payload sizes (aggregatable)
Summary
- No (counts/sums increase)
- Direct quantile query
- Request latencies (pre-calculated on client)
Container Metrics
container_memory_rss
container_memory_rss is a Prometheus metric (exposed by
cAdvisor) that measures a container's
Resident Set Size—specifically, the amount of
physical RAM allocated to non-reclaimable, non-file-backed memory.
Key Technical Breakdown
Under the hood in Linux cgroups, container_memory_rss tracks:
- Anonymous Memory: Process heap allocations, execution stack, and memory allocated via malloc or mmap(MAP_ANONYMOUS).
- Swap Cache: Memory swapped out to disk that is being brought back into RAM.
What it EXCLUDES:
Unlike standard Linux host RSS metrics, container_memory_rss in cAdvisor excludes file-backed page caches (memory used to cache files read from disk). Because of this, it only measures memory that cannot be automatically freed by the Linux kernel under memory pressure.
container_memory_rss vs. Other Container Metrics
To understand its role, it helps to see how it fits into cAdvisor's other core memory metrics:
Metric
- Includes
- Purpose / Key Characteristic
container_memory_rss
- Heap + Stack (Anonymous memory)
- Stable indicator of process memory footprint. Does not fluctuate with disk I/O.
container_memory_working_set_bytes
- Heap + Stack + Active Page Cache
- What Kubernetes actually monitors. Fluctuates with file reads/writes.
container_memory_usage_bytes
- Heap + Stack + Active Cache + Inactive Cache
- Raw total RAM usage. Can be misleading because inactive cache is easily reclaimed by the OS.
Why is container_memory_rss Important?
- Memory Leak Detection: Because it excludes cached file reads, container_memory_rss provides a much cleaner signal for application-level memory leaks. If this metric steadily climbs over time without dropping, your application (e.g., Go heap, JVM heap, Node process) is holding onto unmanaged memory.
- Debugging OOM Kills: While Kubernetes triggers OOMKills based on container_memory_working_set_bytes hitting resource limits, container_memory_rss helps you determine why it happened:
- High RSS + High Working Set --> Application memory leak or underestimated heap limit.
- Low RSS + High Working Set --> Heavy disk I/O / file caching (e.g., reading massive log files or database indexes into memory).
When container_memory_rss is NOT a relevant metric?
Example:
WiredTiger is the default storage engine that MongoDB (here, the Percona Server for MongoDB / PSMDB cluster) uses to actually read and write data to disk. In <ticketID> it matters specifically because of how it uses memory, which is what breaks those four Grafana alert rules.
The relevant behavior:
WiredTiger keeps a large in-memory cache. It maintains its own cache of frequently-accessed data and indexes, and it deliberately sizes that cache to roughly half the container's memory limit (the issue cites ~10 GiB against a 21Gi limit on rs0/rs2, and it's tunable per <ticketID>). WiredTiger runs its own eviction, targeting about 80% cache fill under normal operation and triggering aggressive eviction around 95%.
That cached data shows up as page cache, not RSS. This is the crux of the ticket. container_memory_rss counts anonymous/resident process memory but excludes the OS page cache — and for a WiredTiger workload the file-backed pages (the cache) are the bulk of the real footprint. So rss / limit sits structurally pinned around 50-58% no matter how much memory pressure the pod is actually under. An alert keyed on container_memory_rss > 0.90 can therefore never fire for a WiredTiger container. It's dead. (alert is defined as: container_memory_rss{container="mongod"} / limit > 0.90)
But you can't just switch to working-set either. container_memory_working_set_bytes does include those active file pages, so on the busy replicas it reads 95-96%. That looks alarming but is expected and healthy: it's the WiredTiger cache sitting at its designed ~half-of-limit size and WT's own eviction watermark. The kernel reclaims those pages under real pressure, and the evidence is that nothing in the mongodb namespace has ever been OOMKilled. Flipping the metric would just move the rule from never-firing to always-firing.
The signal that actually matters is WiredTiger cache fill. Because WT stalls user operations when its cache can't evict fast enough, the meaningful early-warning metric is cache utilization (the existing MongoDB WiredTiger cache fill — above 95% for 30m rule from <ticketID>), not container RSS or working set. That's why the ticket argues the RSS rules are redundant, not just broken, and leans toward deleting them.
So in this issue's context, "WiredTiger" is essentially shorthand for "a workload whose memory lives mostly in a large, self-managed, page-cache-backed database cache" — which is exactly the profile that makes RSS-based memory alerting meaningless and makes cache-fill the correct signal instead. (The same logic applies to the Elasticsearch rules, where the JVM heap plays the analogous role.)
Example PromQL Query
To track application heap growth without the noise of filesystem caching, query RSS like this:
# Rate of container RSS growth over 5-minute intervals
rate(container_memory_rss{namespace="production", container!=""}[5m])
container_memory_working_set_bytes
...
container_memory_usage_bytes
...
