The 4 Core Prometheus Metric Types
1. Gauge
- Behavior: Can increase, decrease, or stay the same.
- Use Case: Current state, temperatures, memory usage, concurrent connections, replica counts.
- PromQL Functions: avg_over_time(), max_over_time(), direct evaluation.
Understanding the Gauge Type
- Example Real-World Analogy: A car's speedometer (can go from 0 to 70 mph, back to 30 mph), ambient temperature, or fuel tank level.
- Kubernetes/KSM Examples:
- kube_cronjob_status_last_successful_time: Holds a Unix timestamp (moves forward or stays fixed).
- kube_deployment_status_replicas_available: Number of ready pods (e.g., 3 → 5 → 2).
- node_memory_MemAvailable_bytes: Available system memory in bytes.
Key Feature: Instantaneous Operations
2. Counter
- Behavior: A cumulative metric that only increases (or resets to 0 upon process restart). It never decreases naturally.
- Use Case: Counting total occurrences of events (e.g., total HTTP requests served, total pipeline errors, network bytes transmitted).
- PromQL Functions: rate(), irate(), increase().
3. Histogram
- Behavior: Samples observations (usually things like request durations or payload sizes) and counts them in configurable buckets. It also provides a sum of all observed values.
- Exposed Data:
- <basename>_bucket{le="<upper_bound>"}: Counter of observations with value <= upper bound.
- <basename>_count: Total number of observations (Counter).
- <basename>_sum: Sum of all observed values (Counter).
- Use Case: Measuring latencies, response times, or request sizes where you want to calculate percentiles (e.g., p95, p99) on the server side using PromQL (histogram_quantile()).
4. Summary
- Behavior: Similar to a histogram, but calculates configurable quantiles (e.g., 0.50, 0.90, 0.99) directly on the client application side over a sliding time window.
- Exposed Data:
- <basename>{quantile="0.95"}: The 95th percentile value.
- <basename>_count: Total observations.
- <basename>_sum: Sum of observations.
- Use Case: When client-side percentile calculation is required and you don't need to aggregate quantiles across multiple instances in Prometheus.
Summary Comparison Matrix
- Can Decrease?
- Typical PromQL Functions
- Best Used For
- Yes
- Direct value, avg_over_time()
- Current state, memory usage, counts of current objects
- No (only on restart)
- rate(), increase()
- Total count of events over time, request volume
- No (bucket counts increase)
- histogram_quantile(), rate()
- Request latencies, payload sizes (aggregatable)
- No (counts/sums increase)
- Direct quantile query
- Request latencies (pre-calculated on client)
Container Metrics
container_memory_rss
Key Technical Breakdown
- Anonymous Memory: Process heap allocations, execution stack, and memory allocated via malloc or mmap(MAP_ANONYMOUS).
- Swap Cache: Memory swapped out to disk that is being brought back into RAM.
What it EXCLUDES:
container_memory_rss vs. Other Container Metrics
- Includes
- Purpose / Key Characteristic
- Heap + Stack (Anonymous memory)
- Stable indicator of process memory footprint. Does not fluctuate with disk I/O.
- Heap + Stack + Active Page Cache
- What Kubernetes actually monitors. Fluctuates with file reads/writes.
- Heap + Stack + Active Cache + Inactive Cache
- Raw total RAM usage. Can be misleading because inactive cache is easily reclaimed by the OS.
Why is container_memory_rss Important?
- Memory Leak Detection: Because it excludes cached file reads, container_memory_rss provides a much cleaner signal for application-level memory leaks. If this metric steadily climbs over time without dropping, your application (e.g., Go heap, JVM heap, Node process) is holding onto unmanaged memory.
- Debugging OOM Kills: While Kubernetes triggers OOMKills based on container_memory_working_set_bytes hitting resource limits, container_memory_rss helps you determine why it happened:
- High RSS + High Working Set --> Application memory leak or underestimated heap limit.
- Low RSS + High Working Set --> Heavy disk I/O / file caching (e.g., reading massive log files or database indexes into memory).
When container_memory_rss is NOT a relevant metric?
WiredTiger is the default storage engine that MongoDB (here, the Percona Server for MongoDB / PSMDB cluster) uses to actually read and write data to disk. In <ticketID> it matters specifically because of how it uses memory, which is what breaks those four Grafana alert rules.
The relevant behavior:
WiredTiger keeps a large in-memory cache. It maintains its own cache of frequently-accessed data and indexes, and it deliberately sizes that cache to roughly half the container's memory limit (the issue cites ~10 GiB against a 21Gi limit on rs0/rs2, and it's tunable per <ticketID>). WiredTiger runs its own eviction, targeting about 80% cache fill under normal operation and triggering aggressive eviction around 95%.
That cached data shows up as page cache, not RSS. This is the crux of the ticket. container_memory_rss counts anonymous/resident process memory but excludes the OS page cache — and for a WiredTiger workload the file-backed pages (the cache) are the bulk of the real footprint. So rss / limit sits structurally pinned around 50-58% no matter how much memory pressure the pod is actually under. An alert keyed on container_memory_rss > 0.90 can therefore never fire for a WiredTiger container. It's dead. (alert is defined as: container_memory_rss{container="mongod"} / limit > 0.90)
But you can't just switch to working-set either. container_memory_working_set_bytes does include those active file pages, so on the busy replicas it reads 95-96%. That looks alarming but is expected and healthy: it's the WiredTiger cache sitting at its designed ~half-of-limit size and WT's own eviction watermark. The kernel reclaims those pages under real pressure, and the evidence is that nothing in the mongodb namespace has ever been OOMKilled. Flipping the metric would just move the rule from never-firing to always-firing.
The signal that actually matters is WiredTiger cache fill. Because WT stalls user operations when its cache can't evict fast enough, the meaningful early-warning metric is cache utilization (the existing MongoDB WiredTiger cache fill — above 95% for 30m rule from <ticketID>), not container RSS or working set. That's why the ticket argues the RSS rules are redundant, not just broken, and leans toward deleting them.
So in this issue's context, "WiredTiger" is essentially shorthand for "a workload whose memory lives mostly in a large, self-managed, page-cache-backed database cache" — which is exactly the profile that makes RSS-based memory alerting meaningless and makes cache-fill the correct signal instead. (The same logic applies to the Elasticsearch rules, where the JVM heap plays the analogous role.)
