Kubernetes v1.37: Native Histograms Graduates to Beta (kubernetes.io)

by rss-bot · 2 months ago · 0 comments

I'm excited to announce that native histogram support for Kubernetes metrics is graduating to Beta and is enabled by default in Kubernetes v1.37!
Native histograms (previously introduced as Alpha in Kubernetes v1.36 under <a href="https://www.kubernetes.dev/resources/keps/5808/">KEP-5808</a>)
bring high-resolution, low-cardinality observability to Kubernetes metrics.
By adopting <a href="https://prometheus.io/docs/specs/native_histograms/">Prometheus Native Histograms</a>, Kubernetes components now expose latency and duration metrics with far greater accuracy while significantly reducing telemetry storage and scraping overhead.
<h2 id="why-move-beyond-classic-histograms">Why move beyond classic histograms?<a class="td-heading-self-link" href="#why-move-beyond-classic-histograms" aria-label="Heading self-link"></a></h2>Since the early days of Kubernetes observability, duration and latency metrics (such as API server request latencies or scheduling durations) have relied on classic Prometheus histograms.
Classic histograms require metric authors to define a static list of cumulative bucket boundaries (<code>le</code> labels), such as <code>0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10</code>. While familiar, this approach introduces three major challenges:
<ol>
<li>The Bucket Guessing Game: If a workload's latency profile changes, for example, shifting into microsecond ranges or experiencing long-tail tail latencies beyond the highest bucket, the histogram loses visibility. Specifying bucket boundaries upfront requires knowing the distribution before observing it</li>
<li>High Cardinality & Storage Cost: With classic histograms, each bucket boundary is exported as a separate time series (<code>_bucket{le="..."}</code>). A histogram with 10 buckets across multiple labels multiplies the number of time series by 10, increasing memory consumption in Prometheus and inflating time series database (TSDB) storage costs</li>
<li>Interpolation Error in Quantiles: Calculating percentiles using <code>histogram_quantile()</code> relies on linear interpolation between static bucket boundaries. When bucket spans are coarse, quantile calculations can suffer from significant estimation error</li>
</ol>
<h2 id="what-are-prometheus-native-histograms">What are Prometheus native histograms?<a class="td-heading-self-link" href="#what-are-prometheus-native-histograms" aria-label="Heading self-link"></a></h2><a href="https://prometheus.io/docs/specs/native_histograms/">Prometheus Native Histograms</a> replace static user-defined buckets with dynamic, exponential buckets.
Instead of emitting a separate time series for every single bucket boundary, a native histogram is stored as a single time series containing a rich schema of positive and negative spans, zero thresholds, and exponential scaling factors.
<ul>
<li>High Resolution Automatically: Exponential buckets dynamically adjust to any value range — from nanoseconds to hours — without requiring pre-configured bucket boundaries</li>
<li>Up to 90% Fewer Time Series: By consolidating buckets into structured spans within a single time series, scraping and storage overhead are dramatically reduced</li>
<li>Accurate Quantile Calculation: Quantiles can be calculated with mathematical bounds on error (≃5% worst-case relative error under default settings) across the entire spectrum of observations</li>
</ul>
<h2 id="how-native-histograms-work-in-kubernetes">How native histograms work in Kubernetes<a class="td-heading-self-link" href="#how-native-histograms-work-in-kubernetes" aria-label="Heading self-link"></a></h2>In Kubernetes, native histogram support is implemented directly inside the shared metrics subsystem (<code>k8s.io/component-base/metrics</code>).
Figure 1 illustrates how native histogram metrics are processed and exposed across Kubernetes components.
<figure class="diagram-large "><a href="https://kubernetes.io/images/blog/2026-kubernetes-1-37-native-histograms-beta/native-histograms-flow.svg">
<img src="https://kubernetes.io/images/blog/2026-kubernetes-1-37-native-histograms-beta/native-histograms-flow.svg"
alt="Diagram showing native histogram metric registration, exponential options configuration, and dual exposition flow in Kubernetes components"/> </a><figcaption>
Figure 1. Native histogram processing and dual exposition flow in Kubernetes.
</figcaption>
</figure>
<h3 id="1-dual-exposition-for-zero-breaking-changes">1. Dual exposition for zero breaking changes<a class="td-heading-self-link" href="#1-dual-exposition-for-zero-breaking-changes" aria-label="Heading self-link"></a></h3>A primary design requirement for KEP-5808 was zero disruption for existing observability stacks. When the <code>NativeHistograms</code> feature gate is enabled, Kubernetes components use dual exposition:
<ul>
<li>Classic buckets (<code>h.Bucket</code>) are still emitted alongside native spans. Existing Prometheus servers, dashboards, and alerting rules that rely on traditional text scraping or classic bucket labels continue to work unmodified</li>
<li>Native spans (<code>h.Schema</code>, <code>h.PositiveSpan</code>) are included in the same Protobuf payload for collectors that understand native histograms</li>
</ul>
<h3 id="2-tuned-default-exponential-configuration">2. Tuned default exponential configuration<a class="td-heading-self-link" href="#2-tuned-default-exponential-configuration" aria-label="Heading self-link"></a></h3>When <code>NativeHistograms</code> is enabled, the <code>k8s.io/component-base/metrics</code> package automatically applies standardized exponential options to all histogram metrics:
<ul>
<li><code>BucketFactor: 1.1</code>: Configures exponential buckets where each bucket is at most 10% wider than the preceding one. This guarantees a mathematically bounded worst-case relative error of at most ~5% for quantile calculations regardless of whether an operation takes 1 millisecond or 10 seconds.</li>
<li><code>MaxBucketNumber: 160</code>: Caps the maximum number of buckets per histogram to 160. Following OpenTelemetry SDK recommendations for base-2 exponential histogram aggregation, this limit protects component memory usage even under extreme outlier distributions.</li>
</ul>
<h3 id="3-broad-component-support">3. Broad component support<a class="td-heading-self-link" href="#3-broad-component-support" aria-label="Heading self-link"></a></h3>Because native histograms are integrated into <code>component-base/metrics</code>, all major Kubernetes control plane and node components inherit support automatically, including:
<ul>
<li><code>kube-apiserver</code> (e.g., <code>apiserver_request_duration_seconds</code>, authentication/authorization metrics, validation latencies)</li>
<li><code>kube-scheduler</code> (e.g., <code>scheduler_plugin_execution_duration_seconds</code>, <code>scheduler_scheduling_algorithm_duration_seconds</code>)</li>
<li><code>kubelet</code> (node-level container runtime and pod lifecycle metrics)</li>
<li><code>kube-controller-manager</code> and <code>kube-proxy</code></li>
</ul>
<h2 id="how-to-scrape-native-histograms">How to scrape native histograms<a class="td-heading-self-link" href="#how-to-scrape-native-histograms" aria-label="Heading self-link"></a></h2>The simple answer: upgrade to Kubernetes v1.37, and it works.
Because Kubernetes v1.37 enables <code>NativeHistograms</code> by default, your cluster is already emitting dual-exposition metrics. How you configure Prometheus to scrape native histograms depends on your Prometheus version:
<h3 id="1-prometheus-scrape-configuration-by-version">1. Prometheus scrape configuration by version<a class="td-heading-self-link" href="#1-prometheus-scrape-configuration-by-version" aria-label="Heading self-link"></a></h3><ul>
<li>
Prometheus 3.0+ (Recommended): Use explicit per-job configuration in your <code>scrape_configs</code> rather than global flags (the global <code>--enable-feature=native-histograms</code> flag is deprecated in Prometheus 3.9+):
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml">scrape_configs:
 - job_name: 'kubernetes-apiservers'
 scrape_native_histograms: true
 always_scrape_classic_histograms: true # Recommended during transition
</code></pre></div>You must read the caution in <a href="https://kubernetes.io/docs/reference/instrumentation/native-histograms/#migrating-dashboards-and-alerts">Migrating dashboards and alerts</a> in the Native Histograms documentation. In summary: always set <code>always_scrape_classic_histograms: true</code> during your transition period. Without this setting, Prometheus will only ingest the native format and stop ingesting classic <code>_bucket</code>, <code>_count</code>, and <code>_sum</code> series. Setting <code>always_scrape_classic_histograms: true</code> ensures existing dashboards (<code>histogram_quantile(..._bucket...)</code>) and alerts continue to work while you migrate them to native histograms.
</li>
<li>
Prometheus 2.40 – 2.x: Enable Native Histograms globally by starting Prometheus with the feature flag:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash">prometheus --enable-feature=native-histograms
</code></pre></div>Note that in Prometheus 2.x, this is an all-or-nothing setting for all scrape targets.
</li>
</ul>
<h3 id="2-verify-protobuf-dual-exposition">2. Verify Protobuf dual exposition<a class="td-heading-self-link" href="#2-verify-protobuf-dual-exposition" aria-label="Heading self-link"></a></h3>Standard Prometheus text scraping (<code>application/openmetrics-text</code> or plain text format) only transfers classic buckets. When <code>scrape_native_histograms</code> is enabled, Prometheus automatically negotiates Protobuf format with Kubernetes endpoints.
You can verify that a Kubernetes component is exporting native histograms using <code>curl</code> with an <code>Accept</code> header specifying Protobuf. For example:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash">## THIS IS NOT SECURE. ONLY DO THIS IN A TEST CONTEXT.
curl --insecure \
 -H "Accept: application/vnd.google.protobuf;proto=io.prometheus.client.MetricFamily;encoding=delimited" \
 --header "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
 https://localhost:6443/metrics
</code></pre></div>When decoded, the returned <code>MetricFamily</code> for histogram metrics (like <code>apiserver_request_duration_seconds</code>) will contain both traditional <code>bucket</code> entries and populated <code>schema</code> / <code>positive_span</code> fields.
<h2 id="querying-native-histograms-in-promql">Querying native histograms in PromQL<a class="td-heading-self-link" href="#querying-native-histograms-in-promql" aria-label="Heading self-link"></a></h2>Once native histograms are ingested into Prometheus, you can query them using standard PromQL histogram functions without needing static <code>le</code> bucket labels or <code>_bucket</code> suffixes:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-promql" data-lang="promql"># 1. Calculating P99 latency for a single target:
# Classic histogram (requires _bucket suffix):
histogram_quantile(0.99, rate(apiserver_request_duration_seconds_bucket[5m]))

# Native histogram (operates directly on the metric name):
histogram_quantile(0.99, rate(apiserver_request_duration_seconds[5m]))

# 2. Aggregating across multiple instances (e.g., all API servers):
# Classic histogram (requires sum by (le) to preserve bucket boundaries):
histogram_quantile(0.99, sum by (le) (rate(apiserver_request_duration_seconds_bucket[5m])))

# Native histogram (no grouping by le required!):
histogram_quantile(0.99, sum(rate(apiserver_request_duration_seconds[5m])))
</code></pre></div>With native histograms, functions like <code>histogram_quantile()</code> operate directly on the dynamic exponential spans inside the time series, producing highly accurate quantiles without static bucket interpolation error.
For official documentation on querying Native Histograms in PromQL, see:
<ul>
<li><a href="https://prometheus.io/docs/prometheus/latest/querying/functions/#histogram_quantile">PromQL <code>histogram_quantile</code> documentation</a></li>
<li><a href="https://prometheus.io/docs/prometheus/latest/querying/functions/#histogram_fraction">PromQL <code>histogram_fraction</code> documentation</a></li>
<li><a href="https://prometheus.io/docs/prometheus/latest/querying/functions/#histogram_count-and-histogram_sum">PromQL <code>histogram_sum</code> documentation</a></li>
<li><a href="https://prometheus.io/docs/prometheus/latest/querying/functions/#histogram_count-and-histogram_sum">PromQL <code>histogram_count</code> documentation</a></li>
<li><a href="https://prometheus.io/docs/prometheus/latest/querying/functions/#histogram_count">PromQL <code>histogram_count</code> documentation</a></li>
<li><a href="https://prometheus.io/docs/prometheus/latest/querying/functions/#histogram_stddev-and-histogram_stdvar">PromQL <code>histogram_stddev</code> and <code>histogram_stdvar</code> documentation</a></li>
</ul>
<h2 id="dashboard-migration-rollback-strategy">Dashboard migration & rollback strategy<a class="td-heading-self-link" href="#dashboard-migration-rollback-strategy" aria-label="Heading self-link"></a></h2><h3 id="recommended-migration-workflow">Recommended migration workflow<a class="td-heading-self-link" href="#recommended-migration-workflow" aria-label="Heading self-link"></a></h3>To safely transition your monitoring infrastructure to Native Histograms without breaking existing alerts or dashboards, I recommend a four-step migration workflow:
<ol>
<li>Enable Both Formats: In your Prometheus 3.x scrape config, set <code>scrape_native_histograms: true</code> AND <code>always_scrape_classic_histograms: true</code> so both formats are collected safely during transition</li>
<li>Migrate Queries: Update your Grafana dashboards and Prometheus alerting rules from classic quantile queries (<code>histogram_quantile(..._bucket...)</code>) to native histogram queries (<code>histogram_quantile(...)</code>), and replace references to classic <code>_count</code> and <code>_sum</code> series with <code>histogram_count(...)</code> and <code>histogram_sum(...)</code></li>
<li>Verify in Staging/Production: Validate that all dashboards and SLO alerts fire and graph correctly using the new native histogram queries</li>
<li>Unlock ~10x Storage Savings: Once migration is complete, set <code>always_scrape_classic_histograms: false</code>. Prometheus will stop ingesting the static <code>_bucket</code>, <code>_count</code>, and <code>_sum</code> time series, reducing your histogram time series count by up to 90%!</li>
</ol>
<h3 id="opt-out-and-rollback-flexibility">Opt-out and rollback flexibility<a class="td-heading-self-link" href="#opt-out-and-rollback-flexibility" aria-label="Heading self-link"></a></h3>Because native histograms are dual-exposed, using them is entirely opt-in from a collector perspective:
<ul>
<li>Instant Collector Rollback: If you need to stop ingesting native histograms, simply set <code>scrape_native_histograms: false</code> in your Prometheus job configuration. No Kubernetes restart is required, and Prometheus will immediately resume scraping only the classic format without data loss</li>
<li>Component Feature Gate Rollback: Administrators can also disable the feature gate on Kubernetes components using <code>--feature-gates=NativeHistograms=false</code> (requires component restart)</li>
</ul>
<h2 id="what-s-next-how-to-get-involved">What's next & how to get involved<a class="td-heading-self-link" href="#what-s-next-how-to-get-involved" aria-label="Heading self-link"></a></h2>As native histograms progress toward General Availability (GA) in future Kubernetes releases, SIG Instrumentation will continue evaluating ecosystem readiness, performance characteristics, and long-term plans for eventually deprecating static classic buckets once native histogram adoption becomes ubiquitous across the monitoring community.
<ul>
<li>Read the <a href="https://www.kubernetes.dev/resources/keps/5808/">KEP-5808 page</a> or the <a href="https://kep.k8s.io/5808">KEP GitHub issue</a> to learn more.</li>
<li>Read the <a href="https://prometheus.io/docs/specs/native_histograms/">Prometheus Native Histograms specification</a> and <a href="https://prometheus.io/docs/prometheus/latest/querying/functions/#histogram_quantile">PromQL querying functions documentation</a></li>
<li>Get involved with <a href="https://github.com/kubernetes/community/tree/main/sig-instrumentation">SIG Instrumentation</a> on Slack in #sig-instrumentation or join the weekly SIG meetings</li>
</ul>
<h2 id="acknowledgements">Acknowledgements<a class="td-heading-self-link" href="#acknowledgements" aria-label="Heading self-link"></a></h2>A huge thank you to contributors across SIG Instrumentation and component owners who collaborated on the design, implementation, testing, and review of native histograms in Kubernetes!

AI News

Kubernetes v1.37: Native Histograms Graduates to Beta (kubernetes.io)

Comments