Native Histograms in Kubernetes 1.37: Migration Results and SLI Design

Native histograms can remove thousands of _bucket series from a Kubernetes monitoring stack. The harder part is making sure the dashboards, alerts, and remote storage still see the same requests afterward.

I compared four Prometheus configurations against the same single-node kind cluster. The native-preferred configuration reduced head series by 76.4% and RSS by 34.7%. I also found queries that returned only one component's data, aggregations that returned an empty result, and histogram samples that never reached remote storage.

This article assumes familiarity with Prometheus scraping and basic PromQL. It covers Kubernetes v1.37.0 and Prometheus v3.13.2, using the measurements recorded in September 2026. The remote storage results apply to the specific versions listed below.

What changed, and what the experiment measured

Kubernetes 1.37 makes the NativeHistograms feature gate beta and enabled by default. Supported histograms expose both classic and native representations; Prometheus decides which to ingest. The gate does not make every Kubernetes histogram native. See the Kubernetes feature gate definition and KEP-5808.

Native histograms became stable in Prometheus 3.8.0. In the version used here, collection is controlled through scrape configuration, without a native-histogram feature flag.

All four Prometheus instances scraped the same kube-apiserver and kubelet jobs every 15 seconds:

After approximately one hour:

The block measurements came from a snapshot with skip_head=false, which writes the Head to a block. They were not measured after normal two-hour Head compaction or subsequent block compaction. WAL is the usage at that instant, not a long-term storage estimate.

These are single-point measurements from a small cluster. Scrape timing, changes in the source metrics, GC, and process state introduce variation. In particular, P2's slightly lower RSS and chunk usage do not establish that storing both formats saves resources.

For P3, the reductions relative to P1 were:

• Head series: 76.4%
• Block index: 76.0%
• Block chunks: 65.2%
• RSS: 34.7%

Series reduction is useful, but it is not a memory or storage sizing formula.

Why the series count falls

A classic histogram stores one float series per bucket boundary, plus _sum and _count. For a histogram with 20 bucket boundaries and 100,000 label combinations, that is:

label_combinations = 100_000
bucket_boundaries = 20
classic_series = label_combinations * (bucket_boundaries + 2)  # 2,200,000

A native histogram stores the distribution as one histogram sample per label combination:

native_series = label_combinations  # 100,000

The bucket counts move inside the sample. Increasing resolution therefore increases sample size, rather than creating more time series.

This explains the difference between index and chunk savings. The index contains series labels and postings; fewer series remove much of that overhead. Chunks store the samples themselves, and a histogram sample is larger than a float. WAL, network traffic, and in-memory chunks also depend on sample size.

Ordinary labels still matter. A route label with 5,000 distinct values continues to create 5,000 label combinations. Native histograms do not fix unbounded URLs, user IDs, or request IDs in labels.

What the distributions look like

Grafana Labs illustrates the difference using response-time heatmaps and histograms from the same application.

Native

Classic

Source: Grafana Labs — Prometheus native histograms in Grafana Cloud. These images come from Grafana's example, not the kind experiment. Axis ranges, scales, and displayed time windows differ, so compare the visible structure rather than bar heights or colors numerically. Classic histograms can also use finer boundaries, at the cost of more series.

Exponential buckets provide consistent relative resolution

For standard exponential native histograms, adjacent boundaries have the ratio:

g = 2 ** (2 ** (-schema))

Kubernetes configures a bucket factor of 1.1, which client_golang maps to schema 3. Its actual growth factor is approximately 1.0905: eight buckets per doubling, with a relative width of about 9.05% measured from the lower boundary.

The boundaries nest. Combining adjacent pairs of schema 3 buckets produces schema 2 buckets by adding their counts, without interpolation. You can reduce resolution exactly at the bucket-count level, but you cannot recover the finer distribution afterward.

Illustrative counts, with rounded boundaries. The diagram follows the Prometheus schema specification and OpenTelemetry exponential scale definition.

Sparse encoding avoids materializing the entire theoretical range. In the Mimir measurements, 1,259 active native histogram series contained 17,064 active buckets: an average of 13.6 buckets per series.

The Prometheus native histogram specification explains the data model, schemas, and encoding. NHCB uses a different schema: it retains explicitly supplied bucket boundaries rather than exponential ones.

Enable collection, then rewrite the queries

For an existing job, P3 requires this change:

scrape_configs:
  - job_name: kubernetes-apiservers
    scrape_native_histograms: true
    always_scrape_classic_histograms: false
    # Keep the existing discovery, authentication, and relabeling settings.

To keep classic series during migration, set always_scrape_classic_histograms: true for P2. To convert classic-only histograms to NHCB, also set convert_classic_histograms_to_nhcb: true for P4.

The first three switches can be set globally or per job. Starting with a job limits the scope of the change. Prometheus also supports target-level overrides through relabeling. The options are documented in the v3.13.2 configuration reference.

For the exporter libraries and exposition formats used in this experiment, native histogram collection requires protobuf negotiation. Enabling scrape_native_histograms makes protobuf preferred by default. If you explicitly configure scrape_protocols, check that PrometheusProto is preferred. A text response can contain only the classic representation even when the exporter supports native histograms.

The base name replaces the suffix series

When classic storage is disabled for a native-capable metric, the queryable series change:

For example, preserve the job grouping while removing le:

# Classic p99 per job
histogram_quantile(
  0.99,
  sum by (job, le) (rate(apiserver_request_duration_seconds_bucket[5m]))
)

# Native p99 per job
histogram_quantile(
  0.99,
  sum by (job) (rate(apiserver_request_duration_seconds[5m]))
)

Other common replacements are:

# Request rate
histogram_count(rate(http_request_duration_seconds[5m]))

# Mean duration per input series
histogram_avg(rate(http_request_duration_seconds[5m]))

# Fraction at or below 500 ms, aggregated across the selected series
histogram_fraction(
  -Inf, 0.5,
  sum(rate(http_request_duration_seconds[5m]))
)

Apply rate() before aggregation so resets are handled per series. Preserve the labels that define the service or operation you intend to measure.

A histogram is not a float. This comparison produces no usable alert result:

rate(http_request_duration_seconds[5m]) > 1

Extract a float first:

histogram_quantile(0.99, rate(http_request_duration_seconds[5m])) > 1

Audit dashboards, recording rules, alert rules, and queries against remote storage. A syntactically valid query can still return nothing after its _bucket, _sum, or _count series disappear.

The subtle failure: a query returns only one component

Of the 74 classic histogram metric families observed in the two jobs, 47 became native and 27 remained classic in P3.

Some remaining histograms use a different implementation, such as TimingHistogram; others come from the Go collector rather than Kubernetes component-base. A third group is affected by initialization order.

The same metric name can therefore have different representations in different components:

This query returns kubelet data only:

sum(rate(workqueue_work_duration_seconds[10m]))

The kube-apiserver observations live in workqueue_work_duration_seconds_bucket and its companion series. Prometheus has no reason to warn that the native query omitted a component you expected to include.

Check coverage before replacing a query:

# Jobs exposing the base-name series
count by (job) (workqueue_work_duration_seconds)

# Jobs exposing classic buckets
count by (job) (workqueue_work_duration_seconds_bucket)

In kube-apiserver v1.37.0, informers and REST clients can be constructed before ApplyFeatureGates() propagates the gate state to the metrics subsystem. Histograms initialized at that point remain classic, while later ones become native. I reported this in kubernetes/kubernetes#142379.

NHCB changes the failure mode

NHCB packages classic bucket counts into native histogram samples while retaining their original boundaries. It reduces series count, but does not improve resolution.

P4 converts the remaining kube-apiserver workqueue histogram to custom buckets, while kubelet still exposes exponential buckets. Aggregating the two produces a warning and an empty result:

sum(rate(workqueue_work_duration_seconds[10m]))

{
  "status": "success",
  "data": { "resultType": "vector", "result": [] },
  "warnings": [
    "PromQL warning: vector contains a mix of histograms with exponential and custom buckets schemas for metric name \"\" (1:5)"
  ]
}

The original kind experiment found eight metric names with this problem. All eight returned an empty result when aggregated across the incompatible schemas.

Keep the aggregation groups separate, for example by job, and inspect API warnings. NHCB is useful for classic-only exporters you cannot change, provided the aggregation and storage paths support it. It is not an automatic final step for every migration.

Remote storage can undo the migration

There are two separate questions: did Prometheus send the samples, and how did the receiver store them?

Local collection and long-term availability require separate checks.

Remote write v1 does not send native histograms by default

For the default v1 protocol, explicitly enable sending:

remote_write:
  - url: http://thanos-receive:19291/api/v1/receive
    send_native_histograms: true

Without this option, Prometheus can store histograms locally while sending none of them to remote storage. The v3.13.2 queue implementation returns early without a log or a dropped-sample increment for this disabled path.

If classic storage has also been disabled, the long-term store loses that histogram entirely.

NHCB requires remote write v2. For a receiver that supports it:

remote_write:
  - url: http://mimir:8080/api/v1/push
    protobuf_message: io.prometheus.write.v2.Request

With v2, native histogram sending is always enabled; send_native_histograms is a no-op. With v1 and native sending enabled, NHCB samples are dropped with a warning and the reason nhcb_in_rw1_not_supported.

Receiver behavior differs

These results are scoped to the tested versions, rather than a claim about every release:

The P4-to-Thanos path shows the additional v1 experiment: exponential samples were sent, while NHCB samples were dropped by Prometheus.

Thanos Receive ingested 12,580 histogram samples without an additional native histogram flag. Its v0.41.0 Receive implementation uses the v1 request type.

Mimir reported 1,259 native histogram series among 7,435 active series. When using query sharding in this version, its help text requires -query-frontend.query-result-response-format=protobuf for native histogram queries. This is an internal response-format requirement, separate from the public JSON query API.

VictoriaMetrics expanded the distributions back into bucket series. Comparing separate receivers fed classic and native data gave:

Conversion during ingestion reintroduces multiple float series per distribution.

The receiver's series reduction was 20.5%, compared with 76.4% in the Prometheus Head. Some metrics gained bucket series because more converted buckets were populated than the original classic definition contained. The conversion is visible in the v1.152.0 ingestion implementation.

Head series and totalSeries cover different populations and time spans. These counts also do not measure stored bytes. Measure the receiving system's storage and query behavior before projecting long-term savings.

These choices apply to the tested versions. For P4, also verify aggregation compatibility between exponential and custom bucket schemas.

Better resolution does not make p99.9 stable

The migration changes what distributions you can retain economically. It does not create more requests.

There are two different sources of uncertainty:

Increasing schema addresses the second. It cannot resolve the first.

Count the observations in the tail

For N requests and quantile p, the expected number above a fixed population quantile is:

k = N * (1 - p)

With 1,000 requests, p99.9 is effectively tracking the very slowest observations. A finer bucket layout cannot make those observations repeat consistently.

Under independent observations from a fixed distribution, the count above a fixed population quantile is binomial. Its relative standard deviation is sqrt(p/k), approximately 1/sqrt(k) for high quantiles. At k = 1, that is about 100%; at k = 100, about 10%. See the NIST description of the binomial distribution.

Those percentages describe count variation, not percentage error in the latency value. They cannot be directly compared with a 9.05% bucket width. The value uncertainty also depends on the density of observations near the quantile.

Using 100 expected tail observations as a planning heuristic gives:

This is a way to reason about window length, not an accuracy guarantee. Correlated traffic or a changing workload can still make the result unstable.

Scraping is not random request sampling: counters and histograms normally accumulate events between scrapes. Keep finite-window uncertainty separate from bucket approximation, missed scrapes, resets, and rate() extrapolation.

Threshold SLIs still need an error model

A threshold SLI expresses the fraction of requests that meet a latency target:

histogram_fraction(
  -Inf, 0.5,
  sum(rate(http_request_duration_seconds[1h]))
)

This fits the good events / total events model used for error budgets. It does not eliminate low-traffic variation.

It also introduces interpolation when the threshold is inside a bucket. A factor of 1.1 does not guarantee that the estimated good-event fraction is within 10% of the true fraction. If observations cluster near the threshold, even a narrow bucket can contain enough requests to materially change the result.

For exact threshold counts, retain a classic boundary at the SLO threshold or instrument dedicated good/total counters. Prometheus v3.13.2 also supports histogram_fraction() for classic histograms, so estimating arbitrary thresholds is not exclusive to native histograms. Native exponential buckets make relatively fine boundaries available across a broad range.

The Prometheus histogram guidance explains value-domain and quantile-domain errors. The Google SRE Workbook discusses multiple latency thresholds to capture both typical experience and the long tail.

A larger p99 does not automatically mean worse performance

In the P2 experiment, apiserver_request_duration_seconds produced:

The classic definition's largest finite boundary was 60 seconds. When the requested quantile falls in +Inf, histogram_quantile() returns the previous bucket's upper bound. Native buckets could represent observations beyond that boundary.

This is evidence of a representation limit, not a comparison against raw-event ground truth. The aggregate also included long-running requests such as watches. It is not an example of an appropriate SLI for ordinary API calls. Define the operations and labels first, then compare the same scope and observation window.

Instrumenting a Go exporter

For client_golang, native support starts in v1.14.0. The example below uses the options in v1.24.0:

var requestDuration = prometheus.NewHistogramVec(
    prometheus.HistogramOpts{
        Name:    "myapp_request_duration_seconds",
        Help:    "Request duration in seconds.",
        Buckets: existingBuckets,

        NativeHistogramBucketFactor:     1.1,
        NativeHistogramMaxBucketNumber:  160,
        NativeHistogramMinResetDuration: time.Hour,
    },
    []string{"method", "route", "status"},
)

Keep the existing classic buckets during migration. Observe each event once; the histogram maintains both representations. promhttp.Handler() can negotiate the protobuf response.

If native histograms are enabled and Buckets is omitted, client_golang does not fill in its default finite classic buckets. Text exposition may still expose _sum, _count, and +Inf; do not assume every classic-related series disappears from the endpoint.

Choose the bucket limit from the observed range

For positive values, a rough estimate of the buckets needed to densely cover a range is:

from math import log, log2

# Approximate bucket count; both expressions are equivalent.
N = log(Vmax / Vmin) / log(g)
N = log2(Vmax / Vmin) * (2 ** schema)

At schema 3:

Sparse observations may occupy far fewer buckets. The estimate helps identify distributions where a limit of 160 could force coarser resolution, especially size metrics spanning many orders of magnitude.

Recovery from reduced resolution has a cost

In client_golang v1.24.0, exceeding NativeHistogramMaxBucketNumber can trigger a reset, widening of the zero bucket, or reduction of the schema. With NativeHistogramMinResetDuration left at zero, the reset-based recovery path is disabled. A reduced schema does not recover through that mechanism while the same histogram instance remains alive.

Setting time.Hour allows a reset after the minimum interval when needed; it is not an unconditional hourly reset. The reset affects the whole histogram, including classic cumulative values. rate() can correct an observed reset, but cannot recover observations accumulated after the preceding scrape and lost before the reset was scraped.

Kubernetes v1.37.0 component-base configures factor 1.1 and limit 160, but leaves the minimum reset duration unset. I raised the tradeoff in kubernetes/kubernetes#142377. For your own exporter, choose the limit and reset policy together, using the client_golang v1.24.0 option documentation and implementation.

A rollout that checks data coverage

The first migration task is an inventory of metric names and jobs, followed by queries and remote storage.

1. Find classic families with count by (__name__) ({__name__=~".+_bucket"}) and locate their consumers in dashboards, recording rules, and alerts.
2. Collect both formats for a bounded comparison period if the impact is broad. Compare counts, sums, and means over matching labels and windows. Expect quantiles to differ when boundaries or interpolation differ.
3. Check which jobs expose each representation. Verify that rewritten aggregations still include every intended component.
4. Check native sending and query the receiver directly. For NHCB, verify v2 support and keep incompatible schemas out of the same aggregation group.
5. Disable classic storage for native-capable metrics once those checks pass. Keep access to the classic queries for historical data until it ages out.

A direct P1-to-P3 switch can also work when the affected queries and storage paths are already known and checked. Configuration reloads change future collection; they do not convert existing samples.

For SLI design, decide the user-facing latency thresholds and required success fractions first. Then choose a window with enough observations and a bucket layout that meets the required approximation accuracy. Native histograms make broad distributions cheaper to retain, but neither tail traffic nor threshold accuracy comes for free.

kubectl -n nh-verify exec deploy/prom-native -- \
  wget -qO- 'http://localhost:9090/api/v1/status/tsdb'

kubectl -n nh-verify exec deploy/prom-native -- \
  wget -qO- --post-data='' \
  'http://localhost:9090/api/v1/admin/tsdb/snapshot?skip_head=false'