Why P95 Latencies Cannot Be Averaged: Quantiles, Histogram Merging and Timeout Checks

Define the timed object before choosing a quantile algorithm

Enterprise AI observability and failure diagnosis separates business outcomes from runtime signals. Latency also has distinct objects: a complete user task, one model call, first output and complete output have different start and end points. This article considers nonnegative durations for an explicitly defined population and timing boundary. Mixing these objects does not produce one business-meaningful P95.

The main example uses the discrete nearest-rank definition: sort N valid durations and select position ceil(0.95×N), counting from one. It describes the observed dataset. It does not establish a 95% probability that the next request will fall below that value. Window choice, sample size and traffic composition all affect interpretation.

PostgreSQL 18 distinguishes percentile_disc from percentile_cont, which can interpolate between adjacent values; its ordered-set aggregates ignore null inputs.[1] For the constructed durations 1, 2, 3 and 4 seconds, discrete P95 is four seconds and the continuous result is 3.85 seconds. A different algorithm is not itself a fault, but cross-system comparisons need an agreed definition.

Document four choices: timing boundaries, inclusion rules, window assignment and quantile algorithm. Assigning tasks by acceptance time requires time to collect outcomes and a visible count of unfinished work. Assigning by completion time describes tasks completed during that window. Both can be useful. Neither justifies silently using accepted tasks as the denominator while retaining only completed durations.

A counterexample that request weighting cannot repair

All data here are constructed. Instance A completes 100 requests, each taking one second. Instance B completes ten requests, each taking ten seconds. Their P95 values are one and ten seconds. The simple average is 5.5 seconds; request weighting gives (100×1+10×10)/110, approximately 1.818 seconds.

The combined dataset has 110 observations. Its discrete P95 is at position ceil(104.5)=105. Positions 101 through 110 all contain ten seconds, so the global P95 is ten seconds. In this special example, 1.818 seconds happens to equal the overall mean duration, but it remains the wrong P95. Every value within each group is identical here; that condition cannot be generalized.

A quantile retains a value near one ordering position, not the complete distribution. Local P95 values and sample counts generally cannot reconstruct the combined ordering. The same problem applies to regions, model versions or five-minute windows. Daily P95 is not the average of five-minute P95 values.

Either compute centrally from raw samples or preserve distribution summaries with explicit merge rules. Raw data support reconciliation and regrouping at a storage and query cost. Summaries save resources but introduce accuracy and compatibility constraints. If historical data contain only local P95 values, adding weights cannot retroactively produce an accurate global result.

Merge bucket counts before estimating the combined quantile

OpenTelemetry describes a histogram as multiple counters representing a distribution rather than one statistic.[2] Our fixed-bucket example uses inclusive upper bounds of 0.5, 1, 2, 5 and 10 seconds, plus positive infinity. For the 110 observations above, cumulative counts are 0, 100, 100, 100, 110 and 110. Do not add cumulative buckets to obtain the population count; the final bucket already covers all observations.

For Prometheus data already collected as classic bucket series, the following expression illustrates aggregation. Selected data must share units, timing objects, inclusion rules and bucket boundaries. Scope the selector to one intended service and population before use. The metric name is illustrative and application-defined, not an assumed built-in metric.

histogram_quantile(0.95, sum by (le) (rate(ai_request_duration_seconds_bucket[5m])))

Prometheus documents merging classic histogram buckets before estimating a quantile; precomputed summary quantiles cannot be combined this way.[3] Its function documentation requires rate before sum when combining counter rates and aggregation so that individual resets can be handled.[4] Retaining le preserves bucket boundaries. To separate business classes, also retain their dimensions through selection or grouping.

This yields a distribution estimate from bucket rates over five minutes, not an exact ordering of individual log records. Missing scrapes, extrapolation at window edges and low traffic affect comparisons with logs. Verify the mathematics using complete, aligned fixed counts first, then inspect the production collection pipeline rather than conflating both sources of error.

Compatible buckets do not make the answer exact

Continue with the same dataset. P95 falls above five seconds and at or below ten seconds. Classic linear bucket interpolation gives 5+(104.5−100)/(110−100)×5=7.25 seconds, while the discrete P95 from raw observations is ten seconds. This is a formula calculation for constructed data, not a measurement of a production Prometheus deployment.

The error arises from estimating an unknown within-bucket distribution linearly when all ten observations actually lie at the upper boundary. The function documentation states the interpolation assumption for classic buckets.[4] If the business question is whether at least 95% of requests take no more than five seconds, and every instance has that exact boundary, the cumulative bucket fraction is more direct: 100/110, approximately 90.91%, which misses the target.

That fraction still applies only to included observations and depends on complete collection. Without a boundary at the threshold, a neighboring bucket is not an exact substitute. If some instances lack the five-second bucket, its numerator covers only part of the population while the denominator covers all of it. Extra decimal places cannot repair that structural mismatch.

More buckets can improve resolution in selected ranges while increasing classic time-series storage and query costs. The classic example supports troubleshooting existing systems; it is not a universal recommendation for new projects. With native histograms, verify compatibility across the chosen client, server and storage path instead of copying this bucket-series query. Isolate configuration versions during a bucket change, or establish compatibility before merging them.

Timeouts and retries can silently change the population

Recording duration only after successful completion may remove the slowest timed-out requests entirely. Consider 100 constructed tasks: 90 succeed in one second and ten time out after 20 seconds. Successful-only P95 is one second, but only 90% of accepted tasks succeeded. This does not support a claim that 95% of users received a result within one second.

A timeout record establishes when waiting ended. It generally does not establish when the backend would have completed successfully if waiting had continued. Filling that unknown duration with zero, or treating the timeout limit as actual successful completion time, changes the question. If measuring user waiting until any terminal outcome, record the observed wait at timeout but show outcome status too; fast failure is not fast success.

Report the successful-delivery duration distribution, timeout and cancellation counts, deadline-success fraction over accepted tasks, and unfinished task count separately. For cohorts assigned by acceptance time, assess deadline success after deadlines have elapsed or clearly mark provisional results. Do not force unknown durations into an apparently complete successful-latency quantile.

Retries introduce two denominators. A user task triggering three model attempts can legitimately produce three model-call durations, but the user-level record should capture one complete wait and final outcome. Mixing these gives tasks with more attempts extra weight. Link task and attempt identifiers in controlled logs; do not turn every request identifier into a metric label with unbounded series growth.

Validate the dashboard with partition invariance and failure samples

First, retain the fixed 100 one-second and ten ten-second observations. Verify total count 110, discrete raw P95 ten, bucket counts and the hand-derived estimate 7.25. Redistribute the same observations across three instances. With unchanged population and buckets, the merged answer should remain unchanged. If instance count changes the global result, check for an average of local quantiles.

Second, partition the dataset into non-overlapping smaller windows, merge their incremental bucket counts and calculate the larger-window result. Do not sum cumulative counter snapshots or repeatedly include overlapping windows. This check concerns fixed data; production rate queries need separate scrutiny of scrape intervals and reset handling.

Third, add explicit timeouts, cancellations, null durations and duplicate records, checking which population receives each. Look for silently excluded nulls, unclear deduplication ownership and zero latency displayed when no samples exist. Fourth, simulate instance restarts and incompatible bucket settings. Require a visible anomaly or isolation policy instead of an unexplained aggregate value.

Fifth, compare raw data and dashboards by version and task class. Overall P95 may improve because easy tasks became more common while a critical class slowed down. Keep useful business groups while controlling label counts. Acceptance evidence should include scope, window, sample count, algorithm, boundaries, missing-data fraction and known limitations.

The deliverable is a reproducible calculation path: define the population from acceptance and outcome records, build its duration distribution, apply a declared algorithm and make decisions against a business threshold. Without raw samples or a mergeable distribution, retaining an honest unknown is more useful than making an incorrect global P95 look precise.