p95 and p99 tell you what your slowest users experience; the average tells you almost nothing useful about latency.
A percentile is the value below which a given share of observations fall. p95 latency is the value 95% of requests are faster than; p99 is the value 99% are faster than. Percentiles capture the tail of the distribution that averages hide.
An average latency can look healthy while a meaningful fraction of users suffer, because a few very slow requests barely move the mean. p95 and p99 expose that tail: p99 latency is the experience of your unluckiest one in a hundred requests, which is often where churn and complaints come from.
You cannot average percentiles: the p99 of two services is not the average of their p99s. Percentiles must be computed from the underlying distribution (which is why latency is recorded as a histogram). Aggregating pre-computed percentiles gives wrong answers, a frequent and costly mistake.
The how-to dashboards on this site graph p95 and p99, not averages, and TelemetryOps preserves the distributions Sentinel AI needs to reason about tail latency.
The latency value that 99% of requests are faster than; only 1% of requests are slower, representing the tail of the experience.
No. Percentiles cannot be averaged; they must be computed from the underlying distribution, which is why latency is stored as a histogram.
Ops Singularity turns open telemetry into autonomous, governed resolution. See it on your own stack.