Skip to main content

Latency percentiles

A mean latency line is the chart that hides the outage. The percentile envelope is its replacement: the median as a line, and the distance to the tail drawn as a band around it. When the distance opens up, something is wrong for some of your users — and that is the thing a single line cannot say.

The chart​

Find the place where the envelope tears open while the median line barely moves: p99 goes up ×5.7 and p50 only ×1.23. Ninety-nine percent of the service is fine. One percent is having an outage.

The stall is so much bigger than everything else that it sets the axis, which is why the normal envelope looks like a thin ribbon along the floor. Drag across a quiet hour to zoom — the two tones separate immediately. That compression is itself the argument for the log axis in "Options to try".

(Clock times below are UTC — the fixture's own frame. The axis renders in your local timezone, so the stall will sit at a different hour on your screen.)

The data​

Modelled, not measured — latency percentiles at a useful resolution aren't something anyone publishes. Modelled on the process: a diurnal load curve with noise that scales with the signal, plus one discrete event — a downstream cache going cold at 11:12 and staying cold for nine minutes.

The series is already wide, which is how percentile data usually arrives from a metrics store:

time | p50 | p90 | p99

That matters for the chart shape below — a <BandChart> takes a lower and an upper column, so pre-aggregated percentiles need no further work. If your percentiles arrive long (one row per quantile), reshape first; if you have raw events rather than percentiles, rolling with quantile reducers gets you here.

The generator is seeded and epoch-fixed, so the chart is byte-identical every visit.

Build it​

One band and one line is the whole idea:

<Layers>
  <BandChart series={series} lower="p50" upper="p99" as="outer" axis="ms" />
  <LineChart series={series} column="p50" axis="ms" />
</Layers>

That's the envelope. It has one weakness: a single band can't distinguish "the 90th is drifting up" from "the 99th alone blew out", and those want different people woken up.

Nesting a second band fixes it, and costs one line:

<BandChart series={series} lower="p50" upper="p99" as="outer" axis="ms" />
<BandChart series={series} lower="p50" upper="p90" as="inner" axis="ms" />

Two-tone for free: outer and inner are tints of the same hue, and the inner one paints over the outer, so the darker core is p50–p90 and the lighter halo is the tail beyond it. Draw order is the only thing making that work — inner after outer.

Last, the number the envelope is being judged against:

<Baseline value={400} axis="ms" label="p99 objective" />

A <Baseline> is an annotation, drawn in the annotation register — one hue no series ever takes — so a placed mark is never mistaken for data.

Options to try​

OptionWhat it doesReach for it when
A third nested band (p50–p75)Finer tail resolutionYou genuinely read the mid-quantiles; three is the ceiling
lower="p01" upper="p99" symmetric about p50A true envelope rather than an upward-only oneLatency that can be anomalously fast too (cache hits)
<Baseline onChange>Drag the objectiveThe SLO is under discussion — see SLA & incidents
Log-scaled <YAxis>Compresses the tail so the median stays readablep99/p50 exceeds ~10× and the median flattens to the floor
cursor="crosshair"Reads one point rather than the whole sliceReading a single value off the tail

See also​