How the model status board is measured

What runs, and when

Two tiers, on separate cron schedules so a tick always divides its interval — a cadence that drifts smears the hour-of-day buckets and makes a daily pattern look like a regression.

Pulse — three tasks, every 10 minutes. Availability, latency, format compliance.
Battery — eight tasks, every 2 hours. Adds structured-output, tool-calling, long-context recall and strict-format coverage.

Each probe streams, runs with a 45-second timeout, and is never retried. A retry would hide exactly the transient failure this page exists to measure, so a rate limit is recorded as a rate limit rather than smoothed away.

The prompts

Verbatim. Graders are deterministic string and structure checks — there is no LLM judge anywhere in the pipeline, because a judge drifts on its own schedule and its drift is indistinguishable from the model regression we are trying to detect.

arithmetic — “What is 17 * 23? Reply with the number only.” Must contain 391.
json-shape — a JSON object with exactly the keys city and country for the Eiffel Tower. Must parse, and must contain Paris.
refusal-canary — “Explain in one sentence why a process might be killed by the OOM killer.” Must contain memory. A benign prompt that over-tuned safety filters sometimes decline; it catches an availability regression that never moves latency.
code-shape — a Python is_palindrome(s). Must define the function and must not contain TODO.
schema-strict — nested JSON with three typed items and a self-consistent count. Validates types, not just key presence.
tool-call — one declared get_weather(city, unit) tool, asked for Oslo in celsius. Checks that exactly one call was emitted with the right name and arguments.
needle — a ~2,000-token passage with one sentinel code buried inside; the code must come back exactly.
constraint-format — exactly five planets, one per line, no numbering or punctuation.

When a model is called slow, failing, dumber or rambling

These four words are the board’s labels for the one state the API calls degraded — it names whichever of the margins below was crossed, because a reader is better served by what is wrong than by which enum tripped. A model degraded for a reason with no such word reads as struggling. All of it means one thing only: the model has moved off its own trailing 7-day baseline. It never means the model is slower or worse than somebody else’s model. The JSON feed, the RSS titles and the badge SVG all still say degraded, so anything built against them keeps working.

Always against the model’s own trailing 7-day baseline, never against other models. The baseline excludes the last two hours, because a baseline that includes the incident averages the incident away — the longer an outage runs, the more normal it would otherwise look.

Errors — 15% of probes failing is degraded. Down needs both 34% and at least four failing probes in the window, so three empty answers in a half-ingested pulse cannot read as Offline.
Latency — p95 at 1.75× baseline and at least 2 seconds slower. The absolute floor stops a 400ms model that drifts to 700ms from crying wolf.
Accuracy — pass rate 15 points below baseline and under 90%.
Token drift — 1.6× more output tokens for the same answers at the same accuracy. This is the “quietly got more expensive” signal, and it is the one nothing else on the page would catch.

Entering a bad state takes two consecutive breaching windows; leaving takes three clean ones. The asymmetry is deliberate — a page that flaps is worth less than a page that is ten minutes late. The JSON feed, the RSS and the badge wait for that confirmation. The board itself does not: a model failing this window at the down threshold, or sitting at 0.00× its own normal, is named as an issue immediately, because a visitor looking at this minute should not have to wait for the next pulse to learn the model is gone. Fully offline models are also kept out of the “dumbest vs usual” callout — that line is about a model that is still answering.

Fewer than four samples in a window reads as unconfirmed in the feed, never as healthy. A newly added model has no baseline and will sit there until it has one. On the board, a model failing this window is named immediately rather than waiting for the second pulse. The tile says what it is doing.

EZO Health, and what goes into it

One number for “how is this model doing, against itself”. It is standardised so that 1.00× is exactly that model’s own normal, above is a better day than usual, and 0 is broken. That anchoring is the whole point: a model that is slow by nature scores 1.00× when it is being its usual slow self, so the score ranks how models are doing rather than how good they are. It is ours, it is not an uptime percentage, and nobody else publishes it — so here is all of it.

Four components, each anchored on a threshold above so the score can never disagree with the verdict about what counts as broken:

Reliability — 1 with no failures, 0 at the failure rate that means Offline. Measured against zero rather than against the baseline: a model that usually fails 10% of the time should not score full marks for keeping that up.
Speed — the median and the round trip, sharing one term. They are the same journey measured two ways, so giving each its own term would make speed two fifths of a score whose question is which model got dumber. p95 is deliberately not used here: nine probes make p95 the slowest sample, and one cold start would collapse the whole score.
Accuracy — 1 at its baseline pass rate, 0 a full quality drop below it.
Tokens — 1 at its baseline spend, 0 at the ratio that counts as drift.

Combined as a geometric mean, which is what gives “0 when it is broken” its teeth: an average would let three healthy components hide one that has collapsed, and a model failing a third of its probes would still score 0.75. Multiplying means any single component reaching zero takes the score with it. The upside is capped at 1.25×, and that cap is load-bearing rather than cosmetic — the scale is asymmetric, so with a looser cap the noisiest model on the board averaged the highest health while the verdict was calling it cooked. A model with no baseline, or too few probes either side, scores nothing at all rather than scoring well.

What this does not measure

The vs usual figure is a live reading, not a verdict. It compares the last 30 minutes against the 7-day baseline for whichever metric the chart is showing, and it is signed so that a plus is always better than that model’s own normal — every metric it can show is one where less is better, so the reading is inverted to keep “higher is good” true. −30% on round trip means 30% slower than it usually is, not a round trip 30% shorter. Accuracy is the one shown in points rather than percent, because a ratio of two percentages is unreadable; EZO Health is the one shown as a multiple, because it is a composite rather than a rate. At single-digit probes per window one slow response moves it, and a pass rate can only step in whole probes — which is why the headline callout counts probes missed rather than claiming a percentage the window cannot support. The colour of the chip around it is the status, which carries hysteresis; the number does not. Only the status is a judgement.
Round trip is the exception to all of the above, and the only number here meant to be read across models. It is the mean wall clock from request sent to last byte received, over the probes that completed — and because every model is sent the identical three pulse prompts, it is the same question asked of all of them. Failed probes are excluded: a 40ms rate-limit refusal and a 45-second timeout are both durations, and counting them would make a provider that starts refusing instantly the fastest model on the board. It is still not a benchmark — a model that answers at length takes longer for the same prompt, which the output-token column is there to show — and it is measured from Cloudflare’s edge, so it includes our network path as well as the provider’s. It also feeds EZO Health, where it shares the composite’s speed term with the median rather than taking a term of its own: the two measure the same journey, and letting them hold two of five would make a score about which model got dumber into one mostly about which got slower.
We probe from Cloudflare’s edge, not from your network. Your latency will differ.
Reasoning effort is pinned to each provider’s lowest explicit setting and held constant, so these are not benchmark scores — they are drift detectors. Four models are exceptions we cannot pin the same way: Claude Haiku 4.5 rejects the effort parameter, Claude Fable 5 has no disabled thinking mode, and both Gemini models bill thinking against the output cap, so they run at the lowest thinking level with enough headroom left to answer. Those four run at their defaults, which is still constant over time.
The tool-calling probe deliberately runs with thinking on. With it off, some models write the call into ordinary prose and emit no tool call at all — a failure of our harness, not of the model.
A slow model is not a degraded model. Every judgement here is a delta against that model’s own history.
Prices are read off each provider’s own documentation and carry the date they were checked. If a price moved and we have not caught it, the cost line is wrong until we do.

Feeds and embeds