A benchmark is not an SLO: production needs task rate, tails, and abstention

The signal and the facts that change the decision

New models arrive with benchmarks, but production operates with its own distribution, dependencies, tool failures, and service objectives. A public score informs selection; it does not define the application SLO.

A benchmark measures a sample under a protocol; an SLO measures a real service over time. Quality needs sample audit and indirect signals because not every answer has an immediate label.

Correct abstention may be success where inventing an answer is worse than not answering. The system can retain a high benchmark while failing on current data, local language, or partial tool results.

Decision, limits, and stop condition

Convert product value into observable SLOs: accepted task, grounding, latency, cost, safety, abstention, and recovery.

The system can retain a high benchmark while failing on current data, local language, or partial tool results.

Execution and proof of completion

Execution should start by defining task, population, severity, and accepted outcome. Then, instrumenting sampled quality, tail latency, cost, and dependency failures. Closure requires linking slo breach to fallback, scope limitation, or model rollback.

The service publishes SLO, measurement method, coverage, error budget, and decisions taken when budget is exceeded.

• defining task, population, severity, and accepted outcome.
• instrumenting sampled quality, tail latency, cost, and dependency failures.
• linking SLO breach to fallback, scope limitation, or model rollback.
• The service publishes SLO, measurement method, coverage, error budget, and decisions taken when budget is exceeded.