Test with confidence, code with clarity

• Public benchmarks saturated and the leaderboards stopped carrying ship signal. The Stanford AI Index 2026 and a saturation review across 60 benchmarks flagged 29 as high or very high on a saturation index. Variance between frontier models approaches measurement noise. Harder benchmarks are built, saturate, and get replaced on the same curve.
• Gaming is now a documented leaderboard failure mode. A null model (fixed-string output) topped AlpacaEval 2.0 under automated judging. Multilingual contamination studies report leakage up to 91.8% on popular suites. Hugging Face retired its Open LLM Leaderboard in 2025 over the compounding of these failures. Public numbers describe style and contamination as well as capability.
• A private eval built from your domain vocabulary is the ship signal. Public benchmarks tell a team capability shape, usefully as triage. Your suite tells the team what the model will do on the team's own tasks, in the team's own vocabulary, against the team's own invariants. One number sorts the shortlist. The other decides the shortlist.
• The eval suite is the test suite in domain language. The evals-wearing-a-lab-coat post called the eval-versus-test split a role artifact. The private eval is the operational consequence: it lives in the same vocabulary as the production tests, assembled from the same builders, asserting on the same invariants.

Read the argument: Your Private Eval Is the Only Benchmark That Steers You