MODEL EVALUATION: BENCHMARKS LIE, PRODUCTION DOESN'T

Model evaluation is the practice of measuring whether a given model actually works for your specific use case, rather than trusting the benchmark scores its makers publish. It is the layer of an AI system that actually works that tells you which model to trust, and when.

Why Benchmarks Mislead

Every model ships with a headline number. One scores 92% on a public test set, another 91%, and the leaderboard invites you to conclude the first is better. For your use case, that conclusion is unfounded. A benchmark is curated, so it never contains the queries that arrive at three in the morning when the API is overloaded, or the ones where the model is confident and wrong, or the ones riddled with typos and half-formed grammar that real users produce.

Benchmarks are also adversarial by design, built to probe edge cases rather than the average request. A model that scores 92% against an adversarial set might score 99% on your actual traffic, or 60% — the number alone does not tell you which. And benchmarks are static: the set was frozen at some point in the past, while your product and its language have kept moving. Treat published scores as marketing, because that is largely what they are.

Measuring On Your Own Distribution

The only reliable evaluation runs the candidate models against queries drawn from your own product. A working test set is 100 to 200 real queries, each paired with a known-correct answer taken from your documentation, a human judgment, or established ground truth. That set is a compact model of your actual use case, and it is worth far more than any leaderboard because it measures the thing you actually care about.

Running two models against the same set is the same A/B discipline you already use in prompt engineering, where identical inputs isolate the effect of a single change. Scoring is domain-specific: for code generation, does it run; for summarization, does it cover the main points; for question answering, is the response grounded in the source documents, which is the core test of any retrieval-augmented generation system. Two models that post nearly identical totals often diverge sharply on the handful of queries where they disagree, and that is where the real difference between them lives.

Cost-Adjusted Evaluation

Raw accuracy is only half of a decision. A model that answers correctly 87% of the time at thirty cents a query is not obviously better than one that manages 85% at ten cents. Dividing accuracy by cost per query reframes the comparison, and it frequently flips which model wins: a two-point accuracy loss against a threefold cost saving is a trade most systems should take.

Once you hold that cost-adjusted number, the cheaper model tends to become the sensible default, with the expensive one reserved for the hard queries that genuinely need it. Deciding which queries deserve the costlier model is its own discipline, closely tied to inference optimization and worked through in the cost decision tree for routing queries to models.

What Real Evaluation Looks Like

The table below sets the two approaches side by side. It is less a scorecard than a reminder of what a benchmark quietly leaves out.

Dimension	Benchmarks	Real Evaluation
Data	Curated	Your actual distribution
Diversity	Adversarial, edge cases	Average case + edge cases
Freshness	Static, old	Current, updated monthly
Cost	Ignored	Measured, factored in
Time	Single snapshot	Trending over time

Real evaluation is more work, and that is the point. It is the only method that answers the question a benchmark cannot: does this model work for you, on your data, at a price you can live with.

Degradation And The Red Flags

A model that scored 95% in testing can lose ground steadily in production — 87% user acceptance in the first week, then 84%, then 79% — not because anything broke but because the benchmark was measured on clean, formatted data while production data is full of typos, abbreviations, and context-dependent language it never saw. This kind of drift is normal, and it is exactly what a one-time evaluation gate is blind to.

Some of the most dangerous signals are metrics that improve while users grow unhappier: benchmark accuracy rising as acceptance falls, output length growing as people read less, speed improving as the error rate climbs, or confidence increasing in step with hallucination. When confidence and hallucination rise together, scoring alone will not save you; you need verification as a separate gate on every output.

Evaluation As A Cadence

Because models degrade quietly, evaluation is not a gate you pass once but a feedback loop you run on a schedule. A workable rhythm is monthly, quarterly, and yearly: each month, rerun your existing test set on the current model to catch drift; each quarter, compare it against one or two competitors to check it is still the right choice; each year, rebuild the test set, because your distribution has moved out from under the old one.

The same cadence governs the decision to switch models. When a new model appears, the honest answer is to measure it — run both against a batch of recent production queries and compare accuracy, user preference, cost, and latency — rather than to trust the release notes. A cycle of this costs only a few hours, and it is what keeps a slowly degrading model from running unnoticed for months.

Open Questions

• As models increasingly train on public benchmarks, how do you build test sets that stay uncontaminated and genuinely predictive of production behavior?
• What is the right way to evaluate open-ended generation — reasoning, long-form writing, multi-step judgment — where there is no single correct answer to score against?
• When degradation comes from a shifting user distribution rather than the model itself, how do you tell the two apart quickly enough to act before acceptance erodes?

Part of the knowledge graph at The Best Blog Ever — reference definitions for ideas that matter.