Can You Trust Your Own Eval?
A framework and controlled validation for measuring exploitability and contamination in LLM evaluation harnesses.
Public benchmarks have been shown to be gameable (an exploit agent scored 100% on five popular agent benchmarks without solving a single task), but the evaluations that actually gate enterprise model selection are private internal harnesses, and those have never been audited for the same failures. This paper separates three failure families (contamination, exploitability, defectiveness), defines metrics for each, and validates them by differential testing over eight open-weight models and three harness archetypes: a deliberately non-capable exploit captures all attainable score on the leaky harnesses and little or none once they are sealed, a model fine-tuned on a leaked set shows a +0.78 perturbation gap that clean and held-out controls do not, and whether a defect changes which model gets picked depends on its type.
The gap
The last twelve months produced a run of demonstrations that a high benchmark score can measure the benchmark's machinery rather than the model. METR found frontier reasoning models reward-hacking about 30% of runs on an agentic coding suite, and continuing to do so under explicit instructions not to. OpenAI retired SWE-bench Verified after an audit found a majority of a sampled subset had defective tests. A Berkeley team reached perfect or near-perfect scores on most of eight agent benchmarks by attacking graders, exposed reference solutions, and under-specified oracles. Every one of those results concerns a public benchmark built and scrutinized by specialists. The harnesses that decide procurement and release are private, heterogeneous, and assembled under deadline by teams whose expertise is their domain rather than adversarial measurement, and there is no reason to expect they hold up better.
How you measure it
The method is differential testing: hold the models fixed and vary the harness between a leaky configuration representing common production defaults and a sealed one hardened against each vulnerability class. Three metrics come off that comparison. HES is the fraction of attainable score a deliberately non-capable exploit policy captures: capability is held at chance by construction, so anything it scores is the harness leaking. DFR is the rate at which pairwise model rankings reverse between the two configurations, under a bootstrap indifference zone so ties are not counted as flips; because the sealed oracle is hand-verified tests on authored items, a disagreement is the leaky harness being wrong, not merely different. PG is the drop from a semantically preserving rewrite of the eval items, read as memorization only against a clean model and a difficulty-matched held-out control.
What the validation shows
On leaky code grading the non-capable exploit reaches HES = 1.00, and 0.06 sealed. An adaptive version that probes the sandbox and fires several vectors does no better than the fixed one, so the seal is not just blocking the single attack it was built for. A constant "list every option" response scores 1.00 on the leaky multiple-choice grader and 0.17 sealed, no better than random guessing. A weak judge accepts an injected wrong answer half the time leaky, a quarter sealed, and its judging accuracy goes from 0.71 to 0.89. Decision impact turns out to be defect-dependent, which was not assumed: weak tests and pass@k inflation compress scores toward the ceiling without reordering anything (DFR = 0.00 in both), whereas judge attacks and contamination change which candidate wins: a contaminated model scores 100% and would be selected, then drops to 22.5% once the harness is sealed.
Test-strength and aggregation defects distort absolute scores without reordering; judge attacks and contamination change which candidate is selected.
What it doesn't show
This validates the instruments on controlled archetypes, not deployed harnesses. The archetypes are reconstructions of documented practice, so prevalence in the wild stays an estimate. The reported HES is a lower bound given the released exploit policy; a more creative one raises it. And PG is one-sided: a large gap is strong evidence of contamination, a small one certifies nothing. The paper closes with a sealed-harness reference design and audit checklist that is deliberately mundane (no novel cryptography, just hygiene), and all code, prompts, seeds, and the contaminated adapter are in the Zenodo deposit, with the runnable code on GitHub, so anyone with a real harness can point the same instruments at it.