Can You Trust Your Own Eval?
A framework and controlled validation for measuring exploitability and contamination in LLM evaluation harnesses.
The evaluations that decide which model an enterprise buys, promotes, or ships are private internal harnesses, and no one has audited them for the failures now thoroughly documented on public benchmarks. This paper separates three failure families — contamination, exploitability, defectiveness — defines a metric for each, and validates them by sealed-versus-leaky differential testing across three harness archetypes and up to seventeen open-weight models (0.5B–32B). A deliberately non-capable exploit takes all attainable score on the leaky harnesses and drops to the non-capable floor once they are mechanically sealed. A model fine-tuned on a leaked set shows a +0.78 perturbation gap that a clean model and a difficulty-matched control do not. Whether a defect changes which model wins turns out to depend on the defect. And the same compression pattern shows up in lm-eval-harness's GSM8K filters — infrastructure we did not build.
The gap
The last twelve months produced a run of demonstrations that a high benchmark score can measure the benchmark's machinery rather than the model. METR found frontier reasoning models reward-hacking about 30% of runs on an agentic coding suite, and continuing to do so under explicit instructions not to. OpenAI retired SWE-bench Verified after an audit found a majority of a sampled subset had defective tests. A Berkeley team reached perfect or near-perfect scores on most of eight agent benchmarks by attacking graders, exposed reference solutions, and under-specified oracles. Every one of those results concerns a public benchmark built and scrutinized by specialists. The harnesses that decide procurement and release are private, heterogeneous, and assembled under deadline by teams whose expertise is their domain rather than adversarial measurement, and there is no reason to expect they hold up better.
How you measure it
Hold the models fixed and vary the harness: a leaky configuration built from common production defaults against a sealed one hardened per vulnerability class. Four readings come off that comparison.
HES is the fraction of attainable score a deliberately non-capable exploit policy captures. Capability is pinned at chance by construction, so whatever it scores is the harness leaking. DFR is how often pairwise model rankings reverse between configurations, under a bootstrap indifference zone so ties aren't counted as flips; the sealed oracle is hand-verified tests on authored items, so a disagreement means the leaky harness is wrong, not merely different. EDR covers what DFR structurally can't see — defects that compress instead of reorder — by measuring the share of pairs the sealed harness orders decisively that the leaky one collapses into ties. PG is the score drop from a semantically preserving rewrite of the eval items, read as memorization only when a clean model and a difficulty-matched control stay flat.
What the validation shows
The non-capable exploit reaches HES = 1.00 on leaky code grading and lands at the non-capable floor sealed — 0.06 on the easy tier, 0.10 on a 53-problem hard tier. An adaptive version that probes the sandbox and fires four vectors does no better than the fixed one, so the seal isn't just blocking the attack it was designed against. That seal is mechanical rather than conventional: candidate code runs in a subprocess that never holds the expected outputs, and a released test_seal.py enforces the property against the exploits that beat the previous in-process grader 1.000. Ablating the leak vectors one at a time answers the obvious strawman worry — trusting self-reported results and leaving the reference solution reachable each independently yield HES = 1.0, while weak tests alone yield roughly zero. Elsewhere: a constant "list every option" response scores 1.00 leaky and 0.17 sealed on the multiple-choice grader, no better than guessing; a weak judge accepts an injected wrong answer half the time leaky and a quarter sealed, while a stronger judge resists it even leaky.
Decision impact is where the result stopped matching the assumption. On a de-saturated hard tier built to remove the score ceiling, the weak-test defect flips zero of 89 decisively ordered pairs — a powered null (one-sided 95% bound 0.033 pair-level, 0.16 under the most conservative model clustering), not an underpowered shrug. But the same defect silently erases 18% of the distinctions the sealed harness could make, and 38% back at the easy tier's ceiling. Judge attacks and contamination behave differently: they change the winner. A contaminated model scores 100% and gets selected, then falls to 22.5% once the harness is sealed.
Test-strength and aggregation defects don't reorder the field — they blind the harness to it. Judge attacks and contamination change which candidate is selected.
Someone else's harness
The sharpest objection is that all of the above runs on harnesses we wrote. So this version audits one we didn't: the answer-extraction stage of EleutherAI's lm-evaluation-harness on GSM8K, both filters reproduced verbatim from upstream with per-file SHA-256s recorded. The compression pattern replicates — flexible-extract inflates honest accuracy over strict-match by up to +0.105 — with near-zero non-capable exploitability under either filter, because GSM8K's open numeric target can't be null-modeled the way a multiple-choice grader or a judge can. One finding worth reporting rather than burying: strict-match's robustness is entirely its sentinel requirement, and an attacker who prepends #### to a constant guess recovers the same 0.04.
What it doesn't show
The instruments are validated on controlled archetypes, not on deployed harnesses. The lm-eval audit is one stage of one public harness — a step toward external validity, not its resolution. The archetypes are reconstructions of documented practice, so prevalence in the wild stays an estimate. HES is a lower bound given the released exploit policy; a more creative one raises it. PG is one-sided: a large gap is strong evidence of contamination, a small one certifies nothing. The paper closes with a sealed-harness reference design and audit checklist that is deliberately mundane — no novel cryptography, just hygiene. Code, prompts, seeds, the seventeen-model campaign with pinned checkpoint digests, and the contaminated adapter are in the Zenodo deposit and on GitHub, so anyone with a real harness can point the same instruments at it.