In 2020, the team that built GPT-3 did something most benchmark papers skip: they checked their own homework for cheating. GPT-3 had been trained on a compressed slice of the internet — hundreds of billions of tokens scraped from Common Crawl, books, and Wikipedia — and the paper's authors, Tom Brown and his colleagues at OpenAI, knew that a scrape that size had a real chance of accidentally swallowing the exact test questions the model would later be graded on. So before publishing a single benchmark score, they ran every evaluation set against the training data looking for overlap, using a method that flagged any 13-word sequence in a test example that also showed up somewhere in what GPT-3 had already read.
What they found was uncomfortable. Roughly a quarter of the benchmarks they checked had more than half of their test examples flagged as overlapping with the training set in some way. For most of those benchmarks, once they re-scored the model on only the clean, non-overlapping examples, the accuracy barely moved — evidence that the overlap was mostly incidental phrasing, not the model having memorized the actual answers. But not every benchmark got off that easy. On the reading-comprehension benchmark SQuAD, 94% of the test set turned out to overlap with the training data, and scoring GPT-3 on just the clean remainder dropped its accuracy by about six points. On the Winograd Schema Challenge, a test of commonsense pronoun resolution, 45% of examples were flagged, and manual review found 132 of the actual Winograd schemas sitting in the training data in a slightly reformatted form — not just similar phrasing, the real questions. The published scores for both benchmarks got an asterisk.
Every benchmark that gets used for more than a year or two eventually ends up copied, quoted, discussed, and re-posted somewhere on the open web — in a paper, a GitHub repo, a blog post walking through the answers, a forum thread. Train a model on "the internet," and you are, whether you intend to or not, also training it on fragments of the very tests the field uses to measure it. This is train-test contamination, and at internet scale it isn't a rare accident — it's closer to a background rate. The GPT-3 team's finding that a quarter of their benchmarks showed heavy overlap wasn't a story about sloppiness; it was a story about what happens whenever a training corpus and an evaluation corpus are both built by scraping the same public internet.
The more useful part of the finding is what it implies about benchmark scores in general: a high score on a leaked benchmark and a high score on a genuinely unseen one can look identical in a results table, and nothing about the number itself tells you which one you're looking at. The only way to tell the difference is to actually check for overlap and then re-score on what's left — which is exactly the extra step most benchmark comparisons, then and now, don't take. GPT-3's team happened to do it and happened to publish what they found, including the cases where it didn't matter and the two where it clearly did.
Don't stop at the headline benchmark score. Ask the question GPT-3's own team asked about itself: has this benchmark been checked for overlap with the training data, and does the score hold up on the examples confirmed to be clean? A model that's never had that check run on it might be exactly as good as its score says — or it might be getting graded on a test it already had the answer key for, and there's no way to tell which from the score alone.
If you're evaluating a model, comparing benchmark claims across vendors, or building an eval pipeline of your own, talk to us before you take a leaderboard number at face value.
One email whenever we publish something new. No spam, unsubscribe anytime.
{{ subscribeErrorMsg }}