← Back to blog
AI Evaluation

Part of our AI Eval Watch series · Nov 11–17, 2026

AI Eval Watch, Nov 11–17: Unfinished Agent Work, Untested Claims, Evidence-Seeking Judges

By Dr. Nedim Yel, Co-Founder & Lead Statistician · Updated
6 min read

This week's AI-evaluation reading was about the distance between looking finished and being finished, and about whether the claims attached to new models have been tested at all. Below are four preprints posted between November 13 and 15. They are not peer reviewed, and for each one we describe what the authors report in their abstract; we have not read the full papers or checked the results ourselves.

Coding agents that pass tests but rarely finish the job (Lei and colleagues, preprint, Nov 15)

SWE-CoWork drops a coding agent into a simulated software company. It has 101 assignments, each taken from a different real repository, covering 11 programming languages and 12 application domains. The requirements are not handed over at the start: they are scattered across 26 simulated colleagues, each with a defined role and limited knowledge, and across workplace records, so the agent has to go and ask. Hidden executable tests check the result. Across the eleven models evaluated, the two strongest passed roughly 63% of the fail-to-pass tests, yet fully completed fewer than 8% of the assignments. When the authors gave agents the complete requirements up front, the test pass rate rose by about 16 percentage points, but the share of fully completed assignments did not move.

Why it matters. The same runs produce two very different numbers depending on what counts as the unit of success. Scored per test, the best agents look more than half competent; scored per assignment, they almost never finish. Neither number is wrong, but they answer different questions, and a report showing only the per-test rate would overstate how often the agent's work could actually ship. On validity, hidden executable tests anchor the score to a known outcome, which is firmer ground than a model-graded score. The limit is that the company and its colleagues are simulated, and the abstract does not say how closely they resemble real teams. The abstract also does not report run-to-run variation, so it does not tell us how reliable the 63% and 8% figures are. Our guide on evaluating an LLM beyond average accuracy covers how the choice of unit and summary changes the story.

Most novelty claims in model reports go untested (Türkoğlu, preprint, Nov 15)

This single-author paper audits what recent open-weight language models actually changed, and whether their reports back the changes with evidence. In the Hugging Face transformers library, 39 of the 48 architectures added during 2025 borrow at least two of their three core building blocks from an earlier model, most often Llama. Across a sample of 40 technical reports making 73 architectural or training novelty claims, 19 claims are supported by a controlled comparison of the model with and without the change. Only 4 reports state that more than one training run was done, and only 1 discloses the data mixture, token count and compute together. The author proposes seven reporting rules and applies them to two reports. The paper is explicit that it does not rate the models themselves, and that a claim lacking such a test could still turn out to be correct.

Why it matters. A with-and-without comparison is the basic design for attributing an improvement to a specific change, and a single training run gives no estimate of how far the result would shift on a rerun with a different random seed. Without that spread, there is no way to say whether a reported gain is larger than training noise. That is a reliability question, and it comes before any question of validity. The author's caution is the right one: missing evidence is not evidence against a claim; it means the claim has not been tested. The three questions in our guide to vetting an AI vendor's accuracy claims apply to technical reports too: what was compared, how it was measured, and against what baseline.

Spreadsheet work graded by expert rubrics (Garre and colleagues, preprint, Nov 13)

GDP.xlsx is a benchmark of 70 spreadsheet requests written by professionals in 13 fields. Each request comes with its workbook or CSV file and an expert rubric of yes-or-no criteria. An agent judge that can open the file applies the rubric, and some criteria require a list-style answer to be complete with nothing extra included. Models are ranked by strict pass@1, so a single attempt has to satisfy the rubric. Across 20 model configurations, the best passed 38.3% of attempts and the median configuration passed 19.3%. The authors release the evaluation harness and three tasks, and keep the remaining 67 private to limit contamination.

Why it matters. Rubrics written by experts for real requests are a credible route to validity, because the criteria describe what a professional would accept. But an AI judge applies the rubric, so the score is only as valid as that judge's reading of it, and the abstract does not say whether the judge's verdicts were checked against human graders. With 70 tasks, sampling error is also large: by our calculation, a pass rate near 38% on 70 items carries a 95% margin of error of about 11 percentage points, before any run-to-run variation, so close rankings deserve caution. Holding most tasks back helps against contamination, but it also means outsiders cannot rerun the full evaluation. Our guide on how many test items an AI eval needs shows how slowly that margin shrinks, and our guide on telling whether one LLM is really better than another covers comparing two models scored on the same items. Our free model-comparison calculator runs the paired test for that simple case.

Judges that gather evidence before ruling (You and colleagues, preprint, Nov 13)

This survey starts from a weakness of standard LLM judges: they often score a transcript without the evidence needed to judge it, such as the state of an environment an agent changed, or facts that need outside checking. The authors group evaluators that collect evidence before deciding under the name Agent-as-a-Judge, and organize their design around four elements: several cooperating agents, tool use, memory and planning. They compile published results, rerun key agentic judges where results were missing, and report that the benefit of the agentic approach is uneven. They close with open problems in cost, reliability, safety and governance.

Why it matters. Letting a judge see the evidence addresses validity directly: a judge that cannot tell whether a file was actually changed cannot score the task correctly, however consistent it is. More moving parts, though, means more sources of variation, so an agentic judge needs its own reliability check (does it reach the same verdict twice?) and its own validity check against careful human grading. The uneven gains the authors report are a reason to test an agentic judge on your own tasks rather than assume it beats a simpler one. Our guide on what makes an AI eval defensible to auditors covers checking a scoring method against a human baseline.

The week in three lines

  • Report agent results at the level a user cares about, such as finished assignments, next to any partial-credit score.
  • Treat a model claim with no with-and-without comparison and only one training run as untested, not as proven.
  • Small benchmarks and AI-applied rubrics both add uncertainty: report the margin of error and check the judge against people.

If you are building evaluation metrics for an AI product and need them to hold up under real scrutiny, this is squarely what we do.

Sources

  • Lei, X., Liu, M., Zhu, R., Hu, X., et al. SWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations. arXiv:2611.20218 (preprint).
  • Türkoğlu, B. The novelty exposed: ablation, reproducibility and transparency in 40 technical reports of open-weight large language models (LLMs). arXiv:2611.20003 (preprint).
  • Garre, S., Mehta, S., & Chen, E. GDP.xlsx: Benchmarking Frontier Models on Professional Workbooks. arXiv:2611.17766 (preprint).
  • You, R., Cai, H., Zhang, C., Xu, Q., et al. Agent-as-a-Judge. arXiv:2611.18067 (preprint).
Get new posts by email

One email whenever we publish something new. No spam, unsubscribe anytime.

Check your inbox — click the confirmation link to finish subscribing.

{{ subscribeErrorMsg }}