← Back to blog
AI Evaluation

The Study That Found AI Made Developers Slower

By Dr. Nedim Yel, Co-Founder & Lead Statistician · Updated
3 min read

In early 2025, the research group METR set out to measure something many people assumed they already knew: whether AI coding tools make experienced programmers faster. Joel Becker, Nate Rush, Beth Barnes and David Rein recruited 16 developers who had worked for years on large open-source projects, repositories that averaged more than a million lines of code. Each developer brought real issues from their own project, such as bug fixes, features and refactors, 246 in all, each averaging about two hours.

The design was the strong part. Every issue was randomly assigned to one of two conditions: AI tools allowed or AI tools not allowed. When AI was allowed, developers used whatever they liked, mostly Cursor Pro with Claude 3.5 and 3.7 Sonnet. Because the same developer worked under both conditions on comparable real work, the comparison is much cleaner than asking people whether the tools helped. Developers were paid an hourly rate for their time, and the authors recorded their screens while they worked.

The result went against nearly everyone's expectation. With AI allowed, developers took 19% longer to finish an issue. Before the study, they had predicted AI would make them about 24% faster. After the study, having just experienced the slowdown, they still estimated that AI had made them about 20% faster.

The number that matters is the gap

The headline is the slowdown, but the more durable finding is the distance between what the developers felt and what the clock recorded. The people in the study were skilled, motivated, and watching their own work closely. They still could not tell. This is why self-reported benefit is a weak form of evidence for any tool, program or training. It measures how the experience felt, and the experience of getting help is pleasant whether or not the help saves time.

What the study does not show

The authors are careful here, and the care is part of the lesson. Sixteen developers is a small group, and the authors acknowledge that the people who volunteered may differ from developers in general. Completion time was self-reported alongside screen recordings, which leaves room for measurement error. The result describes early-2025 tools in one setting: experienced developers in codebases they knew well. The authors say they are not concluding that AI slows down most developers, that it fails in other fields, or that no better way of using it would help. They note it may help people who are less experienced or working in unfamiliar code. They describe the result as a snapshot of early-2025 capabilities and have since published follow-up work.

Where else this shows up

  • AI evaluation. A user satisfaction score for an AI assistant measures how people feel about it. Whether the work got faster or more accurate needs a separate measurement, ideally from a comparison like this one.
  • Program evaluation. Participants who say a program helped them are reporting a perception. Pair it with a measured outcome and a comparison group before treating it as an effect.
  • Academic research. Studies of "perceived learning" or "perceived productivity" can run opposite to tested learning and measured output, as they did here.

What to ask for instead

When someone says a tool made their team faster, ask how speed was measured and against what comparison. A good answer names an outcome you can check from records, not from memory, and a comparison where the same work was done with and without the tool. If the evidence is a survey, treat it as a hypothesis worth testing rather than a result.

If you are trying to find out whether an AI tool or a program really works for your team, send us what you have. Working out a fair comparison is usually a short conversation.

Related reading

Sources and further reading

Get new posts by email

One email whenever we publish something new. No spam, unsubscribe anytime.

Check your inbox — click the confirmation link to finish subscribing.

{{ subscribeErrorMsg }}