In early 2025, the research group METR set out to measure something many people assumed they already knew: whether AI coding tools make experienced programmers faster. Joel Becker, Nate Rush, Beth Barnes and David Rein recruited 16 developers who had worked for years on large open-source projects, repositories that averaged more than a million lines of code. Each developer brought real issues from their own project, such as bug fixes, features and refactors, 246 in all, each averaging about two hours.
The design was the strong part. Every issue was randomly assigned to one of two conditions: AI tools allowed or AI tools not allowed. When AI was allowed, developers used whatever they liked, mostly Cursor Pro with Claude 3.5 and 3.7 Sonnet. Because the same developer worked under both conditions on comparable real work, the comparison is much cleaner than asking people whether the tools helped. Developers were paid an hourly rate for their time, and the authors recorded their screens while they worked.
The result went against nearly everyone's expectation. With AI allowed, developers took 19% longer to finish an issue. Before the study, they had predicted AI would make them about 24% faster. After the study, having just experienced the slowdown, they still estimated that AI had made them about 20% faster.
The headline is the slowdown, but the more durable finding is the distance between what the developers felt and what the clock recorded. The people in the study were skilled, motivated, and watching their own work closely. They still could not tell. This is why self-reported benefit is a weak form of evidence for any tool, program or training. It measures how the experience felt, and the experience of getting help is pleasant whether or not the help saves time.
The authors are careful here, and the care is part of the lesson. Sixteen developers is a small group, and the authors acknowledge that the people who volunteered may differ from developers in general. Completion time was self-reported alongside screen recordings, which leaves room for measurement error. The result describes early-2025 tools in one setting: experienced developers in codebases they knew well. The authors say they are not concluding that AI slows down most developers, that it fails in other fields, or that no better way of using it would help. They note it may help people who are less experienced or working in unfamiliar code. They describe the result as a snapshot of early-2025 capabilities and have since published follow-up work.
When someone says a tool made their team faster, ask how speed was measured and against what comparison. A good answer names an outcome you can check from records, not from memory, and a comparison where the same work was done with and without the tool. If the evidence is a survey, treat it as a hypothesis worth testing rather than a result.
If you are trying to find out whether an AI tool or a program really works for your team, send us what you have. Working out a fair comparison is usually a short conversation.
One email whenever we publish something new. No spam, unsubscribe anytime.
{{ subscribeErrorMsg }}