Who Gains Most From AI? Heterogeneous Treatment Effects

By Dr. Nedim Yel, Co-Founder & Lead Statistician · Updated November 6, 2026

3 min read ▶ Listen to this post

"The tool raised productivity by 15%" is a useful sentence and an incomplete one. It describes a typical worker who may not exist. Two large studies of AI assistants show why the average is only the start. Both found a solid overall gain, and both found that the gain was very uneven across people.

In a study of customer-support agents at one software company, Erik Brynjolfsson, Danielle Li and Lindsey Raymond followed about 5,000 agents. Access to an AI assistant raised the number of issues resolved per hour by about 15% on average. For newer, less experienced agents the gain was much larger, 34% in the Bureau of Labor Statistics summary of the paper, while the most experienced and highest-skilled agents saw small gains in speed and small declines in quality. In a separate set of three field experiments with software developers at Microsoft, Accenture and a Fortune 100 manufacturer, covering 4,867 developers, a team led by Cui, with Demirer, Jaffe, Musolff, Peng and Salz, found a 26.08% increase in completed tasks among developers given an AI coding assistant, with less experienced developers adopting the tool more and gaining more.

Why the average is not the answer

If a tool helps novices a lot and experts very little, the average depends on how many of each you happen to have. A team full of experts would see a small average gain, a team of new hires a large one, and neither would match the headline number. The same logic applies to a tutoring program, a training course or a clinical protocol: the effect is not one number but a pattern across people. Statisticians call this a heterogeneous treatment effect, and finding out who benefits is often the most useful result of an evaluation.

How to look for it without fooling yourself

• Decide the groups before you see the data. Pick the few subgroups that your theory says matter, such as experience level, baseline score or site, and write them down in the analysis plan. A subgroup chosen after the fact is a forking path.
• Test the difference, not the two groups separately. The question is whether the effect differs between groups. "Significant for novices, not significant for experts" does not show that. Test the interaction between the treatment and the grouping variable, and report the effect in each group with its interval.
• Expect low power. A study sized to detect an average effect is usually too small to detect a difference between subgroups. If the interaction is the point, plan the sample for it, or describe the result as a lead for the next study.
• Count your comparisons. The more subgroups you check, the more likely one looks striking by chance. State how many you looked at, and treat unplanned findings as hypotheses.
• Use a continuous variable when you have one. Splitting people into "high" and "low" experience throws information away. Modeling the effect as a function of experience shows the whole pattern.
• Check for a baseline artifact. People who start low have more room to gain, and the lowest scorers drift upward on their own. A larger gain among novices is partly built in unless the comparison group is handled carefully.

What to ask for instead

When someone reports an average effect, ask who it applies to. Ask which subgroups were planned in advance, how many were examined, and whether the difference between them was tested directly. For a tool you are deciding whether to adopt, the question that matters is the gain for the people you actually have.

If you are designing an evaluation that needs to say who benefits, the sample size has to be planned for it, and our power analysis tools are a place to start. If you would rather talk it through, send us the design.

Related reading

• How to Interpret Interaction Effects in Regression — the model behind a test of whether an effect differs between groups.
• The Bias That Disappeared When You Looked Closer — Simpson's paradox: what a pooled number hides about the groups inside it.

Sources and further reading

• Brynjolfsson, E., Li, D., & Raymond, L. R. (2025). Generative AI at work. The Quarterly Journal of Economics, 140(2), 889–942.
• U.S. Bureau of Labor Statistics, Monthly Labor Review. AI improved the productivity of a Fortune 500 software company (summary of the study above).
• Cui, Demirer, Jaffe, Musolff, Peng, & Salz. The effects of generative AI on high-skilled work: Evidence from three field experiments with software developers. Management Science.

One email whenever we publish something new. No spam, unsubscribe anytime.