← All posts
This post is written in English.
METR's 2025 study made headlines by finding experienced open-source developers took 19% longer with AI. Its follow-up post from February 2026 has been cited often in recent months, usually without the most interesting part: the experiment stopped working, because developers didn't want to take part.
The numbers
The later study covered 57 developers, 143 repositories and over 800 tasks.
| Group | Change in task time | 95% interval |
|---|---|---|
| Early-2025 study | 19% slower | +2% to +39% |
| Returning developers, late 2025 | 18% faster | -38% to +9% |
| New developers, late 2025 | 4% faster | -15% to +9% |
Neither late-2025 interval excludes zero. METR calls this "only very weak evidence" that speedups grew.
Why it's unreliable
- Developers increasingly declined to join because they didn't want to work without AI.
- When surveyed, 30% to 50% said they had held back tasks they didn't want to do without AI.
- Pay dropped from $150 to $50 an hour, which likely affected who took part.
- Time tracking was unreliable for people running several agents at once.
METR says this likely biases the estimate downward, so the true speedup could be higher, but it can't say how much.
My read
"AI makes developers X% faster" is a claim I've stopped trusting from almost any source, in either direction. Good measurement here is really hard, as METR has shown by being honest about its own failure.
Yet the failure itself tells you something. If many skilled developers refuse to go without a tool, even for pay, they believe it helps them. That belief isn't proof, since the original study found people misjudge their own speed. It's still a real shift in a little over a year, and I'm looking forward to seeing whatever METR's new design finds.
