What it is
Against the backdrop of the agent hype, an important counter-trend emerged: a sober check on productivity. A randomized controlled trial (RCT) by METR showed that experienced open-source developers working on mature projects they knew well were 19% slower with AI tools than without them - even though they themselves were convinced AI had sped them up.
Where it came from
- 10 July 2025 - METR publishes "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity" (an RCT - the gold standard of evidence).
- 16 experienced developers, 246 tasks from their own repositories, with AI randomly allowed or forbidden (Cursor Pro + Claude 3.5/3.7 Sonnet).
- The result clashed with intuition and set off a wave of debate about whether AI actually speeds you up.
Why it matters
- The perception gap: developers expected a 24% speedup and estimated 20% afterwards - while the real number was -19%. Subjectively "faster" is not objectively faster.
- Context matters: the effect was measured on mature projects with high standards, where the author is already deep in the material - which is exactly where AI helps least.
- It's a healthy counterweight to marketing: "a 10x engineer thanks to AI" isn't a universal truth.
How to apply it now
- Measure your own productivity with facts (time, throughput), not with the feeling that it "seems faster".
- Use AI where the win is real: unfamiliar code or stack, boilerplate, prototypes - not on code you know by heart.
- Learn to use the tool: METR named weak prompts and unfamiliarity with the interface among the causes of the slowdown.
What to watch out for
- This is a snapshot of early 2025 on a narrow group (experienced OSS maintainers) - it doesn't cancel out the gains for beginners and it isn't a verdict on agents; don't over-generalize.
- Tools have improved a lot since then - rerun the measurement on current models.
- The takeaway isn't "AI is useless" but "the effect is context-dependent and easily overestimated" - keep that in mind when you evaluate ROI.