Faster output is only one part of better work. The harder question is what happens between the first answer and something another person can depend on.
The feeling of speed
A useful AI session can feel productive. Suggestions arrive quickly. An empty file becomes a plausible implementation. There is visible movement, and movement is satisfying.
But an active screen is not a complete measurement. The draft still needs to fit the codebase, survive review, and behave correctly outside the example that produced it. A reading of AI productivity should include that quieter work.
A gap worth noticing
In an early-2025 randomized trial, METR studied 16 experienced open-source developers completing 246 real tasks in repositories they knew well. Participants expected AI access to reduce completion time by 24%. Afterward, they believed it had reduced time by 20%. The study instead estimated a 19% increase. Original research
Expectation meets evidence.
Change in completion time vs. no AI. Negative means less time; positive means more. METR early-2025 trial.
Source: METR · Early-2025 trial
View the data
| Group | Value (%) |
|---|---|
| Expected before | -24% |
| Believed after | -20% |
| Measured | +19% |
These are changes in completion time, not productivity rates. The measured estimate used self-reported implementation time, including work before and after review. This particular setting does not represent all developers or today’s tools.
Keep the update attached
METR’s February 2026 follow-up complicates the picture. The researchers believed developers were probably benefiting more from newer tools, but judged their later data unreliable for estimating how much. Participation and task selection had changed; concurrent agent work also made time harder to measure. Research update
A memorable number can outlive the conditions that produced it. Good reading includes noticing when the measurement needs to change.
The work moves
A CHI 2025 survey describes critical thinking shifting toward verification, integration, and oversight. It reports people’s experiences; it does not establish a loss of thinking ability. Lee and colleagues
DORA’s 2025 research adds an organisational perspective: AI amplifies strengths and weaknesses in the surrounding system. DORA report
Make the whole task visible
The following is an editorial framework, not a measured result from these studies.
| Moment | Useful question | Evidence to keep |
|---|---|---|
| Before generation | What would a correct result look like? | Acceptance criteria |
| During review | What has actually been checked? | Tests and review notes |
| After release | Did the change help? | Errors, feedback, outcomes |
A team can try a similar habit on comparable tasks. Record the starting conditions. Include review and rework. Keep quality expectations steady. Separate the experience of using the tool from the result of the work.
That creates an account of where help appears, where effort moves, and where judgment matters. The account can change as tools improve.
A question to carry forward
If generating a plausible answer becomes easier, what makes a finished result worth trusting?
A useful answer may begin with the person who can explain the decisions, show the checks, and remain responsible when reality disagrees.