Controlled trials find large gains on well-defined tasks such as writing, and none on others, such as medical advice. Across whole jobs the measured effect is small, and many companies abandon most of their AI projects.
24
37%
In a preregistered online experiment in early 2023 with 444 college-educated professionals, those randomly given ChatGPT finished an occupation-specific writing task 10 minutes (37%) faster than the control group, and evaluators graded their work 0.45 standard deviations higher.
Who was asked: 444 college-educated professionals recruited online
Short online writing tasks with an early version of ChatGPT.
Shakked Noy and Whitney Zhang (MIT), Experimental Evidence on the Productivity Effects of Generative Artificial Intelligence (working paper) · early 2023 · Grade A
25
0.37 standard deviation
In a randomized field experiment with 776 Procter & Gamble professionals in 2024, individuals working with AI showed a 0.37 standard deviation gain in the rated quality of their solutions to real product-innovation challenges over individuals working without it, and two-person teams with AI a 0.39 standard deviation gain; the authors say individuals with AI performed at a level comparable to teams without AI.
Who was asked: 776 Procter & Gamble professionals (R&D and commercial)
A one-day virtual workshop at one company.
Disclosure: P&G gave gifts to a Harvard Business School institute, and one author was a paid consultant to P&G.
NBER (Dell'Acqua, Ayoubi, Lifshitz, Sadun, Mollick and others), The Cybernetic Teammate: A Field Experiment on Generative AI Reshaping Teamwork and Expertise (NBER Working Paper 33641) · 2025-04 · Grade A
26
24.3%
In a 2025 study of 48 freelance workers, those who chose to use AI for single steps finished tasks 24.3% faster than workers without AI, while the few who handed AI whole tasks were 17.7% slower, partly from checking and fixing its work and partly because less expert workers chose to hand tasks over.
Who was asked: 48 human workers
Not randomized: workers chose how to use AI, and only a handful handed it whole tasks.
Carnegie Mellon University and Stanford University (Wang, Shao, Fried, Neubig, Shaikh and Yang), arXiv, How Do AI Agents Do Human Work? Comparing AI and Human Workflows Across Diverse Occupations (arXiv 2510.22780v2) · 2025-11-06 · Grade C
27
34.5%
In a randomized study in 2024 with 1,298 UK adults working through doctor-written medical scenarios, those given GPT-4o, Llama 3 or Command R+ identified the relevant conditions in fewer than 34.5% of cases and the right course of action in fewer than 44.2%, no better than a control group free to use any source; tested alone, the same models identified the conditions in 94.9% of cases.
Who was asked: 1,298 UK adults recruited on Prolific (stratified to UK demographics)
Written medical scenarios, not real illness.
Nature Medicine (Bean et al., University of Oxford), Reliability of LLMs as medical assistants for the general public: a randomized preregistered study · August 21 to October 14, 2024 · Grade A
28
1%
Linking two surveys of Danish workers in 11 AI-exposed occupations to payroll records through June 2024, economists found AI chatbots had no significant effect on earnings or recorded hours in any occupation; confidence intervals rule out average effects larger than 1%.
Who was asked: About 25,000 workers per survey round
Not randomized; covers the first year and a half after ChatGPT's launch.
Becker Friedman Institute, University of Chicago (Humlum and Vestergaard), Large Language Models, Small Labor Market Effects (BFI Working Paper 2025-56) · late 2023 and 2024 · Grade B
29
42%
In S&P Global Market Intelligence's 2025 survey of more than 1,000 respondents in North America and Europe, as CIO Dive reports it, 42% of companies were abandoning most of their AI initiatives, up from 17% a year earlier, and the average organization scrapped 46% of its AI proofs of concept before production.
Who was asked: More than 1,000 respondents in North America and Europe
As a trade outlet reported it; S&P's own wording isn't quoted.
Cited secondary · S&P Global Market Intelligence (via CIO Dive), AI project failure rates are on the rise: report · 2025 survey · Grade C
30
RAND's 2024 report on why AI projects fail repeats an outside estimate that more than 80 percent of AI projects fail, twice the rate for IT projects that don't involve AI; RAND cites a Fortune article for it and did not measure it.
Widely quoted as RAND's finding, but RAND took it from other sources and measured none of it.
Caution · RAND Corporation (Ryseff, De Bruhl and Newberry), The Root Causes of Failure for Artificial Intelligence Projects and How They Can Succeed (RR-A2680-1) · 2024 · Grade C