🔍 Read the full analysis: AI Assistance And Data Science: What Consultants Retained After ChatGPT on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
A randomized trial at Boston Consulting Group found that consultants using ChatGPT scored higher than a control group on coding, statistics and prediction tasks. After the chatbot was removed, researchers found no evidence that the ChatGPT group performed better on technical questions without it.
A randomized trial involving Boston Consulting Group consultants found that access to ChatGPT improved their performance on coding, statistics and prediction tasks outside their usual expertise, but researchers found no evidence the advantage remained when the chatbot was removed. The study, published October 26 in Nature Human Behaviour, bears on whether workplace AI use should be counted as employee skill development or as assistance that works only while the tool is available.
The trial allocated 986 consultants to either a ChatGPT group or a control group that received guidance on using Google and sites including Stack Overflow and Khan Academy. The main analysis included 487 participants who submitted work on both assigned tasks. Participants completed two 90-minute exercises from a set of three: Python data cleaning, statistical analysis, and predicting the predictability of international football matches. Most participants reported no coding experience, according to the authors.
Compared with the control group, consultants using ChatGPT scored 49 percentage points higher on coding, 20 points higher on statistics and 17 points higher on prediction. They also completed coding and prediction faster on average; the authors report differences of about nine and 14 minutes, respectively. Time spent on the statistics task did not differ significantly. The ChatGPT group nevertheless scored below a benchmark of 44 BCG data scientists who completed the same tasks without AI.
After the task exercises, both groups answered five technical questions without ChatGPT. The researchers report no evidence that the ChatGPT group did better on this follow-up test. The paper describes the performance boost as temporary. It also reports that participants who used ChatGPT became more confident that it could solve problems it could not; the authors characterize this confidence result as exploratory.
The trial in motion · BCG consultants · 2024
With ChatGPT: close to the experts. Without it: no measurable gain.
A randomized trial with a control group that used Google and Stack Overflow instead of ChatGPT.
- 1 · Training15–20 minutes on prompting ChatGPT
- 2 · TasksTwo 90-minute data science tasks, with ChatGPT
- 3 · ScoresGap to 44 data scientists closes, not fully
- 4 · Tool removedNo ChatGPT from here on
- 5 · TestFive technical questions
Coding task score
0 = data scientists’ average
ChatGPT on+49 pp on codingStill below the data scientists on average
Knowledge test without ChatGPT
ChatGPT group vs. control, five questions
ChatGPT off−7.7 to +4.8 ppNone significant · joint test p = 0.531
Source: Wiles, Krayer et al., “Generative AI and the temporary upskilling of knowledge workers”, Nature Human Behaviour (2026); authors’ manuscript, Sections 2.1 and 2.3, Tables B7 and B9. Bars scaled to the five published differences.
Graphic: Powered by Thorsten Meyer AI
AI Performance Is Not Retained Skill
The findings distinguish between doing more with AI and learning to do more independently. For employers, that difference affects how they interpret productivity, training outcomes and an employee’s ability to complete work when a model is unavailable or gives a faulty answer.
The trial suggests that short instruction in prompting can help workers take on unfamiliar technical tasks while the tool is at hand. It does not establish that workers have acquired the underlying skills. Companies evaluating AI-assisted output may therefore need to assess the work and the human contribution separately, rather than treating improved task scores as proof of lasting training.
The confidence finding adds a separate caution, but it should be read with the authors’ qualification that this part of the analysis was exploratory. The study does not show that every AI user will overestimate a chatbot’s abilities, nor does it test long-term workplace consequences. It does indicate that access to assistance and the ability to judge its limits are distinct questions.
Products Worth Considering
Apple iPad 11-inch (A16), 128GB, Wi-Fi, Silver
Samsung Galaxy Tab A11 8.7-Inch Wi-Fi Tablet, 64GB, 4GB RAM, 90Hz, Gray, with 20W Charger
Samsung Odyssey G3 G30D 27-Inch FHD Gaming Monitor
As an affiliate, we earn on qualifying purchases.
How the BCG Trial Was Designed
The researchers preregistered the study on March 13, 2024, before data collection, which took place at BCG in late March and early April that year. The treatment group received 15 to 20 minutes of instruction on using ChatGPT for technical problems and then worked with GPT-4. The control group received comparable instruction on web searches and online learning resources.
BCG data scientists and OpenAI’s Economic Impacts Research team designed the tasks so ChatGPT could offer useful guidance but could not independently answer every question correctly. Participants had to evaluate and apply the assistance rather than simply copy a complete answer. The researchers compared task results with those of the 44 data scientists, then tested participants’ technical knowledge without AI.
The paper, titled Generative AI and the temporary upskilling of knowledge workers, appeared in Nature Human Behaviour on October 26, 2026. Its conclusions are based on this particular set of consultants, tasks, training and model access; they are not a direct measurement of every workplace or AI system.
““find no evidence” that the ChatGPT group performed better on the technical questions completed without the tool.”
— The study authors
Limits Beyond This Workplace Trial
The results do not establish whether repeated use over a longer period could build lasting skills. The follow-up used five technical questions after the task exercises, so it measures a limited set of knowledge rather than every possible form of learning or future job performance.
The trial also involved BCG consultants, a particular GPT-4 setup and tasks selected to test work beyond participants’ stated skill levels. It remains unclear how the results would vary across other occupations, levels of prior expertise, AI tools, training formats or real-world projects. The authors’ finding of no evidence of an advantage on the follow-up should not be read as proof that AI can never support learning.
The confidence result is described as exploratory, and the study summary does not establish how durable or widespread that effect might be. The available findings also do not show whether different supervision or verification practices would change either task performance or confidence in the tool.
Testing Longer-Term Learning Effects
The published results provide a short-term comparison, not a long-term study of employee development. A next step for research would be to test whether extended practice, different training or feedback helps workers retain technical knowledge after AI access ends, and whether those results hold across other jobs and tools.
For organizations applying the findings, the immediate practical question is how to evaluate AI-assisted work without mistaking tool-supported output for independent expertise. The trial does not prescribe a specific workplace policy. It leaves employers to determine how they will check accuracy, measure retained skills and account for the possibility that an AI system may be unavailable or unable to answer a particular question.
Key Questions
What did the BCG study find?
Consultants using ChatGPT scored higher than a control group on coding, statistics and prediction tasks. Researchers found no evidence that they performed better on a subsequent set of technical questions completed without ChatGPT.
How large were the task-score differences?
The ChatGPT group scored 49 percentage points higher on coding, 20 points higher on statistics and 17 points higher on prediction than the control group. These are differences between the study groups on those tasks, not measures of long-term learning.
Did ChatGPT users match the data scientists?
No. The ChatGPT group remained below the average of 44 BCG data scientists across all three tasks. The paper reports that some participants with coding experience came close to the expert benchmark on coding and statistics, but that result does not describe the group as a whole.
Does the study prove AI cannot teach people skills?
No. The trial found no evidence of an advantage on a limited follow-up test without the tool after these tasks and this training. It did not test whether longer-term use or other teaching approaches could produce retained learning.
Source: ThorstenMeyerAI.com
