AIThis post was created with the assistance of artificial intelligence (AI).

TL;DR

In a randomized trial at Boston Consulting Group, consultants using ChatGPT scored higher than a comparison group on three data science tasks. After ChatGPT was removed, the researchers found no evidence that the users performed better on a technical knowledge test, and the authors describe the gains as temporary.

A randomized trial involving consultants at Boston Consulting Group found that access to ChatGPT raised scores on three data science tasks, but researchers found no evidence of a corresponding advantage when participants later answered technical questions without the tool. Published in Nature Human Behaviour on October 26, 2026, the study examines whether AI assistance builds workers’ independent skills or mainly helps them perform while it is available.

Buying for a business?Offer from Amazon

Get business pricing on office and shipping supplies

  • Business-only prices and quantity discounts
  • Tax-exempt purchasing
  • Multiple users, one account, clear invoices
As an affiliate, we earn on qualifying purchases.

The trial assigned consultants to use either ChatGPT or a comparison set of online resources, including Google and Stack Overflow. The ChatGPT group received 15 to 20 minutes of prompting instruction and then used GPT-4. Participants in the comparison group received instruction on searching the web and using resources such as Stack Overflow and Khan Academy. The study took place at BCG in late March and early April 2024 and was preregistered before data collection.

Participants completed two 90-minute tasks drawn from three areas: Python coding, statistics and prediction. The tasks were designed with BCG data scientists and OpenAI’s Economic Impacts Research team so that ChatGPT could provide useful guidance but could not independently produce correct answers to all questions. The researchers compared workers’ results with those of 44 BCG data scientists who completed the tasks without AI.

Among the 986 consultants allocated to groups, 573 began the survey and 487 submitted work on both assigned tasks, forming the main analysis sample. The source says the ChatGPT group scored 49 percentage points higher on coding, 20 points higher on statistics and 17 points higher on prediction than the control group. Those task gains did not mean the consultants surpassed the expert benchmark: on average, the ChatGPT group remained below the data scientists.

At a glance
reportWhen: Study published October 26, 2026; trial…
The developmentA study published in Nature Human Behaviour reports that ChatGPT helped BCG consultants complete unfamiliar data science tasks, but the trial found no measurable knowledge advantage after the tool was taken away.
How the experiment worked, in five steps: 1, 986 consultants allocated and split equally into treatment and control, 573 began, 487 submitted both tasks. 2, a short training: treatment ChatGPT with GPT-4, control Google, Stack Overflow and Khan Academy. 3, two of three 90-minute tasks in Python coding, statistics and prediction, built so ChatGPT alone gets them wrong. 4, benchmark of 44 data scientists working without AI, whose average is the zero line. 5, highlighted: then the tool is gone, five technical questions after the tasks without ChatGPT
The design of the trial at BCG in late March and early April 2024. Source: Wiles, Krayer et al., Nature Human Behaviour (2026), authors' manuscript. Graphic: Thorsten Meyer AI.

Task Gains Did Not Persist Alone

The results bear on how employers describe AI at work. A worker may complete tasks outside their usual expertise with help from a chatbot, yet that performance does not by itself show the worker has learned to do the work independently. In this trial, the advantage appeared during assisted tasks, while the later test did not show a measurable knowledge gain for the ChatGPT group.

That distinction affects decisions about training, staffing and quality control. If companies treat assisted output as proof of newly acquired skill, they could overestimate what employees can do when tools are unavailable or when a task requires them to detect an AI error. The study does not establish how employers should make those decisions, but it gives a controlled example of why performance with an AI system and learning without it are different outcomes.

The authors characterize the observed upskilling as temporary and compare generative AI to an exoskeleton: it can extend what a person can do while in use. That framing reflects the study’s interpretation of its findings. It does not establish that AI can never support learning; the experiment tested a particular short training, tool and set of tasks.

Products Worth Considering

Brother MFC-L2820DW Wireless Monochrome All-in-One Laser Printer

Brother MFC-L2820DW Wireless Monochrome All-in-One Laser Printer

FLEXISPOT 36-Inch Height-Adjustable Standing Desk Converter with Keyboard Tray, Black

FLEXISPOT 36-Inch Height-Adjustable Standing Desk Converter with Keyboard Tray, Black

As an affiliate, we earn on qualifying purchases.

Inside the BCG Trial

The study, “Generative AI and the temporary upskilling of knowledge workers,” was conducted in 2024 and published in 2026. The authors preregistered the trial on March 13, 2024. Participants had at least a bachelor’s degree, and the authors report that most said they had no coding experience. The main analysis included only those who submitted work on both tasks, rather than everyone initially allocated to a group.

The three task types tested different kinds of work. The coding exercise involved cleaning, merging and filtering datasets in Python. The statistics exercise asked participants to identify problems in a proposed model, interpret residual plots and calculate probabilities. The prediction exercise involved building a model related to the predictability of international football matches. Each participant completed two of the three tasks, with 90 minutes for each.

For the follow-up, participants answered five technical questions on similar topics without ChatGPT. The source reports that the estimated differences between the ChatGPT and control groups ranged from −7.7 to +4.8 percentage points across the questions, with none statistically significant. The joint test reported a p-value of 0.531. The authors say they “find no evidence” that the ChatGPT group did better on that test.

““find no evidence””

— Wiles, Krayer and co-authors, as quoted in the study

Limits of the Follow-Up Test

The follow-up consisted of five technical questions, so it measures performance on that test rather than every way workers might learn from using AI. The supplied study summary does not report whether a longer training period, repeated practice, different instruction or other workplace tasks would produce lasting gains. It also does not establish how the findings apply beyond this BCG sample and these data science exercises.

The study summary describes the confidence result as exploratory: the ChatGPT group emerged more confident that the tool could solve problems it could not actually solve. The material provided does not give the size of that effect or enough detail to assess its scope. It should not be treated as a definitive estimate of how workers generally judge AI reliability.

Questions for Workplace Training

The study’s immediate implication is a measurement question for employers: do training programs improve workers’ independent skills, or only their ability to produce results with AI assistance? The trial does not prescribe a particular policy. It does show why evaluations may need to test both assisted performance and what workers can do after the tool is removed.

Further evidence would be needed to determine whether different forms of instruction or longer practice can turn short-term assistance into retained knowledge. The source material does not identify a planned follow-up study or a next research milestone. For now, the reported findings apply to the trial’s participants, tasks and testing conditions.

Key Questions

What did ChatGPT change in the BCG trial?

Compared with a group trained to use Google and other online resources, the ChatGPT group scored 49 percentage points higher on coding, 20 points higher on statistics and 17 points higher on prediction tasks.

Did participants keep the advantage after ChatGPT was removed?

The researchers found no statistically significant advantage for the ChatGPT group on a five-question technical test completed without the tool. The authors say they found no evidence of better performance on that test.

Did the ChatGPT group match the BCG data scientists?

No. The consultants using ChatGPT remained below the average of 44 BCG data scientists who completed the tasks without AI, although the gap narrowed on the measured tasks.

How many consultants were included in the main analysis?

Of 986 consultants allocated to groups, 573 began the survey and 487 submitted work on both tasks. Those 487 formed the main analysis sample described in the source.

Does the study show that AI cannot teach workers skills?

No. It tested a specific short training, GPT-4 access, BCG consultants and a set of data science tasks. The findings show no measurable advantage on the study’s immediate test without ChatGPT; they do not settle what different training or longer practice might achieve.

Source: Thorsten Meyer AI

This content is for general information only and is not financial, tax or legal advice. Consult a qualified professional for decisions about your money.
You May Also Like

Nocopi Technologies Reports Second Quarter Results

Nocopi Technologies announced its second quarter financial results, showing a decline in revenue but a slight increase in net profit compared to the previous quarter.

Chartbook 476 Walled-in Wealth #2: US-China Oligarchic Divergence.

Search and coverage interest is spiking in Chartbook newsletter issue 476, ‘Walled-in wealth #2: US-China oligarchic divergence.’ The trigger is unconfirmed.

Find An SMB Acquisition Opportunity That Fits Your Skills

A proposed marketplace would match small-business listings to buyers’ operator skills and send brokers fit-scored inquiries. Its results remain untested.

Daily Dental Checks: Using Photos To Keep Gaps Away

A new approach uses daily gum-line photos to track inflammation and prevent gaps between teeth, offering a preventive tool for dental care.