Researchers at Bocconi University, in collaboration with OpenAI Economic Research, conducted a randomized experiment that examined how access to ChatGPT (GPT‑4o) and a short causal-reasoning exercise affected student work. The study involved over 1,000 first-year undergraduates tasked with developing marketing recommendations for the university’s merchandise store.
Study design and measurement
Students were randomly assigned by class period to one of four groups: access to ChatGPT (GPT‑4o), a causal-reasoning training, both interventions, or neither. The causal-reasoning training—unrelated to AI—taught students to link causes and effects through a game, examples, questions, and feedback.
Submissions were evaluated by trained human graders using a five-point rubric focused on two standard marketing goals: increasing awareness and use of the university store. Researchers also applied automated text analysis to assess number and variety of ideas, signs of causal reasoning, and similarity to recommendations from three experts.
Findings
According to the researchers, students with access to ChatGPT scored almost a full point higher on the five-point rubric. Their submissions contained more ideas, clearer logical structure, and higher similarity to expert recommendations. The report notes students did not simply hand over assignments to the model: they had to decide prompts, evaluate outputs, and select material for final submission.
The causal-reasoning exercise produced different effects. Students who completed the training provided clearer explanations of why proposals might work or fail, and automated analysis found they generated a broader and more distinct set of ideas compared with peers. However, the training did not raise rubric scores, which focused on the two conventional marketing goals used for grading.
Students who received both ChatGPT access and the causal-reasoning training showed benefits associated with each intervention: rubric scores and idea counts similar to the ChatGPT-only group, and idea variety matching the training-only group, along with stronger logical coherence and evidence of questioning assumptions.
The researchers cite the randomized design as enabling separation of the effects of ChatGPT access and causal-reasoning training, and they report the results as a contribution to evidence on how AI and critical-thinking instruction interact in classroom tasks measured by human grading and automated text metrics.
Original source: OpenAI News