AIED 2025 · Main track · with Kasia Muldner
Do engagement prompts still work when the tutor is ChatGPT?
In a controlled experiment with 101 programming novices, three simple prompts (show your work, don't just copy, reflect on wrong answers) significantly increased learning, whether students had ChatGPT or traditional materials.
What I did
A 2×2 between-subjects experiment: engagement prompts (present/absent) × instructional resource (ChatGPT/traditional materials). Novices solved Python code-tracing problems; learning was measured by pretest→posttest gain. I also hand-coded screen recordings of every ChatGPT interaction.
What I found
Prompted participants learned significantly more (26.1% vs 10.6% gain, p = .040). The resource itself made no significant difference. Strikingly, about half the participants given ChatGPT never used it, and those who tried the problem before asking ChatGPT gained the most.
Why it matters
The cheapest intervention, a few lines of instructional text, beat the choice of technology. For anyone building AI tutoring products, the design lesson is that scaffolding how students engage matters more than which resource you hand them.
Learning gains by condition
Mean pretest→posttest change for the four experimental groups (N = 88: 13 of the 101 participants who scored 100% on the pretest are excluded). Hover any bar for details.
The prompts main effect: F(1,84) = 4.3, p = .040, η²ₚ = .05. Resource main effect: n.s. (p = .31). Data from Table 1 of the paper.
How students actually used ChatGPT
From hand-coded screen recordings of 43 participants in the ChatGPT conditions.