Artificial intelligence makes many tasks faster and easier. But a new study suggests that students may learn more when AI makes them slow down.
In an experiment involving more than 6,000 middle schoolers in Tennessee, students learned slightly more math when an AI tutor walked them through their mistakes and then required them to demonstrate the same skill correctly three times in a row before moving on.
Academic researchers tested four approaches to practicing fractions. Students were randomly assigned to receive either conventional computer-based instruction or the same software with an AI tutor. Within each of those groups, half the students had to correctly answer practice questions covering the same skill three times in a row if they made a mistake.
Students used software built by researchers, which was similar to Khan Academy’s videos and exercises, once for 50 minutes during math class. A week later, they took a 15-minute test to measure how much they retained.
The winning combination wasn’t the addition of AI tutoring alone, but AI tutoring plus repetition, with the idea that students needed to stick with the skill to demonstrate some level of mastery. The students who practiced math with this AI-enhanced “mastery learning” approach scored about 3 percentage points higher than students receiving conventional computerized instruction. The advantage was small.
“I don’t want to jump out and say we’ve demonstrated that AI is going to be the game changer that we hope it is,” said Philip Oreopoulos, lead author of the study and an economist at the University of Toronto. “But it might be the first kind of evidence that shows there’s at least some hints that it has some positive value against no AI at all.”
That’s significant because there’s mounting evidence that AI is often harming learning, spitting out answers for students and short-circuiting the learning process.
The study, “Making AI Tutoring Productive: Evidence from a Mastery-Based Math Practice Experiment,” was conducted by researchers from the University of Toronto and the University of Pennsylvania’s Wharton School. A working paper is scheduled to be circulated by the National Bureau of Economic Research on Aug. 17 and has not yet been published in a peer-reviewed journal. The authors provided The Hechinger Report with a draft.
Making students slow down
The researchers think AI helped because it walked students through their mistakes instead of simply showing them a solution.
Without AI, students could see a step-by-step example after getting a problem wrong. But a student can easily skim through the steps to a solution and move on, without figuring out what went wrong.
Numi, an AI tutor, guides students step by step
Right after a mistake, an AI tutor called “Numi” walks a student through a solution step-by-step for dividing two numbers: 3 6/7 ÷ 4 5/2.
Here Numi, the AI tutor, encourages a student to pick a step to review in more detail. The original problem was 8 1/4 ÷ 3 3/4.
Source: Appendix of Oreopoulos et al., “Making AI Tutoring Productive: Evidence from a Mastery-Based Math Practice Experiment.”
The AI tutor, by contrast, could respond directly to a student’s work and guide the student through the mistake.
The students who used AI combined with mastery learning spent more time per question than students in any of the other groups — a sign that they were engaging more with the material. These students were also more likely to get the next question right after making a mistake.
That’s important because requiring students to answer three questions correctly in a row is common in educational software. But hitting a short mastery threshold doesn’t necessarily mean a student has developed a deep understanding. Students can guess their way to three correct answers or succeed from repeated exposure without really learning the skill.
There was a limit to the benefits for students in this study. Students in the AI-plus-mastery group performed better primarily on the easiest fraction questions — the ones most similar to what they had practiced. The advantage did not extend to more challenging problems.
In this study, AI didn’t produce a deeper or more transferable understanding of fractions. Then again, it was only a 50-minute intervention and it’s unknown how this combination of AI plus mastery learning might improve student learning over the course of many months.
Oreopoulos cautions against concluding that mastery learning is the best way to use AI in learning math. The researchers tested only the four combinations in their study; there could be better ways to enhance computer-assisted learning and make practice work more effective. His larger ambition is to keep testing different features against one another, continually improving the software.
For now, he said, the goal was more modest: to show that AI “has a little bit of benefit, and talk about its potential.”
Part of that potential may lie not in helping kids learn math faster, but in helping them slow down.
Kristin Fasiang is a graduate student in computer science and learning sciences at Northwestern University. Fasiang reported and wrote this story with The Hechinger Report’s Jill Barshay.
This story about AI and mastery learning was produced by The Hechinger Report, a nonprofit, independent news organization that covers education. Sign up for Proof Points and other Hechinger newsletters.
A randomized experiment involved more than 6,000 Tennessee middle school students learning fractions.{}