Soon after the introduction of ChatGPT in November 2022, students figured out how to use the chatbot as an answer machine, getting it to do their school work for them. Not surprisingly, that shortcut harmed student learning and led to lower test scores.
AI enthusiasts sought a solution. The chatbots could easily be instructed to act more like a human tutor, withholding answers, providing hints and guiding students with questions. Khan Academy created one of these “Socratic-method” chatbots, christened it “Khanmigo,” and released it in 2023.
But a new study that came out last month suggests that creating an AI tutor is only one hurdle among many in helping children improve their math skills. Students have to actually want to use it.
Researchers from the University of Toronto followed students in 18 Tennessee middle schools for two years, from 2024 to 2026. Almost all the students tried Khanmigo at first. And many tried to get the AI assistant to spit out the answer. But when Khanmigo refused and instead asked questions or offered hints, students largely stopped using it.
“We observe how students actually used the AI tutor, and the answer is: not much,” Philip Oreopoulos and Nina Low, the researchers, wrote in their draft paper, circulated by the National Bureau of Economic Research in August. “Seeking help with one’s own confusion remained a choice, and most students declined it most of the time.”
The researchers were testing whether Khanmigo could improve the performance of low-achieving students who were at least one grade level behind their peers. These students had an extra remedial math period built into their school day. Some were randomly assigned to use Khan Academy, with Khanmigo available as an AI assistant. The others received their usual remedial math work, which often included online tools such as Waggle, IXL, Zearn and DeltaMath.
The students who received the Khan Academy intervention did better in math than those in the regular remediation classes. But the gains were modest, and no greater than what previous studies had found from Khan Academy practice sessions without an AI assistant.
In other words, Khan Academy appeared to help. But the Khanmigo tutor didn’t add anything.
Sal Khan, the founder and CEO of Khan Academy, praised the study and wrote a detailed blog post about it, explaining the randomized controlled trial (RCT) just the way he explains a math problem in his videos. In an interview, Khan said the findings confirmed what Khan Academy was seeing internally: Students weren’t engaging much with Khanmigo.
Khan said he doesn’t regret releasing the AI tutor before knowing whether it would be effective. He is confident the early release did “no harm” to students, saying his organization went to great lengths to protect student data, make chatbot conversations safe, prevent cheating and minimize the risk of the bot hallucinating and giving students incorrect instruction. “It’s allowed us to learn and hopefully make the new version even more helpful,” he said.
The version of Khanmigo tested in Tennessee was released in 2023. Since then, Khan said, the company has changed how the AI assistant works. Khanmigo is now integrated directly into Khan Academy rather than sitting in a separate tab that students have to click on. If students ask for help before trying a problem, Khanmigo is deliberately taciturn, offering little more than a hint and a pep talk. But after a student gets a problem wrong, the AI assistant automatically pops up to offer help and work through the problem together with the student. Schools can also choose to turn the AI assistant off completely.
Khan Academy is experimenting with more ways to encourage students to use the AI tutor. Khan said he plans to give students “credit” for using Khanmigo after a mistake, allowing a bot-assisted redo to count toward one of the five correct answers in a row that students need before moving on to the next skill.
It will be interesting to see whether these changes get students to use Khanmigo more. But use and engagement are only one piece of the puzzle. We still need evidence that when students regularly use an AI tutor, they actually learn more. I’ll be watching.
This story about Khanmigo, an AI math tutor, was produced by The Hechinger Report, a nonprofit, independent news organization that covers education. Sign up for Proof Points and other Hechinger newsletters.
A large study of Khan Academy’s math program found no extra benefits AI.{}