Subscribe
Analysis
7 min read

Six Randomized Trials, Three Verdicts: What AI Tutoring Evidence Actually Shows

Between September 2025 and October 2026, at least six randomized controlled trials reviewed AI tutoring in real schools and universities, with results ranging from a doubling of learning gains to a 0.37 standard deviation drop in final grades.

Chart comparing learning gains vs losses in six randomized controlled trials of AI tutoring systems
Chart comparing learning gains vs losses in six randomized controlled trials of AI tutoring systems

Recent randomized controlled trials indicate that AI tutoring's impact ranges from significant learning gains to measurable grade drops, depending entirely on instructional design rather than the underlying AI model. Effective systems elicit reasoning and require mastery before progression, whereas open-ended answer provision often leads to student disengagement and lower performance.

Between September 2025 and October 2026, at least six randomized controlled trials tested AI tutoring in real schools and universities, with results ranging from a doubling of learning gains to a 0.37 standard deviation drop in final grades. Edtech vendors cite the first number. Researchers who audit the literature keep landing on the second. Both are real, and the difference between them is not the model. It is the design.

Split scene: a student fed an answer at desk level, a student climbing steps to earn each one

This is what the trials measured, and the pattern that emerges when you line them up.

The results that made headlines

The most-cited positive study comes from Harvard. In a randomized crossover trial, 194 undergraduate physics students learned two topics twice: once in an active-learning classroom, once at home with a custom AI tutor built around the same pedagogical principles. On matched post-tests, the median score after the AI condition was 4.5 versus 3.5 after the classroom, against a shared baseline of 2.75. The median learning gain in the AI group was more than double the classroom group's (rank-sum test, z = -5.6, p < 10^-8), and students got there in a median 49 minutes instead of a 60-minute class period. The comparison group was not a bad lecture. It was a well-run active-learning class, which is the hardest bar education research can offer.

In England, a four-week micro-randomized trial of the GCSE science revision platform Medly, designed with Durham University researchers and evaluated independently, assigned 644 post-test completers (of 929 who took the pre-test) to either 30 minutes a week with the AI tutor or their usual self-directed revision. The intention-to-treat effect was 0.33 standard deviations, about 2.56 extra marks out of 35, significant in all three subjects and largest in Biology (0.52). Attrition near 31% is a genuine limitation, and a four-week revision tool is not a general education, but the direction was consistent.

Google and the charity Eedi ran an exploratory RCT in UK secondary math classrooms comparing static hints, remote human tutors, and a LearnLM-based AI tutor. For correcting a student's misconception on the spot, a human tutor raised the odds of resolution over static hints by a factor of 2.9. The AI tutor raised them by 3.2. Statistically, the machine matched the human at the actual tutoring task.

Finally, one of the cleanest results is not about AI tutors at all. In the Tutor CoPilot trial, 386 human tutors in a math program for students below proficiency got an AI assistant coaching their responses in real time; 396 continued as normal. Students with AI-assisted tutors hit 66% end-of-session mastery versus 62%, a gain concentrated among the least-skilled tutors (up to 9 points), and the mechanism was visible: assisted tutors were 10 percentage points more likely to ask a student to explain their thinking.

The results the press releases skip

Princeton's Bastani and colleagues ran a field experiment with roughly 1,000 high-school math students under three arms: no AI, a plain GPT-4 chatbot, and a purpose-built GPT tutor with guardrails. During practice, both AI arms looked like a triumph: 48% more problems solved for the plain chatbot, 127% more for the guarded tutor. One week later, on a closed-book exam with no AI, the plain-chatbot students scored 17% below the control group. The guarded-tutor students finished statistically indistinguishable from controls. The students, notably, did not perceive this. Self-reported learning in the chatbot arm was not lower despite the worse exam scores.

The largest trial so far, at the University of Maryland, tested a GPT-4o tutor integrated into a course platform across 2,379 undergraduates and 30 instructors in fall 2025. Among sections of the same course, students given tutor access ended around four percentage points lower in final grades, a drop of 0.37 standard deviations, and their participation in the learning management system fell by 0.90 standard deviations. Estimated losses were larger among first-generation students. The tutor did not fail as a chatbot. It succeeded as a substitute: students asked it instead of doing the work the course assigned.

A machine holding its speech bubble closed while a student works the problem

The meta-evidence is itself contested

A 2025 meta-analysis out of East China Normal University pooled 337 effect sizes from 68 experimental studies (2022-2025) and reported a moderate overall positive effect of generative AI on learning, SMD 0.45. A 2026 meta-meta-analysis by Bartoš and colleagues took it apart: they pooled 1,840 effect sizes from 67 meta-analyses and found severe publication bias. Adjusted estimates land near one third of the headline numbers, about SMD 0.196 with a credible interval touching zero, and prediction intervals spanning large negative to large positive effects. A separate audit cited by the Hechinger Report found that published meta-analyses of AI in education report positive effects without exception, which is the signature of a literature that does not publish its failures. Stanford's Susanna Loeb team screened close to 900 studies and judged about 20 rigorous enough to summarize.

Treat vendor-cited effect sizes of "0.4 to 1.3 standard deviations" as the optimistic tail of a biased distribution.

What separates the wins from the losses

Line the trials up and the dividing line is not model quality. The Maryland tutor ran GPT-4o. So did the Princeton chatbot. The design variable is what the system does when a student asks for the answer.

  1. Elicit, do not provide. The Harvard tutor was built with expert-authored scaffolds that withheld solutions and turned mistakes into further practice. The plain chatbot handed over answers, and 17% of exam performance evaporated with them. Bastani's own framing: the vast majority of students use a chatbot to obtain solutions, not to work through problems.
  2. Gate on mastery, not time. A Tennessee trial of 6,000-plus middle-schoolers (NBER, August 2026) found AI-assisted practice slowed students down: they attempted fewer problems but were more accurate conditional on reaching an attempt, and after a mistake they needed fewer tries to get back to a correct answer. Gains showed up where the program required demonstrated mastery to move on.
  3. Watch substitution. The Maryland effect sizes look like a tutoring failure and are more precisely an engagement displacement: the LMS participation drop (0.90 SD) is nearly three times the grade drop. A tutor that replaces the homework is not a tutor.
  4. Test unaided, later. Every positive result in this literature rests on a test, a delayed exam, or an exit ticket taken without unrestricted AI access. Sessions measured with a chatbot open measure the chatbot, not the student.

The honest bottom line

The evidence does not say AI tutoring works, and it does not say it fails. Across the trials it says something narrower and more useful: an AI system that forces a student to retrieve, reason and retry produces learning gains comparable to good human instruction at a fraction of the cost, and the same underlying model offered as an open-ended answer machine produces measurable losses while the student feels they learned more. The 2026 state of the field is that the second product is the one most schools bought.

Buyers of AI tutoring, in school or at home, should ask three questions no vendor brochure answers by default. Does the system withhold the answer and require the student to produce the next step? What does the delayed, unaided post-test show, not the in-session performance? And what does the vendor's own engagement data say about whether the tool is supplementing the coursework or replacing it? If a vendor cannot answer all three, the evidence base says you are buying the 17%, not the 2x.

Sources

Frequently asked questions

What was the outcome of the Harvard AI tutoring trial?

In a randomized crossover trial at Harvard, undergraduate physics students using an AI tutor showed median learning gains more than double those of the active-learning classroom group (4.5 vs 3.5) while requiring less time.

Why did the University of Maryland AI tutor trial result in lower grades?

The University of Maryland trial found that students using a GPT-4o integrated tutor scored four percentage points lower because the tool served as a substitute for course work, leading to a significant drop in learning management system participation.

How did the Princeton field experiment compare plain chatbots to guarded tutors?

While both AI arms performed well during practice, plain chatbot students scored 17% below controls on a closed-book exam one week later, whereas students using a purpose-built guarded tutor scored indistinguishable from the control group.

What did the meta-meta-analysis by Bartoš and colleagues find regarding AI in education?

The 2026 meta-meta-analysis pooled 1,840 effect sizes and found severe publication bias, with adjusted estimates landing near SMD 0.196 and prediction intervals spanning large negative to large positive effects.

AnalysisEducation