AI, AI everywhere and not a proof in sight. My apologies to Coleridge, but it certainly feels as though we are adrift in a sea of AI applications. Even as I type this, Copilot repeatedly attempts to force its way into my writing. But is any of it actually making us better at anything? In this week’s issue, we look at some of the research behind the claims of AI’s benefits for learning and find that everything may not be what it seems. We then close with three Sparks about international students, a famous college completion program, and the agentic professor.   

Who Is It Actually Benefiting? 

From The ‘Nonexistent’ Research on AI’s Benefits for Education | Inside Higher Ed 

Kathryn Palmer explores just how little research we have on AI’s benefits or harms in education.  

Our Thoughts  

Educational research and educational technology have often been unhappy bedfellows. With their Silicon Valley mantra to move fast and break things, ed tech companies rarely have the patience required for rigorous academic study about whether their products work. Of course, this is the very thing that would make their products more appealing in the market—evidence that their products actually made a difference in how students learn. Now, higher education is facing one of its greatest challenges yet in the form of AI that threatens to disrupt the very nature of its business model. And again, technology companies are selling us on adoption of this new technology to improve learning and because employers will expect AI skills from graduates. So I ask, where’s the evidence? Turns out that might be the wrong question.  

When Inside Higher Ed says the evidence is nonexistent, they are being generous. When you look deeper, you see researchers pooling questionable studies into reviews that claim more than the underlying research can support. Back in May, Springer Nature retracted one of the most well-known papers, which found a positive impact on student learning from the use of ChatGPT. The paper was a meta-analysis of previous research on ChatGPT use in education and was published in 2025, a little more than two years after ChatGPT was made available to most users. How many rigorous studies of ChatGPT use in education could have been conducted and published in those two years? Probably not very many given how long it takes to conduct research, write it up, go through the publication process, and finally have a paper in print. Although the paper has been retracted, the challenge is its legacy will probably outlive the retraction. The paper has been cited hundreds of times since its initial publication, and some of those papers are using it as support for something it can no longer claim. 

Now, before you say “Allen...one retraction. That’s all you got?”, let's look at some more recent studies. Lawson and colleagues, publishing in Computers & Education, analyzed the individual studies in two ChatGPT meta-analyses and found that most of the underlying research never established that the instructional methods were controlled for between the groups under study, meaning it's difficult to attribute any learning differences to the use of ChatGPT. Basically, we don’t know if ChatGPT made the difference or if it was something else in the instruction. Although not yet peer-reviewed, we also have two pre-print articles, which draw similar conclusions. First, concluding that “the current evidence is insufficient to support robust policy or practice recommendations,” František Bartoš and his coauthors conducted a meta-meta-analysis of the studies, finding the literature varies too much from study to study for comparison. Second, Patrick O’Neill conducted an audit of a sample of the meta-analyses and reported that due to chronic methodological failures in the literature, none of them provide a valid basis for the claims they are making.  
 
So, if we go back to my original question, you can see it’s not quite the right one. We have a pile of poorly constructed meta-analyses all claiming to find proof that isn’t present. And that’s exactly the citation that an AI vendor will present to your campus during a sales demo. Hopefully, now you feel armed with evidence to ask the right follow-up questions and help your campus make the best decisions it can with the actual evidence that is available because you can’t wait three years for a better study. Educational research takes time and money, two things in short supply these days. Instead, what you can do is treat the research findings as presented in a sales deck as a best guess as to how AI might benefit learning on your campus. Then, ask the hard questions about how your campus will assess the tools that you’re buying. If in three years when the contract is due for renewal, that decision will be based on license counts and session volumes, you’re buying based on adoption metrics and not learning outcomes. Use of a tool is not learning; plenty of students will log in once to see what it’s like. 

If you do plan to use usage metrics in your decision-making, ensure that the contract explicitly states that student-level usage data ends up with you in your data warehouse. Dashboards in a vendor AI tool are a black box. You can’t run your own analysis inside of their interface that you don't control. You can’t check their definitions. You can’t run the analysis that someone else asks for in three years. And, perhaps most importantly, you want that data for longitudinal comparisons and research if you decide to switch vendors after three years. 

The toothpaste is out of the tube for AI. People on your campus are going to buy AI tools for learning and probably already have. But that doesn’t mean that you have to accept the learning claims that come with them, and nothing stops you from asking now for the data access that you’ll need later to find out if those claims prove true for your institution.  

Sparks 

• Judge Blocks Controversial International Student Restrictions (Inside Higher Ed) - A judge has blocked the new international student visa cap. Given the impacts of international enrollment on many institutional budgets, this is a good thing.
• More degrees but not higher earnings: Puzzling data from CUNY’s famed ASAP program (The Hechinger Report) - A new report on CUNY’s ASAP college completion program finds that the additional completions don’t always translate into higher earnings. While I have some problems with the report design, it’s still worth reading about the assessment of this program that has been duplicated dozens of times at other institutions.
• The Agentic Professor: Exploring GenAI-Supported Futures in Higher Education (EDUCAUSE Review) - The Agentic Professor is a construct that helps colleges and universities explore how AI agents providing scalable, long-term personalized instruction could reshape teaching, assessment, and ultimately the role of faculty. A really interesting thought experiment with an example you can try yourself!