While it sometimes feels like data analysis can bring you close to tears when the numbers just won’t add up, federal reporting shouldn’t produce actual tears. And yet, that's exactly what happened to one focus group participant, who ended up crying on the phone with the IPEDS Help Desk over a financial aid figure that was off by about three dollars. That story comes from a new report on the first ACTS collection, and it says a lot about the process and how it went for institutions. We’ll look more closely at those experiences this week followed by three Sparks on AI integration, a Gallup poll, and Michigan’s pause on pass/no credit.   

Is Bad Data Better Than No Data? 

From How Trump’s Admissions-Data Push Burdened Colleges | The Chronicle of Higher Education 

A report from the Association for Institutional Research (AIR) and the Urban Institute details the experiences of colleges and universities as they worked to submit the new Admissions and Consumer Transparency Supplement (ACTS). 

Our Thoughts  

I don't know if it's because I've had a busy week or I'm wondering how it could already be October, but the irony in the Admissions and Consumer Transparency Supplement (ACTS) hit me harder than usual, given all the questionable data wrangling institutions had to do to submit it. It takes a special kind of talent to put "transparency" in the name and end up with data that is suspect...not because institutions were purposely deceptive, but because the data sought may not exist.  

I, too, want richer admissions data in IPEDS, and I’m far from the first to say so publicly. Numerous higher education organizations and researchers were requesting this expansion years before ACTS existed. The difference is that they asked for this data under the standard IPEDS process, with a technical review panel and enough time for institutions to collect the data and build the data sets before being asked to report on it. Any administration that skipped that process, especially after higher education associations warned during the public comment period that much of the requested data might not exist, would deserve the same scrutiny.  

So, let’s dig into the actual report and see what happened. Upon submission, ACTS validation checked what an institution submitted against what it had already reported to IPEDS. On the surface, this seems to make sense until you discover that ACTS didn’t always define its cohorts the way IPEDS does. ????Data analysis 101, right? If you want to compare rates, the denominators have to be the same. You can’t use different definitions for cohort comparisons. In the report, participants across all three focus groups said that in order to clear ACTS flags, they had to go back and change data they had already submitted to IPEDS, meaning prior-year comparisons may no longer be possible. Somehow a collection designed to make admissions more transparent may have made the federal higher education dataset less dependable. 

Of course, we know some of the problems—many of our systems are transactional. A student’s demographic detail captured three years ago gets updated in the transactional system when the student sees it’s an error. Data from five years ago on applicants who didn’t enroll has already been archived (or even deleted) based upon institutional record retention policies. When ACTS asked for seven years of data that no one was purposely snapshotting, IR offices had to rebuild history from current records and archived files. According to the report, only 17 percent of respondents had all the undergraduate data ACTS requested. That’s a lot of institutions recreating data trails and connections for a data collection with questionable motivations. 

All of this matters even more when the data goes public, which is expected later this fall, because some people are going to want to rifle through the data to find support for their conclusions about college admissions. But I forgot to mention that the data is even more incomplete because earlier this year a federal judge blocked ACTS submission requirements for public institutions in 17 states. This means the public dataset will be missing a large and nonrandom slice of higher education institutions, meaning comparisons will be skewed, and the conclusions drawn from them unreliable. Even the authors of the report conclude that this first cycle of ACTS data should be interpreted with caution, a warning I would take seriously.   

 At this point, assuming you’re one of the institutions that had to submit ACTS data, I think the most useful thing you can do is document everything related to ACTS. Data changes you had to make. Definitional calls made at the eleventh hour. Basically, you want a record and explanation that you can hand off to your communications person or team so that when a reporter or legislator compares your figures to a peer’s, your institution is ready to explain itself. After that, if you haven’t already, it’s time to revisit your data retention policies and data freeze plans, so you’re prepared for the future. Even if ACTS goes away tomorrow, the calls for this type of data aren’t going anywhere. 

Sparks 

• Higher Ed’s AI Integration Overhyped, New Data Shows (Inside Higher Ed) - Instructure analyzed the external learning tools integrated into Canvas and found that the most widely adopted AI tool in Canvas didn’t even crack the top 100 tools used.
• Share of Americans who say college is ‘very important’ hits a new low (Higher Ed Dive) - A new Gallup poll found that only 31% of Americans think a college education is very important. Maybe not the best news as we enter the demographic cliff.
• U. of Michigan Leaders Hit the Brakes on Pass/Fail Plan (The Chronicle of Higher Education) - The University of Michigan’s College of Literature, Science, and the Arts has paused its plans to use pass/no credit grading for first-year students. Paused because of media coverage and its portrayal of a UMich education...hmmm.