Conventional wisdom states that the future of higher education lies online. However, few studies tell us whether this is necessarily a good thing. Indeed, both the detractors and supporters of online education tend to rely on anecdotes rather than data. So a recent report by William Bowen, Matthew Chingos, Kelly Lack, and Thomas Nygren of Ithaka S+R, a non-profit organization devoted to furthering online education, is a welcome addition to the discussion.

The report, \”Interactive Learning Online at Public Universities: Evidence from Randomized Trials,\” summarizes the results of an experiment the team conducted to rigorously compare Interactive Learning Online (ILO), with traditional classroom-based learning (CBL). Bowen et. al. randomly assigned students wanting to take introductory statistics into two groups. The first enrolled in a traditional classroom-based course, while the second took a prototype ILO course developed at Carnegie Mellon University and met face-to-face once a week. Both groups were subsequently tested for their mastery of the material.

The study found that there was no statistically significant difference in outcomes for students in their sample overall, and none for any particular subgroup (by gender, ethnicity, language spoken at home, year in college, or income level). It appears that students learned just as well from ILO as from CBL. When you consider how inexpensive ILO is (or is likely to be in the future) compared to CBL, this is a major finding. If confirmed, it means that ILO is likely to be far more productive.

We should commend Bowen et al for using rigorous tests to discover what actually works in higher education. While randomized control group experiments are the norm in medical research, they are uncommon in educational research. One of the chief reasons I tend to be skeptical of \”great new innovations\” in education is that they are rarely rigorously tested. On the college level, I\’ve seen the \”audio-visual age\” come and go, followed by the \”computer-assisted instruction age.\” And over the decades I\’ve dealt with the results of educational \”reforms\” such as the New Math, the New New Math, Bilingual Education, and the \”whole word\” approach to reading in my own classes.

But I have some issues with the Bowen group study. For one, the report itself observes that \”Levels of educational attainment in this country have been stagnant for almost three decades, while many other countries have been making great progress in educating larger numbers of their citizens.\” True enough–but have any of the countries that have shown progress used ILO to achieve it? If not, perhaps our focus should be on adopting what actually works in other countries first.

Also, questions remain about the study\’s scope. It only looks at an intro statistics course–a basic math course. It would be nice to see similar experiments with arts, humanities and lab science classes to see if the effects work there as well. Moreover, the study\’s sample size was not large–605 students total, to be exact. That is larger than is common in much educational research, but that is hardly a high standard. There is another issue with the students mentioned in the report: they all chose to enroll in the ILO. This means that that ILO might only be effective with highly motivated students. Finally, the study did not include data from community colleges. But the community college students are generally the weakest. This might mean that ILO works only or mostly for students with good academic backgrounds.

To their great merit, the authors concede the study\’s limitations. They warn against both accepting online education willy-nilly and discounting it entirely. As they put it, \”We do not mean to suggest–because we do not believe–that ILO systems are some kind of panacea for this country\’s deep-seated educational problems, which are rooted in fiscal dilemmas and changing national priorities as well as historical practices.\” In short, this study is intriguing, but is it only a first step towards comprehensively testing the usefulness of ILO.

Author Bio:Gary Jason is an academic philosopher, a senior editor of Liberty, and author of Dangerous Thoughts.