On June 1st, a group of professors and researchers from MIT’s RELATE Lab will launch an edX course called Mechanics ReView. What makes this course unique is that, while many MOOC developers discuss the potential for using student data to improve courses, Mechanics ReView has already crunched five years’ worth of data to optimize the learning experience. The thousands of students who took prior online versions of the course have allowed the RELATE team to refine the focus and phrasing of every question and explanation in the course. It’s a great illustration of how scientific method can improve the way science is taught.

The course was first taught at MIT in 2009 to help MIT freshmen who were having trouble with 8.01 Physics I: Classical Mechanics. Unlike traditional lecture courses, Mechanics ReView focused on problem solving. It taught students a style of strategic thinking that made it easier to solve mechanical engineering problems. They learned how to frame the interactions between objects and decide what systems and models govern them.

From the start, the RELATE Lab designed the course as an experiment. They wanted to see how effectively they could improve the exam scores of underperforming MIT students, bringing their grades up to the level of their peers. They “flipped” the traditional classroom model, using class time to teach advanced problem-solving skills, while developing a series of sophisticated online supports as homework to replace lectures.

The course was an immediate success. Students showed measurably better performance on their final exams, and their grades surpassed expectations for later courses such as 8.02 Electricity and Magnetism, so the instructors turned it into a full-blown online course. They expanded its breadth and added more questions, homework problems, and other online components, then offered the course twice more during the spring and summer of 2012.

Each iteration provided more student data. Running statistical queries against the pool of students’ answers revealed which questions were the most effective at teaching a sequence of skills. “There’s nothing in the course that hasn’t been tested,” said Dr. Colin Fredericks, a member of the RELATE team who earned his doctorate in Physics Education from UMass Amherst, “We have a total of about 1100 items in the course, about half of which are problems. The problems are divided into two general types—checkpoint problems that are embedded in the reading to simply test understanding, and homework problems that are designed to test more strategic thinking.”

As Fredericks explains, perfecting a physics problem is a combination of art and science. Even the most advanced number crunching only yields correlations and groupings between questions. The computer highlights which questions best teach a certain skill—but it can never name the skills. “As educators, we have to look closely at the problems that share the same skills, and identify exactly what it’s teaching,” explains Fredericks, “Is it teaching a surface-level feature or something more conceptual? We ask ourselves if the question is teaching the skills we want the students to understand. Do we want to emphasize something else? Can we change a feature of the problem to reduce its complexity and de-emphasize some part of the question that’s causing problems?”

This data-centric approach to course design has already surfaced several best practices for improving outcomes in online learning. For example, research has proven that students are much more likely to read the course material and finish a course when they are frequently tested with checkpoint questions and weekly quizzes. The RELATE team also discovered that adopting a more flexible schedule, with more open deadlines, can increase retention rates.

For the RELATE team, the prospect of presenting Mechanics ReView on edX is especially exciting. Beyond the opportunity to present their course to a much wider audience, more student data will open new avenues of research. “Getting enough data to explore deeper levels of course design can take a long time. It’s not something you can do with 100 students. As you get to 1000 students, it becomes more feasible. So working with the number of students that MITx might reach is very attractive.”

One of the RELATE team’s goals, for example, is to better sequence the delivery of course material. “Right now, we don’t really know if we’re teaching ideas in the right order,” says Fredericks, “We’re doing it as experts and teachers, in a way that makes sense to us. But that doesn’t mean it’s the most effective way. We want to reach a point where we can feed students exactly the problem they need at exactly the right time in order to maximize their learning.”

While it hasn’t yet reached that level of sophistication, Mechanics ReView offers a living example of the next generation of MOOC, where the careful and methodical use of student data is designed to continuously improve course delivery, increase student retention, and enhance learning.