By David Wiley, Chief Academic Officer

Originally posted on the opencontent.org blog, February 9, 2018 

In the spirit of iterating toward openness, I’ve recently had the opportunity to revisit some of my earlier thinking about how to measure the impact of OER-related work. Some of this seemed interesting enough that I thought I would share.

I have previously written about metrics I call the educational golden ratio and the OER impact factor. These are ways of thinking about the learning-related return on investment students get from their purchases of learning materials. Here’s an example from the 2014 essay linked above:

For example, beginning in 2011 [Lumen Learning] helped a college in the northeast move their College Algebra course away from a $180 MyMathLab bundle to an open textbook, open videos, and a hosted and supported version of MyOpenMath – an open source platform for providing online, interactive homework practice. In Spring Semester 2011, when every section of the course used the $180 bundle, 48.4% of students passed the course. In Fall Semester 2013, after all sections of the course had transitioned to the OER and open source practice system (which Lumen Learning hosts and supports for $5 per student, paid by institutions and not students, for institutions who don’t want to host it themselves), the percentage of students passing the course grew to 68.9%.

So for a scenario like this one, the two ratios would be:

  • Old model: rg = (48.4% pass rate) / ($180 required textbook cost) = 0.27 percent passing per required textbook dollar
  • New model: rg = (68.9% pass rate) / ($5 required textbook cost) = 13.78 percent passing per required textbook dollar

The golden ratio provides a simple, intuitive way to talk about the overall impact of an educational product. It also provides a similarly straightforward way to compare the overall impact of two products…. We can also calculate an “OER impact factor” which I’ll designate w (omega for open) – the overall effect of switching from publisher materials to OER – by dividing the golden ratio for OER by the golden ratio for the previously used publisher materials:

w = 13.78 / 0.27 = 51.03

I think this would be an extremely interesting metric for open initiatives to explore and report.

I discussed a similar issue – why we should think about the cost of materials in addition to thinking about the degree to which they support student learning – in a brief essay about how the FDA thinks about the difference between efficacy (which is measured in the lab) and effectiveness (which is measured in the real world). In that essay, I suggested that we should think of effectiveness as

effectiveness = efficacy x affordability

because when you can’t afford a textbook or online homework system (or a cancer treatment), it doesn’t matter how well it works in the lab. Its practical effect is to be perfectly ineffective.

Maybe it’s because I’m getting older, but I find my thinking moving beyond the “learning outcomes per dollar” impact of a set of learning materials and increasingly including thoughts about how you would measure the impact of a person or organization. As I reflect back on 20 years of work in open education, sincerely hoping to be blessed with another 20 years to work in the field, I guess it’s natural to ask questions about the kind and scale of impact one is having. Is this the kind of difference I want to be making? Am I making as much of a difference as I hoped to make?

As I’ve tried to think about the simplest possible model for answering this question in my personal context (keep in mind that your context is different and you will want to develop a different model for yourself), I’ve felt the need to add one more variable to the learning outcomes per dollar equation: the number of people affected by the work. A somewhat ridiculous example may serve to make the point: even if you could create learning materials whose use resulted in instant and complete mastery and you were able to pay people for using them rather than charging for them, if only 17 people ever used those materials you probably wouldn’t have had the impact you wanted.

As I thought through these three goals – improving learning, saving money, and benefiting more students – and the ways they interact with one another, something obvious but interesting occurred to me:

  1. These three individual goals can be grouped into three distinct pairs of goals.
  2. You generally engage in activities that directly support your goals.
  3. Consequently, there are probably some identifiable patterns of activity associated with each pair of goals.

To try to describe this picture in less than 1,000 words,

  • If you choose to optimize your work for number of students and cost savings (bottom of the triangle) – that is, if you’re trying to save the most students the most money possible, ignoring learning – then a disseminate-OER-as-PDF strategy makes a lot of sense.
  • If you’re trying to optimize your work for number of students and learning gains (left side of the triangle) – that is, if you’re trying to help as many students as possible learn as much as they can, ignoring cost – the field’s response (alas, misunderstanding Bloom’s 2 sigma problem) has historically been expensive intelligent tutoring systems, adaptive systems, and personalized learning systems.
  • If you’re trying to optimize for cost savings and learning gains (right side of the triangle) – that is, to help your own students save as much money as they can and learn as much as they can, ignoring the broader applicability or scalability of your approach – then using open pedagogy in your class might work well.

I don’t mean to imply or suggest that any of these pairs of goals is more inherently valuable than another. I don’t see it that way at all. Rather than judging between these pairs of goals, my interest is in what lies at the center – what types of activities could simultaneously create all three kinds of impact, specifically in higher education settings? My current efforts in this area happen as part of the Lumen Learning team and are best exemplified in our work on Waymaker. But thinking about how to optimize for all three kinds of impact is a design problem – meaning there are many potential solutions. Our designs will improve over time, as will those proposed by others. If you’re interested in K-12, informal, or other settings beyond higher ed, you will likely end up thinking about the underlying problem differently, too.

My current best thinking is that this is the simplest way of modeling the kind of impact I want to create over the next 20 years:

impact = learning gains x cost savings x number of students

Open approaches are an absolutely indispensable tool for creating this impact. However, another happy result of my recent reflections on impact was the reminder that openness is a means not an end. We should adopt OER and use OER-enabled pedagogies in higher education because they can improve learning, decrease costs, and improve participation. Obviously, the degree to which open approaches help accomplish these goals will be a function of which OER we adopt and how we ask students to interact with them. Should adopting a specific open approach turn out to decrease learning, increase costs, or lower participation, we should abandon or drastically revise it immediately. All of that to say, to myself as much as anyone else, remember – openness is not the end goal – it is a critically important means to achieving our actual goals.