By David Wiley, PhD, Chief Academic Officer and Robert Bodily, PhD, Senior Data Scientist
A substantive body of research demonstrates the incredible impact teachers can have on student learning when they know where their students are in the learning process and adapt their teaching to meet their learners’ specific needs. These adaptations are most effective when grounded in relationships of care and trust between teachers and students. (We provide a high-level discussion of this research in the final section of this article.)
Lumen’s Waymaker courseware applies these principles in course materials developed using open educational resources (OER). Waymaker courses curate learning outcome-aligned text, video, practice questions, simulations, and other learning activities designed to help students master the content. They also include features specifically designed to help catalyze meaningful teacher-student relationships and to support teachers in developing accurate understandings of where their students are – while there’s still time for teachers to influence student learning.
While these features are grounded in research, that doesn’t guarantee that they’ve been designed and implemented in a way that will actually result in improved learning for students! Below, we describe our implementation of these features in Waymaker courses used by thousands of students between August 2017 and August 2019, and the results of an empirical test of its influence on student learning.
How Waymaker Supports Teacher-Student Relationships
In Waymaker’s mastery-based model, all students are given at least two opportunities to take each end-of-module quiz. This quiz is comprised of outcome-aligned items that are drawn from a pool, so that a student’s second attempt is on a quiz equivalent to their first quiz, but not the same quiz.
Waymaker’s messaging tools help faculty track how students are doing and greatly simplify the process of reaching out to struggling students for one-on-one follow up. When a student fails to achieve mastery (set by default at 80% correct) on their first end-of-module quiz attempt AND that student has made use of the practice opportunities built into the courseware, they are brought to their faculty’s attention in this dashboard:
When the faculty clicks MESSAGE in the Actions column, they see a templated message filled in with information specific to the student, including their name and the specific learning outcomes they struggled with on their first quiz attempt. The message invites students into a conversation with their faculty member, either face-to-face during office hours or synchronously in some other medium. Faculty can review the message, edit it if they choose, and then send it on to the student.
But Does It Impact Student Learning?
We examined the amount of improvement 30,678 students saw between their first and second quiz attempts over a total of 164,354 quizzes. We divided students who failed to achieve mastery on their first quiz attempt into two groups: (1) those who were sent a message like the one above by their faculty and (2) those who were not sent the message.
The graph below shows the amount of improvement each group of students saw between their first and second quiz attempts. As students’ scores on their first quiz attempt gets higher, the amount they improve on their second attempt decreases – because they have less room to improve. The size of the difference in improvements between the two groups also decreases similarly.
The difference in the amount of improvement the two groups saw from their first to their second quiz attempt is statistically significant, with students who were sent messages by their faculty improving significantly more than students who were not sent messages.
There is an interesting issue in these data which deserves further study. Some students begin their second quiz attempt immediately following their first quiz attempt. This student behavior does not allow faculty time to reach out and help them. Consequently, the faculty behavior of interest (sending messages) is inversely correlated with student behavior (not waiting between quiz attempts) which may have its own independent relationship with improvement between attempts. Additional research into this relationship is needed.
Previous Research Exploring Teacher Impact on Learning
Hattie (2015) describes lessons learned from synthesizing the findings of 1200 meta-analyses relating to influences on student achievement. (This is called a meta-meta-analysis!) These 1200 meta-analyses include more than 65,000 studies, which in turn include around a quarter of a billion students. In the summary section of the article, he writes:
The estimates from the synthesis are that about 50% of the variance in learning is a function of what the student brings to the lecture room or classroom… About 20% to 25% of the total learning variance is in the hands of teachers (p.87).
Of course the learner – who is the one doing the learning – plays the largest role in this process. But the role of the teacher can also be massive! We all know the importance of teachers intuitively, but there are a lot of things people intuitively “know” about education that are totally and completely wrong (looking at you, learning styles). So, having empirically confirmed the magnitude of the influence teachers can have on student learning, we are naturally led to ask – how can teachers maximize their influence on student learning?
On Hattie’s website there’s a running list of the 195 influences he and his team have reviewed in their meta-meta-analyses. This list is sortable by effect size (the degree of impact of the influence on student learning) and each influence is categorized by domain and subdomain. One of these groupings is “Teacher” (others are School, Student, Curricula, and Teaching). The highest impact practice in the Teacher category, and the third most impactful influence overall (third out of 195), is “Teacher estimates of achievement” (d = 1.29). This means situations in which a teacher has an accurate sense of where each learner is in their learning, and proactively uses that knowledge to decide how to adapt their teaching.
The notion of “teacher estimates of achievement” is closely related to many other high ranking influences on Hattie’s list. In this article, we focus on “Teacher – student relationships.” This is another high ranking influence within the Teacher category (d = 0.52). In 2009, Hattie wrote:
Building relations with students implies agency, efficacy, respect by the teacher for what the [learner] brings to the class (from home, culture, peers)… Developing relationships requires skills by the teacher – such as the skills of listening, empathy, caring, and having positive regard for others.
We share this quote from Hattie’s first book to demonstrate that these relationships are not simply instrumental. That is, they’re not just a way for teachers to continuously update their estimates of student achievement. As Hattie and Zierer (2019) wrote later, “The teacher-student relationship has a huge impact on student achievement. Without a foundation of trust, learning and teaching are virtually impossible” (p.101).
Hattie’s exhaustive research demonstrates the incredible influence teachers can have on student learning when teachers are aware of how their students are doing and they proactively engage those students based on that knowledge. Faculty use of Waymaker’s messaging tools, which are designed to support the development of teacher-student relationships and improve “teacher estimates of achievement,” is associated with statistically significant improvements in student learning. We’re always working to make Waymaker better, but it’s great to know we’re on the right track.
Interested in Learning More?
Digital courseware provides an exciting opportunity to try out evidence-based learning tools, assess their effectiveness, and make improvements to increase their positive impact on learning and teaching.
Visit Lumen’s website to learn more about Waymaker and the variety of ways it is designed to support effective learning. If you’d like to try out how Waymaker courses work, let us know and we’ll set you up to explore any course(s) of interest.
Over the coming months, keep an eye on Lumen’s blog, where we’ll continue sharing insights about what we’re analyzing, seeing and doing to further improve Waymaker’s influence on learning and student success.