Better Use of System Log Data: Shared Learnings from the Forum and New Priorities
2016 Pathways Forum Blog Series 2 of 4
Carnegie and SRI International—a non-profit research institute based in Menlo Park, CA—are collaborating to learn from data that students generate as they use one of the Pathways online learning systems. For background purposes, as students log into a course’s website, whether a Quantway or Statway course, certain pieces of information are collected and stored by the system (i.e., system log data, or “log data”), such as which readings, assessments, and practice activities students accessed and when. These data offer a great deal of potential in that they are collected during students’ learning activities without interrupting their learning. Moreover, these data can be analyzed longitudinally, providing unique opportunities to better understand how specific learning behaviors unfold over time. System log data is well suited for improvement purposes when aligned to a working theory for improvement and displayed using tools like run charts and Shewhart control charts. We took a deeper dive into the analysis of Pathways online data at this year’s Forum, and we look forward to building on this work through ongoing testing in classrooms across the network and sharing out successful interventions in the coming year.
Faculty who use online systems as part of their instruction typically have access to dashboards and data displays on how well a student did on a particular assignment along with the number of attempts he or she had on a given assignment. In some cases, faculty might also be able to view in aggregated form the number of readings and practice activities individual students completed along with the total amount of time that a student spent logged in. All of this information can be incredibly valuable to faculty: at any point in the semester, an instructor can see how well students have done up to a given point in the course.
A key limitation of traditional visualizations of students’ use of online systems, however, is that most data displays only capture aggregate information that can hide patterns of behavior that manifest and change over time. When used to better understand specific learning behaviors, time can be helpful in dis-aggregating data. For example, what did students do before they took an assessment? What did students do after they took an assessment? These questions, while simple, are only tractable when looking at behaviors over time.
As SRI and Carnegie worked to understand the potential benefits of data generated by students through their use of online systems in the Math Pathways, the partnership collectively clarified goals, strove to understand the environments in which students and instructors worked along with potential connections to prior research, and jointly developed and interpreted data products. For example, building off of the work of the Starting and Staying Strong teams as well as additional research scans, we identified high leverage measurement opportunities for both Quantway and Statway online platforms, such as what students do before and after an assessment especially after experiencing an initial “low” score.
As our understanding of the instructional context, research findings, and data analyses mutually reinforced one another, SRI and Carnegie—together—began sharing what was learned with faculty. The latest Pathways Forum represented our largest audience to date, where across two presentations and a design workshop session we shared what we learned from our collective analyses of Canvas data.
We outlined the importance of students doing well early (i.e., starting strong), both in terms of early assessment performances as well as engagement with learning resources such as online homework and practice opportunities. Also, students who persisted on assessments until they were successful tended to earn higher grades in the course as compared to students who did not retry an assessment after earning a “low” score.
A key theme that we have observed across multiple analyses is that there is a great deal of variation within courses and between courses in terms of how faculty and students use the online systems in the Math Pathways. As we move into the next phase of our improvement efforts, we want to find ways to highlight bright spots and reduce variation over time. An important part of this work will involve working with more faculty to test out promising ideas and build knowledge as a network.
Over the next 12 months, SRI and Carnegie will be working to analyze more data from the Pathways’ online systems. Our joint work will shift toward co-designing interventions with faculty throughout the network and using data from the online system to answer the key improvement question: “How will we know that a change is an improvement?”
If you are interested in joining, email Pathways@carnegie.org.