Understanding why students stopout will help in understanding how students learn in MOOCs. In this report, part of a 3 unit compendium, we describe how we build accurate predictive models of MOOC student stopout. We document a scalable, stopout prediction methodology, end to end, from raw source data to model analysis. We attempted to predict stopout for the Fall 2012 offering of 6.002x. This involved the meticulous and crowd-sourced engineering of over 25 predictive features extracted for thousands of students, the creation of temporal and non-temporal data representations for use in predictive modeling, the derivation of over 10 thousand models with a variety of state-of-the-art machine learning techniques and the analysis of feature importance by examining over 70000 models. We found that stop out prediction is a tractable problem. Our models achieved an AUC (receiver operating characteristic area-under-the-curve) as high as 0.95 (and generally 0.88) when predicting one week in advance. Even with more difficult prediction problems, such as predicting stop out at the end of the course with only one weeks' data, the models attained AUCs of 0.7.
Research studies performed on the same data as ours in this paper show a steady progression in how variables are assembled and progress is made on this data. @cite_10 identify the sources of data in MOOCs and discuss the influences of different factors on persistence and achievement. @cite_16 identifies the demographic and background information about students that is related to performance. @cite_12 assembles 20 different variables that capture aggregate student behavior for the entire course. @cite_49 posits variables on a per week basis and correlates with achievement, thus forming a basis for longitudinal study. Our work, takes a leap forward and forms complex longitudinal variables on a . Later, we attribute the success of our predictive models to the formation of the variables.
@cite_16 @cite_10 @cite_12 @cite_49
2036049175 2153378020 2171634212 2115903838
We hypothesized that college major persistence would be predicted by first-year academic performance and an interest-major composite score that is derived from a student’s entering major and two work task scores. Using a large data set representing 25 four-year institutions and nearly 50,000 students, we randomly split the sample into an estimation sample and a validation sample. Using the estimation sample, we found major-specific coefficients corresponding to the two work task scores that optimized the prediction of major persistence. Then, we applied the estimated coefficients to the validation sample to form an interest-major composite score representing the likelihood of persisting in entering major. Using the validation sample, we then tested a theoretical model for major persistence that incorporated academic preparation, the interest-major composite score, and first-year academic performance. The results suggest that (1) interest-major fit and first-year academic performance work to independently predict whether a student will stay in their entering major and (2) the relative importance of two work task scores in predicting major persistence depends on the entering major. The results support Holland’s theory of person-environment fit and suggest that academic performance and interest-major fit are key constructs for understanding major persistence behavior. Earlier studies have suggested that higher education institutions could harness the predictive power of Learning Management System (LMS) data to develop reporting tools that identify at-risk students and allow for more timely pedagogical interventions. This paper confirms and extends this proposition by providing data from an international research project investigating which student online activities accurately predict academic achievement. Analysis of LMS tracking data from a Blackboard Vista-supported course identified 15 variables demonstrating a significant simple correlation with student final grade. Regression modelling generated a best-fit predictive model for this course which incorporates key variables such as total number of discussion messages posted, total number of mail messages sent, and total number of assessments completed and which explains more than 30 of the variation in student final grade. Logistic modelling demonstrated the predictive power of this model, which correctly identified 81 of students who achieved a failing grade. Moreover, network analysis of course discussion forums afforded insight into the development of the student learning community by identifying disconnected students, patterns of student-to-student communication, and instructor positioning within the network. This study affirms that pedagogically meaningful information can be extracted from LMS-generated student tracking data, and discusses how these findings are informing the development of a customizable dashboard-like reporting tool for educators that will extract and visualize real-time data on student engagement and likelihood of success. Longitudinal behavioral data generally contains a significant amount of structure. In this work, we identify the structure inherent in daily behavior with models that can accurately analyze, predict, and cluster multimodal data from individuals and communities within the social network of a population. We represent this behavioral structure by the principal components of the complete behavioral dataset, a set of characteristic vectors we have termed eigenbehaviors. In our model, an individual’s behavior over a specific day can be approximated by a weighted sum of his or her primary eigenbehaviors. When these weights are calculated halfway through a day, they can be used to predict the day’s remaining behaviors with 79 accuracy for our test subjects. Additionally, we demonstrate the potential for this dimensionality reduction technique to infer community affiliations within the subjects’ social network by clustering individuals into a “behavior space” spanned by a set of their aggregate eigenbehaviors. These behavior spaces make it possible to determine the behavioral similarity between both individuals and groups, enabling 96 classification accuracy of community affiliations within the population-level social network. Additionally, the distance between individuals in the behavior space can be used as an estimate for relational ties such as friendship, suggesting strong behavioral homophily amongst the subjects. This approach capitalizes on the large amount of rich data previously captured during the Reality Mining study from mobile phones continuously logging location, proximate phones, and communication of 100 subjects at MIT over the course of 9 months. As wearable sensors continue to generate these types of rich, longitudinal datasets, dimensionality reduction techniques such as eigenbehaviors will play an increasingly important role in behavioral research. In massive open online courses (MOOCs), low barriers to registration attract large numbers of students with diverse interests and backgrounds, and student use of course content is asynchronous and unconstrained. The authors argue that MOOC data are not only plentiful and different in kind but require reconceptualization—new educational variables or different interpretations of existing variables. The authors illustrate this by demonstrating the inadequacy or insufficiency of conventional interpretations of four variables for quantitative analysis and reporting: enrollment, participation, curriculum, and achievement. Drawing from 230 million clicks from 154,763 registrants for a prototypical MOOC offering in 2012, the authors present new approaches to describing and understanding user behavior in this emerging educational context.