Professional Documents
Culture Documents
SUCCESS IN MOOCs
Robert H. Bryant
Dissertation
Regent University
February 2017
ProQuest Number: 10283615
All rights reserved
INFORMATION TO ALL USERS
The quality of this reproduction is dependent upon the quality of the copy submitted.
In the unlikely event that the author did not send a complete manuscript
and there are missing pages, these will be noted. Also, if material had to be removed,
a note will indicate the deletion.
ProQuest 10283615
Published by ProQuest LLC (2017 ). Copyright of the Dissertation is held by the Author.
All rights reserved.
This work is protected against unauthorized copying under Title 17, United States Code
Microform Edition ProQuest LLC.
ProQuest LLC.
789 East Eisenhower Parkway
P.O. Box 1346
Ann Arbor, MI 48106 - 1346
IDENTIFYING PREDICTIVE VARIABLES THAT FORECAST STUDENT
SUCCESS IN MOOCs
Robert H. Bryant
This dissertation has been approved for the degree of Doctor of Education by:
April, 2017
iii
ABSTRACT
Over the past 10 years, massive open online courses (MOOCs) have gone through
significant changes in terms of expectations and use. During this transition, much
research has been conducted regarding student retention. What has resulted from these
different from any other form of learning modality, including that of distance learning.
This has moved research toward the use of MOOCs to harvest big data for the purpose of
testing various theories regarding learning, instruction, and curriculum design. Little
research has been conducted that would identify variables that could predict a likelihood
algorithm, to identify variables that could predict the performance of participants who
enrolled in MOOC courses at Regent University. Based on the machine learning analysis,
predictive power increased in strength the more activities that were performed by the
retention
iv
DEDICATION
sacrifice, support, and encouragement, I would not have been able to complete the
journey. I consider this her accomplishment as much as my own. Second, I also dedicate
this to my family who sacrificed much time away from their father or grandfather. I
dedicate this with hopes that they will fully realize that seasons come and go, those who
obstruct may pile up like corn-wood, self-doubt may breathe its icy chill to the bone, the
last pedal may fall from a flowering youth, but their loftiest aspirations can be achieved
through Christ who strengthens them (Philippians 4:13, English Standard Version).
v
ACKNOWLEDGEMENTS
I have been blessed to have much support and encouragement from many special
people during my journey toward an Ed.D. To them and others too many to count, I offer
this acknowledgement.
I want to thank Dr. Jason Baker, without whose tutelage and expertise this could
not have been done. Mentorship is a rare and extraordinary commodity. From when I first
realized his depth and devotion to Christ and the Reformed faith, love for classical
and all things geeky, he was imprinted on my mind as the one I most admired and wanted
Glenn Brown. Even though data science was a new journey for all of us, you challenged
worldview with the rigors of each discipline have prepared me in ways that I could only
Learning Innovation and Networked Knowledge Research Lab at the University of Texas
at Arlington, for taking the time to answer many of the critical questions I had regarding
the latest research in the study of MOOCs. Thank you to Dr. Andrew Ng, co-founder of
Coursera and Adjunct Professor at Stanford University, along with Dr. Daphne Koller,
datasets.
A special thank you to Dr. Dan Brown, Dean of the University College and PACE
Center at Texas State University, for the interview on using analytics for college
retention. A thank you goes to Dr. Sinda Vanderpool, Associate Vice-Provost for
Dr. Scott Hamm, Assistant Professor of Education and Director of Online Education at
Further gratitude is extended to my friends and colleagues who gave their time
and talents to afford me the time and space I needed at critical junctures. A special thank
you goes to Dr. John Garrison. Had he not come to serve as the Academy president and
convinced me of the importance of advanced degrees, I would not have likely started the
journey.
Thank you to Mr. Jimmie Scott, president of San Marcos Academy, for
encouragement and support. Thank you to Mr. Jeff Baergen for his friendship and
willingness to do whatever was necessary to help me finish. Thanks also to Dr. Brian
Guenther and Mrs. Cindy Brooks for the encouragement and added duties they assumed
Above all, thanks be to God through our Lord Jesus Christ for whom and
through whom everything exists for the many benefices afforded by His providence that
TABLE OF CONTENTS
ABSTRACT iii
DEDICATION iv
ACKNOWLEDGEMENTS . v
LIST OF FIGURES .. x
LIST OF ABBREVIATIONS . xi
CHAPTER 3: METHODOLOGY .. 54
Setting . 54
Population, Sampling Procedures ... 57
Instrumentation, Apparatus, and/or Materials 57
Instrumentation ... 57
viii
Apparatus/Materials 58
Procedures .. 58
Threats to Validity .. 64
Internal Threats to Validity . 64
External Threats to Validity 64
CHAPTER 4: RESULTS 65
Overview of Luxvera MOOCs ... 65
Demographics in the Who is Jesus? Course 69
Preprocessing the Data 70
Construction of the Two-Class Random Forest Model .. 78
Classifying Predictors of Completion Rates ... 82
Classifying Predictors Relative to a Benchmark Score .. 89
Conclusions . 95
CHAPTER 5: DISCUSSION . 97
The Future of MOOCs 97
Areas of Consideration for Future Marketing and Development .. 100
Adding Value to the Luxvera System ... 101
Adding Value to the Courses 104
Adding Benefit to the Student ... 106
Design Improvements Based Upon MOOC Research .. 109
Limitations and Improvements to the Study . 111
Further Research Opportunities .... 112
Conclusion 113
REFERENCES .. 114
ix
LIST OF TABLES
Table 1. Conversion for Preprocessing the Data in the Course ...... 59
Table 2. Course Enrollments in Luxvera .... 66
Table 3. Features Selected for the Course ... 71
Table 4. User Groups in the Course .... 73
Table 5. Predictive Values for Completion in the Course .. 84
Table 6. Variables Scaled Down by Correlation Values in the Course .. 88
Table 7. Predictive Values Relative to Benchmark Score in the Course .... 91
x
LIST OF FIGURES
Figure 1. String conversion in the course .. 76
Figure 2. Missing data in the course . 79
Figure 3. Outliers in the course .. 80
Figure 4. Evaluation with three high features in the course ... 85
Figure 5. Participation rates by activity in the course . 89
Figure 6. Evaluation of video watching as predictor in the course .. 92
xi
LIST OF ABBREVIATIONS
ACE American Council on Education
ACSI Association of Christian Schools International
AUC area under the curve
Azure ML Azure Machine Learning Studio
CDC Centers for Disease Control and Prevention
CEU continuing education unit
CSV comma-separated value
cMOOC connective massive open online course
DOCC distributed open collaborative course
DV dependent variable
FICO Fair Isaac Corporation
FN False Negative
FP False Positive
IPI Information Processing Index
IV independent variable
GTS Great Talk Series
hMOOC hybrid massive open online course
LOOC little open online course
MIT Massachusetts Institute of Technology
MMOG massively multiplayer online game
MMORPG Massively multiplayer online role playing game
MOOC massive open online course
OE open education
OER open educational resources
PA predictive analytics
ROC receiver operating characteristic
SMOTE synthetic minority oversampling technique
SOOC small open online course
SPOC small private online course
TN True Negative
TP True Positive
xMOOC extended massive open online course
1
CHAPTER 1: BACKGROUND
Massive open online courses (MOOCs) are no longer perceived as the demise of
traditional higher education. Neither have they risen to the fever pitch that utopian
innovators aspired. None of the prognostications of alarmists have come to fruition. They
have not replaced faculty (Holmgren, 2013). They have not destroyed the economic
underpinnings of higher education. They have not knocked academia from its perch
(Waldrop, 2013). At the same time, MOOCs have not fulfilled the promise of high
quality education that is equivalent in value to a college degree and free to the masses.
Neither have they empowered disenfranchised people groups, nor created a worldwide
middle class.
Even though the hype on both sides has largely passed, it has become clear that in
spite of their early development and disruption to the status quo MOOCs have been
sustained as an important part of the academic landscape (Christensen & Weise, 2014).
Since about mid-2013, the hype surrounding MOOCs has diminished. Some have taken
the posture that MOOCs are little more than a comfortable pair of slippers. For example,
Kolowich (2015) said, The conventional wisdom now is that free online courses offer a
promising recruiting tool and an interesting (but not essential) research tool for colleges
that can afford the upkeep (para. 8). On the other hand, those in the adaptive learning
movement have affirmed, We can now do the kind of rapid evolution in education that is
common at companies like Google, which A/B test their ad positions and user interface
Higher education institutions are having to grapple with their existence, whether
they value them as a means of delivering quality education or not. Many have largely
2
disillusionment and are well along the plateau of productivity (Fenn & Linden, 2005).
Even some ardent critics begrudgingly acknowledge that MOOCs are here to stay in
some form (Zemsky, 2014). As reflected in the Kamenetz (2010) article, We can howl
in protest, but the question is no longer whether computer-based, intelligent agents can
prompt learning of some material at least as well as instructor-focused courses (p. 91).
Instead, he resigned himself to the following: The question is whether the computer-
based version can become even more effective than traditional models, and the
strategic plans (Davis et al., 2014). College administrators are affirming that those who
neglect them will not escape the widespread influence (Nanfito, 2013, Ch. 9, para. 10).
Arthur Kirk, the president of Saint Leo University that has 3,000 online courses, affirmed
their importance when he asserted that MOOCs are one of a number of things that are
As White, Leon, and White (2015) noted, many have presumed MOOCs to be a
threat in various ways. The researchers conducted a deep dive into the higher education
analysis of MOOC literature with grounded theory to determine resonating themes. They
discovered that despite the educators reluctance, the foregone conclusion among higher
education professionals is that MOOCs will be sustainable into the foreseeable future.
phenomenon into its proper role alongside existing higher education programs.
3
The current configuration of MOOCs, however, may change from its current
stand-alone posture. Bonk, Lee, Reeves, and Reynolds (2015) concluded, Higher
education can blend MOOCs into their educational ecosystem without major disruptions
and expand its ability to serve growing and diverse student needs for alternative modes of
Even though the evolution of MOOC research flowed along the banks of
Nevertheless, MOOCs are indeed changing the online education landscape. Major
companies are circumventing college degrees and training employees through their own
creation of certain credible badges to award technical certifications. The largest and most
reputable organization to attribute college credit for life experience, the American
Council on Education (ACE), now includes the successful completion of some prescribed
MOOCs in their determination (Masterson, 2013). The ACE membership includes almost
2,000 colleges and universities and about 600 organizations and institutions that use their
recommendations to award credit for college courses. The organization has already
credit for successfully testing based upon MOOC prep classes, much like the College
Level Examination Program system (Sturgis, Rath, Weisstein, & Patrick, 2010). All the
while, MOOCs are morphing into variant strands to meet the needs of the market.
distributed open collaborative courses (DOCCs), and small private online courses
It can be safely asserted that MOOCs have survived an onslaught of critics who
were intent on their elimination. Early studies and reports complained of the MOOCs
surface treatment of subject matter. Some complained that they were too expensive to
such, precious resources were being routed to their development that could have been
used to improve more proven pedagogical methods. Cynics sat passively by while
asserting that this is just another technology fad that will fall of its own weight at some
point.
more affordable and provide a way for students to complete their undergraduate degrees
in 4 years (Bonk et al., 2015). Among other cost-saving measures, Bonk et al. (2015)
noted that MOOC-integrated universities would save by eliminating the need for each
university to employ similar content instructors (p. 16). This, of course, plays into the
fear that faculty will lose jobs to be replaced by MOOC course designers.
Even from those sectors of higher education that normally herald all forms of
innovation, criticisms were being levied against MOOCs. Those deeply invested in the
open educational resources (OER) movement claimed that the likes of Coursera and
Udacity, the two largest early platforms for offering MOOCs, had betrayed and in effect
5
redefined the use of the term open (Bonk et al., 2015). For example, David Wiley, Chief
Academic Officer and Co-founder of Lumen Learning, objected to the way some MOOC
providers have interpreted the idea of open. To Wiley, true openness requires open
licenses and not just open enrollment. To be truly open, Wiley contended that the content
hindrance by the user (Bonk et al., 2015, p. 9). Wiley noted, MOOCs have started to
fall back toward earth under the pull of registration requirements, start dates and end
dates, fees charged for credentials, and draconian terms of use (Bonk et al., 2015, pp. 6-
7). By limiting enrollees to those who can pay, slapping on certifications and
endorsements, issuing restrictive licenses for use upon the users, the big MOOC
providers have in essence moved over from an open architecture to one that is closed. To
him, they have now migrated to distance education courses as opposed to true MOOCs.
To bolster the attacks on MOOCs from many sides and at the center of this
firestorm of criticism was the pitifully low, less than 10%, retention rate that was
consistently being identified. At the time, theorists on both sides of the debate equated
retention with a participants completion of all of the assignments in the MOOC course
and earning a certificate of completion at the end (Jordan, 2014, p. 136). This caused
grave consternation as to the value of MOOCs altogether, when college retention rates
were a mean of 74.3% (Habley, Valiga, McClanahan, & Burkum, 2010) and completion
rates of MOOCs were averaging about 4% (Stein, 2013). The fact that so few participants
completed the course to earn the certificate gave much fodder to critics. The credibility of
the most ardent supporters for MOOCs was being compromised by the mounting data
Anxious MOOCs advocates conducted research to try to discover the cause and
remedy for this factor. The early research was characterized by descriptive statistics,
surveys, and a click-through method used by a new field of data science called learning
analytics. Little quantitative research was done, as the field was still nascent to most
researchers (Raffaghelli, Cucchiara, & Persico, 2015). At the same time, a fundamental
error was being made by conflating completion rates with retention. As George Siemens,
Director of the Learning Innovation and Networked Knowledge Research Lab at the
terms of investment costs (personal communication, January 19, 2016). He argued that
someone who has no consequence whatsoever. Dr. Siemens opined that the fallacy occurs
when we are assessing the outputs, but are assuming that the input variables are the
same, but theyre not. The input variable for a MOOC is that you click a button
Per the early research, retention should not be equated with completion rates. The
expectation factors are vastly different between retention and completion rates (Nanfito,
2013). Nanfito (2013) cited research regarding demographics of MOOC users that more
Participants, and Active Participants (pp. 442-443). This has a definite effect on
determining completion rates. Although 180,000 is the largest recorded MOOC ever, the
7
typical enrollment is about 50,000, with a 90% dropout rate. Nanfito chronicled that most
who are enrolled do not participate beyond watching a video or two and abandoning the
This led to a major paradigm shift when the retention research of MOOCs was no
longer equated with forms of traditional education. MOOCs were hereafter largely
studied as its own medium of instructional delivery. Continuing the evolution of MOOC
for various forms of MOOC development. Because of the nascence in the field, this has
The relative newness of the MOOC phenomenon has also led research to
primarily focus on qualifying terms, descriptive statistics, and utilitarian uses of the
medium. Although the term MOOC has become a part of the nomenclature of education
technology, there remains some ambiguity even at that rudimentary level. For example,
there are no precise benchmarks for enrollment scale to determine what is massive versus
what is not. How many students does it have to enroll to become a MOOC? Must the
Siemens (2013) readily admitted that years later he is unable to adequately define the
Since MOOCs have morphed into varied designs, researchers and theorists are
scrambling to categorize, define, and delineate the one from the other. Even at the time of
the writing of this dissertation, emerging models are evolving as new technology fuses
new delivery systems to push the technological envelope. This further complicates the
issues of generalizability of results from one quantitative research project to the next,
8
The simple notion of reframing the argument, however, did not resolve the issue
of why in the MOOC environment low completion rates continued to plague them, much
less begin to constitute what qualifies as a successful completion. As Kizilcec, Piech, and
Schneider (2013) noted, the focus on completion rates results in a monolithic view of
disengagement that does not allow MOOC designers to target interventions or develop
adaptive course features for particular subpopulations of learners (p. 1). This is because
changing ethos. Some researchers are identifying lines of demarcation between cMOOCs
and xMOOCs (Daniel, 2012). Others are engaged in identifying kinds of motivation
Still others are qualifying what constitutes a completion rate altogether (Khalil & Ebner,
2014). Others are determining what terminologies describe those who drop out at what
point of a course (Sinha, Jermann, Li, & Dillenbourg, 2014). Almost simultaneously,
the increasing value being created by universities and industry. S. Kolowich (2012)
attested that more universities are looking to MOOCs for college credits, for a fee. The
completion of certain MOOCs, and that the University of Massachusetts online is also
looking to award credit for MOOC learning. This is in tandem with Colorado State
9
University who became the first school to offer brick-and-mortar credits for MOOC
course completion (Nanfito, 2013, Ch. 1, Sec. 1, para. 2). With the Babson Group
Survey attesting that 70 percent of public and for-profit colleges now offer online
coursework, Nanfito (2013) advised, MOOCs must be discussed, planned for, and
provides flexibility and choice to students trying to navigate a higher education system in
Consider also the online standard of Mozillas Open Badges, where more than
14,000 organizations and educational institutions are now using them to authenticate
The next iteration of their use is translating badges as certifications for competency-based
learning into college credits, to which the director of the New America Foundation,
Kevin Casey, contended, MOOCs will ultimately (inevitably) be considered for credit
(Nanfito, 2013, Ch. 6, Sec. 3, para. 3). Already a California bill sought to award college
credit for students taking faculty-approved MOOC courses online (Levin, 2013).
Add to this the increasing pressure upon higher education, as major institutions
are using MOOCs for extending reach and access, building and maintaining brand,
outcomes, innovation in teaching and learning, and research on teaching and learning
(Blackmon, 2016, p. 88). It then becomes obvious that colleges and universities should
MOOC.
Further, this increasing value for MOOCs is not limited to universities. Dr. Karen
10
corporate recruiters and managers, as well as legislators, clamor for graduates with
particularly well suited to provide such training (Bonk et al., 2015, p. 17). There is a
growing threat to colleges and universities of major corporations using MOOCs to train
and certify their own technicians, circumventing the requirement for a college degree
Two other fields of study have evolved rapidly that are playing a major role in the
study of MOOCs. One is the proliferation and use of massive databases to harvest
unprecedented amounts of data, like Hadoop and MapReduce. Although somewhat more
mature than the MOOCs themselves, big data, as it is now called, allows for researchers
and course designers to engage with massive amounts of data, create interventions, and
conduct experiments all in real time. O'Reilly and Veeramachaneni (2014) asserted,
MOOC big data is a gold mine for analytics (p. 30). The researchers attested, We can
now build a reliable predictor for which students will exit the course before completion
(p. 30).
education, Finlay (2015) attested, This ability to pick out new data items that are
predictive of something, that an expert might not have considered important before, is
one of the big strengths of predictive analytics (Ch. 5, para. 24). The literature provides
many such examples. For example, PA has grown in its ability to provide actionable
11
intelligence for military logistics, crime prevention, and financial risk analysis (Mayer-
Schonberger & Cukier, 2013, Ch. 10, para 1-3). Many companies like Amazon,
Facebook, and major retail chains attribute PA for online suggestive sales techniques.
Major retail stores and shipping companies like Wal-Mart, Federal Express, and United
new flu virus named H1N1 promised to release a pandemic worldwide. In the United
States, the Centers for Disease Control and Prevention (CDC) frantically tried to get
ahead of its spread but to no avail. By the time patients reported to doctors, who in turn
reported to the CDC, a week or two had lapsed. As a result, the disease was spreading
A few weeks before the H1N1 virus made headlines, engineers at Google published
an article in the scientific journal Nature. In it, Ginzberg et al. (2009, as cited in Mayer-
Schonberger & Cukier, 2013) claimed that Google could predict the spread of winter flu
in the United States down to the specific state and area. CDC officials contacted Google
to see if they could help. Since Google receives about three billion search queries per day
and saves them all, it had plenty of data with which to work. As such, the Google
engineers gathered 50 million of the most common search terms and compared them to
the seasonal flu data between 2003 and 2008. In total, they processed 450 million
different mathematical models in order to identify the best 45 search terms related to the
flu (Mayer-Schonberger & Cukier, 2013, Ch. 1, para. 5). Then, the analytics went to
work and began to identify the spread of H1N1 within hours of transmission. Thus, the
partnership between Google and the CDC was able to stop the spread of the deadly virus.
12
Now, take the power of analytics with big data and move it to education. Many
are familiar with the story of Sal Khan, founder of Khan Academy. By using PA and
massive amounts of data, analysts can identify weak areas in the curriculum and
Because of this new capacity, data science and particularly analytics have enjoyed
unprecedented and rapid development. Various algorithms have been developed and
tested to provide results that boast up to the 97% predictive accuracy (Siegel, 2013, Ch.
2, Sec. 10, Subsection 5, para. 4). This has moved many MOOC course designers, like
Stanfords Andrew Ng, into using these to make design improvements that improve
completion rates in real time (Mayer-Schonberger & Cukier, 2014, Ch. 1, para. 8).
online curricula. This has lent itself to a new field of course design called adaptive
learning. In the adaptive learning design, the course reshapes itself to best address the
performance level of the learner. As in other fields of inquiry, this too is being addressed
through the use of MOOCs. It is yet to be determined whether this will evolve into a
separate type of MOOC or can be embedded in other forms of MOOC development. This
MOOC research.
What this has created in terms of education research is to place MOOCs at the
change quite dramatically in order to benefit from big data (Mayer-Schonberger &
Cukier, 2014, Ch. 3, para 29). This is why research universities are clamoring to develop
and gain access to data and analytics of MOOC courses. Not only does this promise new
13
understanding related to MOOC course design, but the potential of testing other forms of
have leveraged the massive datasets and analytics to understand learner behaviors in
relation to design. Just as many other aspects of the evolving research, however, much of
With the advent of big data and PA, vistas of opportunities are available to
measure entire populations of users rather than being limited to sample sizes (Mayer-
Schonberger & Cukier, 2014). At the same time, the research provides cautions related to
MOOCs begin to statistically quantify those results for the generalizability of their
findings.
As such, the problem that is being addressed by this study is to identify those
learning behaviors that are performed by those who successfully complete MOOC
courses that can predict the successful traits of other students, which may lead to student
important to precisely frame the problem when working with PA. As Finlay (2015)
noted, Identifying the right problem is critical. One limitation of predictive analytics is
that it is very specific (Ch. 8, para. 19). Davenport and Kim (2013) noted, By the end
of it [the PA process] youll need to have created a clear statement of the problem, with
concrete definitions of the key items or variables you want to study (p. 42).
14
Although there are studies that use analytics to address the well-publicized
problem with a low completion rate, the state of MOOC research necessitates studies that
quantify learning outcomes other than just completion rates. As Raffaghelli et al. (2015)
noted regarding the study of MOOCs, data analysis does not devote much attention to
need for quantifying performance has never been greater. As such, this dissertation is
Research Questions
The research question the dissertation attempts to answer is: What performance
MOOC environment? and Is there a stronger correlation between certain behaviors and
reading articles and video watching are analyzed to determine whether they are predictors
Null Hypotheses
The null hypothesis for the study is that there are no variables that predict a
between students who successfully perform on a MOOC quiz and those who do not.
There is no correlation that is stronger between certain behaviors and the predicted
15
Assumptions
Several assumptions are made in the study. It is assumed that the database
information is accurate. Because of the volume and granularity of the data, it is assumed
that all of the relevant variables that play a part in the results have been accounted for or
are compensated by the sheer size of the population and/or design of the study. To test
the veracity of the assumptions and validity of the data, I use the standard practices of
data science. One such problem that too many assumptions can create fallacious results is
called overlearning or overfitting (Siegel, 2013). The first set, or training data, undergoes
the two-class decision forest algorithm, which is designed to minimize overfitting. Then,
Delimitations
There are several delimitations contained in the study. Because the focus of the
study was to identify predictive variables related to student success, it was decided not to
include variables that seem tangential to the performance outcomes on the quizzes or
identified. For example, behaviors recorded in the activity log that are procedural or
tertiary to that primary dynamic of content delivery and learning were purged from the
dataset.
Because over half of all registrants in the six available Luxvera MOOC courses
occurred in a single course, it was decided that its data would adequately reflect the
performance data of the other courses as well. This course alone provides adequate
granularity to conduct an analytic-based study. As such, this study focuses on the course,
16
Who is Jesus?
Since the video viewing and reading of articles were the primary teaching and
learning activities, those two measures along with their correlated time spans from the
time of enrollment are the only ones considered. Because the data were limited to either
entrance or completion time-stamped activities, it was decided to not use them to analyze
the learning behaviors themselves. As such, some of the broader issues of how much time
was dedicated to reading articles, reading behaviors, and video watching were not
Limitations
One of the limitations of the study has to do with the current development of
researchers and provides ample evidence that it is just now beginning to make the
transition into the second phase mentioned by Ebben and Murphy (2014).
defining the phenomena, and issues of design for the purposes of marketing. The research
has only recently begun to focus more on quantitative pedagogical and learning research.
Although I was able to acquire age and gender information, the lack of additional
demographic information is another limitation. This is true of most MOOC studies that
The study applies PA to the Luxvera database at Regent University (Regent). The
Ministries, one course in Business and Economics, and two courses in the Humanities.
The Luxvera courses have been active for differing lengths of time, basically from 1 to 4
years.
The courses are primarily designed as xMOOCs, meaning they are primarily
directed by instructors to students who are recipients of content. Furthermore, the value
of studying these xMOOCs is that There has not yet been extensive published research
on xMOOCs, partly because they are so new, and partly because of their proprietary
Meanwhile, the Luxvera courses have varied features, making them a rich
reservoir to analyze student success relative to these comparative features. For instance,
the length of videos varies widely. Some courses contain guest lectures with quiz
components following the videos. Some videos have creative animations. Some are long
oratory with blank backgrounds. Others have interview formats. All have reading
Once the data were acquired, a process of cleansing occurred. Cleansing the data
the data were clean, they were uploaded into Microsoft Azure Studio, which was then
used to conduct the analytics study. A two-class decision forest was used to identify the
Through this process, students were identified who completed the course and/or
Performance variables that were common to those students were identified, and the
Definition of Terms
Big data. Big data contain the following properties: (a) it contains datasets where
the volume allows it to extrapolate reliable results by analytic formulas that would
require prescribed research methods and statistical formulas using smaller datasets to get
a comparable result, (b) the volume of datasets contains variables beyond the horizon of
inquiry that may contribute toward the results, and (c) it contains complex storage and
retrieval architecture.
Connectivist MOOC (cMOOC). The design of this is based on the learning theory
of connectivism coined by Downes and Siemens (Downes, 2005; Siemens, 2005). The
and the coherence of the course and the progression is constructed by the learner
commonly used metaphor for depicting digging around through data in one fashion or
another, it is often used more broadly as well (Siegel, 2013, Intro., Sec. 6, para. 6).
Data scientist. A data scientist is an expert not only at analyzing data, but at
Feature engineering, This is the process of transforming raw data into features
that better represent the underlying problem to the machine learning algorithm, resulting
in improved model accuracy on unseen data (Barga, Fontama, & Tok, 2015, p. 67).
Massive open online courses (MOOCs). These are courses designed for large
numbers of participants, that can be accessed by anyone anywhere as long as they have
an internet connection, are open to everyone without entry qualifications, and offer a
19
full/complete course experience online for free ("Definition of massive open online
Node. Although the term is used in various ways, depending on the discipline, in
data science, nodes are connection points in a data structure that contain specific records,
Researchers can organize nodes into hierarchiesmoving from more general topics (the
parent node) to more specific topics (child nodes)in order to support their particular
research needs (p. 82). The primary node upon which all other nodes are dependent is
predict future behavior or events based upon big data. To differentiate it from other forms
of analytics, it has the following aspects: (a) it uses various forms of statistical regression
to predict the likelihood of future events or behaviors, and (b) it identifies associations
among the variables and then predicts the likelihood of a phenomenon (Davenport &
Kim, 2013).
takes characteristics (variables) of the individual as input, and provides a predictive score
as output. The higher the score, the more likely it is that the individual will exhibit the
Preprocessing. A data mining technique that involves transforming raw data into
Small open online courses (SOOCs). This is a newer term for a particular kind of
MOOC design. In a SOOC, all students wishing to take the course are obliged to take a
20
Small private open courses (SPOCs). A SPOC is a MOOC designed for a class
of students who are registered at a university in the conventional manner (Pomerol et al.,
2015, p. 16).
To provide clarity regarding MOOC research, one must forge through the morass
of oblique terminology that is endemic to the field. Once that occurs, the research is
better positioned to derive the meanings and results of prior research that can be
extrapolated to the study at hand. After a scientific theory has been applied to the terms,
the evolution of the research literature is studied to ascertain the current state of the
research.
like MOOCs, testability of language plays an important role. This is not a purely
philosophical or abstract exercise. It goes to the heart of the value for this study. For
course, the applicability of the results will be limited to those taking MOOCs. This is also
true of the variant MOOCscMOOCs, xMOOCs, etc. If those aspects that differentiate
one form of MOOC from another are ill-defined, further research will suffer.
their results. One of the most telling examples of this occurred in the interview with
Knowledge Research Lab at the University of Texas at Arlingtonand the one who
reputedly coined the term MOOC. When asked, When does a MOOC become a non-
MOOC?, Dr. Siemens attested that a course ceases to be a MOOC when it does not have
a large number of learners, although he could not define what constituted large. He also
22
qualified that the course had to be primarily online. It had to be open to anyone who had
an Internet connection. And, it had to have a course structure. At the same time, he could
not identify precise demarcations when drilling down each of those aspects; instead, he
faculty have a vested interest in branding and developing a name identity (personal
communication, January 19, 2016). He then disclosed the economic drivers that are at
play in the MOOC terms as well. According to him, a recognized term is a useful
citation, and increases credit to you, like writing a book. Those who are successful, he
Dr. Osvaldo Rodriguez (2012) was among the first to separate the kinds of
MOOC structures by learning theory. The two most prominent are xMOOCs and
cMOOCs. The xMOOCs are based upon cognitivebehavior learning theory and are
does not stand for an eXtended MOOC, but that it stands for an eXtension of something
else. The xMOOC is designed as information transmission modalities. They contain high-
quality content delivery and automation as it pertains to a vast majority of the student
interaction.
On the other hand, the cMOOC predates the xMOOC as an outgrowth of the
connectivist work of Siemens and Downes (Haber, 2013). It is designed with high value
for learner autonomy, use of a diversity of tools, interactivity, and openness in terms of
23
access (Bates, 2014). Kop, Fournier, and Mak (2011) identified four types of activities
that are part of a cMOOC: aggregation, relation, creation, and sharing. Whereas, Downes
(2013a) narrowed the essential components down to autonomy, openness, diversity, and
interactivity. By aggregation, Kop et al. (2011)meant that the cMOOC provides a lot of
content resources to the participants. The relation component attributes the generous use
generate new ideas rather than regurgitate those from the sage on the stage.
Subcategories have evolved within the MOOC architecture. There are SPOCs, LOOCs,
SOOCs, hybrid open online courses (HOOCs), DOCCs, and the list goes on. This view of
the use of the technical language seems to be rather widespread, resulting in imprecise
terms and missing demarcations. For the purposes of this study, the entire superstructure
Upon further study of how other MOOC researchers use language, there seems to
reference knowledge directly from postmodern authors. For example, Kop and Fournier
gave evidence of direct knowledge of the writings of Foucault (Bonk et al., 2015, p. 305).
George Siemens noted Wittgenstein in his grappling with the opaque nature of definitions
and researchers like deWaard et al. (2011) cited Derridas ideas of cultural context as a
Those MOOC researchers who do not provide direct citations to postmodern ideas
seem to demonstrate a use of language that comes out of that ethos in academia. For
example, Karen Head, Director of the Communication Center at the Georgia Institute of
Technology, noted the pains she and her colleagues went through to scrub the images and
language from cultural biases to be understood by the masses. Additionally, she equated
attributions of Western culture as a new form of colonialism (Bonk et al., 2015, p. 13).
Her attempts to scrub her MOOC course from any cultural context is clearly reflective of
Jacques Derridas (1997) idea of deconstruction, even though she did not reference him
in the narrative.
Others are not as socially adept as Head in the use of anti-Western victimary
narratives, which are a prominent theme in postmodern literature. At the same time, just
as Dr. Siemens pointed out, many MOOC authors and researchers reflect Heideggers
(1971) and the latter Wittgensteins (1968) postulates of language equating to use,
It is not only the being-as-use philosophy that permeates the terminology and is
creating a muddled view of MOOC terms, but the language games that it spawns
(Wittgenstein, 1968, p. 5). Dr. John Warwick Montgomery (2012) referenced this as a
problem by noting the four basic themes of postmodernism from D. E. Polkinghorne: (1)
6).
Observing the term MOOC in the aggregate, several associations can be readily
researcher is struck with its lack of attribution to its roots. As such, from what foundation
25
does the MOOC derive its unique aspects? What is its genus?
Like the other terms in this study, so also the term MOOC has varied etymologies
as to its origin. The casual etymologist may resort to the Oxford English Dictionary and
claim that perhaps it is derived from either of the acronyms MMOG (massively
game), two online game constructs developed in the early 1990s ("Oxford Dictionaries,"
2016). Most often in the literature, Alexander and David Cormier coined the term in
reference to the Downes and Siemens course, Connectivism and Connective Knowledge;
yet the authors provided no association between them (de Freitas, Morgan, & Gibson,
from essential components of other innovations, some of which go as far back as the
early 1960s (Nanfito, 2013). This occurs in rather sketchy association, like
educational technology (p. 2). In this case, there is an opaque reference to massive scale.
Perhaps the genus comes from other essential aspects of a MOOC. For example,
perhaps from the fact that a MOOC must be delivered online or that they require large
database capacities to implement. So, does the origination of the term MOOC start with
the creation of databases, distributed file systems, Map-Reduce mapping programs, or the
aspects seen around 2001, with the production of OpenCourseWare online from the
university attempted to provide permanent online content that allowed open use,
modification, and redistribution. Soon to follow MITs lead, Britains Open University
launched the OpenLearn Project, followed by Carnegie Mellon University with its Open
Learning Initiative.
Although these had features of what would later be termed as MOOCs, there were
some important differences. Although they satisfied the online content criteria, the
massiveness and self-contained coursework was not a part of these particular launches.
(p. 203).
Some argue that MOOCs are simply the natural evolution of the open education
(OE) movement. Bonk et al. (2015) said, The history of the development of OER and
OES [open educational services] indicates that MOOC technologies and MOOC-style
pedagogy are more than a fad, but less than a revolution (p. 38). At the same time,
MOOCs have fundamentally changed the definition of openness from its roots. For the
advocates tend to reject the notion that MOOC platforms are open, because they only
The issues at hand are tantamount to asking when does a chair attain its chair-
ness. Wooden legs without a seat fall short of chair-ness; a seat without a back is a stool
not a chair. What are the essential components of the MOOC, and at what point does the
the etymology of the term, much less the current state of research in this particular field
as a whole. The literature affirms that the root etymology of the word occurred at a time
when all four components were implemented, regardless of the start date of the
technology.
changing expectations, needs, and interests. For instance, it was not until 2008 that the
(Liyanagunawardena et al., 2013). In less than a decade, large MOOC platforms like
Stanford Universitys Coursera, edX from MIT, Udacity, P2P University, and the Open
Universitys Futurelearn, among others, dotted the landscape and enrolled many hundreds
of thousands of participants.
Not only has the relative newness of MOOC studies impacted the language used,
but the sudden popularity and success has generated euphoric expectations that have
further complicated the language. From the time that 160,000 students enrolled in
Udacity, to Daphne Kollers and Andrew Ngs start-up of Coursera and beyond, a
utopian idea has surrounded the notion of MOOCs. During the 2011-2012 time frame,
literature was rife with accounts of MOOCs revolutionizing higher education as we know
it (Pappano, 2012). Dr. Siemens noted that the MOOC phenomenon reached its hype
popularization in the Fall of 2011 (Bonk et al., 2015, p. xiii). Promises of free and open
access to top-notch education from the worlds most prestigious universities were
disassociate the terminology of MOOCs from its higher education roots was an effort to
depressurize threats to the early institutions upon which they relied. Ebben and Murphy
(2014) stressed, While leveraging the mystique of top-tier universities, MOOCs exist
well outside of the space and cultures of these hallowed halls (p. 342). The researchers
also noted the shift away from terms like professor, which according to them is rarely
used. Instead, the terms instructor or facilitator have become most frequently used.
Obviously, these are not equivalent terms. They also noted that the term student is rarely
culture, the newness in MOOC studies, as well as the disinformation projected from the
unrealized hype of the early development (Young, 2013). This is hardly the background
for an easily obtained and objective scientific inquiry into an instructional modality.
are attempting to identify precise categories and terms. Obviously, if a MOOC instructor
is the same as a college professor, then there are a host of expectations that accompany
that arrangement. Yet, are the differences substantial? Are they differences in degree or
in kind? Can it be readily acknowledged that the MOOC instructor is not doing
equivalent work as the professor on campus? How about equating a MOOC instructor
with that of an online college professor? If MOOC instructors are equally qualified,
playing out in faculty lounges as detailed by Siemens, it is not enough to simply lament
29
recognizing the effect an ethos may have on perceptions, there should be extra caution
about creating artificial constructs or cavalier associations between aspects that should be
precisely defined and separated. A cogent argument for identifying clear categories and
causal relationships, much like the genus and specie in biology, needs to be constructed.
(2010) earlier work that attempts to achieve a mathematical precision in the use of
language. Specifically, with earned doctorates in theology, history, and law, Montgomery
(2012) argued the case for the use of evidence where truth claims are imprisoned by
culture. He asserted that all scholars serve as a scientist in the sense that he or she uses
logic, collects facts, sets forth explanatory constructs to explain the facts, tests the
constructs against the facts, and accepts those explanations which best accord with the
asserted that all claims, including purely formal ones, are testable. Montgomery refuted
Lessings ditch, the assertion that historical knowledge can never provide us with the
necessary truths of reason, by affirming that all such claims are delving into the realm of
For Montgomery (2005), the testability is not limited to empirical data found in
test tubes. He did this by acknowledging that knowledge never rises above probabilities,
30
2005, p. 67). The testability is equated to legal evidence that is provided in a court of law.
He defined this as evidence having any tendency to make the existence of any fact that
is of consequence to the determination of the action more probable or less probable than
It comprises artifacts that can provide empirical evidence. It also comprises sworn
testimony. The substance of the truth claim is based upon the weight of the evidence, not
clarity) and verankern (groundedness). Regarding the order of clarity, the weight
descends from the most observable without obfuscation to the least. In terms of
A second step toward moving the discussion regarding MOOCs in the direction of
the hard sciences is the work of Herman Dooyeweerd (1984). Dooyeweerd traversed the
their aspects. For Dooyeweerd, aspects are ways in which things can be meaningful. Like
fragmentation errors of the postmodernists concept of being as use. Entities are more
than what they do, as they may contain spatial, quantitative, organic, analytic, formative,
or aesthetic aspects. These aspects may or may not contain certain functions. In addition,
these properties contain certain weights of factuality that are testable by empirical data
and weight of evidence. To move the study of MOOCs toward more objectively precise
terms, which will yield a more credible study, Montgomery (2005) and Dooyeweerd
31
(1984) would ask, Are there evidentiary and aspectual differences sufficient to warrant
One of the first attempts to construct an official definition of MOOCs that could
provide aspectual and evidentiary elements occurred in March 2014 from the HOME
November of that same year, OpenupEd partners, a European MOOC higher education
provider, published a commonly agreed-upon definition. This, then, was tested by a large
survey, which solidified the definition in February 2015. The value of the proposed
definition was not only that it went through the rigors of testability but included the
MOOCs are courses designed for large numbers of participants, that can be
(MOOCs)," 2015)
Likewise, some qualifiers are needed to further ensure precision of the definition.
To determine if something meets the criterion for being massive, it must be scalable and
enroll participants above what would be constituted a normal campus class size.
Regarding the feature of it being scalable, it means that the amount of resources
the increase in enrollment. This, of course, ensures that the MOOC warrants a separate
32
more tenuous. By asserting that the enrollment exceeds the normal class size, Downes
participants. This would be precise, assuming that the MOOC was designed to maximize
the number of participants that could be maintained with whom the presenter could
interact.
massive number of students is that number of participants that exceeds the capacity of the
teacher to instruct students. This would differentiate the term massive as it pertains to
MOOCs from both the number of students on campus as well as through distance
There are the same obvious difficulties as it pertains to identifying big data. At
what point does it go from small data to big data? Is it in volume alone or, as Davenport
(2014) contended, does it also include the kind of data being used? If big data has to
include variations of data like pictures, videos, audio files, text, and other media, at what
point does it become big data? Is it with 10%, 40%, or 60%, or what is the breakpoint
This same precision needs to be brought to bear on the language related to what is
commonly termed as big data. The reason this is important is that tests applied to data
may change as the qualities and quantities of the dataset change. It is theoretically
33
possible that the use of decision trees, algorithms, and analytics may be the best tests of a
certain size and kind of data, whereas traditional statistics may be best for another.
When one tries to arrive at a definition for big data that meets the aspectual and
evidentiary criteria, it is met with the same flux that is found with the term MOOCs.
Waller and Fawcett (2013) wrote, There is a lack of agreement regarding the meanings
of these terms, and because there is a dearth of articles on how these terms apply to the
logistics (p. 78). What are the lines of demarcation that separate one size and kind of
data from another? Or, are all data to be treated equally, regardless of size or kind?
The likely ground zero for the current use of the term came from John Mashey,
who was the chief scientist at Silicon Graphics in about the 1990 time frame (Lohr,
2013). Since Mashey framed the term to reference the huge data files accumulated by
Usenix, the term has been vulgarized to mean many different things.
The etymological trail is convoluted and requires careful scrutiny, as one sashays
between populist generics with only tertiary reference to a separate field of study to
serious research that uses the term within a range of meanings. Like the etymology of
MOOCs, however, Gil Press (2013) claimed the origin much earlier, 1941 to be precise,
where the first attempts to quantify the growth rate of a volume of data occurred by the
Oxford English Dictionary. He further noted that as early as 1944, Wesleyan University
published an article claiming that libraries double in size every 16 years. Press (2013)
discovered a 1961 article that approximated the doubling of scientific journals every 15
years.
Press (2013) next noted that in 1967 there was an article that referred to the
information explosion and called for a fully automatic and rapid three-part compressor
34
of information (Press, 2013, para. 4). He documented and summarized many primary
documents that address the concerns of stockpiling data from 1971 to 2011 as evidence of
an ever-growing concern and call for new hardware and processing capabilities to
manage data. Of particular note is a September 1990 article by Peter J. Denning (as cited
in Press, 2013); this is one of the first published records to predict that the volume of data
will overwhelm the human capacity for comprehension. As cited in Press (2013), the
origin of the term big data occurred in an October 1997 report by Michael Cox and David
Ellsworth. Cox and Ellsworth referred to big data in relation to the volume of data being
In the Press (2013) article, the primary documentation seems to indicate that by
the turn of the century, the term big data is a part of the parlance of data scientists and
other technology insiders. However, the exact size of what constitutes big data increases
as the technology improves. For example, Press referenced an August 1999 article by
Bryson, Kenwright, Cox, Ellsworth, and Haimes (1999). In it, the researchers
acknowledged big data was a term that in previous times was considered in megabytes,
whereas at the time of the article they were scaling Big Data at 300 gigabyte (Bryson
et al., 1999, as cited in Press, 2013, para. 17). Since that time, data gathering has grown
Beyond the mere acquisition of binary data found in databases, Press (2013)
referred to an October 2000 article by Lyman and Varian as the first comprehensive
study to quantifythe total amount of new and original information (para. 18). Upon
reviewing the original report, Lyman and Varian attempted to answer the question, If we
wanted to store everything, how much storage would it take? They determined that the
35
world produces between 1 and 2 exabytes of unique information per year (Abstract).
According to Lyman and Varian, this is about 250 megabytes per individual. What is
more significant is that the data they analyzed included four forms of mediapaper, film,
Three years later, Lyman and Varian (2004) conducted more research on the
subject and found that data accumulation had grown 30% a year between 1999 and
2002, indicating almost double the amount discovered in the prior report (Executive
Summary, Sect. 2). In a later study, Short, Bohn, and Baru (2011) noted that in 2008 the
worlds computers processed 9.57 zettabytes of information. What is also of note is that
while Lyman and Varian (2000) noted a large cache of data by individuals, what they
called democratization of data, no such issue was noted in subsequent reports. This
What this means in terms of defining the term for the purposes of this study is that
there are no lines of demarcation that divide small, medium, and large amounts of data. In
the Press (2013) article that traces the etymology for the term, links can be made to
references of size projections all the way back to the 1940s. However, the most stable
definition as to what constitutes big data was a February 2001 report by Doug Laney
qualifying the term as containing unique volume, velocity, and variety. According to
Press, A decade later, the 3Vs have become the generally-accepted three defining
dimensions of big data (para. 20). This definition has not remained the standard bearer;
however, as Davenport (2014) noted, some current research has added other components
stored information grows four times faster than the world economy while processing nine
times faster, any attempt to differentiate data according to size or speed would be difficult
However, the amount of data is now so large that it no longer fits into the memory that
computers use for processing. As such, engineers needed to invent new tools and
processes to just retrieve and use the data. Davenport (2014) noted,
In effect, big data is not just a large volume of unstructured data, but also the
technologies that make processing and analyzing it possible. Specific big data
technologies analyze textual, video, and audio content. When big data is fast
moving, technologies like machine learning allow for the rapid creation of
statistical models that fit, optimize, and predict the data. (Davenport, 2014, Ch. 5,
para. 1)
problems that are at play when it comes to the term big data. First, as has been noted, the
term big is relative, what is big today will not be so tomorrow. Second, the volume of
data is but one aspect of what is distinctive about the new forms of data. Furthermore, it
is not the most important characteristic in terms of use. A third objection, which
reinforces the malleable nature of terms in the education technology sector, is that some
people use the term big data to mean any use of analytics, or in extreme cases even
reporting and conventional business intelligence (p. 166). As such, Davenport predicted
that the term big data is going to have a relatively short life span.
37
scientists reframe the terminology in terms of what are the intentions related to the data.
For example, financial analysts may analyze video data at their automatic teller machines
combine electronic medical records with genome data to create personalized treatment
plans. In each case, the analysts would be engaged in what has been termed big data
research, but it would be more precise as to what kind of data they were using and for
what purpose.
Although Davenports (2014) use of the term resorts to the being-as-use model, in
another sense, it is being more precise. Another redeeming quality of his perspective is
the evidentiary element of consistent definitions and standards across the data we use for
analysis (Appendix, Sec. 1, num. 4). To better define the term big data, Mayer-
Schonberger and Cukier (2013) framed it in terms of the results; big data are things one
can do at a large scale that cannot be done at a smaller one, to extract new insights or
create new forms of value, in ways that change markets, organizations, the relationship
Although the idea of changing markets and organizations seems to fall within the
pale of neo-pragmatism, it does encapsulate more of an aspectual element to the term big
data than that produced by Davenport (2014). Further filling the aspectual components,
Moses and Chan (2014) viewed the term as more of a catchall that covers the many new
tools that can quickly process larger volumes of data coming from diverse sources with
different data structures (p. 650). It is a term that deliberately moves with technological
advances over time (p. 650). This was apparently the original intent of Mashey around
38
1990. He apparently intended to coin a phrase that would encapsulate all of the rapid
On the other hand, Boyd and Crawfords (2012) attempt to define big data is far
less utilitarian. They defined big data as a phenomenon that rests on the interplay of
gather, analyze, link, and compare large data sets. (2) Analysis: drawing on large
data sets to identify patterns in order to make economic, social, technical, and
legal claims. (3) Mythology: the widespread belief that large data sets offer a
higher form of intelligence and knowledge that can generate insights that were
previously impossible, with the aura of truth, objectivity, and accuracy. (Boyd &
As trained and practicing lawyers, Moses and Chans (2014) critique of the
big data. They accentuated the notion that careful and accurate processes in data mining
are still occurring in a social context. As such, the results from analytics are still
perceived by people with biases. The same skepticism that lawyers have about doctrinal
rules that necessarily predict outcomes should also be applied to big data, because they
Additional problems with the Boyd and Crawford (2012) definition remain. They
did not qualify the term large in the first aspect of the definition. Once again, it is left to
ambiguity. The problem with the second aspect of the definition is with large still being
ill-defined, the use for drawing on datasets may include those items listed, or some other,
39
and can also be said to be a function of a single database on an individuals home laptop.
For Davenport and Kim (2013), data that are easily retrieved, usually contained
within a single database, captured in rows and columns, quantitative, and in relatively
small volumes of a terabyte or two are considered small data. For them, big data
(p. 6).
Realizing the term big data is probably a placeholder for other terms that will be
more precise, to synthesize the perspectives of noted authorities, and to move the
result by applying aspectual and evidentiary concepts to the definition, the following
Primarily, the idea is to capture those aspects that differentiate its capabilities
above and beyond datasets of a lesser volume. To differentiate aspects between big and
small data, the study proposes that big data be understood as to contain the following
properties:
2. The volume of datasets contains variables beyond the horizon of inquiry that
contrasted the traditional sampling method with data analytics: Using all the data lets us
see details we never could when we were limited to smaller quantities. Big data gives us
an especially clear view of the granular: subcategories and submarkets that samples cant
assess (Section 3, para. 2). Whereas, with the smaller datasets, the researcher can
prescribe and control all of what he or she perceives to be relevant, big data move beyond
the controls of the researcher to provide a granularity of information that may provide
PA carries similar baggage of rapid change and popularity as that of the terms
MOOC and big data. It too has become a buzz word and thus suffered a certain
ambiguity. Therefore, the same process is afforded the notion of PA, as the same
obfuscating dynamics are at play. In order to derive a definition that clearly serves the
attributes to contributing toward that ultimate design. A few attribute it as far back as
1689, by Edward Lloyd of the famed Lloyds of London; crude data-gathering and
statistical algorithms were used to determine risk assessments and construct actuarial
tables (Wood, 2013). Some refer all the way back to the dawn of the computer age in the
late 1930s, particularly as to the use in decoding German messages, Kerrison Predictor
automating anti-aircraft targeting, and the Manhattan Project (Holsapple, Lee-Post, &
Pakath, 2014). Some ascribe the beginning to more commercial enterprises in the 1950s
41
and 1960sthe Electronic Numerical Integrator And Computer that forecasts weather,
the use of analytics to solve the shortest path problem in air travel and logistics, Fair
Isaac Corporation (FICO) applying predictive modeling to credit risk decisions, or the
Agriculture ("The analytics big bang: Predictive analytics reaches critical mass as big
science. In the schema for related terms, PA is one form of a broader heading of
analytics. Bichsel (2012) surveyed some research organizations and conducted seven
professionals, one purpose being to define analytics. They were to respond to the
definition, Analytics is the use of data, statistical analysis, and explanatory and
predictive models to gain insights and act on complex issues (p. 6). Interestingly, the
predictive, or prescriptive according to their methods and purpose (p. 3). Add to these
other types of analytics visual (Keim et al., 2008), learning (Picciano, 2012), posterior
(Charles, 2000), and a growing list of other such terms, and one can readily see a growing
ambiguity and conflation in how the terms are used. Van Barneveld, Arnold, and
Campbell (2012) clarified some of the ambiguity: The term [analytics] may reflect
In trying to carefully trace an etymology that may clarify the use of the terms,
researchers meet the same difficulty as was found in other tech terms. For example,
claiming the first to be as old as the first actuaries in insurance or underwriting in banking
could hardly be essentially the same as todays PA. Although some have tried, todays
PA could hardly be traced to the 1940s when the U.S. government began predicting the
outcome of nuclear chain reactions and other computer simulations. Again, whereas these
for the age, the modern PA processes more information on a single laptop, much less
By the early 1970s through the 1990s, analytics were clearly gaining definition
and identity. One of the best known occurred in 1973, where a PA model known as the
Black-Scholes was created to predict optimal stock prices. Less well-known occurred
earlier, where some attribute the first use of PA to John Elder who also used it to
anticipate stock values (Siegel, 2013, pp. 1-3). About 1995, FICO started using analytics
to fight credit card fraud. That same year, Amazon and eBay raced to personalize the
online buying experience using analytics. A few years later, Google applied algorithms to
searches in order to maximize results and relevance ("The analytics big bang: Predictive
analytics reaches critical mass as big data and new technologies collide," 2015). Those
who chronicled these events adopted the term PA to refer to the algorithms applied to
large datasets that predicted future behavior or events based upon that prior information.
For example, one such definition is by noted analytics expert, Eric Siegel (2013),
43
who says that PA are a technology that learns from experience (data) to predict the
future behavior of individuals in order to drive better decisions (Intro., Sec. 5, para. 2).
A similar one by Davenport and Kim (2013) defined PA as using data from the past to
predict the future. They first identify the associations among the variables and then
predict the likelihood of a phenomenon (p. 3). Van Barneveld et al. (2012) defined it as
a process thatacts as a connector between the data collected, intelligent action that can
be taken as a result of the analysis, and, ultimately, informed decision making (p. 6).
Waller and Fawcett (2013) ascribed PA as a subset to data science, which they defined as
the application of quantitative and qualitative methods to solve relevant problems and
The problem with the Waller and Fawcett (2013) definition is that although data
science can use some traditional research methods and statistical formulas, it is conducted
in a different manner than that of traditional sampling and research design. Constructing
an etymology of the term brings one to the same components repeatedly. Simply put, the
literature refers to it as using historical data to predict future behaviors and events.
Rightly did Cooper (2012) object to the notion of Van Barneveld et al. (2012) that
the defining aspect of simple analytics is data-driven decision making. But to correct this
through problem definition and the application of statistical models and analysis against
In addition, only the Siegel (2013) definition accounts for the recent advent of
machine learning in creating an automaticity relative to PA. Most of the other definitions
44
inadvertently retain the being-as-use philosophy. For example, Picciano (2014) wrote,
Essentially it is the science of examining data to draw conclusions and, when used in
decision making, to present paths or courses of action (p. 38). In understanding the use
combine the science of quantitative analysis with the art of sound reasoning (Intro.,
Once again, to synthesize the literature and define PA with its aspectual and
process used in data science to predict future behavior or events based upon big data. To
differentiate it from other forms of analytics, it has the following aspects: (a) it uses
behaviors, (b) it identifies associations among the variables and then predicts the
Another significant term that needs to be clarified for the purposes of this study is
what constitutes student success. Of all terms clarified in this study, this one seems to be
the most subjective and difficult to quantify. Some MOOCs are designed on the outset to
clarify success as completing the assignments and earning a certificate. Others nix the
certification and focus on completion. Calvert (2014) noted that Open University as well
as other external agencies in the United Kingdom define student success in relation to
the students study/qualification aim (p. 161). Those of the cMOOCs design are more
metric. Since the focus of this study is a course designed in xMOOC fashion, it was
45
decided to use participation in respect to activity logs that are correlated to passing
Theoretical Framework
articles pertaining to MOOCs research. Not only did the study critique the methodology
but provided a framework for the progression of MOOC research development based
upon the research of others. This provides researchers a framework by which to gauge the
There were several items of note in the Raffaghelli et al. (2015) research that
indicate a need to be able to predict student success in MOOCs. First is the comparatively
few studies dedicated to this. Only 16 of the 60 articles were oriented toward the learning
processes in MOOCs. Even then, almost all focused on survey data, descriptions, and
concept generations. Raffaghelli et al. constructed six categories used in the research
design. By far, the most common design found in the journals was that of the theoretical
researchers ascribed the term theoreticalconceptual to the studies because they either
which conceptualization of dimension was the most often used. It is here that the research
teams contention that a review of research methodology needs more thorough study
becomes most obvious. Of those 22 journal studies that fell into the conceptualization of
46
support a thesis or substantiate a claim of the authors (Raffaghelli et al., 2015, p. 498).
In the remaining 48 journals studied, Raffaghelli et al. (2015) noted, Little use
was observed of other, more specic and systematic methods aligned with the qualitative
controlled experiment) (p. 498). Similarly, Raffaghelli et al. identified nine categories of
data analyses used in these journals. Of those, the most commonly used method was that
Raffaghelli et al. (2015) acknowledged that the research designs of the peer-
reviewed studies they analyzed were subject to different scientic backgrounds and
skills of the researchers involved in research nature of educational problems (p. 489). Of
note, their study inadvertently chronicled the changes in the research of MOOCs
correspondingly to the increasing use of data science over time. With the advent of big
data and analytics, more research in the field was being conducted by using these tools in
One of the more cogent insights derived by Raffaghelli et al. (2015) was the entire
research field as it relates to MOOCs should undergo a methodological tuning (p. 490).
This tuning, they asserted, should result in producing awareness and agreement between
which the methodological choices are based (p. 490). Thus, this is a clear indicator of
the need I attempt to address in this study to provide clear concept framing and definition,
Later in the study, Raffaghelli et al. (2015) critiqued even those conducting
47
quantitative research, that they were primarily descriptive statistics that did not follow up
with the likes of correlation, factor analysis[or] inferential statistics (p. 499). As
such, the research team viewed them more as exploratory, rather than experimental (p.
499). Beyond this, they alleged this to be a common practice, understandably due to the
nascence of the field in general. They theorized this may be due to the fact that
researchers in the field of MOOCs are making an effort to define the main constructs
needed to drive empirical research (p. 502). All of those studied tended to describe
499). They evinced this claim by noting that only two out of the 60 studies analysed
field of MOOC development. Raffaghelli et al. (2015) noted that the studies that used
analytics identify clicks on specific resources and records of interactions, etc. As reported
by them, these techniques are used in early works as well as in the more recent cases of
scholarly literature on MOOCs (p. 499). They made the caveat, however, that in these
studies the connections between the constructs discussed were loose, and the data were
not elaborated further than descriptive statistics (p. 499). They noted, Few papers adopt
predictive models (p. 499). Of those studied by Raffaghelli et al., only four papers
Raffaghelli et al. (2015) then built their case that research of MOOCs is still in the
early life cycle of educational research by pulling from the framework built by Stephen
48
Gorard (Gorard & Cook, 2007). Gorard and Cook (2007) identified seven phases where
education research tends to evolve. Raffaghelli et al. compared the dates published with
that evolution and found a sufficient pattern in the writings to determine that MOOC
conceptualizing the phenomena and identifying more clearly and systematically the main
In the Raffaghelli et al. (2015) study, a number of areas are in dire need of
research as it pertains to MOOCs; among them are those that foster positive learning
outcomes. More importantly, however, these researchers advocated for a close alignment
and tailoring of the mathematical and statistical procedures to extract and suitably
represent significant data (p. 502). The framing of this need captures the intent of the
research at hand.
To trace the genus and specie of MOOCs, Ebben and Murphy (2014) broke down
the scholarship relative to MOOCs into two distinct phases. In the first phase, they
asserted that the focus was on the rise of connectivist cMOOCs, engagement, and
creativity. In the second phase, the focus shifted to xMOOCs, further development of
MOOC pedagogy, growth of learning analytics and assessment, and the emergence of a
In their second phase of the evolution of MOOC research, Ebben and Murphy
(2014) noted that the scholarly literature aims in different directions. Among the insights,
they shared the need for development of learning analytics based on student
characteristics and behaviors that are recursively applied in MOOCs (p. 342). What
49
Raffaghelli et al. (2015) and Ebben and Murphy showed by this, in terms of the
theoretical underpinnings of this research, is that the field as a whole is still very much in
altogether. That said, it is also clear that a need exists to do much quantitative research
with substantial statistical validation, focused on learning outcomes among a few other
items.
Current Findings
toward learning analytics, using them to improve teaching and learning through big data
and PA. Several studies are precursors and foundational to the current study. Calvert
(2014) used a predictive model to predict student success in distance learning classes.
The researcher identified numerous milestones spanning several years. These milestones
covered not just academic success but retention The milestones included specific
modules of study at specific time periods to give data of a sufficient level of granularity
Although Calvert (2014) was able to acquire more demographic information than
is found in the typical MOOC, there were also many more extrinsic motivators that
would influence completion; the value of this study as a precursor, however, is that the
research established that a relatively small set of routinely collected data that is
associated with student success at various points could be identified (p. 172). Second, it
established that by using PA a person can predict the probabilities that students who pass
the milestones can be used to determine total numbers expected to pass the milestones
(p. 173). Last, the importance of specific variables in generating the predicted
50
A second study was conducted by Smith, Lange, and Huston (2012) of Rio Salado
College. Again, this study was designed for online courses rather than specifically for
MOOCs. A difference lies in that the college provides enrollment every Monday, is
asynchronous, provides the coursework in 8-week intensives, and was facilitated entirely
online. This makes the working environment a little more MOOC-like, although they
have all the extrinsic motivators of a traditional online course that would be absent in the
MOOC. Another factor that makes it foundational to the current study is they chose to
predicting student success. Like many of the completion rate studies of MOOCs, Smith et
al. used the activity tracking methods to determine the predictive variables. They
compiled the top 10 performances of successful students. The researchers used a random
sampling cross-validation method to test the accuracy of the predictive models. Pearson r
correlations were used to quantify the correlation between the course outcomes and
the results of the Smith et al. was applicable to this study in that, This study
demonstrated the strong correlation that exists between LMS [learning management
al. (2015), have mainly been focused on completion rates and attrition issues. These were
characterized more around descriptive statistics, surveys, and raw numbers of results.
51
During this second phase of research, more quantitative studies on teaching and
learning are evolving (Ebben & Murphy, 2014). A recent centerpiece MOOC research is
focused once again on MOOC completion, it quantifies the kind of certificate participants
receive or whether they receive one at all during performance metrics identified during
the first week. This is a useful study because Evans, Baker, and Dee (2016) already
confirmed a significant attrition rate after the first week in MOOCs. Jiang, Warschauer, et
al. (2014)found that participation in peer assessments during that first week yielded the
odds of getting a higher certification at seven times greater than normal. This was
performance and social interaction during Week 1 within the MOOCs to predict their
Jiang et al. (2014) affirmed that participants who are well connected in forums are
more likely to receive distinction than normal certificates. The team used predictive
modeling with logistic regression along with a tenfold cross-validation that confirmed a
92.6% accuracy in the predictions (between certificates earned). For predicting normal
certificates over no certification at all, their model showed 79.6% accuracy. Items for
continued experimentation that is recommended by the authors were research into the
relationship between quality of online courses and student engagement and performance
is recommended (p. 275). Again, this affirms the need for ongoing research in MOOCs
to be able to identify and predict those performance behaviors that lead to success.
behaviors that they termed an Information Processing Index (IPI). Each behavior was
52
weighted according to the strength of the behavior related to engagement. They used
learning analytics and machine learning to determine the outcomes. Particularly, they
used Millers (2011) survival analysis to determine the level of sensitivity of the
clickstream action vectors, IPI, and student dropout rates. They reported that a student
was 37% less likely to drop the MOOC, if in their video viewing profile, they have one
standard deviation greater IPI than the average. If a students rewatching behavior
changed from low to high, they were 33% less likely to drop out. In addition, if they
started watching more proportion of the video lectures, they were 37% likely to drop out
of the MOOC.
Conclusion
In conclusion, the research project adopts practices that were successful in prior
research. It adopts a variation of the Calvert (2014) study of collecting a set of data that is
associated with student success at varying points of performance. Like Calvert, these
The Smith et al. (2012) study was instrumental in that although they were true
distance education courses, they nonetheless shared practical similarities with MOOCs in
several important ways. They too used PA to forecast a successful outcome if students
met certain benchmarks that could be tracked through activity logs. Also, it was shown
that research is appropriate to set a number that is based upon common practice and
institutional interest as to what defines success. In their case, they used a letter grade, C
The Jiang et al. (2014) study was instrumental in providing a backdrop of the
drop-in performance that occurs after the first week, which was confirmed by
53
Balakrishnan (2013). How this contributed to the current work is that there was analytic
work to determine if the predictive variables held true after the first week.
related to video viewing. Since the Luxvera MOOCs are strongly designed as xMOOCs,
it was essential to drill down to these metrics. Whereas Sinha, Jermann, et al. reported
that students performed better if they have one standard deviation than average, I
disaggregated video watching as a possible variable that would lead to student success as
Summary
conduct quantitative research regarding MOOCs. It has further been demonstrated that
due to the nascence of this field of inquiry, critical terms remain malleable. This can have
an obfuscating effect on the research. Further, the terms need evidentiary and aspectual
descriptive statistics, surveys, and utility to more objective and quantitative research
using big data and PA. Particularly, the fields of learning and adaptive analytics are
learner outcomes. Pieces of prior research have provided a framework for the current
work. As such, this research is an extension of the current status of MOOC research and
CHAPTER 3: METHODOLOGY
Much of the current research adopts the framework of the Calvert (2014) study at
Open University. Although, his setting was in college credit-bearing distance education
performance (Smith et al., 2012). For example, Simpson (2006) defined success as
passing an exam as determined by the college. In concert with these and other studies,
this study uses an 80% average or higher on the first attempt of the quiz grades in the
MOOC course as the benchmark for having successfully understood the content of the
courses.
Setting
The current study applies PA to the Luxvera database at Regent. This database
noted, Very few predictive models need more than a few dozen data items to be able to
generate very good predictions indeed (Ch. 5, para. 28). Currently, Luxvera offers
online video presentations entitled the Great Talk Series (GTS) on 22 topics. These are
the system, there are six actual MOOC courses that cover a range of content areas. Three
are courses in Christian Ministries, one in Business and Economics, and two in the
Humanities. These Luxvera courses have been active for differing lengths of time,
basically from 1 to 4 years. One of the first offerings is the focus of this study, as it has
The Luxvera courses in the study are primarily designed with xMOOC features,
meaning the content is primarily directed by instructors through digital media to students
who are recipients of the content. Furthermore, the value of studying these xMOOC-
styled courses is shown by Clow (2013): There has not yet been extensive published
research on xMOOCs, partly because they are so new, and partly because of their
threads, activities, quizzes, and many variations to the structure that allow the student to
alter the order, pace, or trajectory of their learning. However, in most of these features
only beginning or completion time stamps are archived. The clearest exception is the
Should future design allow for more robust data gathering, the panoply of
Luxvera courses have varied design features, making them a rich reservoir to analyze
student success relative to these comparative features. For instance, the length of videos
varies widely. Some videos contain guest lectures with quiz components following the
videos, some videos have creative animations, some are long oratory with blank
backgrounds, one is a single graphic with only audio for the duration, and others have
interview formats. The video recordings in the six courses vary from 53.5 minutes in the
Asia: Yesterday and Today course to that of 130.8 minutes in Why Did Jesus Live?
All courses have reading components of varying lengths and sophistication. The
database does not capture reading performance behaviors that may contribute toward
student success through activity logs. From course to course, the reading assignments
were more varied than the videos. The smallest amount of reading in the course What Did
56
Jesus Teach? contained 6,292 words, while the largest, Exploring the Ancient Word,
contained 33,094 words. In addition to the variance, some courses were designed with
long it took them to read, or whether they felt they understood the material or not, there
were limited opportunities to drill into the reading behaviors themselves to see if certain
Other features are loosely related to the learning objectives. One such feature,
which is consistent in all six courses, is that of a course introduction. It is a narrative that
is designed to be read by each participant at the beginning of the course. A second feature
consists of introductory readings in each unit. This serves as an anticipatory set and thus
warrants inclusion among possible predictive variables. Again, each introduction varies
in length. Because the design is such that a participant can skip this section altogether and
proceed to other parts of the course, this should also be included as possibly predictive of
performance success.
A third feature is that of the discussion thread. D. Yang, Sinha, Adamson, and
Ros (2013) revealed that those who engaged in earlier discussion threads were less
likely to drop out (p. 3). The researchers also noted other patterns regarding
appropriate use of the discussion thread leads to higher completion rates, some of the
Luxvera courses did not refer them at all, some used them for writing assignments but
provided no feedback from instructors, others like the course in this study simply
57
encouraged the students to use them to self-monitor their own growth. The database
provided did not have records of performance in the discussion thread. Regardless, the
data were only tertiary to any learning performance in the Luxvera MOOCs. So, it was
Like almost all other MOOCs, the Luxvera MOOCs do not gather much
survey from Duke University, it may be safe to assume that they are primarily
nontraditional student populations, meaning not college students in their early 20s
(Bolkan, 2015).
The study consists of the entire population of those who have taken the course
entitled Who is Jesus? and/or perhaps other of the six Luxvera courses. The training and
testing data were implemented at a 70/30, respectively. These datasets were configured
Instrumentation
MOOCs by analyzing activity data in the Luxvera MOOCs. To perform this, a two-class
When running some form of decision tree analytic, the algorithms consist of
hierarchical structure that works by splitting the dataset iteratively based on certain
58
statistical criteria (Barga et al., 2015, p. 17). The intersection of these splits is called a
node. The term node takes on many definitions, depending on the context and field of
study. In data science, a node is a connection point at which the data intersect and/or
branch off. Nodes act as the centers through which data are routed. They may contain a
The primary starting point for data processing is called the root node,
intermediary connections that converge or branch off data are called interior nodes, and
the final points of a decision tree are called leaf nodes. In the data structure, there are
parent and child (or sub) nodes. The parent node is that connection that transmits the
Apparatus/Materials
The materials of the study consisted of a MySQL database of six Luxvera MOOC
courses. Since about half of the enrollment occurred in a single course, that course was
selected as the sole source for this study. The data were converted into comma-separated
value (CSV) files using Microsoft Excel 2016. To manipulate the data into the most
useful format, the R programming language software was utilized. To analyze the data,
Procedure
This is a quantitative study using standard data science protocols and a two-class
decision forest to identify those performance variables that lead to student success in the
research] does not depend on statistical assumptions, but instead they learn from the
As noted earlier, some aspects of this study are comparable to the Calvert (2014)
Some of the variables identified in Calvert (2014) as well as Smith et al. (2012)
were includeddays between logged in to the course [time between enrollment and
activities], viewing the course syllabus [viewing the introduction], opening a lesson,
completing a quiz, reading assignments, and the like. The encoding system was aligned
with the nodes, which has become a fairly common practice in the study of performance
data in MOOCs (Sinha, Jermann, et al., 2014). Similar to the Sinha, Jermann, et al.
(2014) study, categories were analyzed by students clicks. This in turn was followed by
interpretive data.
Upon acquiring access to the Luxvera SQL database, a 70/30 training to test data
model was implemented (Kloft, Stiehler, Zheng, & Pinkwart, 2014). The metrics were
Table 1
As George Siemens noted, Overfitting and research design are prominent in any
data analysis work (personal communication, January 19, 2016). Therefore, it was
successful in accomplishing the course were rather straightforward or not. First was
regarding whether the student completed the course or not. Second, the study follows the
Breslow et al. (2013) study, which operationalized achievement to use the course grade
(p. 31). In concert with Breslow et al., participants were required to achieve an 80% or
However, other areas were not as apparent. Xu and Yang (2016) affirmed that
watching the videos provides a much stronger learning result. Since the videos are a
completion, the two-class decision forest algorithm was used. Upon succeeding runs of
62
the algorithm, features were added to ensure that all of the variables that may contribute
readings for time or cognition, it was decided to rely on the quiz grades to be the metric
for reading activities. Again, the two-class decision forest provides information as to the
The design of the course allows for multiple attempts to make a good grade on the
quizzes. There are no time-lapse restrictions. So, a participant can perform multiple
was decided that a node was feature engineered that would determine whether the
participant made more than one attempt at a quiz. If there is more than one attempt, only
the first attempt would be selected to ensure that it is a more accurate reflection of what
was learned from the content, rather than just multiple guesswork.
Three other nodes indicate the consistency of participation with the course.
Tracking the path to success of the individual student by including the date and time
stamps of each may also provide predictive results. As Calvert (2014) noted, The most
exible, transparent and sustainable approach to forecasting was to build a model based
on individual student journeys (p. 161). These nodes help map out the path that the
successful student takes. As such, these were included in the list of variables considered
to be predictive.
At the same time, it is well documented that not all participants in MOOCs intend
to complete the entire course or earn a certificate (Evans & Baker, 2016). My focus is to
63
identify those participants who performed successfully in the MOOCs, identify those
performance variables that accompany them, and recommend a design that can be used
The first node related to participation was to determine if there was a lag time
between when the participant started the course and date/time he or she exited. Because
The second node related to participation may measure whether the participant
jumped to a unit or not. It is possible for a participant to jump over some units and skip
viewing them altogether. Although it may be interesting to note the number of times this
occurs, the base measure is to determine whether the course was followed as designed. If
the participant skipped over units, discovering that pattern is sufficient for my purposes.
This too may be an interaction among the participation measures that would affect their
For the purposes of this study, success is defined by scoring 80% or better
average on the quizzes in the MOOC course. The intent of the study is to identify those
variables that predict student success when participating in a MOOC course. Part of the
reaching the defined benchmark. Participating is determined by the evidence from the
activity log that the student has watched the video, read the materials, and taken the
Threats to Validity
By using analytics and big data, selection and sampling errors are averted by
having the software randomly place the data appropriately. However, there are several
possible internal threats to validity in this study. The most obvious is that of mortality.
MOOCs have a well-established track record of high attrition (Evans & Baker, 2016). It
is also true that the performance data from the first week of participants enrollment is
significantly different than the rest of the weeks of their performance (Jiang, Warschauer,
et al., 2014). However, research shows that these effects are ameliorated by using
analytics and big data. The quantity of participants and variables in a MOOC offset the
bias. Differential selection is another possible internal threat in this study. By using the
two-step clustering component, the risk should be minimized. A third threat to internal
validity lies in the tendency for analytics to overfit the data. However, by using a random
There is a threat to the ability to generalize the outcomes of this study that is
incumbent upon all MOOC designs that lack demographic information. There is the very
real possibility that confounding factors that are not identified in the general population
may confound the results of the study. For example, participants of low socioeconomic
status may have different predictive variables that result in success. This may be true of
CHAPTER 4: RESULTS
As mentioned, the study of MOOCs has primarily been based upon qualitative
studies involving items like course satisfaction surveys. The most recent breech into
data and the like (Raffaghelli et al., 2015). Mostly, this has focused on completion rates
aspects of MOOCs. To move this research forward, this study provides a two-fold benefit
working model. By chronologically marking the path, citing research that justifies
various decisions in creating the model, and referencing experts in the field who have
used this method, it is intended to serve as a reference point for those who seek to use
Second, this study intends to identify certain variables as predictors that likely
increase the potential for success in a MOOC environment. Because of the variables
available from the Luxvera MOOC database, watching videos and reading articles are
Upon observing the raw data from the database, one is struck by the volume. In
addition, the records were segmented by a relational database into multiple tables, and
thus not readily organized to conduct analysis from a spreadsheet. A few missteps in
calculations adds many hours and several days of work to prepare the data into a useful
form. The study of data analytics requires a fundamental paradigm shift in how data are
66
handled. Trying to replicate simple procedures used by traditional research methods with
random samples in a big data environment costs many hours and days of additional work.
For example, the raw data from which this study was extrapolated included
17,024 users who registered to participate in one of Luxveras online offerings. Although
6,141 enrolled in a separate video-watching series (i.e., GTS), 10,883 enrolled into one or
more of the six MOOC courses (see Table 2). The disparity between total participants
and those enrolled in a MOOC course is accounted for by the fact that some participants
Table 2
Because over half of all Luxvera MOOC users enrolled into a single course
entitled (i.e., Who is Jesus?), it was determined that the data contained adequate
in several aspects. Other studies utilized PA to record the efficacy of certain behaviors of
video watching and analyzed aggregate performance, demographic and grade benchmark
correlations (Sinha, Jermann, et al., 2014; Smith, Lange, & Huston, 2012; Calvert, 2014).
Differences lie in that although the research problems being addressed are similar, two of
those studies occurred in traditional online credit-bearing courses. Although Sinha, Li,
Jermann, and Dillenbourg (2014) addressed MOOCs, they too focused on attrition related
The database records consisted of MySQL tables that were converted into seven
large spreadsheets that covered the time span from January 22, 2014 to January 4, 2016.
Prior to receipt, the records were scrubbed of any personal identifiers to produce an
anonymized dataset for analysis. One spreadsheet contained 12,597 registered users
attributed with a unique identification number, status as to whether they activated one of
the courses or GTS, registration time stamp, birth year, and gender. All time stamps for
all spreadsheets recorded each time the participant initiated or completed the activity but
not both.
Another spreadsheet contained 653 separate activities, each with unique activity
identifiers as well as a title and type of each activity. This spreadsheet contained 316
articles, 216 videos, and 121 quizzes. A similar activity spreadsheet provided a unit
identification, name, and whether the activity remained active or not. Still another
spreadsheet was correlated to students and provided 67,722 noted activity performances
by the users, each with an associated user and activity identification number, along with
the date and time stamp that the participant closed or left the activity. Another
spreadsheet disaggregated the data by units and logged user interactions with a
There was also a spreadsheet that contained course identification numbers, codes,
and descriptions of all 23 GTS lectures and six MOOC courses. One spreadsheet
contained 17,026 participation records with course numbers and corresponding user
identifiers. Each contained time stamps for enrollment into the courses, status as to
whether they have completed the courses or not, as well as the dates and times
69
completing the courses. Finally, a spreadsheet with quiz activity data was provided. Each
quiz identifier corresponded to a user identifier. Date and time stamps for completion
Demographic data were harvested from both the GTS and MOOCs, as the
database was designed to require logins with these features in both. The median age of
those who submitted demographic data in the entire database was 54 years old (Mdn =
54). The largest population of 408 users was 52 years old. The oldest participant was 102
years old. There were 5,925 from ages 53 to 72. There were 4,380 enrollees from ages 33
to 52. Below 32 were 1,382 enrollees, with the youngest being 12 years old. The
population of those who declared gender was 8,219 females and 4,249 males.
In the Who Is Jesus? course database, the oldest participant was 102 years old.
The approximate age quartile breakdown of participants was 100 from ages 79 to 102.
There were 2,346 from ages 59 to 78. There were 3,131 enrollees from ages 36 to 58.
Below 36 were 943 enrollees, with the youngest being 12 years old. Of those who
Of the 8,420 students who enrolled into the course, 6,521 were considered active.
Of those deemed active, 1,337 enrolled into the course, but failed to perform either
quizzes or activities. This amounts to a drop in active users or what data scientists call
phenomenal retention rate of 61.6%. In the field of MOOC studies, this would be unheard
of in that they have usually maintained about 10% or less retention rates (Hone & El
70
Said, 2016). If the 356 users who completed the course are considered those who are
retained, and all those who enrolled but failed to complete the course are considered
churn, the data show a retention rate of 4.2%. Since the acquired data ended in 2014, did
not account for current activity, and there is no indication of a metric whereby users
clearly exit or drop the course, it is inconclusive as to whether that is the case.
Prior to implementing the analytic tools, the first step in the analytic process is
what data scientists call preprocessing the data (Barga et al., 2015, p. 28). Particularly,
this involves preparing the data to be used by the analytic software. In this particular
case, it required merging the separate massive spreadsheets into a single file that was to
in preprocessing the data (Barga et al., 2015, p. 12). In this study, a comparable amount
of time was used to identify and merge data into a single CSV file to serve as the datasets
upon which Azure Machine Learning Studio (Azure ML) would derive its data.
The first step to preprocessing was to purge the data unrelated to performance in
the course in the respective spreadsheets. For example, many of the features had
identification numbers that were assigned by the Luxvera SQL database that were simply
to identify locations in the database. Those types of items were the first purging that
occurred.
Because the Luxvera SQL database provided the data in columns. One of the
transform routine. After more than 6 hours of mindless iterations, it became evident that
71
these kinds of feats were better served by transforming the data using a programming
language. This is demonstrative of the paradigm shift required to work with large
datasets. Ultimately, the six spreadsheets were aggregated into a single CSV file to where
the users records were in rows and the demographics, categories, and performance items
of each user, called features in data science, were in columns (see Table 3).
Table 3
sufficiently familiarize themselves with the data as to ensure that the most reliable data
are utilized in the study. As a result, the data were categorized into several representative
and Yang (2016) classified participants by their motivation. In conjunction with their
self-reported motivation, grade predictions were made. They theorized that the precision
of these predictions, however, occurs only as the motivation classifications become more
sophisticated and accurate, not limited to aspirations to earn a certificate. At the same
course, like that of course satisfaction, which also needs to acquire metrics.
users, I referred those who registered for Luxvera and enrolled into the MOOC course,
but neither performed any activities nor attempted any of the quizzes as visitors. A
second group, viewers, performed activities but did not take any of the quizzes. A third
group, participants, took quizzes and activities, but did not complete the course. Another
group, completers, were those who performed activities, took the quizzes, and completed
The visitor group contained 1,362 users (n = 1,362). All but one claimed gender.
The gender demographic for the group was 932 females and 429 males. The median age
for the group was 52 years old. The mode was 57 years old, with 50 users reporting that
age. The oldest was 87 years old and youngest was 12.
73
Table 4
For the participation group who completed some quizzes and performed some
activities, the median age was 54 years old. This was the largest group of enrolled users
(n = 3,024). The mean age was 52 years old. The oldest participant was 92 years old; the
youngest was 12. Of those who declared, there were 4,384 females and 2,219 males.
Based on the data, it is clear that only 4.2% of those who enrolled into the course
completed it from January 22, 2014 to January 4, 2016. Of those who completed, 232
were female with an average age of 55 years old (M = 55, Mdn = 54). There were 124
males whose average age was 53.5 years old (M = 53.5, Mdn = 64). The database for this
group showed 356 users who completed the entire course by completing 6,924 quiz
attempts and 9,127 performance activities. Three were marked as completed in the data
but did not take any quizzes and no activities. Therefore, it was concluded that these were
anomalous and tertiary to the study and were purged from the dataset.
Although the longest span between registration and enrollment into the course
was almost 2 years, 1,322 of the visitors group typically registered 6 seconds prior to
enrolling into the course (Mdn = 0:06). One of the anomalies of the program was that the
design allowed for a user to enroll into a course prior to registration with Luxvera. This
led to about 40 visitors enrolled into the course prior to registering with Luxvera.
The viewers group consisted of 3,138 users (n = 3,138). The median age of this
74
population was 53 years old. Of those who self-identified, the gender make-up was 2,123
females to 1,013 males. Although the largest span from the time a visitor enrolled into the
course until completing the last activity was over 441 days, the time lapse soon thereafter,
measured in days, was a mean of 2.82 days, mode was .001 day, and median was .004
day (M = 2.82, Mode = .001, Mdn = .004). At the same time, the standard deviation
indicated a wide dispersal of time lapses performed by the users (SD = 21.31).
These primarily engaged in watching video lectures. There were 1,753 who
watched the Course Introduction video, 874 watched the video introduction to Unit 1, 22
watched the Unit 2 Introduction video, 11 watched the introduction to Unit 3, two
watched the Unit 4 Introduction video, and three watched the video on Jesus, Son of
David. The same person, only one, read articles on Jesus the Promised Healer, Jesus the
Foretold King, and the article on Unit 4 Introduction. Five read the brief Course
Conclusion article.
Of the participant group, 3,382 typically enrolled in the course soon after
registering with Luxvera (Mdn = 0.07). The data also show that six users enrolled into the
course, took quizzes, failed to perform any activities, and never registered with Luxvera
at all.
engineering to better identify the performance outcomes relative to the quizzes. Twelve
engineering is the process of transforming raw data into features that better represent the
accuracy on unseen data (Barga et al., 2015, p. 67). Feature engineering is at times
75
essential in data science. In this instance, the raw quiz data showed a series of scores
provide aspects of the testing data that would not have been readily available in the raw
data.
For example, it was important to differentiate between the first attempt at a quiz
as well as retesting because one of the DVs was to determine a score of 80 or better on
the first time taking an exam. It was also important to provide that analytic software with
how many exams each participant took, how many retakes, the lowest grade, the highest
grade, and other such items that would not be readily available in the raw data. These
were used to identify significant characteristics of quiz performance that may yield
predictive results.
As such, I calculated the number of quizzes completed by each user, identified the
number of quizzes retaken, minimum quiz score of quizzes taken, maximum number of
quizzes taken, number of quizzes who met the 80% benchmark, score of the first attempt,
score of the last attempt, and average quiz score per individual. These were then
One of the first obstacles to preprocessing the data to comport with Azure ML
occurred with the custom formatted dates and times provided by the Luxvera database
(mm/dd/yy hh:mm). To include these as IVs that may have an impact on successful
student completion, they had to be changed into numeric values. After uploading the data
into Azure ML, it was discovered that this Excel formatting was being formatted as an
Optical Feature. Several attempts were made to change the formatting to other date
configurations that could be converted by Azure ML into numeric values, but it continued
76
Despite using the Edit Metadata module in the analytic software to reformat the
cells as numbers, the program continued to read the cells as text files. The first attempt to
convert the cells for Azure ML to read them as numbers was to convert them by adding a
paste special add routine with the number 0 to all date/time formats in Excel prior to
converting it to a CSV file and uploading it into Azure ML. The analytics continued to
Upon reviewing the massive datafile in R, it was discovered that row 3,235, user
5,159 had an enrollment time stamp but never registered in Luxvera. As such, it skewed
all of the data in that line. After spending almost an entire day trying to solve the
problem, this again was a paradigm shift in preprocessing big data. Rather than assuming
that the analytics software has faulty design features, it is better to use R or Python
variables. By detecting the single faulty line and removing it from the dataset, the
77
Luxvera were also optional in the course design. As a result, a small number of users did
not self-identify either by birth year or gender. To address the issue, the missing data
were replaced in birth year by the median year. As it was decided that the missing data in
the category of gender would not adversely affect the results in light of the substantial
Another hurdle to correctly transfer the data from the Excel CSV file to a dataset
that is recognized correctly by Azure was to account for time lapses in the various
quizzes and activities. Associating time sequences with its impact on the IV is often
problematic, as noted by Q. Yang and Wu (2006), Many time series used for predictions
The only data provided from the MySQL database were the same aforementioned
custom date/time stamps. This posed an additional problem besides converting them into
numeric values that the analytics software could process. The dilemma was how to
standardize the date/time stamps so the data associated with one another sufficiently to
perform data analysis. For example, if user_12 performed quiz_255 on 2/01/14 at 13:22,
what does that have to do with user_17 performing quiz_255 on 12/16/13 at 06:37?
Because I converted these custom date/time stamps to time lapsed from the
date/time of enrollment, this allowed the Azure ML to be able to make the association.
Consequently, both users could be ascribed 4.88 depending the time lapse between the
The metric used to determine the time lapse was in number of days from the time
of enrollment. This allowed the longest lapse to be performed by user_id 3253 in 577.790
days from the time of enrollment to watching the first video in the Who is Jesus? course
to user_id 90, who started it within .008 of the same day he or she enrolled.
After the features were identified and feature engineering conducted, the finished
CSV file served as the foundation upon which each aspect of the model was constructed.
At this point in the process, the literature confirms that most data contain elements that
are incomplete, noisy, or inconsistent. Incomplete data contain missing data or lack the
proper values configuration. Noisy data contains extraneous, erroneous, or outlier data.
One of the prerequisites to building the model required another review of the data
to ensure accuracy in Azure ML. By doing so, several things became evident. Some of
the features in the preprocessed file contained different values than in the original file
Once the correct data have been uploaded into Azure ML, there are several other
items that need to be addressed, particularly missing data, outliers in the data, and data
transformation (Barga et al., 2015, p. 11). Upon adding a module that performed
descriptive statistics to the data, what Azure calls Summary Data, missing data were
40% of data, all features that were deemed ancillary to the study had already been deleted
prior to uploading the preprocessed dataset. To retain the data with missing data, two
Clean Missing Data modules were added to the model. The first module was used for all
string and categorical data, which replaced the missing values with a custom substitution
value of 0. The second missing values module was used for all numeric data, to where the
missing values were replaced with the median score of each feature because of the
tendency toward skewness (Han, Pei, & Kamber, 2011). Also in Figure 2, it can be
readily seen that the feature uid has a large collection of variables in mode. This is
because several of these features had not yet been converted into categories. This was
Upon further review of the dataset, visualizing the boxplot showed many of the
features with outliers (see Figure 3). As Barga et al. (2015) noted, Outliers can skew the
results of your experiments, leading to suboptimal results (p. 57). To address this issue,
Azure ML provides a module entitled Clip Values. To ensure these variables did not
80
skew the outcome, the Clip Data module was used to address the issue.
work behind the scene, such that the researcher cannot see what exact math formulas are
being implemented, there is no easy way to see whether other features are needed to gain
the precise results or not. It is here that good data science takes on an artistic aspect rather
As one studies the data science literature related to selection of mean, median, or
zero to replace missing data, several criteria govern. It depends on the skewness of the
data, why the data are missing in the first place, and what it is you are studying. In the
end, data analysts rely on experience and training to make decisions that are to a degree
subjective.
In dealing with outliers, the choices from the literature seem even more opaque.
Barga et al. (2015) said to either drop the outlier or transform the variable into a log
function. However, the Apply Math Operation module that allows for the Log10
81
algorithm they suggested reduces the input and output to a single variable and does not fit
well in a design with many features being tested at once. Besides the logistics difficulty,
other data scientists attribute some evidence of outliers to noise, by which you
compensate in differing methods (Sunithaa, Rajua, & Srinivas, 2013). Still others
advocate allowing a limited number of outliers depending on the size of the data and
robustness of the algorithm. Since it is common for the literature to affirm random forest
algorithms as robust in relation to outliers, it was decided to allow for the Clip Values
One of the more daunting tasks can be transforming the data such that the
metadata is correctly ascribed to each bit of data. For those newer to the field, the terms
data transformation and feature engineering may be confused. As noted before, feature
engineering adds variables, in data science called features, which enables the analytic
software to identify aspects of the dataset not available in the raw data. Data
transformation does not add variables to the dataset. Instead, data transformation makes
sure that each dataset contains the correct attributions known as metadata.
The term metadata is used to describe the attributes of the actual data itself. For
example, the number one can be read as a string (text), category, or numeric. In the
current study, most of the metadata transferred from the CSV file into Azure ML without
change. However, some of the metadata changed when being added to the model.
Therefore, it was essential to correct the metadata so that the analytic software could read
When there are a variety of metrics as are found in this dataset, converting them
into a normalized form is a best practice in data science (Guyon & Elisseeff, 2003).
82
Barga et al. (2015) wrote, By transforming the values so that they are on a common
scale, yet maintain their general distribution and ratios, you can generally get better
results when modeling (p. 59). Azure MLs Normalize Data module offers several
options to normalize the data. Of the options available, it was decided to convert to z
scores.
To ensure that the model did not overfit the data, a commonly used practice of
splitting the data between 70% training data and 30% testing data was implemented.
Overfitting is where the model is drawing too fine of conclusions from the data that we
have (Hartshorn, 2016, pp. Ch. 7, para. 3). When a model overfits the data, it tends to
create false positives. Standard data science practice is to split the data so that the training
set of data is cross-checked with a separate set of testing data to ensure they are accurate.
To determine student success in the MOOC, the research first led to quantifying
the completion rate of users and try to discover if certain variables or combination of
variables may increase the likelihood of completion. A problem in managing big data was
that the number of users who completed were extremely disproportionate to those who
did not (n = 356, n = 4,828). As such, to be able to find any features that would lend itself
(SMOTE) module provided algorithmic adjustments to the data. Barga et al. (2015)
explained the process that the module deals with class imbalance by a combination of
removing samples from the population until the minority class becomes more
proportional to the majority class. It then oversamples the minority class, not by
replicating data instances but by constructing new minority class data instance via an
Because this study attempts to classify variables that may lead to binary results,
two-class decision forest was used. This algorithm capitalizes on the latest research of
ensemble modeling. An ensemble model uses varied sets of classifiers and panels of
algorithms instead of a single one to solve classification problems. There are two
examples in the training set more important during training, the bagging algorithm uses
different subsets of the data to train each model. As great care was taken in the early
preprocessing phase to ensure all classification would be correct, using different subsets
correlation to those who scored at the accepted rate on the first quiz and those who
completed the course, particularly if by optimizing the data to yield a completion result
there was any measurable difference in those who scored. The difficulty in big data is the
obvious volume of possible variables and combinations of variables that may yield the
greatest impact on the DV. To accomplish this, Azure ML provides an important module
called Feature Selection. As Barga et al. (2015) noted, feature selection is the process of
finding the right variables to use in the predictive model (p. 159).
To determine student success in the MOOCs, the first area of investigation was
84
excessive skewing and kurtosis, and no evidence was provided to assume the distribution
was parametric, there was a decision to use Spearmans rho in the Feature Selection
module. This resulted in a succession of correlative values in order that would it be used
Table 5
The data also showed that there were no correlations to completion from
85
To see what features would likely improve the completion rate, the two-class
decision forest was applied to the top three strongest features that were identified by the
activity_id_26 (rs = 0.749), and activity_id_23 (rs = 0.747). The Evaluation module
confirmed .954 area under the curve (AUC), with an accuracy score of .965, precision
Usually, accuracy is the metric you look to first to determine the predictive
validity. However, in cases where the test data are unbalanced, this is not usually as
effective a classifier (Barga et al., 2015). With such a drastic disparity between those who
were enrolled in the course and those who completed (n = 4,828, n = 356), it was
As such, the literature enjoins the researcher to look at other metrics as well, the
86
receiver operating characteristic (ROC) curve, the AUC, and the relationships associated
with the confusion matrix. This includes True Positives (TP), False Positives (FP), False
Negatives (FN), True Negatives (TN), as well as the accuracy, precision, recall, and F1
scores. In data science, scores in the high seventies and low eighties are often acceptable
The terms recall and precision in data science have an inverse relationship and
unique definitions. Precision is the fraction of retrieved elements that are relevant. Recall
is the fraction of relevant elements that are retrieved. Accuracy is simply a ratio of
In the case of the FP in the Evaluation report, the model incorrectly predicted that
39 elements would lead to successful completion of the course when in fact they did not.
This is the equivalent of a Type I error. The FN indicates the model incorrectly predicted
20 elements that would not lead to successful completion in the course when in fact they
did. The FN category in the Evaluation report corresponds to Type II error in standard
research practices.
Although in quantitative research the traditional goal is to not have more than .05
chance of a Type I or Type II errors, in data science these numbers are mitigated by the
sheer volume of variables tested and amount and kind of tests being replicated
false classifications depends on the items tested and effect on populations at large.
Conversely, in the current study the model correctly predicted 186 elements that
category, the model correctly predicted 1,417 elements that would not result in successful
87
Where the raw data provide clear evidence of the two-class problem, it is prudent
validity. Because the aforementioned metrics are significantly above the .70 to .80
baseline, the predictive value is likely reliable. At the same time, it is understood in the
context of the research question and impact on those likely to be affected by the study.
For example, if the study was to determine the use of a highly toxic and lethal therapy to
treat cancer, allowing for even 39 FP or recall in the low 90s would likely be
unacceptable.
In this instance, the results of the model showed that even by leveraging the top
three features, there were 225 elements, which were 14% of the total predictions being
made by the model. This was precisely the same with the top 10 features. However, in
this top three the model reported 225 elements, 186 were TP and 39 FP. In addition, the
algorithm predicted 1,430 negative elements, of which 1,417 were TN and 20 FN. The
accuracy score was .965, precision .827, recall .903, F1 score .863, and AUC .954.
The next inquiry was to determine if there was a discernable point in the users
likely. In other words, if users completed the fifth activity, they were more likely to
complete the course. To answer this query, activities were incrementally scaled down in
order of correlation value while retaining the 10 activities (see Table 6).
performed that would likely result in a completed course. The reason the number
increased as the scale reduces by five activities each time is because there were more
88
Table 6
P P
Scale TP, FP, TN, FN Accuracy Precision Recall F1 AUC
elements %
Top 225 14 186, 39, 1417, 20 .965 .827 .903 .863 .959
>5 235 14 190, 46, 1410, 16 .963 .805 .922 .860 .959
> 10 246 15 194, 53, 1403, 12 .960 .785 .937 .854 .960
> 15 263 16 196, 67, 1389, 10 .954 .745 .951 .836 .955
> 20 294 18 198, 96, 1360, 8 .937 .673 .961 .792 .950
> 25 313 19 199, 114, 1342, 7 .927 .636 .966 .767 .945
> 30 345 21 194, 151, 1305, 12 .902 .562 .942 .704 .933
This relationship of the Elements, TP, FP, and precision columns can be readily
seen in the 30-reduced scale Evaluation report. The precision score speaks of the TP
(correct) and FP (incorrect) predictions made of those who would likely complete the
course. In this Evaluation record, it shows 194 TP, or correct predictions, and 151 FP.
That is why the precision score of .562 should be perceived as little more than guessing
by a human.
It should be noted that 5,152 users watched the first video activity. This dropped
to 4,207 for the second activity, 3,113 dropped at the third, and so forth (see Figure 5).
This is similar to Nanfitos (2013) assertion that most enrolled in MOOCs do not
participate beyond watching a video or two and abandoning the course around the second
week. In this study, there is a severe drop in participation until the fourth activity. At that
juncture, there has been a loss of 2,912 participants or roughly 57% of those who started
A secondary drop occurs at the eighth activity, whereby the course loses 45.7% of
the remaining participants. By the ninth activity, the churn occurs at a much slower
decline. It should also be noted that although the Feature Selection module confirmed
89
what common sense would theorize, that those users who completed the latter activities
in the course would be the more likely to complete it, counterintuitively the number of
retests, n_retests (rs = .370) was a stronger correlation to completion than other quiz
6000
5000
4000
3000
2000
1000
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
As one distills the results of the scaling procedure and attempts to report it in a
manner that makes sense to the most people, the following concepts are evident. The
model confirms that the more activities the user completes increases the likelihood they
will complete the class. The closer the user performs activities to the completion tends to
carry greater strength of correlation to completing the course. The closer that activities
are to completion, the more predictive precision and accuracy is provided by the model.
Much of the model was retained to determine what variables may be significant
relative to the benchmark score of 80. One of the adjustments to the model, however, was
relative to retaining the SMOTE module, as the number of users who completed quizzes
90
was 3,382. The users who did not take a quiz were 1,802 users. It was not
determined that the two-class problem does not apply to this instance; thus, the SMOTE
Another anomaly in the design of the Luxvera courses is the ability to retake
quizzes immediately, one after another, as soon as they are completed. In effect, users
could take them repeatedly until they could guess the correct answers. The corresponding
time stamps made it difficult to determine if that was the case or not. Upon review of the
data, there seemed to be only a few instances where that may or may not have been the
case. Because this was inconclusive, they were allowed to remain in the dataset.
However, to better determine whether the actual videos and articles had a positive effect
on learning, it was determined to use only the first account of each students quiz
To determine again which of the variables may contribute toward the benchmark
score, the Feature Selection was run (see Table 7). For reporting purposes, it was
determined to separate out those activities that were watching videos from reading
articles. It was also determined to identify which videos showed the strongest correlation
to the passing first-time scores. Finally, any attributes relative to the quizzes were
considered important.
91
Table 7
To determine whether watching the videos had much predictive value toward
scoring an 80 on the first attempt or not, the model predicted 1,555 elements that would
contribute toward the benchmark, which was 100% of the total predictions. At the 0.5
threshold, the model predicted 1,424 TP, 131 FP, 0 TN, and 0 FN. Since this was not a
two-class problem, the accuracy score usually carries more weight. But as can be seen in
the other metrics, there is more that must be understood. The accuracy score was .916,
precision was .916, recall was 1.00, and F1 score was .956. All measures seem to verify
highly predictive validity. On the other hand, the ROC was only marginally separated
from the zero base, and AUC was .700 (see Figure 6). None of these metrics were
It may be concluded from the model that the dataset for the Luxvera course did
not conclusively classify video watching as a predictive element for reaching the 80
93
benchmark. Although there were some indicators toward that end, it would likely require
conclusively.
Relative to the predictive impact that reading articles may have, the model
ultimately produced the same outcome as that of video watching. An unusual outcome
was produced by the model in that it did not predict any negatives (i.e., that the elements
would not score an 80). At the 0.5 threshold, it predicted 1,424 TP and 131 FP. The
accuracy score was again .916, precision was .916, recall was 1.00, and F1 score was
.956. However, the ROC curve remained midway to the zero base, and the AUC was
.608. This was not changed, even with incremental adjustments to the threshold. As such,
the conclusion is the same for article reading as that of watching videos.
This once again shows how when understanding the results of a two-class
decision forest, the researcher needs to look beyond one metric. Although precision and
accuracy were high, and recall was perfect, ROC was very low, and AUC was below an
acceptable level. Although the accuracy metric is the first indicator of validation of the
scoring, the literature indicates that ROC and AUC are given greater weight.
Relative to the predictive impact that the time span may have on whether it may
improve the benchmark scores, the model ultimately produced the same outcome as that
of prior two categories of features. The model did not predict any negatives. At the 0.5
threshold, it predicted 1,424 TP and 131 FP. The accuracy, precision, recall, and F1 score
all remained the same as the others. The curvature of ROC was midway between the
optimum and zero, and AUC was .703. Neither was this changed, even with incremental
adjustments to the threshold. Since all metrics were virtually identical except for AUC,
94
there is likely little difference between the three in score validity. As such, the conclusion
is the same for watching videos and reading articles. Based upon the results of this
model, the ability to predict a successful score on a benchmark exam based upon the time
In analyzing the quiz performances that were largely feature engineered at the
start of the preprocessing stage, there are some similarities and some differences to the
other three categories of features. Like the others, the model did not predict any
negatives. At the 0.5 threshold, it predicted 1,424 TP and 131 FP. The accuracy score was
.916, precision was .916, recall was 1.00, and F1 score was .956. Where this begins to
differentiate from the others is that the curvature of ROC was substantially closer to the
optimum scores, and AUC was .800. Incremental adjustments to the threshold did not
change the scores. Because of the configuration of ROC and AUC meeting the .800
threshold, there is a probability relative to how students take the quizzes that has a
The final section covers the basic demographic and general information. Again,
the model did not predict any negatives. At the 0.5 threshold, it predicted 1,424 TP and
131 FP. The accuracy, precision, recall, and F1 score were as the others. Where this is a
difference from all the other categories of features was that the curvature of ROC almost
aligned with the zero baseline. AUC was .534, which is basically what score a person
would achieve by randomly guessing. Incremental adjustments to the threshold did not
change the scores. As a result of these scores, there is little to no probability that these
benchmark.
95
Conclusions
To answer the initial research question (What performance variables will predict a
successful completion of a MOOC course?), the results of the PA model showed several
probable variables that would qualify. The first simply verified what would be common
sense to most people. It demonstrated that the longer users continue to engage in assigned
activities, the stronger the correlation between the activity and finishing the course.
Coupled with that, however, there were two areas that showed clear changes in
trajectory as to whether students would be retained in the course or not. By the fourth
activity, 57% of those who started performing activities had dropped out. Then, by the
ninth activity, the MOOC course lost 45.7% of those who had remained. Based upon the
Moving from completion to quiz performance at the benchmark 80, there was
something in the quiz variables that suggested a probability the dynamics of quiz taking
what combination or behaviors resulted in a significant ROC curve and moderately high
AUC.
behave differently in a MOOC environment?) was limited. There was no way to track
behaviors, as that was not captured in the database. The time-span associations were
related to when they either started or completed an activity, but not both. There were no
click streams that revealed behaviors within the activity. For example, there were no data
regarding video rewinds, fast-forwards, pauses, and the like. The same is true for reading
the articles.
96
Regarding the other corollary question (Is there a stronger correlation between
certain behaviors and predicted learning outcomes in a MOOC course?), the answer was
mixed. Some of the activities (videos or articles) did show a stronger correlation than
others. However, because of the limited data captured, the reasons for those correlations
CHAPTER 5: DISCUSSION
As a result of this study, a number of items should be addressed to round out the
Luxvera MOOCs place in the constellation of MOOC studies. The first area that will be
considered is the future of MOOCs. The second area is future marketing and
development. The third area is exploring design features that need to be considered for
future improvements for the Luxvera MOOCs. The fourth area identifies those areas that
could be improved in the study itself. Finally, the last topic speaks to the various areas of
just like in PA one can extrapolate a reasonable macroeconomic trajectory based upon
past events. Notable trends have been evident by studying the history of the field. These
identifying these trends. Some have calculated the outside forces that are likely to impact
their development. For example, Koxvold (2014) noted cheaper and better hardware that
is more user friendly, expanding platforms, market demand for more animation,
visualization, market demand moving away from talking heads and lectures, interactivity,
personalization, and more powerful assessment tools. The aggregate effect of these
inexpensive and user friendly products are enabling the general public to produce high-
quality MOOCs at little to no cost. As a result, basic economic theory affirms that the
ubiquity of information will necessarily drive down the value of the content aspect of
these courses.
98
Some of the prognostications are couched in advisement for reframing the current
MOOC designs. One provocative article by Mintz (2014) enjoined MOOC developers to
integrate successful features from other online services if they are going to remain viable
in the marketplace. Mintz noted the typical lack of meaningful interaction on MOOC
discussion threads that should be reconstructed along the lines of the more successful
connections along the lines of dating sites or listservs. He advocated developing personal
profiles like LinkedIn, course offerings like Netflix, broad applications of analytics for
multiple aspects like at-risk identifiers, and credentialing. Depending how all of these
features are incorporated into future MOOC development, the author speculated will
Clearing the field of speculation and conjecture, it is reasonable to infer that there
from an array of credible and free online sources will devalue the content aspects of
MOOCs. What then comes to prominence is the perceived prestige of the source, design
features not found from other content providers, and how broadly the credential is
Monetization will likely become more mainstream in the United States and yet
complex at the same time. On the one hand, monetized design patterns have already
evolved. It is fairly common to offer courses for free and then charge a nominal fee for a
certificate. On the other hand, Udemy and their followers have already ventured from this
mold and have begun charging fees up front for specialized training courses.
The complexity lies in the market forces surrounding education and global
99
economies at large. Since MOOCs and online education have a universal reach, markets
are impacted by forces outside the native country. Recently, George Siemens spoke to the
(personal communication, January 19, 2016). The president had no interest in monetizing
their MOOC courses. The government was about to infuse $1.3 billion into EPFLs
the Harvards, MITs, and Stanfords because that is what they know, but the next
There are several Western universities whose higher education courses are already
fully funded by the government. For instance, Germany also does not charge tuition to its
whereas Canada, Australia, Britain, and the United States primarily see higher education
Consider the impact of the Chinese economic juggernaut of 1.4 billion people
who in the last 2 years have been training 13 million K-12 teachers to use education
technology through MOOCs (Y. Wang, 2015). For-profit companies in China are using
MOOCs to train tens of thousands of entrepreneurs. Currently, estimates are that Chinas
MOOC industry is about $20 billion (Y. Wang, 2015). The Chinese government sees
MOOCs as leveling the playing field for a greater portion of their population.
markets, the U.S. MOOC providers will likely need to adjust their market strategies
100
accordingly. However, finding ways to monetize the courses at this point is essential to
reasonable to assume that credentialing will not only continue with MOOC designs but
will become a prominent part of MOOC strategies. A team from the University of
Pennsylvania surveyed 35,000 people on why they use MOOCs (Alcorn, Christensen, &
Emanuel, 2014). Of the respondents, 44% sought for help to do their jobs better, 17%
sought skills that might help them land new jobs, and 13% wanted knowledge toward a
degree. The unmistakable conclusion from this survey is that credentials that are
recognized by companies and universities would address the primary demand of current
A third path that does not require much speculation is that MOOCs will likely
study, they have already morphed into various kinds of learning experiences. It is only
reasonable that those will continue to do so with the continued discoveries in technology.
As noted previously, to improve the Luxvera courses to where they are a viable
revenue stream for Regent is going to take some start-up resources and added costs to the
budget. What must be sorted out first, however, is whether the program offers value
are using MOOCs for extending reach and access, building and maintaining brand,
101
outcomes, innovation in teaching and learning, and research on teaching and learning
(Blackmon, 2016, p. 88). By the nature of these facts, this will have a deleterious effect
on the universitys bottom line, if they do not wisely find a niche for responding to the
MOOC phenomenon.
As Davis et al. (2014) noted, many colleges and universities are including MOOC
development in their strategic plans. What seems to need clarification is the role that
Luxvera has in the panoply of Regents offerings. Even a casual observer will note that
little has been done to make Luxvera an integral part of the universitys plans for the
future. As noted previously, Bonk et al. (2015) concluded, Higher education can blend
MOOCs into their educational ecosystem without major disruptions and expand its ability
to serve growing and diverse student needs for alternative modes of instruction (p. 35).
However, this cannot be done without embedding the program into the universitys
Before additions to the value of the program can seriously be considered, the first
consideration is a cost analysis for rebooting the Luxvera program. It is clear from the
lack of development of the program that there are inadequate incentives for faculty or
department participation. Since many smaller universities maintain tight budgets, these
added incentives for developing MOOC courses do not necessarily have to be financial,
although that is usually the cleanest way to get through the bureaucracy. Working with
requirements for publishing to be considered for advancement may be one such initiative.
102
Yet, mere faculty incentives are insufficient for developing online courses much
less MOOCs, as evidenced by Oblinger and Hawkins (2006). They argued, Developing
and delivering effective online courses requires pedagogy and technology expertise
possessed by few faculty (p. 14). If this is true of online courses, it is far more the case
The process would call for some added personnel with certain expertise in the
field. Pomerol et al. (2015) particularly called for involvement from existing teaching
student-test subjects, and project manager. For some teams, dual roles can be fulfilled by
a single person. But this speaks to the idea that MOOC design work is not simply a
teaching professor and cameraman. Assuming that adequate preparation has been done
and the instructional materials are ready for presentation, the process takes at least 12
greater exposure. If Kernohan (2013) is correct, that platforms are not available for 99%
(p. 1, Sec. 4, para. 1), there are a host of institutions that would jump at the chance to
share a MOOC platform for the right arrangement. As such, Luxvera could target other
institutions that would bring added value and make the Regent MOOC courses simply
At the same time, many other institutions would be hesitant to invest in a platform
that is wholly owned by another university. Some form of divestiture would likely have
to transpire as well. To accomplish this, there would be some cost to adjusting the
103
platform design, securing additional infrastructure costs, legal fees for organizational
with an existing platform that already has a large base of users to include these courses in
their offerings. This too has issues in that only Coursera of the major MOOC providers
has a limited offering of religious studies. So, this has a potential of segregating a
allowing professors to use the MOOC coursework in the context of their credit-bearing
degrees in 4 years, there may be a variation of this to where students will complete a
As per the Udacity and San Jose State University model, Luxvera could require
more skin in the game by paying $150 per credit compared to the standard per-credit
fees in the California state university system of $450 to $750 (Nanfito, 2014, p. 562).
This may comport with the press for the entire U.S. higher education sector to reduce
Some strategies require little cost to the institution. For example, the Luxvera
courses are not currently registered with Class Central. From January 21 to January 27,
the number of MOOCs listed on Class Central grew at a rate of greater than 15 courses
per day and a user growth rate greater than 2,000% (Cook, 2016). If not Class Central,
then Luxvera can look to several other metasearch companies that trade in MOOC
104
courses. Because there is no direct path from enrolling into a Luxvera course to enrolling
into Regent, there is no synchronous effects on social media. Although Regent has a
commercial sponsorships. This was done by inserting MOOCs into solving problems in
unexpected ways (Hyatt, 2012). There are evangelical businessmen who own Fortune
500 companies. With the right configuration, they may prefer to outsource their employee
training materials to MOOC structures. The cost savings to personnel alone would be
worth the venture. With the right on-ramp configuration, the Regent School of Business
& Leadership could add some credit of the less technical courses to make a faster-track
To take advantage of some of these ideas will require some major improvements
to the Luxvera courses themselves. Even some of the larger platforms have moved from
individual and standalones and lecture series to that of mastering programs like that of
Open Universitys Future Learn. Some practitioners have provided evidence that
There are many professional workplaces that require recognition of training but
not necessarily a college degree. One such example is that many studies have noted that
those who consume MOOCs have already earned a degree, many of whom are teachers.
Many states require a certain number of continuing education unit (CEU) hours to renew
the teacher certification or licensure. Yet, there are currently no MOOCs designed
institutions as well. Some of those 99% who have been excluded from the major
platforms, like Coursera, have designed fantastic MOOC courses but have struggled to
get the exposure needed to make them viable. There is no law that says that a MOOC
which receive comparable benefit. In fact, Cook (2016) said the third wave of MOOC
development will consist of hybrid MOOCs (hMOOCs) that are parlayed into some
credits or recognitions. The extent to which the hybrids will morph is yet to be fully
realized.
Studies have shown that if the university is without name recognition, MOOC
users tend to enroll in courses based upon subject matter. There does not seem to be a
focus for the selection of courses that were added to the current Luxvera course listing. It
seems to have been populated by willing participants, rather than recruited specialists
who bring expertise to a high-value market. Thus, a concerted effort needs to be made by
specific courses and configurations, to begin the process of creating those courses and
A unique aspect of the Luxvera platform is that of the GTS. This is a product
too seems to suffer from a lack of market relevance and focus. It also is disjointed from
both Regent and the MOOCs. By the way it is presented, it poses a dilution of what the
At the same time, as Luxvera positions itself to perhaps support Christian schools
and companies, it provides vistas of opportunities. Teachers are always looking for
quality single-subject material to present to their students. Human resource officers are
looking for specialized topics to require individual employees to undergo. Nurses look
for single-topic procedures to bolster their competence in the workplace. They are not
At the same time, Luxvera need not rely solely on its own limited resources.
There are growing numbers of orphaned and abandoned MOOC courses and
presentations. Many are moved over to OER areas like MERLOT. Others are hidden
away in college databases. This has lent itself to a treasure trove of quality online
materials that are lying dormant in various locations, often hidden in the midst of low-
quality or dated videos. These could be reclaimed and/or refurbished to add quality and
volume to the offerings provided in GTS. There would be a way to reconfigure the GTS
to where individuals, professionals, and companies could go straight to the topic needed,
receive an assessment to verify competence of the material, and receive certifications for
employers.
There needs to be some value added for the students themselves in order to
increase the marketability of the Luxvera MOOCs. To some degree, there needs to be
some form of validation of competencies. At least it should provide what Nanfito (2013)
component in a broader online learning environment that provides flexibility and choice
107
to students trying to navigate a higher education system in transition (Ch. 3, para. 4).
administrators. Eleven percent of the nations children are educated in Christian schools.
Although some denominations and parochial systems require state licensure, regional
accrediting bodies such as the Southern Association of Colleges and Schools do not. In
those instances, teachers are required to have a bachelors degree that includes at least 24
college credit hours in the subject they are teaching. Thus, many Christian schools have
qualified teachers doing great work with students, but they do not have state certification.
To address the certification needs of those already in the field, are immanently
qualified to teach the subject, yet cannot or will not go back to school to take many more
college hours to earn a certification on the typical salary offered by the schools, there is
Christian Schools International (ACSI). Even ACSI requires CEU credits every 5 years to
This situation is tailor made for Luxvera MOOC courses. Properly redesigned,
this program has the capacity to increase the number of students enrolled into Regents
School of Education, if parlaying the benefits of CEU credits that can then count toward
To build upon this prospect, Luxvera could take a page from Udacity and do some
Placement services at Regent, who would have additional access to the data of needs and
supplies.
Universities are grappling with securing their place in the accreditation landscape.
108
de Freitas et al. (2015) said, The threat of corporate institutions replacing accreditation
powers of higher education is arguably a greater danger (p. 457). They identified the
four most looming issues in the MOOC-o-sphere, of which credentialing, badges, and
develop their own credentialing capabilities, MOOCs are a way by which the universities
can level the playing field. Colleges and universities are using numerous scenarios that
add value to the MOOC by conferring credits. Some are recognizing credits that have
been attributed by reputable organizations like ACE who have added MOOC certificates
to life experience as credit worthy (Masterson, 2013). Others, like Antioch University,
Antioch then offers these MOOCs for credit as part of a bachelor degree program. Or,
perhaps follow the lead of The University of Maryland University College and the
University of Massachusetts to find ways to award credit for MOOC learning online. All
of these are in tandem with Colorado State University who became the first school to
offer brick-and-mortar credits for MOOC course completion in the United States
Cathy Sandeen (2013) noted that UK Universities attributes four evolving means
companies like ACE, the National Credit Review and Recommendation Service, and The
Council for Adult and Experiential Learning, where the former has been servicing
109
military personnel for converting prior learning into college credits for many years. The
first university worldwide to acknowledge MOOCs for college credit was The University
other universities," 2013). In January 2013, Georgia State University announced it would
consider granting credit for MOOCs just as it does transcript credits (Jaschik, 2013).
2013). Here, universities make agreements to license MOOC content for inclusion into
campus-based courses that are eligible for credit. Mostly, universities are importers of
these data, which are in turn used as content for flipped classrooms. Sandeen (2013)
noted both San Jose State University and Antioch University as early models for doing
this (p. 36). But a growing number of universities are developing consortia agreements
One of the standard issues that resurfaces in varied MOOC research is that of the
lack of motivation or intention that is left undisclosed. Some MOOC developers are
beginning to include a survey that identifies what users expect from participating in the
course. This would help to determine course satisfaction and better frame the predictive
model.
Along with this would be better metrics to acquire more than demographics.
Rather than having demographic data gathering as a precursor to the course, follow the
innovation of Udacity and San Jose State University who are building data gathering
of the courses. It would need to be carefully designed to not scare off those with test
anxiety or those who are using the course exclusively for personal enrichment. The
pretest should be correlated with a posttest toward the end of the course that identifies the
Since this study noted that retesting had a higher correlation with completing the
course than some of the other features, it should be kept in the design. However, because
students can do rapid retests over and over until they guess the right answers, even
without going through the material, this feature should be adjusted to ensure test
reliability.
In the current design, the discussion threads are superfluous to the courses and
basically unproductive. In concert with Jiang, Warschauer, et al. (2014), the discussion
threads need to be redesigned to create more peer-to-peer interaction and feedback. Jiang
et al. found, The number of peer assessments taken in week 1 is a strong predictor for
achieving Distinction [certificate] (p. 274). In their study, every unit increase in the
number of peer assessments increased the odds of earning the Distinction certification
over seven times. Particularly, a focus needs to be made on the first week of participation
in the course. As several other previously cited studies have shown, this is critical to
Overarching to many of these issues are the apparent lack of accepted standards
and practices for creating production value in the MOOCs. Looking at the Luxvera
courses as a whole, some of the videos are much longer than others, some have interview
formats interjected for no apparent reason, one uses animation poorly, and one has a
picture with audio for the entire video lecture. Many have visual aspects that will appear
111
quality control person who is conversant in MOOC design and is delegated the requisite
authority to implement the changes needed. This, of course, affects the cost of the
One of the limitations in conducting this study was the kind of data being
harvested by the database. Because the only features archived were those that could
separate the variables by videos, articles, and corresponding time stamps, the ability to
investigate the specific behaviors that lend themselves toward successful completion of
Conversely and in concert with the Raffaghelli et al. (2015) findings, the database
should be redesigned to harvest and archive the data that are designed to extract and
suitably represent significant data (p. 502). There needs to be careful thought into what
the MOOC courses are intended to accomplish and then ensure that the architecture of the
database is acquiring the necessary data to serve the higher interests of the university.
There was an unforeseen pattern that emerged in the second phase of the study
once the DV was changed to that of scoring a benchmark 80 on the quizzes the first time
taken. The model predicted zero negative (TN or FN) outcomes. There was nothing in the
error. As such, another improvement would be to do further research on the design to see
Research opportunities easily flow out of this study. One such opportunity would
be to use other analytic software to explore the same research problem with even larger
datasets to analyze the results. Since such a large percentage of the participants dropped
out at some point from completing the course, a churn-designed model that focuses on
dropouts of MOOCs would be valuable. A study could further explore why retests,
n_retests (rs = .370) had a stronger correlation to completion than other quiz factors
further study on performance differences between videos, articles, and quiz results as it
of researchers in the field of MOOCs. It is tailor made for this environment because of
the massive data being analyzed. There is a likelihood that critical learning behaviors that
have yet to be considered will be discovered, similar to what it has done in other
industries.
study that are likely to produce greater success for the students taking the Luxvera
courses. For example, conducting more research and development on the attrition
coursework, and developing pilot programs that will implement courses with redesigned
Conclusion
It is evident that the MOOC phenomenon will be a viable part of the higher
education ecosystem for the foreseeable future. MOOC course offerings continue to
rapidly change as new technologies and discoveries yield advancements in the field. This
completion. MOOC designers should use the findings of this study to pay special
attention to assessment features in the design work. Because a higher correlation was
References
Abdous, M., He, W., & Yen, C.-J. (2012). Using data mining for predicting relationships
between online question theme and final grade. Educational Technology &
Alcorn, B., Christensen, G., & Emanuel, E. J. (2014). Who takes MOOCs? For online
Amnueypornsakul, B., Bhat, S., & Chinprutthiwong, P. (2014). Predicting attrition along
the way: The UIUC model. Paper presented at the Proceedings of the 2014
Doha, Qatar.
The analytics big bang: Predictive analytics reaches critical mass as big data and new
infographic/438
b9de-c6d04fc8a4b6/9352152ec2ae2b07c3f44edea79d239e (UCB/EECS-2013-
109)
115
Barga, R., Fontama, V., & Tok, W. H. (2015). Predictive analytics with Microsoft Azure
machine learning, second edition (Second ed.). New York, NY: Springer.
mooc/
2015(167), 87-101.
Bogaty, E., & Nelson, J. C. (2013). Moodys: 2013 outlook for entire US Higher
16.
http://campustechnology.com/articles/2015/07/06/survey-moocs-supplement-
traditional-higher-ed.aspx
Bonk, C. J., Lee, M. M., Reeves, T. C., & Reynolds, T. H. (2015). MOOCs and open
Booker, E. (2013). Can big data analytics boos graduation rates? InformationWeek.
big-data-analytics-boost-graduation-rates/d/d-id/1108511?
116
Boyd, D., & Crawford, K. (2012). Critical questions for big data: Provocations for a
Breslow, L., Pritchard, D. E., DeBoer, J., Stump, G. S., Ho, A. D., & Seaton, D. T.
(2013). Studying learning in the worldwide classroom: Research into edX's first
Bryson, S., Kenwright, D., Cox, M., Ellsworth, D., & Haimes, R. (1999). Visually
exploring gigabyte data sets in real time. Communications of the Association for
doi:http://dx.doi.org/10.1145/310930.310977
doi:10.1080/02680513.2014.931805
Chang, R. I., Yu, H. H., & L., C. F. (2015). Survey of learning experiences and influence
Christensen, C. M., & Weise, M. R. (2014, May 9). MOOCs' disruption is only
http://www.bostonglobe.com/opinion/2014/05/09/moocs-disruption-only-
beginning/S2VlsXpK6rzRx4DMrS4ADM/story.html
117
Clow, D. (2013). MOOCs and the funnel of participation. Paper presented at the
Cook, M. (2016). State of the MOOC 2016: A year of massive landscape change for
Cooper, A. (2012). What is analytics? Definition and essential characteristics. Centre for
1(5), 1-10.
Coursera. (2013). Five courses receive college credit recommenations Coursera (Vol.
Daniel, J. (2012). Making sense of MOOCs: Musings in a maze of myth, paradox and
Davenport, T. H. (2014). Big data at work: Dispelling the myths, uncovering the
Davenport, T. H., Harris, J., & Morison, R. (2010). Analytics at work: Smarter decisions,
Davenport, T. H., & Kim, J. (2013). Keeping up with the quants: Your guide to
Davis, H. C., Dickens, K., Leon-Urrutia, M., Vera, S., del Mar, M., & White, S. (2014).
http://eprints.soton.ac.uk/363714/1/DavisEtAl2014MOOCsCSEDUFinal.pdf
118
de Freitas, S. I., Morgan, J., & Gibson, D. (2015). Will MOOCs transform learning and
Definition of massive open online courses (MOOCs). (2015). 1(1). Retrieved from
http://www.openuped.eu/images/docs/Definition_Massive_Open_Online_Courses
deWaard, I., Abajian, S., Gallagher, M. S., Hogue, R., Keskin, N., Koutropoulos, A., &
Ltd.
http://www.downes.ca/post/33034
Downes, S. (2013a). The semantic condition: Connectivism and open learning. Paper
http://www.downes.ca/presentation/323
Downes, S. (2013b). What makes a MOOC massive? Half an Hour: A place to write, half
Downes, S. (Producer). (2013c, April 9). What the 'x' in 'xMOOC' stands for. [Online
https://plus.google.com/+StephenDownes/posts/LEwaKxL2MaM
Downes, S., & Siemens, G. (2008). CCK08The distributed course. The MOOC guide.
Ebben, M., & Murphy, J. S. (2014). Unpacking MOOC scholarly discourse: a review of
Evans, B. J., & Baker, R. B. (2016). MOOCs and persistence: Definitions and predictors.
Evans, B. J., Baker, R. B., & Dee, T. S. (2016). Persistence patterns in massive open
Fenn, J., & Linden, A. (2005). Gartner's hype cycle special report for 2005. Retrieved
January, 7, 2010.
amazon.com.
Gorard, S., & Cook, T. (2007). Where does good evidence come from? International
Guyon, I., & Elisseeff, A. (2003). An introduction to variable and feature selection.
Haber, J. (Producer). (2013, April 29). xMOOC vs. cMOOC. Degree of Freedom: An
http://degreeoffreedom.org/xmooc-vs-cmooc/
Habley, W., Valiga, M., McClanahan, R., & Burkum, K. (2010). What works in student
Haggard, S. (2013). The maturing of the MOOC. Retrieved from London, England:
Han, J., Pei, J., & Kamber, M. (2011). Data mining: Concepts and techniques (Third ed.):
Elsevier.
Hartshorn, S. (2016). Machine learning with random forests and decision trees: A mostly
intuitive guide, but also some python: [Kindle]. Retrieved from amazon.com.
Hayes, S. (2015). MOOCs and Quality: A review of the recent literature. Retrieved from
Gloucester, UK:
http://eprints.aston.ac.uk/26604/1/MOOCs_and_quality_a_review_of_the_recent_
literature.pdf
Heidegger, M. (1971). Poetry, language, thought (1st ed.). New York, NY: Harper &
Row.
Holmgren, R. (2013). The real precipice: Essay on how technology and new ways of
teaching could upend colleges' traditional models. Inside Higher Ed. Retrieved
from https://www.insidehighered.com/views/2013/04/15/essay-how-technology-
and-new-ways-teaching-could-upend-colleges-traditional-models
Holsapple, C., Lee-Post, A., & Pakath, R. (2014). A unified foundation for business
Hone, K. S., & El Said, G. R. (2016). Exploring the factors affecting MOOC retention: A
Hyatt, M. S. (2012). Platform: Get noticed in a noisy world: Thomas Nelson Inc.
Jiang, S., Warschauer, M., Williams, A. E., ODowd, D., & Schenke, K. (2014).
Jiang, S., Williams, A., Schenke, K., Warschauer, M., & O'Dowd, D. (2014, July 4-7).
Jordan, K. (2014). Initial trends in enrolment and completion of massive open online
15(1), 133-160.
Keim, D., Andrienko, G., Fekete, J. D., Grg, C., Kohlhammer, J., & Melanon, G.
(2008). Visual analytics: Definition, process, and challenges. New York, NY:
Kernohan, D. (2013). Content that talks back: what does the MOOC explosion mean for
Khalil, H., & Ebner, M. (2014). MOOCs completion rates and possible methods to
Canada.
Kloft, M., Stiehler, F., Zheng, Z., & Pinkwart, N. (2014). Predicting MOOC dropout over
weeks using machine learning methods. Paper presented at the Proceedings of the
Kolowich, S. (2012). The online pecking order. Inside Higher Ed. Retrieved from
https://www.insidehighered.com/news/2012/08/02/conventional-online-
universities-consider-strategic-response-moocs
Kolowich, S. (2015). The MOOC hype fades, in 3 charts. Wired Campus. Retrieved from
http://chronicle. com/blogs/wiredcampus/the-mooc-fades-in-3-charts/55701
Kop, R., Fournier, H., & Mak, J. S. (2011). A pedagogy of abundance or a pedagogy to
support human beings? Participant support on massive open online courses. The
93.
Lamb, A., Smilack, J., Ho, A. D., & Reich, J. (2015). Addressing common analytic
Levin, T. (2013). California bill seeks campus credit for online study. The New York
http://www.nytimes.com/2013/03/13/education/california-bill-would-force-
colleges-to-hoor-online-classes.html?_r=1&
Lieberman, M. D., & Cunningham, W. A. (2009). Type I and Type II error concerns in
Lohr, S. (2013, February 1, 2013). The origins of 'Big Data': An etymological detective
Lyman, P., & Varian, H. R. (2000). How much information? Retrieved from Berkeley,
CA: http://www2.sims.berkeley.edu/research/projects/how-much-info/how-much-
info.pdf
Lyman, P., & Varian, H. R. (2004). How much information 2003? Retrieved from
2003/printable_report.pdf
Masterson, K. (2013). Giving MOOCs some credit American Council on Education (Vol.
Mayer-Schonberger, V., & Cukier, K. (2013). Big data: A revolution that will transform
how we live, work, and think. New York, NY: Houghton Mifflin Harcourt.
Mayer-Schonberger, V., & Cukier, K. (2014). Learning with big data: The future of
Meister, J. (2013). How MOOCs will revolutionize corporate learning and development.
http://www.forbes.com/sites/jeannemeister/2013/08/13/how-moocs-will-
revolutionize-corporate-learning-development/#4d1f9643e859
https://www.insidehighered.com/blogs/higher-ed-beta/future-moocs
Montgomery, J. W. (2012). A short and easie method with postmodernists. The Journal
Moses, L. B., & Chan, J. (2014). Using big data for legal and law enforcement decisions:
Testing the new tools. University of New South Wales Law Journal, 37(2), 643-
678.
Nanfito, M. (2013). Massive open online courses in colleges and universities. Seattle,
Nietzsche, F. W. (2004). Ecce homo: How one becomes what one is (T. Wayne, Trans.).
O'Reilly, U.-M., & Veeramachaneni, K. (2014). Technology for Mining the Big Data of
Oblinger, D. G., & Hawkins, B. L. (2006). The myth about online course development.
http://www.oxforddictionaries.com/us/definition/american_english/mooc
Pappano, L. (2012, November 2). The year of the MOOC, Massive Open Online Courses
Are Multiplying at a Rapid Pace. The New York Times. Retrieved from
http://edinaschools.org/cms/lib07/MN01909547/Centricity/Domain/272/The%20
Year%20of%20the%20MOOC%20NY%20Times.pdf
Picciano, A. G. (2012). The evolution of big data and learning analytics in American
Picciano, A. G. (2014). Big data and learning analytics in blended learning environments:
Pomerol, J. C., Epelboin, Y., & Thoury, C. (2015). MOOCs: Design, use and business
Press, G. (2013, May 9). A very short history of big data. Forbes.
Pursel, B. K., Zhang, L., Jablokow, K. W., Choi, G. W., & Velegol, D. (2016).
Rodriguez, C. O. (2012). MOOCs and the AI-Stanford like courses: Two successful and
distinct course formats for massive open online courses. European Journal of
Sandeen, C. (2013). Integrating MOOCS into traditional higher education: The emerging
Sharkey, M., & Sanders, R. (2014). A process for predicting MOOC attrition. Paper
Short, J. E., Bohn, R. E., & Baru, C. (2011). How much information? 2010 report on
Siegel, E. (2013). Predictive analytics: The power to predict who will click, buy, lie, or
10(1).
Siemens, G. (2013, December 23). 2013 in MOOCS - Which event best defined the quest
https://allmoocs.wordpress.com/2013/12/23/2013-in-moocs-which-event-best-
defined-the-quest-to-solve-education/
Simpson, O. (2006). Predicting student success in open and distance learning. Open
Sinha, T., Jermann, P., Li, N., & Dillenbourg, P. (2014, October 25-29). Your click
decides your fate: Inferring information processing and attrition behavior from
Sinha, T., Li, N., Jermann, P., & Dillenbourg, P. (2014). Capturing "attrition
Smith, V. C., Lange, A., & Huston, D. R. (2012). Predictive modeling to forecast student
Stein, K. (2013). Penn GSE study shows MOOCs have relatively few active users, with
shows-moocs-have-relatively-few-active-users-only-few-persisting-
badges-matter-to-employers-or-admissions-officers
Strickland, J. (2015). Data science and analytics for ordinary people: Lulu. com.
http://www.cs.helsinki.fi/en/news/68231
128
Sturgis, C., Rath, B., Weisstein, E., & Patrick, S. (2010). Clearing the path: Creating
content/uploads/2015/02/clearing-the-path.pdf
Sunithaa, L., Rajua, M. B., & Srinivas, B. S. (2013). A comparative study between noisy
data and outlier data in data mining. International Journal of Current Engineering
and Technologv.
Van Barneveld, A., Arnold, K. E., & Campbell, J. P. (2012). Analytics in higher
1-11.
Waller, M. A., & Fawcett, S. E. (2013). Data science, predictive analytics, and big data:
A revolution that will transform supply chain design and management. Journal of
Wang, W. K. S. (1981). The Dismantling of Higher Education, Part II: The Beginnings of
White, S., Leon, M., & White, S. (2015). MOOCs inside universities: An analysis of
Lisbon.
129
Williams, J. J., Paunesku, D., Haley, B., & Sohl-Dickstein, J. (2013). Measurably
TN.
Macmillan.
http://www.gutenberg.org/files/5740/5740-pdf.pdf
Analysis Predictive Prescription Analytics (Vol. 2016). Omaha, NE: CAN: Can
Work Smart.
Xu, B., & Yang, D. (2016). Motivation classification and grade prediction for MOOCs
Yang, D., Sinha, T., Adamson, D., & Ros, C. P. (2013). Turn on, tune in, drop out:
the Proceedings of the 2013 Neural Information Processing Systems (NIPS) Data-
Yang, Q., & Wu, X. (2006). 10 challenging problems in data mining research.
604.
Young, J. R. (2013). Beyond the MOOC hype: A guide to higher education's high-tech
Zemsky, R. (2014). With a MOOC MOOC here and a MOOC MOOC there, here a