STUDENT ENGAGEMENT IN EDUCATIONAL APPS

A white paper by Maya Lopuch, Data Scientist at eSpark Learning

Copyright 2013, eSpark Learning

s iPads enter the classroom, they bring with them an unprecedented number of educational resources. Developers have created thousands of highly engaging apps that target specific academic skills. With so many educational apps on the App Store, the Apple website explains, theres no limit to your learning. In practice, one limit may be the capability of educators to select apps for their classrooms. A search for fractions yields 506 different iPad apps alone. Over 100 of those apps have a four star rating or higher. While this level of diversity can be a boon for students, the task for educators is daunting. It is easy to be paralyzed by choice. Educators have some resources to guide them in these decisions. On the App Store, educators can view screenshots of apps and read through reviews before purchasing an app. However,

Figure 2: These two apps, iTooch 5th Grade Math and 5 Dice: Order of Operations Game, teach similar skills but differ dramatically in their approach. iTooch encourages students to practice rote skills and 5 Dice focuses on higher-order thinking.

teachers and parents almost always write these reviews. The views of students, the target audience, are not directly represented. If an app does have high reviews, its not immediately clear whether that is because it works well for parents or if its a good fit for the classroom. Apps with in-app purchases, or apps that require a lot of manual setup and adult directions, may be good for one-on-one but may not work well in a classroom environment. Educators intuition remains the most valuable resource in evaluating the quality of content. Many agree that interactive apps create engaging learning experiences. Apps that have a worksheet-type interface and focus on repetition tend to be less engaging for students. Applying those kinds of heuristics can help reduce a list of over 100 four-star fraction apps to a more manageable list of candidates. New data is enabling educators to take the app selection process several steps further. This paper describes how eSpark Learning is using a unique database of student sentiment to identify highly engaging iPad apps. This data builds upon the subjective expertise of teachers in two important ways. First, eSpark draws from millions of ratings from thousands of students to determine which apps students find more engaging than others. Second, eSpark can identify the specific app characteristics that are predictive of student engagement.

Figure 1: A search for fractions in the App Store yields 506 different iPad apps.

These analytical insights are changing how educators incorporate technology into the classroom. Personalization engines like those

offered by Netflix and Amazon have changed how consumers find products in entertainment and retail. To date, education has largely eschewed personalization. The most prevalent model in K-12 continues to be one instructor teaching many students the same lesson in the same way. Mobile devices now allow for individualized delivery of content, and data on student engagement can help personalize that content to student preferences. Educators are now better equipped to augment traditional classroom instruction with a personalized learning experience.

Figure 3: The eSpark rubric has six categories of app quality.

Building a rich database of educational apps

eSpark Learning is a personalized learning platform for the iPad. Pedagogical experts create personalized learning curricula using curated instructional videos, third-party apps, and assessment tools. Each curriculum is closely aligned to a Common Core domain and grade level. Thousands of students across the country use eSpark to augment typical classroom instruction in math and reading in grades K-8. Students work through a curriculum that is tailored to their academic needs and learning goals. For example, one student may work on second grade level Number and Operations in Base Ten while her classmate may work on fifth grade level Geometry.

To curate this digital content, eSparks curriculum design experts scour the App Store to find the best educational resources that are aligned with the Common Core State Standards. Each potential app is evaluated on a rubric that considers Common Core standard alignment, authenticity of task, scaffolding of learning, intuitiveness, student engagement, and cost of the app. Educational apps that receive high scores on this rubric are incorporated into the eSpark curricula. This process is a rigorous and large-scale solution to the challenges that teachers face when searching for apps for their own classrooms. Once apps are selected to be in the eSpark curriculum, curriculum designers catalogue each app on whether it fits dozens of additional categories. These categories cover a wide variety of app characteristics pertaining to content, pedagogical framework, and interface. For example, each app is examined for social studies applications, text highlighting capabilities, and gyromotion. This extensive attribute tagging process provides a highly granular view into the composition of educational apps. This attribute database is continually expanding as new content enters the eSpark curriculum.

This paper highlights a subset of the 2012-2013 eSpark curriculum. In total, 395 apps covering content from grade levels PK-5 are reviewed in this study. Table 1 shows counts of the most common app attributes in this sample.

Linking apps to student engagement

eSpark Learning links app characteristics to student engagement with ordinary least squares multiple regression models. App attributes are the independent variables in the prediction
Attribute Sound Effects Graphics, colors, and supplemental visuals Multiple-activity app Definition

equation. The dependent variable, student engagement, comes from a database of student sentiment. As students work through their personalized learning plan in eSpark, they are asked to rate each of the curated apps they explore. Students rate each app with a thumbs up or thumbs down. Although this binary indicator of sentiment is an imperfect proxy of student engagement, the large sample sizes of ratings help to capture more accurate information. The 395 apps analyzed in this study have been rated by 1,203 students.1
Apps with Attribute in Sample

The app has interesting, noticeable, and motivating sound effects. Graphics and supplemental visuals are colorful and visually stimulating. The app has multiple activities (i.e. adding fractions, comparing fractions) and requires directions from eSpark. Users are allowed to try again but receive no feedback explaining why their answer was wrong. The user interacts with the app by tapping or swiping in response to an academic question. The app has one activity. Students are able to go right into the app without having to locate the activity. Some or all of the app is narrated by a woman. Cartoon characters (people, animals, monsters) are part of the app; app has an animal graphic. The app has music in the background. At the end of the module, chapter, or game students see an overview of their progress, e.g. how many questions they answered correctly.

241 201 183 182 179 140 122 116 109 94

Moderate scaffolding

Interactive

One-activity app

Female voice

Cartoon/animal character

Background music

Score/progress overview

Table 1: Ten of the most popular attributes in eSpark PK-5 curated apps. The prediction models described in this paper included a more extensive set of regressors.
The apps and students represented in this sample are a subset of all of the apps in the eSpark curriculum and total students using eSpark. This subset was selected to be analyzed first because of the grade level of content. eSpark will extend the sample as the attribute catalogue grows to encompass apps from grade levels 6-8.
1

share of positive ratings to total ratings to determine an engagement metric for each app in the curriculum. This average rating is the dependent variable in the regression models, and app features are the primary regressors. These models also control for the grade level and subject area of the content.
Figure 4: Students are asked to rate each app in their personalized curriculum with a thumbs up or a thumbs down button. The average app in this sample received 82 thumbs up for every 100 student ratings.

Findings
The results from the prediction equations are shown in Table 2. Among the curated set of 395 apps covering content for grade levels PK-5, we show the app characteristics that are most strongly predictive of student engagement. The effect on student engagement
p-value

Among apps analyzed in this sample, the mean student engagement rating is 82%. This high average rating reflects the fact that the apps in this sample have been highly
Attribute Animation Definition

curated with expected student engagement as one criterion. A random sample of educational apps would likely have a lower average engagement rating. eSpark then calculates the

Effect on Student Engagement

App includes animation of graphically created characters. App has music in the background. Students are rewarded for correct answers with badges or rewards. Interface is black/white/green, lowbudget, and does not boast stimulating graphics. Menu of activities has many different functions and may be difficult to navigate. Students practive the skill in a rote and repetitive way. Graphics and supplemental visuals are colorful and visually stimulating. No sound effects, music, or audio are included.

+2.2% +1.9% +1.7% 2.3% 2.5% 2.5% +2.1% 4.6%

0.029 0.021 0.087 0.039 0.055 0.032 0.003 0.001

Background music

Badges or rewards

Bare-bones interface Complex menu of activities Drill and kill Graphics, colors, and supplemental visuals Silent

Table 2: The app characteristics that are most strongly predictive of student engagement among the curated set of 396 apps covering content for grade levels PK-5.

can be interpreted as the contribution of a particular attribute, holding all else equal, to the average rating of apps in this sample. For example, among two otherwise similar apps, an app with background music is predicted to have a mean student rating that is two percentage points higher than an app without music. These results provide empirical support for three main findings. First, students are very sensitive to audio and visual components of apps. Most of the strongest predictors of student engagement relate to the app interface. Compared to otherwise similar apps, apps that have distinguishing animation, music, or graphics are predicted to have higher student engagement ratings. Apps that are silent or have a bare-bones interface are predicted to have significantly lower ratings than average apps. This data suggests that student engagement in iPad apps is in large part driven by whether apps can differentiate the experience from a more typical educational exercise. Students gravitate toward rich media and this is consistently reflected in their engagement ratings. Second, the data on student engagement confirms some common hypotheses on pedagogical strategies. Students respond positively to badges and rewards. Apps that reward students for correct answers are associated with an engagement rating that is two percentage points higher. Apps that follow a Drill and kill

Figure 5: The app Fractions. Smart Pirate engages students with animated cartoons, music, and stimulating graphics.

strategy are associated with ratings that are 2.5 percentage points lower than average apps. Although the teaching medium changes when content is delivered through an iPad app, student responses to pedagogical strategies remain largely consistent. Student motivation remains a central component of the educational experience.

Third, apps that offer a large degree of choice reduce engagement. Apps that offer a complex menu of activities are associated with significantly lower student engagement ratings. This finding may come as a surprise to many educators. Several of the most popular apps in the App Store advertise extensive functionalities. Educators often gravitate to these omnibus

Figure 6: The app Counting Caterpillar, which teaches fundamental counting skills, rewards performance with badges and rewards.

Figure 7: The apps Coin Math and iTooch offer many applications but require students to choose from a complex menu before engaging with activities.

apps because they offer so much content in one package. That these comprehensive apps are associated with reduced engagement suggests that students may feel overwhelmed by the upfront decisions required of these apps. Clarity of use has a strong impact on student engagement.

Whats Next
By linking qualitative characteristics of apps to student engagement, eSpark has identified empirical insights that were otherwise unattainable at scale. This new knowledge has led eSpark to refine its app curation process. A deeper understanding of what kinds of apps resonate with students has enabled eSpark curriculum experts to make more sophisticated design decisions. The newest eSpark curriculum reflects many of the insights described here. eSpark continues to prioritize educational rigor as the foundation of its curriculum, but now curriculum experts are able to select content with a much stronger understanding of how to identify the most engaging apps. eSpark hopes this data will reach further and inform decisions among both educators and app developers. When educators evaluate apps, they would be wise to prioritize content that has a rich interface, adheres to proven pedagogical strategies, and prioritizes intuitiveness of use. App developers can apply these lessons to create more content that meets these best practices.

The attributes discussed here merely scratch the surface of the universe of app characteristics. eSpark is continuing to develop its app database by expanding attribute tagging in both breadth and depth. Newly developed apps are continually added to this catalogue, and apps in the current eSpark curriculum are being tagged for even more extensive attributes. Looking forward, eSpark will use this attribute database to develop a richer understanding of how iPad apps impact student outcomes. This study has explored the effects of app characteristics on student engagement. In the future, eSpark will be able to link specific app characteristics to impacts on student achievement. This data will allow educators to refine app curation even further, enabling them to select the content that has been proven to be most effective for improving student learning. Looking forward even further, eSpark is excited about a future where educators will be able to match apps to students, at scale, based on students interests, preferences, and motivations.

Maya Lopuch
Data Scientist at eSpark Learning
Maya Lopuch leads research and analytics at eSpark Learning. Prior to joining eSpark, she was a researcher at the Harvard Graduate School of Education, where her work investigated how public schools impact inequality and long term student outcomes. Maya holds degrees in Economics and Public Policy from Stanford University and the University of Chicago. She believes in using data to inform and improve education.