Impact of Blended Learning of Student Engagement

A Case Study at the '1 Decembrie University' of Romania

1. Theoretical framework: blended learning and student engagement
Blended learning is an established concept that started more than 20 years ago. Details related to
this concept are still under discussion, but the general meaning is widely accepted to imply some
'combination of traditional learning with web-based online approaches' (Whitelock & Jelfs, 2003).
Simply put, blended learning is the practice of 'mixing online learning with face-to-face' learning
(Oliver & Trigwell, 2005).
Implementing blended-learning must involve some online tools that are used in an educational
context. These tools can be organized in two main categories:

general purpose tools

e-mail and social media platforms can be used for professor student or student
student communication;
blogs and websites may be used to deliver educational content;
online spreadsheet software can be used for classroom management (attendance,
recording observations etc).

specialized platforms such as Learning Management Systems (LMS), Virtual Learning

Environments (VLE), Massive Open Online Courses, Simulation/Virtual Reality
Environments and others, that provide a wide array of tools and methods for computerbased learning and assessment.

Student engagement is also a concept that attracted much debate in the last decades, starting with
the observation that students may or may not take part in educational activities and may or may not
fulfill their assignments (Natriello, 1984). Further exploration of the topic showed that physical
attendance and basic completion of assigned tasks may not be enough to show student engagement.
Thus, the concept has been extended to include cognitive aspects, behavioral aspects and affective
aspects of students' interaction with the educational process (Skinner & Belmont, 1993) that have
been described as follows (Chapman, 2003):

cognitive aspects relate to the mental effort that the students apply in their learning tasks;

behavioral aspects relate to the actual involvement of the students in the learning process
such as asking questions, completing assignments, participating in discussions

affective aspects relate to emotional responses of students towards learning.

Assessing student engagement is not a trivial task since it needs to take into account all the aspects

presented above. Current literature proposes some indicators for assessing student engagement, as
well as some methods that allow to get relevant data for such criteria.
The main indicators defined for student engagement are the level of academic challenge, active and
collaborative learning, student-faculty interaction, enriching education experiences and a
supportive learning environment. (Kenny et al, 1995). On the other side, indicators of lack of
engagement include absences from classes, cheating on tests or incomplete assignments (Chapman,
2003).
To get data on student engagement, researchers may use self-reporting measures such as
questionnaires to determine self-perception and student satisfaction with the learning environment.
Since these methods depend on the students' perception they may be completed through direct
observation to record actual student activity on pre-defined criteria, work sample analysis and
other qualitative methods such as in-depth interviews or focused case studies (Chapman, 2003)

2. Student engagement in Moodle-based classes at the '1 Decembrie 1918'

University
2.1. Context, aim and limitations
The current study looks at student engagement in a set of courses implemented using Moodle - a
free Learning Management System (LMS) / Virtual Learning Environment (VLE) (Moodle, 2014).
Moodle offers a wide range of functionalities for managing students (e.g. monitor student
interaction, register attendance etc) as well as functionalities for aiding instruction (e.g. distribution
of educational content, assignment of tasks, development of quizzes etc).
Most universities in Romania use online tools in their educational processes, yet so far there is no
overview on this issue. The main practice can be described as a non-systematic approach with most
instructors using various tools (Yahoo/ Facebook groups, online document systems etc) as they see
fit. as described in a previous study (Cucu, 2014).
The '1 Decembrie 1918' University does not officially and systematically offer Moodle-based
courses (excepting long distance learning programs), rather these are part of a pilot project
developed to test the system's advantages and limitations, especially its impact on professor
activities (the time and difficulty of managing courses) and on student engagement and results.
A number of 135 economics students have been registered on the Moodle system, each student
being registered to only one (mandatory) course. The aim is to analyze the way that students use the
online platform and draw conclusions on student engagement in this context.
One assumed limitation is focusing on the actual use of the system, on the behavioural aspects of
student engagement, thus we only address how students actually interact with the online platform. A
secondary study will be set up to address affective and cognitive aspects to some extent.

2.2 Methods and metrics

To answer questions about student engagement the self reporting method has been used - a
questionnaire has been filled in (on an optional basis) by the students after the end of the course.
The self-reporting data has been organized in six variables that followed the questionnaire items.
Since the course was mandatory, using the platform during the course (in the classroom) to access
educational resources and complete assignments was also mandatory, therefore less relevant to
student engagement.
Therefore, we firstly looked at optional activities - such as how often the students used the system
to check their attendance and their intermediate grades. These have been modeled using two ordinal
variables AttCheck and GrdCheck that have been mapped on two 3-point Likert-type items with
possible values 1 - 'None', 2 - 'A few times' and 3 - 'Many times'.
The extent to which students used the system for school related communication was the second
theme, with two variables modeled: one for communication between students - SComm and one for
student - professor communication PComm. These have also been mapped on the 3-point Likerttype items shown above.
Though some discussion may arise regarding the decision to use 3-point Likert items (instead of 5
points or more), there is sufficient research (eg. Jacoby & Matel, 1971; Dolnicar et al. 2011) to
suggest this should be 'good enough' in this case.
The frequency of using the system outside the classroom is one of the most relevant indicators of
engagement, since the desired outcome is to motivate students to keep connected to their learning,
i.e. to keep them involved in activities related to the course such as submitting assignments,
reviewing past activities, downloading course contents and even (to a lesser extent) checking their
attendance or their grades.
The frequency of using the system outside the classroom was modeled using an ordinal variable
(usage frequency - UF) mapped on a five point Likert item (1 - 'Very rarely / not at all'; 5 - 'Very
often, at least weekly ').
Data regarding the frequency with wich students used the system was completed with data
regarding the students' purpose for using it - a multiple choice question (with multiple possible
answers) that presented the standard functionalities: checking grades, checking attendance,
downloading course contents, submitting mandatory assignments, submitting optional assignments,
reviewing past activities.
The ways in which the students used the platform allows insight on their engagement on a deeper
level, distinguishing between using the platform as an information point, i.e. information about the
course (such as attendance or intermediate grades) and using it as a learning aid, for the activities
accessing contents, reviewing activities or submitting optional assignments. These data have been
used in two ways:
The first way has been to establish a usage mode by ranking each activity according to the
number of students who checked it.
The second way has been to define a usage intensity, by counting how many options had
been chosen by each student. To better point towards student engagement (not only
behavioral, but also cognitive), the usage intensity has been defined to take into account the
number of learning-aid activities (as defined above - accessing contents, reviewing
activities, submitting optional assignments) that have been performed by students. Thus we
have created a numerical variable with values ranging from 0 to 3 (0 meaning no such
activities had ever been performed, and 3 meaning all these activites having been performed
at least once).

From the usage frequency and the usage intensity variables (as defined above), a general indicator
for engagement has been created (PE - platform engagement), as the product of the two variables
(PE = UF * UI). This variable contains information both from the frequency of using the platform
(UF) and also from what the students used it for (UI) and is used as a categorical variable, to
provide an overview of how connected each student was to the system.
To support the use of the PE variable, we propose the following arguments that might be necessary
to a better understanding of the classes even if a true comparison on a scale is not needed since the
variable is not ordinal:

the data is not complex enough to know exactly how many times a student used the platform
(only that is was more or less often) or to know exactly from those uses how much has been
for learning purposes (we can only know that among the uses, there have been some for
learning, according to UI);
since we start with UI from zero, students who declared not having used the platform for
learning will get an engagement value of zero;
it can be considered that a student who had a good frequency of using the platform had a
better engagement than a fellow students with a lower frequency, if they have the same
intensity excepting the case when the intensity is 0 (eg. UF1 = 4, UI1 = 2 -> PE1 = 8;
UF2=3, UI2=2 -> PE2=6);
it can be considered that a student who had a high intensity had a better engagement than a
student with a lower intensity, at the same frequency (eg. UF1 = 3, UI1 = 3 -> PE1 = 9;
UF2=3, UI2=2 -> PE2=6);
it can be considered that a student who had a high frequency had, to some extent, a similar
engagement with a student with a lower frequency but a higher intensity (eg. UF1=3, UI1=1
-> PE1=3; UF2=1, UI2=3 -> PE2=3);
there are cases where the comparison may be unclear, eg. UF1=4, UI1=2 vs. UF2=3, UI2=3,
yet these differences are not fundamental since the range is high enough and in the end of
the analisys we split the PE data into two bins therefore cancelling potential flaws.

The self-reporting data has been completed with data regarding results for each student, taken from
official student reports:
the grade for the course being evaluated (CG - course grade) is a target variable; one goal of
the current study is to check to what extent the engagement is associated with the course
grade;
the course grade may also depend on the overall student ability (SA); this variable has been
obtained by looking at the student's average for the semester, calculated without the grade
for the evaluated course.

3. Data analysis and interpretation

The main objective of the study has been to analyze how often students use the Moodle system and
for what purpose, and if and how the engagement with the system can be associated with the final
course grade.
From the 135 registered students, 73 answered the questionnaire, but three students have since
either withdrawn from the university or had no course grade and no semester average and have
therefore been discarded, leaving 70 cases to analyze.

Communication through Moodle

When looking at the extent to which the students used the system to communicate, both with fellow
students as well as with the instructor, we observe a radically different picture than the one above:
Having a median of 1 ('None'), the results show students didn't use peer-collaboration through the
Moodle system, with >95% of students stating they have never sent a message to colleagues
through the Moodle system. Student professor communication has been used a few times, but a
vast majority of the students (>87%) still did not use this functionality.
Figure 2a. Student - professor communication -

Figure 2b. Student - student communication -

percentages

percentages

Though further inquiry is needed, possible reasons for the lack of using communication
functionalities may include:
most activities were individual, not collaborative in nature and communication has not been
a specific request;
students may prefer communication through known means such as social media;
the communication functionalities of the platform, though easy to find and use, have not
been explained in detail.
Since there are several studies that show communication and collaboration to be important tools in
increasing student engagement (e.g. Dixson, 2010), this aspect should be prioritized in blended
learning courses.
Usage modes: information point and learning aid
As previously described, students' purpose for using the Moodle system may fall in two categories:
for information regarding their current status and as learning aid. For relevant information we may
look at the choices that students selected in regard to their purpose of using the system outside the
classroom.
The results (table ... ) show that the most used feature was checking the grades (87%), followed by
content download (84%) and checking attendance reports (~83%). Though content download is
related to learning, it may be important to note that these percentages do not entail the frequency of
each use, but are limited to saying if the student used the system in that way at least once.
Looking back at the high number of students who declared they have checked their attendance and

The jittered
scatter plot
mapping PE to
the course grade
(CG) is very
loosely
distributed,
showing only a
slight positive
correlation.

Figure
plot (jittered)distribution
for platform
engagement - course grade
Figure 5.
4. Scattered
Platform engagement
- percentages

Figure 6. Platform engagement (PE) - four bins distribution

To obtain further
insight, we may
bin the data in
equal range bins.
Below we can see
the distributions
when using 4 bins
(0-3: 'Low
engagement'; 4-7:
'medium-low
engagement'; 8-11:
'medium-high
engagement'; 1215: 'high
engagement) and
using two bins (07: 'low
engagement'; 8-15:
'high engagement').

From the first

binning we may
notice that most
students (~36%)
had a medium-high
Figure 7. Platform engagement (PE) - two bins distribution
engagement with
the system, but only 15% had a high engagement. Naturally, when looking at only two bins, we see
that the students are only slightly towards high engagement (~51% high to ~49% low).

Looking at the relationship between PE and the course grade, we used the 2-bins PE to verify if
students who had a higher engagement performed better than low - engagement students and if the
difference is statistically relevant.

Figure 8b.
8a. Normal Q-Q plot for high
low

There were 34 low PE and 36 high PE students

considered. An independent-samples t-test was run
to determine if there were differences in course
results between low PE students and high PE
students. There were no outliers in the data, as
assessed by inspection of a boxplot. Grades for
each level of PE were normally distributed, as
assessed by inspection of Q-Q plots and there was
homogeneity of variances, as assessed by Levene's
test for equality of variances (p = .933). The high
PE group performed better (M = 6.78, SD = 1.78)
engagement students
than the low PE group (M = 5.79, SD = 1.72), a
statistically significant difference, M = 0.98, 95% CI [0.15, 1.81], t(68) = -2.35, p = 0.022, d = 0.57
(medium effect, according to Cohen (1988)).

Group Statistics
PE (Binned)
CG

Mean

Std. Deviation

Std. Error Mean

low

34

5.79

1.719

.295

high

36

6.78

1.775

.296

Table 3. Course grade (CG) statistics grouped by platform engagement (PE)

Table 4. Course grade (CG) t-test for low and high engagement groups
Lastly, we checked to see if general student ability (SA) is the same in the two groups (low PE /
high PE). Again we find that there are no outliers and that SA values were normally distributed.
Though the mean of the first group (M 7.61, SD = 0.98) was lower than the mean of the second
group (M 7.93, SD = 1.14), the difference is not statistically relevant (p=0.22), therefore we fail to
reject the NULL hypothesis.

Group Statistics
PE (Binned)
SA

Mean

Std. Deviation

Std. Error Mean

low

34

7.6135

.97719

.16759

high

36

7.9328

1.14294

.19049

Table 5. Student ability (SA) statistics grouped by platform engagement (PE)

Table 6. Course grade (CG) t-test for low and high engagement groups

4. Conclusions
The current study gives some insight into students' usage patterns of a Moodle System in a blendedlearning context. It may be important to mention students were not familiar with Moodle or any
other LMS.
Results show that students tend to use such systems firstly as an information point, especially to
check their grades and attendance (considering these are available options). This may be considered
a good starting-point for increasing student engagement since the students form a behavior to access
the system, but more work should be done to motivate students to do more, especially to collaborate
and communicate with instructors, review the activities, or to solve optional assignments.
Solutions may include creating specific milestones or activities that involve some review of
contents / assignments and giving more focus to assignment feedback, including feedback for
optional assignments.
Students didn't use the system for communication, either with fellow students, or with the instructor.
This may be explained by the type of activities (individual), the fact that communication was not a
requirement and the functionality itself was not thoroughly explained, but we also should keep in
mind that students already have various communication tools, with social media being at the top of
the list.
Solutions may include putting more focus of the communication functionality of the system, but
integration between social media and Moodle may also be an interesting future direction.
Regarding statistical results, we analyzed two groups of students - one with low engagement and
one with high engagement, both groups having students of similar general ability. We found that
high engagement performed almost one grade point better (M = 6.78, SD = 1.78) than the low
engagement students (M = 5.79, SD = 1.72), a difference we found to be statistically significant M
= 0.98, 95% CI [0.15, 1.81], t(68) = -2.35, p = 0.022. The effect is medium (d=0.57).
Overall, using an LMS such as Moodle provides some good ways to keep students active and

connected to their learning process and has a beneficial impact on students' results.
To get even better results, more effort has to be put in the implementation of the system and into the
management of the class so that students feel motivated to use it more frequently and in a more
intensive manner.
References:
Chapman, Elaine (2003). Alternative approaches to assessing student engagement rates. Practical
Assessment, Research & Evaluation, 8(13). Retrieved Sept. 26, 2015 from
http://PAREonline.net/getvn.asp?v=8&n=13.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ:
Lawrence Erlbaum Associates.
Cucu, C. (2014). Blended learning using Moodle at the '1 Decembrie University' in Romania.
Annals of the University of Petrosani Economics, 14(1).
Dixson, M.D. (2010). Creating effective student engagement in online courses: What do students
find engaging? Journal of the Scholarship of Teaching and Learning, Vol. 10, No. 2, June 2010, pp.
1 13.
Dolnicar, S.; Grn, B.; Leisch, F. and Rossiter, J. (2011). Three good reasons NOT to use five and
seven point Likert items. In Proceedings of the 21st CAUTHE National Conference, Adelaide,
Australia.
Jacobym J. & Matel, M.S. (1971). Three-Point Likert Scales Are Good Enough. Journal of
Marketing Research Vol. VIII .

Kenny, G. Kenny, D. and Dumont, R. (1995) Mission and Place: Strengthening Learning and
Community Through Campus Design. Oryx/Greenwood. p. 37

Moodle, (2014) https://docs.moodle.org/28/en/About_Moodle. Last accessed Sept 26th 2015.

Natriello, G. (1984). Problems in the evaluation of students and student disengagement from
secondary schools. Journal of Research and Development in Education, 17, 14-24.
Oliver, M., & Trigwell, K. (2005). Can blended learning be redeemed? E-learning, 2(1), 17-26.
Skinner & Belmont (1993). Motivation in the classroom: Reciprocal effects of teacher behaviour
and student engagement across the school year. Journal of Educational Psychology, 85(4), 571581.
Whitelock, D. & Jelfs, A. (2003) Editorial: Journal of Educational Media Special Issue on Blended
Learning, Journal of Educational Media, 28(2-3), pp. 99-100.