Measuring impact of e-Learning on Quality and learning outcomes – a

proposal for a pHD project

Authors: Rita Falcão, Alfredo Soeiro

Affiliation: Universidade do Porto

Abstract
In Higher and Continuing Education, the use of eLearning methodologies is becoming a
common resource. At this point, we have probably passed a first stage of eLearning
implementation - experimentation. After a first stage where pilot projects were implemented
and strategies were defined, we come to a phase where eLearning implementation is more
mature. It should now be possible to find consistent examples of the use of eLearning in
Engineering Education, with several years of application. It should be possible, at this time, to
evaluate the impact of eLearning methodologies on the Learning Outcomes. The work here
proposed intends to address the problem of measuring the impact of eLearning on
Education, through the analysis of Learning Outcomes.
The scope of the study includes case-studies in Higher Education, pre and graduate courses.
Also, cases of the use of eLearning in Continuing Education will be addressed. Authors
intend to collect case-studies of courses that have been using eLearning technologies, in
different contexts and Institutions. Data collected should include information about the
implementation and Learning Outcomes of the course, before and after eLearning. Using a
considerable amount of data, this study aims to identify a potential trend of alteration of
results of Learning Outcomes. If possible, this study will try to find a relation between the
increment of Learning Outcomes and the implementation of eLearning.

Keywords: evaluation, Learning Outcomes, quality

Introduction
E-learning is a relatively new field of research. Even though the use of technologies for
learning has been implemented for several decades, the introduction of the Internet has
promoted a considerable shift and innovation. Also, e-Learning is by its nature a
multidisciplinary subject and, consequently, very complex. For these reasons, even though e-
learning feeds from previous research in Technology and Education (among others), e-
learning, as a research subject by itself, is still in a very early but exciting stage of
development.

In the past years, e-learning strategies have been implemented in Institutions of Higher
Education and others. Although these implementations have different objectives and follow
different strategies, the importance of the use of e-learning technologies to improve the
Quality of Education has been widely accepted. The e-learning Programme of the European
Commission, promotes the “effective integration of Information and Communication
Technologies (ICT) in education and training systems in Europe (2004 – 2006)”, and
consolidates this perspective. As stated by the European Commission, ICT, properly used,
should contribute to the quality of education and training and to Europe’s move to a
knowledge-based society.

1
When analyzing the proceedings of most e-learning or e-learning related conferences, one
finds that another trend is becoming frequent related to Quality and Quality Assurance of e-
learning. The implementation of e-learning can be very demanding for an institution, since it
introduces changes in many different processes. For this reason, concerns about the
effectiveness of e-learning focus mainly on organizational and procedural factors, justifying
the increased interest in Quality Assurance. However, quality issues should also address the
impact of e-learning on learning results. When assessing the quality of e-learning or any
education related process, the learning outcomes should be considered as an important
indicator. Even though the implementation of e-learning can be very diverse, technically and
pedagogically, an effort should be done to estimate if these technologies are affecting the
results in a positive or negative way.

This proposal seeks to address these issues of quality in e-learning, and in particular the
impact on the Learning Outcomes. What is the real added value of e-learning to the
Education Process, and is this value dependent of the implementation strategies.

Objectives of the study

This study has two general aims related to the quality in e-Learning, in a Micro and Macro
perspective:

• To measure the impact of the application of eLearning in Education, through the analysis of
the Learning Outcomes.
• To identify success and unsuccess factors for the use of eLearning in Education of
Engineering, based on the selected case-studies.

More specifically, the objectives are:

• To collect and analyze European and Worldwide initiatives and methodologies on the
quality of e-learning, that address the issue of Learning Outcomes.
• To define and test a methodology to evaluate the general quality of e-learning trough the
analysis of Learning Outcomes
• To collect and compile case-studies of e-learning at different institutions, including
information about strategies and Learning Outcomes
• To identify factors of success or failure in e-learning, based on the analysis of provided
case studies
• To develop indicators of quality in e-learning supported by case studies derived from best
practice and taking account of definitions and practices in different countries

Detailed description
This research project intends to evaluate the real added value of e-learning to the
Educational Process, from the perspective of the impact on the Learning Outcomes. To
achieve this goal, a methodology of evaluation will be developed, based in case-study
analysis, pre and post implementation and bibliographic analysis. The first stage of the study
will consist on collecting information on existing methodologies for evaluating e-Learning, in
particular from accreditation agencies and institutions. Based on the information collected, a
methodology will be developed that will address the particular needs of this study.

On this stage, a structure for the case studies will be defined, including the type and amount
of data to collect. Also, a matrix of existing quality indicators will be compiled and used as a
complement to the evaluation of quality. Case studies of courses should include information
about the Learning Outcomes before and after the e-Learning implementation, when
possible. Evaluation will be based on results (outcomes) of similar courses and user
assessment.

2
The second stage of this project consists on selecting case studies based on previously
defined criteria and structure, and statistically analyze them. The case studies will focus on
courses of cooperating Institutions that are involved in e-learning. This is a critical stage of
the study and the longest. The statistic of analysis intended for this stage of the project is
based on large numbers. However, the selection criteria for case studies impose some
restrictions, increasing the difficulty of finding eligible courses and slowing down the pace of
the project. The case study analysis, focusing mainly on the statistic overview of Learning
Outcomes will try to measure the evolution after the e-Learning implementation.

The following stage consists on a deeper analysis. The previous results will be confronted
with different context related factors. This stage aims to identify trends for failure or success
factors in e-learning implementations. It is expected that the conclusions of the first stages of
the project, will provide indicators of quality that support e-learning implementation. Also,
these conclusions will feed a Guide for Good Practice on e-learning implementation, from the
Learning Outcomes perspective.

For each stage of the research project, the following outputs are expected:

Stage I - Preparation
• Report on State of the Art of e-learning Quality and Evaluation: Compilation of existing
initiatives of quality evaluation in e-Learning, in particular the ones addressing Learning
Outcomes. The goal of this activity is to identify trends and tools used in e-learning
evaluation. Also, it will help to identify Institutions active in e-learning that can be asked to
collaborate. This report will support the future decisions of the study.
• Evaluation methodology: Based on the previous outcome, a methodology will be defined to
evaluate how e-learning is affecting the Learning Outcomes of students and trainees.
• Case-study Structure: The structure of the case-study that will be used for this project is
very important and should reflect the specific needs of this study. However, it will benefit from
the research done for the State of the Art report, since case-study analysis is the most
common evaluation method for Quality in e-learning.
• Matrix of quality indicators: The third stage of the project consists on an analysis of the e-
learning implementation based on context. Among other factors, a possible relation between
existing quality indicators for e-learning and the impact of e-learning on the Outcomes should
be addressed.

Stage II – Statistic Analysis

• Database of Institutions with e-Learning initiatives: This is a critical stage of the study, since
the value of the statistical analysis will strongly depend on the amount and quality of the
information gathered. Also, the diversity of institutions and contexts in which e-learning is
being used will enrich the final conclusions and widen the applicability of results.
• Case-studies and statistical analysis of data collected: This analysis will look at the case-
studies as a black-box, not taking in consideration the e-learning processes or methods used
on the on-line courses. The main goal is to measure the alteration of Learning Outcomes on
courses that moved from traditional teaching to e-learning.

Stage III – Context analysis

• Context analysis of data collected: From the previous stage, a selection of case studies will
be analyzed more deeply, based on the results achieved. For this stage, is important to
analyze case studies that represent diverse contexts of Education, a variety of e-learning
implementations and also that presented different behaviors in relation with the Learning
Outcomes.
• Success and failure factors: Based on the statistical and context analysis, an effort will be
made to identify key factors in e-Learning that may affect the Learning Outcomes, in a
positive or negative way.

3
• Good Practice Guide: this output will gather in a structured document, the findings of the
previous activities. It is expected that this guide will help the stakeholders in future decisions
concerning e-learning implementations

Finally, the results and conclusions of the study should be disseminated at different levels, to
promote good practice.

This study intends to establish an innovative approach to e-Learning evaluation and quality.
Focusing on the impact of e-Learning on the Learning Outcomes, at a larger scale, it intends
to identify trends and success/failure factors in existing e-learning implementations. The
major innovation consists on the statistical analysis of a large number of e-learning courses,
ignoring at this stage, the methodologies and strategies used during the course.

Results and conclusions

At this point, results of several case studies from Universidade do Porto were collected. A
preliminary analysis of results predicts a positive tendency of change after the use of e-
Learning. In almost every case analysed, there is a tendency for increasing values of the
general indicators of the course. The rates Passed/Registered, Passed/Evaluated and
Evaluated/Registered increased after the introduction of e-Learning. Also, analysing the
charts of grades distribution, we find that there is a slight positive deviation of the lines.
Below, tables and charts of three case studies are presented.

Even though these results seem to indicate a positive change after e-Learning introduction, it
is to soon to assume this conclusion. This is only one superficial analysis of course results,
from a single institution. Also, teaching and learning are very complex processes, making
very dangerous to relate this positive change with a single factor. A deeper and broader
study is needed, including a context analysis.

Table 1 - Results obtained for a course in statistics, 1st Semester

Statistics (Y1/ S1) Before After
Registered 258 268
Evaluated 125 163
Passed 95 128
Pass/Register 37.0% 49.2%
Pass/Evaluate 75.1% 78.5%
Evaluate/Register 48.5% 62.0%

4
Figure 1 - General indicators for the course Statistics. P/R - passed over registered; P/E - passed over
evaluated; E/R - evaluated over registered

Figure 2 - Average grade distribution for the course Statistics before and after e-Learning

Table 2 - Results obtained for a course in Economics, 7th Semester

Economics (Y4/1S) Before After
Registered 297 371
Evaluated 200 252
Passed 180 229
Pass/Register 60.6% 63%
Pass/Evaluate 90.0% 91%
Evaluate/Register 67.3% 69%

5
Figure 3 - General indicators for the course Economics. P/R - passed over registered; P/E - passed over
evaluated; E/R - evaluated over registered

Figure 4 - Average grade distribution for the course Economics before and after e-Learning

Table 3 - Results obtained for a course in English, 1st Semester

English (Y1/S1) Before After
Registered 187 76
Evaluated 125 50
Passed 85 32
Pass/Register 44% 42%
Pass/Evaluate 67% 63%
Evaluate/Register 66% 67%

6
 Figure 5 - General indicators for the course English. P/R - passed over registered; P/E - passed over
evaluated; E/R - evaluated over registered

Figure 6 - Average grade distribution for the course English before and after e-Learning

References
• CONOLE, Gráinne, OLIVER, Martin. Contemporary perspectives in E-learning
Research: themes, methods and impact on practice. Oxon,UK: Routledge, 2007.
• REIS LIMA, Jorge, CAPITÃO, Zélia. E-learning e e-conteúdos: Aplicação das teorias
e modernas de ensino e aprendizagem à organização e estruturação de e-cursos.
Portugal: Centro Atlântico, 2003, Colecção: Sociedade de Informação
• KOPER, R., 2002, Improving Learning Through Technologies: Opportunities for All,
www.prometeus.org
• Machado, J., E-learning em Portugal. 1 ed. Sociedade Digital. 2001, Lisboa: FCA.
• KOMAROMi, László, MEDINA, José Collado, VAN DEN BRANDEN, Jef, SOEIRO,
Alfredo, PERRY, Brian, Quality guide of open and distance learning – Procedures
and practices, SOCRATES ODL PROJECT (56591-CP-2-1999-1-HU-ODL-ODL),
http://www.szamalk.hu/projects/P010/English.htm
• CARVALHO, C. V. (2003) Defining an evaluation methodology for blended learning in
Higher Education, Evaluating e-learning – Galecia Project