EUROPEAN UNION

Project financed under Phare

MoERI/ NCDTVET-PIU

Activity 04

Training and advice for the staff of the 16 regional support centres for e-learning MONITORING THE IMPLEMENTATION OF ELEARNING MATERIALS IN TVET SCHOOLS

Training and Advice for Further Development of the TVET Sector, Romania, PHARE TVET RO 2006/018-147.04.01.02.01.03.01

Monitoring e-learning implementation in TVET schools

January 2009

2009

The content of this material does not necessarily represent the official position of the European Union

Material produced under Phare 2006 financial support Phare TVET RO 2006/018-147.04.01.02.01.03.01

Monitoring e-learning implementation in TVET schools

January 2009

Content

INTRODUCTION..........................................................................4 WHY MONITORING?....................................................................5 GUIDELINES FOR MONITORING....................................................6 DEVELOPING A MONITORING STRATEGY.......................................................6 DETERMINE QUESTIONS......................................................................................6 NECESSARY METHODS TRAIN
INFORMATION...................................................................................6 AND TOOLS FOR MONITORING...................................................................7 SAMPLE........................................................................................ 7

MONITORING GATHER ENSURE LESSONS

MONITORING PERSONS................................................................................7 BASELINE DATA..................................................................................... 8 MONITORING TARGETS ARE BEING MET............................................................ 8 LEARNED FROM IMPLEMENTATION.................................................................8 AND

REPORTING

DISSEMINATING DATA....................................................................8

ANNEX 1 MONITORING SURVEY...............................................10 ANNEX 2 MONITORING FORM .................................................14

Material produced under Phare 2006 financial support Phare TVET RO 2006/018-147.04.01.02.01.03.01

Monitoring e-learning implementation in TVET schools

January 2009

Introduction
Executive Summary This material is planned to provide guidance on importance and necessity of monitoring the process of implementation of e-learning materials in schools. It is essential that a monitoring plan is put in place to ensure that the activities undertaken using each of the e-learning materials are reviewed for progress, consistency, results and benefits. It is advised that a dedicated results-based monitoring system is devised and introduced to track the effectiveness of the activities and to identify the decisions required to be made by the regional centres on an ongoing basis. Monitoring plan/strategy needs to define necessary information, questions that will be asked, methods and tools to be used in gathering data, who and when will monitor and who will be monitored. Monitoring process can include different activities based on priorities defined to be monitored, but everything should be defined very clearly at the beginning prior monitoring process should start. How the collected data will be analyzed and what will be the presentation document is also important to know at the beginning so that the planning process can be successful and monitoring will give results. A sound monitoring strategy consists not only from documents what has been done (activities, outputs) but pays particular attention to what changes this has brought about (use of output outcomes).

Material produced under Phare 2006 financial support Phare TVET RO 2006/018-147.04.01.02.01.03.01

Monitoring e-learning implementation in TVET schools

January 2009

Why monitoring?
Monitoring is an important tool because it is used in parallel with the monitored process, and can propose corrective measures if necessary. The monitoring process looks at what is being done and how it is done till the monitored moment. It could involve continuous tracking of activities, review of the flow of services and activities provided by the program, people trained, equipment used, frequency of usage, etc. Much of the information gathered through monitoring process is also valuable for the evaluation component. Monitoring and evaluation determine the way of program implementation and its success. A formal evaluation process using predetermined tools and methods for data collection allows objective and complete information about what the programme did and did not accomplish. Monitoring and evaluation are conducted for three primary purposes: To ensure that activities are completed and targets are met. Monitoring the progress and achievements of the programme as it is implemented creates accountability by making sure that programme activities take place as planned. In addition, programme monitoring ensures that the activities reach their targets and that sufficient data are collected for final outcome evaluation. To determine the outcome of the programme. Although informal participant feedback and other anecdotal evidence about the effects of a programme can create perceptions about its success or failure, a true measure of its outcomes can only be made through structured monitoring and evaluation. It is essential to know whether or not the programme makes a difference in order to justify its continuation. To adapt further program activities as needed. Monitoring allow staff to examine which programme components were more successful and which need improvement, and adapt future programme activities accordingly in order to better reach final goals.

Material produced under Phare 2006 financial support Phare TVET RO 2006/018-147.04.01.02.01.03.01

Monitoring e-learning implementation in TVET schools

January 2009

Guidelines for Monitoring

Developing a Monitoring Strategy
Monitoring and evaluating any programme is easier if the monitoring strategy is clearly defined prior to launching the program. To define a monitoring strategy it is necessary to go through a series of steps described in next sections.

Determine questions
To start planning monitoring process you need clearly to decide what programs priorities are and what outcomes the program is aiming to. Consider questions that examine how well the programme is meeting its stated outcomes as well as questions that measure programme effects, both on shorter and longer terms. In this way, whether the programme outcomes are positive or less than anticipated, you can go back and examine the process in order to determine possible reasons for the outcome. And then you can propose corrective measures, so that final outcomes should be as close as possible to the anticipated. Questions about the delivery aspects of the programme are answered by process monitoring. Process monitoring helps you keep a record of what has happened so far. For example, are activities being carried out as planned? How many participants? How many days? What is the level of satisfaction of participants? Are targets being met in terms of numbers of people reached and quantities of educational materials distributed? Process monitoring can also suggest areas of strength or weakness in the current programme activities. For example, are learners satisfied with the information delivered in the e-learning material? Are there immediate changes that can be made to the curriculum based on this information? Monitoring questions will receive answers about programme delivery.

Necessary information
Think about the type of data you need to answer each monitoring question. Generally, there are two types of data that can be collected: Quantitative data are defined in numeric terms, including percentages, averages and increases. Quantitative data answer questions such as how many and how much and are gathered via monitoring forms, surveys and other means. Qualitative data can be described in terms of perceptions, feelings, opinions and reasons. Qualitative data address how and why, and are gathered via conversations with focus groups, in-depth interviews, anecdotes and other means. It is valuable to gather both quantitative and qualitative data when monitoring. Using a variety of methods provides a more complete picture of the outcome and compensates for the biases or other limitations of any single data collection method. For example, surveys with employees who have attended education sessions might reveal that they are now more knowledgeable, but they do not report using the gained knowledge in practice. Focus groups could provide an opportunity to explore more fully the reasons behind this. The ideas generated in focus groups can provide important guidance for modifying the program to make it more effective.

Material produced under Phare 2006 financial support Phare TVET RO 2006/018-147.04.01.02.01.03.01

Monitoring e-learning implementation in TVET schools

January 2009

Methods and tools for monitoring

Monitoring forms keep a record of what has taken place. Because they are used on an ongoing basis throughout the project, monitoring forms should be as simple as possible, often in the form of checklists or fill-in-the-blank. In addition to making sure that planned activities are completed, monitoring keeps track of whether or not targets are met in terms of peer educators trained, employees attending education sessions, and other process indicators. Necessary changes can be made based on this data; for example, if only half of the target number of employees attend sessions during a given period, perhaps the sessions need to be held at a more convenient time or place. Think about the kinds of activities you must carry out in order to obtain the information you need. Monitoring information can be generally collected with methods such as tracking activities against timelines and checklists, observe elearning practice present activities, analysis of documentation and using questionnaires to obtain feedback from participants. Monitoring tools that can be used are: Surveys/questionnaires Monitoring forms, Interviews, Checklist, Tests.

Try to define the most suitable monitoring method and tools that can easier answer your monitor question. Identify necessary number of items in the tool, but remember not to make it too big and impossible to answer.

Monitoring sample
Define how much monitoring information will be enough to draw conclusion. It depends on the number of beneficiaries and target groups for your program. Gather the least amount of information necessary to answer your research questions. Do not overload with the monitoring information that can be very difficult and time consuming to analyse later on. Try to use different monitoring methods and different groups of participants in order to receive as much as possible objective information. The interviews may be conducted individually or in small groups of persons with similar job titles (e.g. trainers, peer educators). In order to ensure that implementers are comfortable speaking frankly about the implementation process, it is recommended that outside researchers conduct the implementer interviews and compile the results

Train monitoring persons

Usually not only one person is conducting monitoring process. Whenever more people are involved in monitoring it is essential for all of them to have common understanding of the items that should be followed and monitored. Training is one way to assure that the information gathered will be against the same criteria and to avoid any bias inadvertently introduced by inexperienced interviewers.

Material produced under Phare 2006 financial support Phare TVET RO 2006/018-147.04.01.02.01.03.01

Monitoring e-learning implementation in TVET schools

January 2009

Using Monitoring and Evaluation Tools and Methods

Gather baseline data
Define a period in which concrete data will be collected. Make it clear to monitoring persons what information and when are needed. Develop a table to summarize quantitative data. Calculate all necessary averages, percentages and sums. Create charts to have better visual presentation. Monitoring surveys may be distributed by coordinators, peer educators or outside evaluators. For monitoring instruments that are designed to measure the individual knowledge, attitudes and practices, the surveys should be completed and returned on time. Doing so will reduce the chances that employees compare and change their answers, creating inaccurate results. Define a report template for qualitative data and gathered all received information in one document.

Ensure monitoring targets are being met

People, who are responsible for tracking the ongoing progress of the intervention, including coordinators and peer educators, should complete monitoring forms as activities take place rather than at the end of the intervention when details will be difficult to remember. Instructions should be provided regarding when and to whom monitoring forms should be submitted.

Lessons learned from implementation

Programme implementers can provide important insight to the challenges and lessons learned about the programme itself. The opinions of trainers, peer educators, outreach staff and others, can be extremely useful in making recommendations for future interventions.

Reporting and Disseminating Data

Gathered data should be analysed right away. Then they will be baseline for comparison with expected outcomes in that period and with results from other monitoring periods. Data gathered from follow-up surveys will then be compared to the baseline to determine what changes have taken place as a result of the activities. Monitoring and evaluation results provide essential information about which aspects of a project are successful and which need modification. A final step in every evaluation activity is to write up the results of the evaluation, make recommendations based on the data, and disseminate the report to those who are interested in the project and/or are responsible for adapting project activities. It is recommended that professional evaluators analyse all data from the programme collectively and present the results, along with recommendations, in user-friendly formats. Monitoring reports can have one of the following possible formats: A detailed report with complete statistical analysis This report serves as the technical foundation for preparing other e-learning materials and may be appropriate for distribution to programme coordinators and partner organizations that are actively involved in programme implementation. Briefing materials for the press or other mass media These materials should avoid all technical language. Each press release or briefing should focus on only one or two key findings and their implications.
Material produced under Phare 2006 financial support Phare TVET RO 2006/018-147.04.01.02.01.03.01

Monitoring e-learning implementation in TVET schools

January 2009

One- or two-page executive summaries These documents, written in non-technical and concise language, are appropriate for distribution to worksite employees and community members who are curious about the results and implications of the programme in which they have or are participating. Visual presentation materials A visual presentation is another format in which employees and community members may be informed of the results and recommendations of the monitoring and/or evaluation. Like the summaries, presentations should avoid technical language and should include positive recommendations for further preventive action.

Material produced under Phare 2006 financial support Phare TVET RO 2006/018-147.04.01.02.01.03.01

Monitoring e-learning implementation in TVET schools

January 2009

Annex 1 Monitoring survey

Monitoring questionnaire for Management of support centers This document will be used for monitoring purposes of the use of e-learning materials. Your open and sincere answer will help in getting clear picture of the situation in your support center and TVET sector in Romania. The answers from this questionnaire will be collected, analyzed and send back to you to be at your disposal. You can use them to improve the usage of e-learning materials in your center. Support center name: ____________________________________________ Date: ______________ How many e-learning courses are you using in your center? __________ How many teachers from your center are engaged in development of eHow many teachers from your center are engaged in using of e-learning How many computers in your center are used in e-learning process? How many hours in a week are computers dedicated for e-learning purposes?

learning materials? ____________ materials? ___________ ____________ _____

If you could, what would you change in existing e-learning materials?

If you could, what would you change in organization of e-learning process?

If you could, what would you change in process of development of e-learning

materials?

What should be taken into consideration in future development of new e-

learning materials?

Material produced under Phare 2006 financial support Phare TVET RO 2006/018-147.04.01.02.01.03.01

10

Monitoring e-learning implementation in TVET schools

January 2009

What other e-learning material you need in your support center?

1. . 2. . 3. . What else do you need to improve the quality of e-learning process? o o o o o Management team training Teachers training Equipment Technical support for equipment and materials _________________________

YES

Do you need additional trainings for management teams? If yes, state which ______________________ ______________________ ______________________ NO

one?

YES

Do you need additional trainings for teachers? If yes, state which one? ______________________ ______________________ ______________________ NO

Material produced under Phare 2006 financial support Phare TVET RO 2006/018-147.04.01.02.01.03.01

11

Monitoring e-learning implementation in TVET schools

January 2009

Monitoring questionnaire for teachers in support centers This document will be used for monitoring purposes of the use of e-learning materials. Your open and sincere answer will help in getting clear picture of the situation in your support center and TVET sector in Romania. The answers from this questionnaire will be collected, analyzed and send back to you to be at your disposal. You can use them to improve the usage of e-learning materials in your center. Support center name: ____________________________________________ Date: ______________ What should be taken into consideration in future development of new e-

learning materials?

What is well/satisfactory in present modules: 1. __________ 2. __________ 3. __________

What is well/satisfactory in present organization of e-learning process: 1. __________ 2. __________ 3. __________

What is well/satisfactory in development process of e-learning materials: 1. __________ 2. __________ 3. __________ Learners are using e-learning materials _______ % of time at home (or

private) and _______ % in your support center.

Material produced under Phare 2006 financial support Phare TVET RO 2006/018-147.04.01.02.01.03.01

12

Monitoring e-learning implementation in TVET schools

January 2009

How many learners have asked for feedback or other type of support/help?

_________

How can learners ask for additional information?

___________________________________________________________

Is developed e-learning material you are using of blended type? YES

NO

Is assessment in e-learning of blended type? NO

YES

What else do you need to improve for better quality of e-learning process? o o o o o Teachers training Equipment Technical support for equipment and materials E-portal adjustment to your center needs _________________________

Do you need additional trainings for teachers? If yes, state which one? YES ______________________ ______________________ ______________________ NO

What is your information about satisfaction of learners after following the course?

What is your information how much e-learning learners have been able to find employment after finishing the course?

Material produced under Phare 2006 financial support Phare TVET RO 2006/018-147.04.01.02.01.03.01

13

Monitoring e-learning implementation in TVET schools

January 2009

Annex 2 Monitoring form

Material produced under Phare 2006 financial support Phare TVET RO 2006/018-147.04.01.02.01.03.01

14