Running Head: BOARD REPORTS MADE EASY TRYOUT PLAN

Board Reports Made Easy

Tryout Plan

Presented to
Yvonne Walker, Ph.D., IPMA-SCP
Learning and Organizational Development
Human Resources Department
County of Monterey, CA
By
Curriculum Manager
Darryl Sink, Ed.D.
IST 626, Advanced Instructional Design Professor
Instructional Designers
Deborah LeAnce, Rouwaida Mahmoud, Nawara Nassan, and Dana Payne
Master of Science in Instructional Science and Technology (MIST)
Graduate Students
California State University - Monterey Bay
August 15, 2016

BOARD REPORTS MADE EASY TRYOUT PLAN

2
Board Reports Made Easy
Tryout Plan
Table of Contents
Introduction
Developmental Testing
Pre-Learner Tryout
Learner Tryout
Special Arrangements
Learner Prerequisites
Facility, Equipment and Supporting Material Requirements
Data Collection Instrument
References
Appendix, BRME Usability Survey
Appendix 2, Post-Training Survey

Tryout Plan
Board Reports Made Easy

BOARD REPORTS MADE EASY TRYOUT PLAN

3
Introduction
Board Reports within the County of Monterey provides a vital, permanent record
that presents issues, solutions, and actions taken by the Board of Supervisors. The Board of
Supervisors faces a problematic issue: the quality of Board Reports presented to the Clerk of
the Board are often of substandard quality. Hence the need for a training program to develop
the skills necessary to create clear, precise, and comprehensive board reports by the
employees entrusted with this task.
Monterey Countys Learning and Organizational Development (LOD) unit has
recently provided California State University Monterey Bay (CSUMB) graduate students in
the Master of Science in Instructional Science and Technology (MIST) curriculum the
opportunity to assist the County of Monterey by creating three pilot learning modules to
enhance Monterey Countys Board Report process. These three e-Learning modules are
pilot modules for the LOD Units next generation, online, interactive, Writing Academy.
This tryout plan presents the developmental testing plan for MIST Team 2s Board Reports
Made Easy (BRME) pilot, e-Learning module. This module consists of five instructional
units and a final unit to evaluate the effectiveness of the module in conveying key aspects of
training.
Developmental testing is planned for two phases, the Pre-Learner Tryout and the
Learner Tryout phase. This Tryout Plan address these two tryout phases, required special
arrangements, Learner prerequisites, Facility, Equipment and Supporting Material
Requirements, and Data Collection Instruments.

Tryout Plan
Board Reports Made Easy

BOARD REPORTS MADE EASY TRYOUT PLAN

4
Developmental Testing
Pre-Learner Tryout
Team 2 will conduct a Pre-Learner Tryout to test-run the Learner Tryout Plan and
gain preliminary Learner feedback to provide an opportunity to make (1) tryout process and
(2) Learner-centric BRME module refinements prior to the Learner Tryout. Team 2 is
fortunate to have a Monterey County Employee for the Pre-Learner Tryout who fits the
general description of the target audience (e.g. the clients Monterey County Employee
Board Report drafters). The BRME Pre-Learner Tryout is planned for Wednesday, August
17, 2016. Although not involved in drafting Board Reports, the Pre-Learner Tryout
Monterey County Employee has experience writing and submitting other types of reports
for the general public, Monterey County officials, and to attorneys presenting his reports a
part of court proceedings.
The Pre-Learner Tryout requires a Learner, a Tester/Observer, a quiet environment
and a computer with internet access running the latest version of the Google Chrome
Browser. Team 2 will provide an on-site Tester/Observer to lead the tryout, assist the
Learner, answer questions and troubleshoot as needed. The Tester/Observer will explain the
modules purpose, estimated time for completion, as well as provide the Learner with paper
and pen for taking notes and for completing the BRME Usability Survey. Throughout this
process, the Tester/Observer will note the learners questions, comments as well as body
and facial language (e.g. collect observed usability data).
The first five units of the BRME module provides instruction. The sixth and final unit
provides an e-Learner evaluation which also serves as the BRME modules criterion test.
Tryout Plan
Board Reports Made Easy

BOARD REPORTS MADE EASY TRYOUT PLAN

5
Upon completion of Unit 6, e-Learning Evaluation, the Learner will take a 5-10 minutes
break before taking the BRME Usability Survey to assess the learners perception of the
modules usability and satisfaction with the experience (e.g. provide perceived usability
data).
Finally, the Tester/Observer will conduct a semi-structured interview with the
Learner to gather additional information related to perceived learning, satisfaction and
usability. This survey, combined with BRME Usability Survey, Tester/Observer
observations and Unit 6 evaluation results will provide Team 2 essential data needed to
assess the modules instructional effectiveness and make revisions prior to conducting the
Learner Tryout.
Learner Tryout
The County of Monterey LOD Unit will provide two employees to participate in the
Learner Tryout. The Learner Tryout is scheduled for Friday, August 19, 2016 between
12:30 and 2:30 PM in the LOD Unit conference room. The plan is for each Learner to
participate using a dedicated computer with internet access running the latest version of the
Google Chrome Browser. Time permitting, Team 2 prefers to administer the Learner Tryout
to the Learners sequentially to permit the Tester/Observer the opportunity to observe each
Learner independently. Team 2 will have at least one team member functioning as an on-site
Tester/Observer to lead the tryout, answer questions, observe the Learners, assist and/or
trouble shoot as needed and administer a semi-structured interview with the Learners upon
completion of the BRME e-Learning module and usability test.

Tryout Plan
Board Reports Made Easy

BOARD REPORTS MADE EASY TRYOUT PLAN

6
As with the Pre-Learner Tryout, the Tester/Observer will explain the modules
purpose, estimated time for completion, as well as provide the Learner with paper and pen
for taking notes and for completing the BRME Usability Survey. Throughout this process,
the Tester/Observer will note the learners questions, comments as well as body and facial
language (e.g. collect observed usability data).
The first five units of the BRME module provides instruction. The sixth and final unit
provides an e-Learner evaluation which also serves as the BRME modules criterion test.
Upon completion of Unit 6, e-Learning Evaluation, the Learner will take a 5 to 10 minute
break before taking the BRME Usability Survey to assess the learners perception of the
modules usability and satisfaction with the experience (e.g. provide perceived usability
data).
Finally, the Tester/Observer will conduct a semi-structured interview with the
Learner to gather additional information related to perceived learning, satisfaction and
usability. This survey, combined with BRME Usability Survey, Tester/Observer
observations and Unit 6 evaluation results will provide Team 2 feedback with the
opportunity to perform further analysis needed to guide module revisions prior to final build
of the the project. Of concern is to assess how usable was the training from a
human/computer interaction perspective? What screen designs need to be improved? Should
anything about the user interface be changed in order to facilitate the Learner's ability to
complete the program, improve their chances of mastering the material, or make the
experience more enjoyable? (Sink, n.d.)

Tryout Plan
Board Reports Made Easy

BOARD REPORTS MADE EASY TRYOUT PLAN

7
Special Arrangements
Team 2 does not anticipate any needed special arrangements for the developmental
test. The LOD unit has been generous with providing their conference room for meetings
related to this project. Tester/observer will make sure all other needed tools, equipment, or
other necessities are available to all parties involved.
Learner Prerequisites
Team 2 requests the LOD Unit provide two county employees for the Learner
Tryout that have experience with and/or understand the importance of Board Reports.
Learners must be fluent in English, achieved at least a high school level of education and
have the basic computer skills needed to navigate the BRME e-Learning module.
Facility, Equipment and Supporting Material Requirements
The Team 2 Tester/Observer will be onsite. The test will be conducted on Friday,
August 19, 2016 between the hours of 12:30pm and 2:30pm in the LOD Unit conference
room. This room is in a quiet part of the training department at the County of Monterey.
Learners will have access to a large table, comfortable seats, and a working computer with
internet access. Access to the BRME e-Learning module during Developmental Testing is
planned via an internet link. The Team 2 Tester/Observer will provide Learners a link to the
live version of the project via a link to an ITCDLAND account. Using the internet based
version of the BRME module minimizes IT requirements at Developmental test sites.
In anticipation of having learner go through the 3 modules provided by CSUMB
graduate students at once, there will be a Zoom link established where each team has a
Tryout Plan
Board Reports Made Easy

BOARD REPORTS MADE EASY TRYOUT PLAN

8
representative to observe learners during tryout. The computer used will have a camera
connection and will allow transmission to observers who are not present onsite.
Data Collection Instruments
The information in the tryout questionnaire is based on the perceived usability
data collected during the tryout. This data will be compared with the observed usability
data gathered by test facilitator/observer.
The analysis resulting from this tryout will address the following areas:
1. Learning: The focus is on the learner and determining if he/she met the
objectives stated at the beginning of the module?
2. Perceived Learning: Finding out how the learner felt about the material, if they
learned it, and if the new skills and knowledge will be applied at work.
3. Satisfaction: Successful eLearning courses need to engage the audience and keep
them interested in the material. Their experience should be enjoyable.
4. Usability: This step focuses on the interaction from the learners perspective.
Was the computer/learner interaction positive? Are there any design items that
need to be modified? Did the designer miss important tools necessary in the
module to assist the learner in mastering the new information?
The above listed analysis is addressed in Kirkpatricks levels of evaluation. In level
one of Kirkpatricks levels of evaluation, perceived learning, satisfaction, and usability are
addressed in level 1. Learning is addressed in level 2. Team 2 data gathering techniques will
be in three parts: An objective test at the end of the module, a usability survey answered by

Tryout Plan
Board Reports Made Easy

BOARD REPORTS MADE EASY TRYOUT PLAN

9
learners after the test, and direct observation. A team member will be present on site with
learners while the test is being conducted to benefit from collecting the necessary variables
in a personal/one on one environment, where learners reaction is measured during and
right after the test. The on-site observation, along with providing all the necessary tests,
tools and instruments needed for the tryout, will allow for gathering of data that will be
essential for the revision phase of the module prior to full release to client.

Tryout Plan
Board Reports Made Easy

BOARD REPORTS MADE EASY TRYOUT PLAN

10
References:
Sink, D. (n.d.). Informal and Informal Evaluation [Pamphlet].

Tryout Plan
Board Reports Made Easy

BOARD REPORTS MADE EASY TRYOUT PLAN

11
Appendices.
Appendix 1: BRME Usability Survey
Appendix 2: Board Reports Approval Tracking Sequence

Tryout Plan
Board Reports Made Easy

BOARD REPORTS MADE EASY TRYOUT PLAN

12

Appendix 1: BRME Usability Survey

(Survey adapted from Darryl L. Sink & Associates, Inc.)

Please circle the number that indicates your preference.

1.

The course met the stated objectives

If you disagree, why?

2.

The material was appropriate to meet my needs

If you disagree, why?

3.

I was able to master the objectives of the course

If you disagree, why?

4.

The structure and flow supported my learning needs

If you disagree, why?

5.

There was sufficient time allotted for the course

If you disagree, why?

6.

The directions provided me with enough information to use

the module properly.
If you disagree, why?

Tryout Plan
Board Reports Made Easy

BOARD REPORTS MADE EASY TRYOUT PLAN

13

Please circle the number that indicates your preference.

7.

I learned new knowledge and skills from this training.

If you disagree, why?

8.

The directions provided me with enough information to use

the module properly.
If you disagree, why?

9.

The prerequisites were sufficient to prepare me for this

module.
If you disagree, why?

10. I was usually sure of exactly what I was supposed to be

learning.
If you disagree, why?
11. Locating the resources/materials was easy and took little
time.
If you disagree, why?
12. The module was well organized.
If you disagree, why?

Tryout Plan
Board Reports Made Easy

BOARD REPORTS MADE EASY TRYOUT PLAN

14

Please circle the number that indicates your preference.

13. Enough examples were provided so that I could learn the

material well.

If you disagree, why?

14. The information did not repeat itself.
If you disagree, why?
15.

I was able to use the materials at my own pace.

If you disagree, why?

16.

The self-test questions did a good job of testing my

knowledge of the main points of the module.
If you disagree, why?

17.

Having the correct answers provided after the self-test was

helpful.
If you disagree, why?

18. The content was at an appropriate level (not too easy or hard).
If you disagree, why?

Tryout Plan
Board Reports Made Easy

BOARD REPORTS MADE EASY TRYOUT PLAN

15

Please circle the number that indicates your preference.

19. I would prefer this type of instruction compared to most

textbook and lecture approaches.

If you disagree, why?

20. I have used or will be able to use the skills and knowledge
learned in this module in my present or future job.
If you disagree, why?
21. The online delivery had the right level of user
interaction/involvement.
If you disagree, why?

22. The delivery method was an effective way for me to learn the
material
If you disagree, why?
23. What were the two best things about this module?

24. For you, what was the most difficult part of this module?

Tryout Plan
Board Reports Made Easy

BOARD REPORTS MADE EASY TRYOUT PLAN

16
Appendix 2, Post-Training Survey
(Survey from BRME Design Document By Team2)

Comment [1]: Should we include?

Post-Training Survey
Course

Yes

The course met the stated objectives

The material was appropriate to meet my
needs
The examples helped me understand the
content
The structure and flow supported my learning
needs
There was sufficient time allotted for the
course
Learning Effectiveness
I was able to master the objectives of the
course
I learned new knowledge and skills from this
training
Job Impact
I will be able to apply the knowledge and
skills learned in this class to my job
Support Tools
The material enhanced my learning and will be
useful on the job

Tryout Plan
Board Reports Made Easy

No

BOARD REPORTS MADE EASY TRYOUT PLAN

17
Post-Training Survey
Course

Yes
Online Delivery

The online delivery had the right level of user

interaction/involvement
The delivery method was an effective way for
me to learn the material
Comments:
What concepts were not covered and/or needed more coverage?

Do you have any other suggestions to improve the course?

Tryout Plan
Board Reports Made Easy

No