Running Head: Multimedia Evaluation Report 1

Multimedia Evaluation Report

Sharon Persad (3200004806)

University of the West Indies (Open Campus)

EDID6508: Developing Instructional Materials

Course Coordinator: Dr. Leroy Hill

Running Head: Multimedia Evaluation Report 2

Executive Summary

Instructional Media Agents created an interactive, intuitive and responsive program

which uses a multi web-browser platform. It was designed and created using the web authoring

application Articulate Storyline 360 which provided many features which are important in

developing a tutorial based software. The responsive player can easily adapt to mobile devices

optimized for every screen size and orientation. It can also facilitate touch screen to give learners

a deep interaction with drag, drop and zoom features. In addition, it can be used with both Flash

and HTML5. These features can provide online instruction to most e-learning environments with

the least hiccups. The instructional software developed by IMA was designed to provide

instruction for the Mathematics lesson-Pythagorean Theorem, a topic which many learners find

very abstract and have difficulties in grasping concepts. The target audience is between the ages

of twelve to fourteen and mainly secondary school children living in the Caribbean.

Evaluation

Software evaluation provides a usability test “of the software in the form of a

reproducible record of experiences. This gives a developer a practical insight into how the

software is approached and any potential technical barriers that prevent adoption” (Jackson et al.,

2011). Feedback from evaluators can give developers insight into how they can make changes to

address shortcomings in quality and design before the system gets implemented. As a result,

Instructional Media Agents decided to invite users to test the software.

Participants

Two users were asked to do the usability evaluation and then submit a completed rubric

with general comments. The participants are Caribbean secondary school educators who teach

Computer Science and Mathematics to children who fall into the age range of the target
Running Head: Multimedia Evaluation Report 3

audience. These participants were carefully chosen as they are both past Instructional Design

students and are currently in the field related to the lesson topic and the target audience. Peers

were not chosen due to course demands and timeliness of the needed evaluation.

Evaluation Results

The users were presented with a questionnaire (See Appendix A) to complete using a

given rubric (See Appendix B) adapted from Hill (2018). The results were correlated into the

following report.

Relevance

Both participants agreed that the Pythagorean Theorem Software adhered to the

Mathematics Curriculum and instructional objectives of the lesson. The content logically

unfolded according to Gagne’s Nine Instructional Events and the multimedia interactivity is

relevant to different learning styles. Relevant videos were used for visual learners, drag and drop

activities were used for tactile learners and a variety of music and audio narration was used for

auditory learners.

Appropriateness

They also agreed that the content was age appropriate using logical steps. Activities and

quizzes were creatively designed for the target audience where they met instructional objectives

in a short, fun and interactive style. One participant commented-”learners can be engaged

without being bored.”

Sufficiency

Unanimously, the participants gave rave reviews for the nature of the content being

cleverly broken down into two lessons. It was sufficient without overwhelming learners. Its
Running Head: Multimedia Evaluation Report 4

design covered the content but also provides learners with choices of covering the content

depending on their level of understanding. Learners can redo the first lesson until mastery is

achieved and then proceed to the next lesson. The feedback during quizzes and activities was

“sufficient without being overbearing” (user comment).

Areas which need to be Addressed

The areas which fell short were in the technical details and they are as follows:

1. Some of the links did not work properly. Some of the practice activities did not have a

working submit button and the quit button was unresponsive.

2. Videos and activities took a while to load leaving the user in a quandary as to the next

step.

3. One user mentioned that the picture of the teacher used should have reflected a Caribbean

personality instead of the one that was used.

4. There was no reference to copyrighted attributions.

Discussion

The design of this software was relevant, appropriate and sufficient in both instruction

and multimedia which was confirmed by the two users. It is understandable that there were

minor details such as links not working properly due to time constraints of the project. These

issues can be easily fixed as they are very minor.

Upon further investigation, the users had a very small bandwidth which could account for

videos taking a while to load. However, further testing would have to be done to determine the

root cause of these problems. The size of the file could be another factor which might have

caused this problem. If future tests reveal that this is a factor then some of the content will have

to be revised or the file would need to be compressed for use on the web to cater to all browsers.
Running Head: Multimedia Evaluation Report 5

A technical consultant will need to be sought to provide the best solution to the problem. This

could negatively affect the triple constraints of the project.

One user made a valid point about the ethnicity of the “asset” used for the software.

Further tests will be conducted to determine the relevance of that factor on learning and usability

where changes will be made accordingly.

Copyright information will need to be addressed before implementing the software into

the public domain. All videos, audio and graphics will undergo a comprehensive examination

and permission from original owners will have to be acquired and attributed appropriately if

applicable.

In conclusion, the design of the software was relevant, appropriate and sufficient in

meeting the objectives of both instruction and multimedia. It covers the content logically in a

fun, interactive and responsive manner. The team worked cohesively and cooperatively in

producing software that is worthy and relevant in today’s instructional market.

Running Head: Multimedia Evaluation Report 6

References

Hill, L. (2018). Assignment 3-Multimedia Project 2018 [Class Handout]. Retrieved from
http://2017.tle.courses.open.uwi.edu/mod/folder/view.php?id=73501

Jackson, M., Crouch, S., Baxter, R. (2011). Software Evaluation:Tutorial Based Assessment.
Software Sustainability Institute. Retrieved from https://software.ac.uk/sites/default/files/SSI-
SoftwareEvaluationTutorial.pdf
Running Head: Multimedia Evaluation Report 7

Appendix A

Evaluation Questionnaire
Running Head: Multimedia Evaluation Report 8
Running Head: Multimedia Evaluation Report 9
Running Head: Multimedia Evaluation Report 10
Running Head: Multimedia Evaluation Report 11
Running Head: Multimedia Evaluation Report 12
 Running Head: Multimedia Evaluation Report 13

Appendix B

Evaluation Rubric- Adapted from Hill (2018)

Criteria/ Points 1 2 3 4
Storyboard or Storyboard is Storyboard is Storyboard is Storyboard is
Planning Sheet incomplete and not complete. somewhat complete.
lacks necessary Includes few complete. Includes
URL’s, formats, assigned Includes many all assigned
and elements or assigned elements, in
resources to planned formats, elements, in addition to
complete necessary addition to most planned formats,
project. URL’s, planned formats, necessary
and resources. necessary URL’s,
URL’s, and resources.
and resources.

Organization of No logical Some logical Logical Logical, intuitive

Content sequence of sequence of sequence sequence of
information; information, but of information. information.
menus and paths menus and paths Menus and paths Menus and paths
to information are confusing or to more to all
are flawed. information are information
not evident. clear and direct. are clear and
direct.

Originality The work is a The work is an The product The product

minimal extensive shows evidence shows
collection collection and of significant
or rehash of rehash of other originality and evidence of
other people’s ideas, inventiveness. originality and
people’s ideas, products, images While based on inventiveness.
products, images and inventions. an The
and inventions. There is no extensive majority of the
There is no evidence of new collection of content and
evidence of new thought or other many
thought. inventiveness. people’s ideas, of the ideas are
products, images fresh, original,
and inventions, inventive, and
the work extends based upon
beyond that logical
collection to conclusions and
offer sound research.
new insights.
 Running Head: Multimedia Evaluation Report 14

Copyright and Sources have not Some sources Most sources All sources and
Documentation been properly have not been and property cited
cited and properly cited property cited according to
permissions have and all according to MLA
not been permissions have MLA style;
received. not been style; Permissions
received. Permissions to use any
to use any graphics from
graphics from web pages or
web pages or other sources
other sources have been
have been received,
received, printed, and
printed, and saved for
saved for future reference.
future reference.

Format and The stack, The stack, Most of the The stack,
Platform presentation, or presentation, or stack, presentation, or
Transferability web page plays web page plays presentation, or web page plays
only on either best on either web page plays easily on both
Mac Mac or PC. easily on both Mac
or PC. There are There Mac and PC. and PC. Care has
problems with are problems Although there been taken in
the with the are minor naming files,
operation of operation of problems with a selecting
some some files and few files, care technologies, or
files and the the project is not has creating
project is not cross-platform. been taken in enhancements to
cross-platform. naming files, produce a final
selecting product that is
technologies, or cross-platform.
creating
enhancements to
produce a final
product that is
cross-platform.

Subject Subject Some subject Subject Subject

Knowledge knowledge knowledge is knowledge is knowledge
is not evident. evident. Some evident in much is evident
Information is information is of throughout
confusing, confusing, the product. (more
incorrect or incorrect or Information is than required).
flawed. flawed. clear, All
appropriate, information is
 Running Head: Multimedia Evaluation Report 15

and correct. clear,

appropriate,
and correct.

Graphical Design Exaggerated Graphical and Design elements The combination

emphasis upon multimedia and content of
graphics and elements combine multimedia
special effects accompany effectively to elements with
weakens the content but there deliver a high words and ideas
message and is little sign of impact message takes
interferes with mutual with the graphics communication
the reinforcement. and the words and persuasion
communication There’s no reinforcing each to
of attention paid to other. a very high
content and visual design level,
ideas. criteria such as superior to what
proportion, could be
balance, and accomplished
harmony with
restraint. There either alone. The
is some tendency mixture brings
toward random about synergy
use of graphics. and
dramatic effects
which reach the
intended
audience.

Mechanics Presentation has Presentation has Presentation has Presentation has

four or more three of more fewer than two no misspellings
spelling errors misspellings misspellings or
and/or and/or and/or grammatical
grammatical grammatical grammatical errors.
errors. errors. errors.

Screen Design Screens are Screens are Screens contain Screens contain
either difficult to adequate all
confusing and navigate, but navigational necessary
cluttered or some buttons tools navigational
barren and navigational and buttons. tools
and stark. tools work. Users can and buttons.
Buttons Users progress through Users
or navigational can navigate a screens in a can progress
tools are absent few screens. logical path to intuitively
or find information. through
confusing screens in a
 Running Head: Multimedia Evaluation Report 16

logical
path to find
information.

Use of No video, audio Limited video, Some Video, Appropriate

Enhancements enhancements audio audio amounts of
are enhancements enhancements Video,
present or use of are present. In are audio
these tools is most instances, used enhancements
inappropriate. use of these tools appropriately to are
is appropriate. entice users to used effectively
learn and to to
enrich the entice users to
experience. In learn and to
some cases, clips enrich
are either too the experience.
long Clips are long
or too short to be enough convey
meaningful. meaning without
being too
lengthy.