Heuristic Evaluation of a Mobile Hand-Writing

Learning Application
Banu Ylmaz

Pnar Onay Durdu

Dept of Computer Enginnering

Human Computer Interaction Laboratory
Kocaeli University, Kocaeli, Turkey
banuyilmaz53@gmail.com

Dept of Computer Enginnering

Human Computer Interaction Laboratory
Kocaeli University, Kocaeli, Turkey
pinar.onaydurdu@kocaeli.edu.tr

Abstract Usability of educational applications is essential for

the effectiveness of the learning environment. Nowadays, these
applications are developed on mobile devices since their use
widespread rapidly. Considering these applications would be
used by children, usability evaluation methods to be used would
be different than conventional methods. In this study usability
evaluation of a mobile hand-writing learning application was
conducted by using heuristic evaluation method called Heuristic
Evaluation for Child e-Learning (HECE) method. Evaluaton was
conducted with three expert and two novice evaluators and
critical usability problems were determined. The results of the
study would guide the development of mobile applications for
children as well as enabled the improvement of the mobile
application.
Index Termsmobile learning,
heuristic evaluation for children

handwriting-recognition,

I. INTRODUCTON
According to a recent research, childrens access to mobile
media has been increased dramatically from 2011 to 2013 in
US and children between ages 0 to 8 are generally using these
media for playing games, watching videos or using apps, etc.
[1]. Although American Academy of Pediatrics [2]
recommends limiting any kind of screen time for children,
Steyer [3] suggested these devices such as tablets and
smartphones should be used for educational purposes
effectively. As in other forms of information technologies like
computers, mobile devices are already considered to support
education by providing interactive mobile learning
opportunities [4][5][6][7].
Developers of these kinds of technologies for children
should also take into account usability of their products to
create successful applications [8][9][10]. Hartson classified
usability evaluation methods as expert based and user based
inspection methods [11]. Heuristic evaluation is one of the
mostly used usability inspection method to evaluate any system
during its development or after its release [12]. It is one of the

discount usability engineering methods that uses small set of

evaluators to examine any interface regarding the usability
principles defined by Nielsen [13].
In this study usability evaluation of a mobile hand-writing
application, which was developed for children who were
learning literacy in primary school, was conducted to make
improvements on its usability problems[14]. User testing with
actual users seemed to provide more accurate results but it
would be difficult in this case because the target users of this
application were the children at age of 5 who were at their first
year at primary school. In the literature, it was reported that
user testing with this age group would be difficult since they
were illiterate and their verbal skills were not developed fully
yet [15]. Therefore, heuristic evaluation method was applied
considering these issues as well as its above mentioned
advantages. However, traditional usability heuristics proposed
by Nielsen would not be adequate since usability evaluation of
these kinds of products for children required other
considerations such as fun and playability rather than merely
task performance or ease of use [16].
II. RELATED WORKS
The traditional heuristics do not conform the additional
features therefore some additional heuristics were proposed by
previous research regarding playability, fun or some
educational issues. For instance, Malone developed initial
heuristic set which was specific to computer games and
afterwards many researchers have studied on game heuristics
development [17][18][19][20][21]. Specific to children, Bauw
proposed checklist based predictive evaluation method called
Structured Expert Evaluation Method (SEEM) for the
playability and fun for childrens computer games [22]. In a
former study, Squires and Preece proposed a checklist based
evaluation of the educational software based on Nielsens

heuristics [23][9]. Reeves developed a heuristic set for the

evaluation of e-learning applications considering both usability
and instructional design issues together [24]. De Angeli have
proposed Systematic Usability Evaluation technique for
hypermedia which used evaluation patterns called abstract
tasks that enabled sharing and transferring the usability
evaluation know-how among the usability inspectors to gather
more effective and efficient results [25]. Proposed evaluation
patterns defined the activities to be performed during
evaluation process.
In a more recent study, a heuristic set that considered the
childrens usability related needs was proposed [10]. This
heuristic set called Heuristic Evaluation for Child e-Learning
(HECE) was a more comprehensive list that it included
Nielsens usability heuristics, child usability heuristics and elearning usability heuristics. Therefore, these heuristics were
implemented in order to determine the usability of the mobile
hand-writing application in this study.

to be processed and it was verified by matching it with predetermined character set as a response. Letter-based or wordbased hand written text could be recognized in the application.
Figure 1 shows the architecture of the system.

Fig. 1. System Architecture [14]

III. METHODS
A. Participants
The study was conducted with five usability inspectors
three of which were experts while the other two were novice.
All of the participants had a previous experience with mobile
applications and their age range was between 20 to 40 years.
In experienced group there were two male and one female
while the novice group included one male and one female.
B. Data Collection Tools and Procedure
Heuristics used in this study were adapted from Alsumait
and Al-Osaimi [10]. Forty nine items were grouped under three
categories categories which are traditional heuristics, child
usability heuristics and mobile application heuristics similar to
the original study. These heuristics were adapted by replacing
e-learning program with mobile hand-writing application.
Participants were required to evaluate hand-writing mobile
application based on the given heuristics in two phases. In the
first phase they were required to determine whether there was a
usability problem related with the heuristics. In the second
phase they defined the severity of the usability problems based
on the severity rating 0-4 scale determined by Nielsen [13]. In
addition at the end of their evaluation, they were asked to make
additional suggestions as well.
Mobile hand writing application evaluated in this study was
developed as Android application which enabled users to write
the letters in the alphabet by their finger on the screen.
Afterwards, the image crated on the device is sent to the server

Fig. 2. Main interface of the application [14]

Application had a simple interface that can be seen in

Figure 2. It showed the interface of the application where users
inputted characters and could verify them by comparing the
image created on the screen with the pre-determine characters
on the server. Application was installed to the mobile devices
of the evaluators and afterwards they were allowed to
familiarize with the application. Then they were asked to
perform the usability heuristic based evaluation of the system.
IV. RESULTS
Four of the heuristics were determined to have serious
usability problems. These were lack of difficulty level for
child, lack of reward system when levels are completed

successfully, lack of visual, animation, audial feedbacks and

lack of stories, activities to help learning for children.
Expert evaluators determined more problems. They
determined 8 problems while novices determined 5. The results
of the heuristic evaluation was summarized as in Table 1.

TABLE 1PROBLEMSWHICH WERE DETERMINED BY PARTICIPANTS

Determined by experts
(

E1 , E 2

E3

Determined by novices

N1 , N2

Determined by all participants

E1 , E 2

E3,

N1 , N2

The screen interface contains information that is irrelevant or rarely needed in the mobile
application.
The mobile application doesnt express error messages in simple language, precisely doesnt
indicates the problem, and not in a friendly way suggests a solution that a child can handle.
All input devices/buttons that have no functionality to prevent user input errors.
The child cant have enough information to start to use the program when he turns it on
The e-learning program cant provide sufficient feedback (audio, video) to the child to provide
corrective directions.
The e-learning program cant provide the instructor with child evaluation and tracking reports
The e-learning program cant incorporate novel characteristics
The e-learning program doesnt use e-stories, games, simulations, role playing, activities, and
case studies to gain the attention and maintain the motivation of learners.
The child does not need to use a manual to use the application.
The e-learning program cant provide the instructor with child evaluation and tracking reports.
The e-learning program doesnt use e-stories, games, simulations, role playing, activities, and
case studies to gain the attention and maintain the motivation of learners.
All input devices/buttons that have no functionality to prevent user input errors.
The childs fatigue is not minimized by varying activities and difficulties during learning
sessions. Challenges are not positive learning experiences, rather than negative experiences;
resulting in the child wanting to learn more, rather than quitting.
The e-learning program cant provide the instructor with child evaluation and tracking reports.
The e-learning program doesnt use e-stories, games, simulations, role playing, activities, and
case studies to gain the attention and maintain the motivation of learners.
All input devices/buttons that have no functionality to prevent user input errors.
The childs fatigue is not minimized by varying activities and difficulties during learning
sessions. Challenges are not positive learning experiences, rather than negative experiences;
resulting in the child wanting to learn more, rather than quitting.

According to numbers of problems were determined,

gender is not important. Common problems which were
determined by all participants are related with child heuristics.
Participants gave some advices about interface of system such
as layout should be like blackboard. This kind of metaphors
from daily life are more useful to children. A tutorial helps to
children how they can use the system. According to
participants, visual and auditory feedbacks have to be on the
system.
Result of these feedbacks showed that, new developments
are necessary on software mobile application. Background of
the application interface is planned to be updated more colorful
to make children ambitious. A tutorial is planned to be added to
the application that would show its use on the first interaction,
so users can comprehend what they should do and how.
Enjoyable sounds and visual items are planned to be added i.e.
when the user wrote a character appropriately, the user heard a
glowing sound as a feedback that understand is true.
V. CONCLUSION
This study reports on a heuristic evaluation of the mobile
application developed for children to teach them hand-writing.
Forty nine heuristics adapted from the literature were used for
the evaluation. Evaluation results show that application did not
comply nine of the heuristics. Among these problems four of

them were determined as severe problems. These were the

design of the interface, lack of visual or sound based feedback,
Therefore these issues should be dealt primarily for the
effectiveness of the application.
In the study gender of the evaluators did not have any affect
regarding usability problem detection. On the other hand,
expert evaluators reported more interface component usability
problems while novice evaluators reported the content related
usability problems. All of the evaluators could determine the
problems related with child usability heuristics section.
The results of the evaluation revealed the usability
problems of the mobile application. Therefore it is planned to
make modifications based on the results of this study and
afterwards user testing based evaluation is planned to be
conducted to make a summative evaluation of the system as a
follow up study.
ACKNOWLEDGMENT
The authors would like to thank to Assoc. Prof Dr Tugay
Turgay Bilgin for his supervision during the development of
mobile hand-writing application as well as the evaluators for
their valuable effort.

REFERENCES
[1] A Common Sense, Zero to Eight: Childrens Media Use in
America
2013,https://www.commonsensemedia.org/research/zero-toeight-childrens-media-use-in-america-2013, 2013
[2] American Academy of Pediatrics (n.d.). Media and Children. ,
https://www.aap.org/en-us/advocacy-and-policy/aap-healthinitiatives/Pages/Media-and-Children.aspx
[3] Mashable, 38% of Children under 2 Use Mobile Media, Study
Says http://mashable.com/2013/10/28/children-under-2-mobilemedia-study/,2013
[4] Berggren, J., & Hedler, C.,CamQuest: design and evaluation of
a tablet application for educational use in preschools. In
Proceedings of the 2014 conference on Interaction design and
children (pp. 185-188). ACM, June 2014
[5] Chen, C. M., Hsu, S. H., Li, Y. L., & Peng, C. J., Personalized
intelligent m-learning system for supporting effective English
learning ,2006, InSystems, Man and Cybernetics, 2006.
SMC'06. IEEE International Conference on (Vol. 6, pp. 48984903). IEEE.Tao, S. (2013). English Learning, 109, 48984903.
[6] Cho, S. J., Kim, J., & Lee, S., Mobile computer-assisted
language learning courseware for Korean language
learners,2004,In Web Information SystemsWISE 2004
Workshops (pp. 173-178). Springer Berlin Heidelberg.
[7] Zare, S. , Intelligent mobile learning interaction system
(IMLIS): a personalized learning system for people with mental
disabilities,Unpublished dissertation, Bremen, Univ., Diss.,
2010.
[8] Markopoulos, P., Read, J., Hosniemi, J., & MacFarlane, S.,
Child computer interaction: Advances in methodological
research: Introduction to the special issue of cognition
technology and work, Cognition, Technology and Work, 10(2),
7981. doi:10.1007/s10111-007-0065-0, 2008
[9] Khanum, M. A., & Trivedi, M. C. Take Care: A Study on
Usability Evaluation Methods for Children, 2012, The
Computing Research Repository, (i).
[10] Alsumait, A., & Al-Osaimi, A., Usability heuristics evaluation
for child e-learning applications, Journal of Software, 5(6),
654661, 2010
[11] Hartson, H. R., Andre, T. S. T., &Williges, R. R. C. Criteria for
evaluating usability evaluation methods,2001, International
Journal of Human-Computer Interaction, 13(4), 135.
doi:10.1207/S15327590IJHC1304_03
[12] Molich, R., & Nielsen, J.,Improving a human-computer
dialogue,Communications of the ACM, 33(3), 338-348, 1990
[13] Nielsen, J., Usability engineering. Elsevier,1994
[14] Ylmaz B., renme Gl eken ocuklar in El Yazs
Tanma le renmeyi Kolaylatrc Bir Mobil renme
Uygulamas Tasarm, Unpublished dissertation , Maltepe
niversitesi,2014.
[15] Markopoulos, P., &Bekker, M., On the assessment of usability
testing methods for children, Interacting with Computers, 15(2
SPEC.), 227243. doi:10.1016/S0953-5438(03)00009-2,2003
[16] Read, J. C., &Bekker, M. M., The Nature of Child Computer
Interaction,2011, (BCS-HCI 11 Proceedings of the 25th BCS
Conference on Human-Computer Interaction, (1994), 163170)
[17] Malone, T. W., Heuristics for designing enjoyable user
interfaces: Lessons from computer games, 1982, In

[18]

[19]

[20]

[21]

[22]

[23]

[24]

[25]

Proceedings of the 1982 conference on Human factors in

computing systems (pp. 63-68). ACM.
Federoff, M. A., Heuristics and usability guidelines for the
creation and evaluation of fun in video games, 2002, (Doctoral
dissertation, Indiana University).
Desurvire, H., Caplan, M., &Toth, J. A., Using heuristics to
evaluate the playability of games, April 2004, In CHI'04
extended abstracts on Human factors in computing systems (pp.
1509-1512). ACM.
Korhonen, H., &Koivisto, E. M., Playability heuristics for
mobile games, September 2006, In Proceedings of the 8th
conference on Human-computer interaction with mobile devices
and services (pp. 9-16). ACM.
Pinelle, D., Wong, N., &Stach, T., Heuristic evaluation for
games: usability principles for video game design. In
Proceedings of the SIGCHI Conference on Human Factors in
Computing Systems (pp. 1453-1462). ACM,2008
Baauw, E., Bekker, M. a, &Barendregt, W., A structured expert
evaluation method for the evaluation of childrens computer
games..Human-Computer Interaction - Interact 2005,
Proceedings, 3585, 457469,2005
Squires, D., &Preece, J. , Usability and learning: Evaluating the
potential of educational software. Computers and Education,
27(1), 1522. doi:10.1016/0360-1315(96)00010-3, 1996
Reeves, T. C., Benson, L., Elliott, D., Grant, M., Holschuh, D.,
Kim, B., Loh, S., Usability and Instructional Design Heuristics
for E-Learning Evaluation,2002
De Angeli, a., Matera, M., Costabile, M. F., Garzotto, F.,
&Paolini, P., On the Advantages of a Systematic Inspection for
Evaluating Hypermedia Usability,2003, International Journal
of
Human-Computer
Interaction,
15(3),
315335.
doi:10.1207/S15327590IJHC1503_01