Professional Documents
Culture Documents
Applied Ergonomics
journal homepage: www.elsevier.com/locate/apergo
a r t i c l e i n f o a b s t r a c t
Article history: This study aimed to quantify differences in 7 touchscreen gestures. Eighteen participants performed
Received 12 August 2015 index nger tapping, sliding in 4 orthogonal directions, and index nger and thumb pinch and stretch
Received in revised form gestures on a touchscreen tablet computer. We hypothesized that two nger gestures would require
25 May 2016
longer task completion time and greater nger joint excursions than sliding gestures using only the index
Accepted 13 June 2016
Available online 29 June 2016
nger. We measured task completion times and nger joint kinematics. Tapping showed the fastest
average (SD) task completion time, 567(190) ms, of all gestures (p < 0.001). Pinch had faster task
completion time, 765(277) ms, than all single-nger sliding gestures (p < 0.001). Stretch was faster to
Keywords:
Mobile technology
complete at 843(317) ms (p < 0.001) than all sliding gestures except slide right. Stretch demonstrated
Pinch greater mean index nger metacarpophalangeal exion/extension joint excursions, 63(16) , compared to
Thumb sliding gestures, 34(10) , and tapping, 27(13) (p < 0.01). Overall, two-nger gestures were faster to
complete and showed greater joint excursions than single-nger sliding gestures.
2016 Elsevier Ltd. All rights reserved.
http://dx.doi.org/10.1016/j.apergo.2016.06.004
0003-6870/ 2016 Elsevier Ltd. All rights reserved.
D.S. Asakawa et al. / Applied Ergonomics 58 (2017) 176e181 177
only the index nger. Finally, we hypothesized that there would be touchscreen with the index nger (right, left, down, and up), pinch
differences among the four different directions of index nger (pinching the index nger and thumb together as used for zoom
sliding when comparing measured parameters. out) and stretch (spreading the index nger and thumb apart as is
used for zoom in). The 6-cm square buttons were centered in the
middle of the touchscreen. Completion of a slide right gesture, for
2. Methods example, started on the left edge of the button, 3 cm left of the
center of the screen, and ended on the right edge of the button
We recruited eighteen unimpaired right-handed participants (9 which was 3 cm to the right of the center of the screen. The pinch
female, 9 male; age 19e42 years) for the study. Subjects were and stretch gestures were completed with the index nger and
recruited by word of mouth. No subject had a current upper limb thumb moving along a line that was at a 45 angle from the long
injury or history of repetitive strain injury of the wrist, hand or edge of the tablet computer. The participants completed all eleven
ngers. All subjects had previously used touchscreen computing repetitions of each gesture before moving on to the next gesture to
technology. The participants average (SD) height was minimize motion of the upper limb joints between repetitions.
1.67 0.08 m and average mass was 65.20 8.63 kg. For all par- Participants were verbally instructed to complete each gesture as
ticipants we measured hand length as the distance from the base of quickly as possible to make certain there were similar experimental
the palm to the tip of the third digit and hand width as the distance conditions among subjects. No participant reported or was
from the radial side of the second metacarpophalangeal joint to the observed to have any fatigue after completing the study gestures.
ulnar side of the fth metacarpophalangeal joint. Hand length and The custom Android software application recorded task completion
hand width were on average 18.0 1.1 cm and 8.6 0.8 cm time. Task completion times were measured bare-handed to ensure
respectively. We obtained informed written consent from all par- the experimental measurements were as close to real touchscreen
ticipants using forms and protocols approved by California State use as possible.
University San Marcos Internal Review Board. Participants performed all gestures in two conditions: 1) bare-
All gestures were completed on a Samsung Galaxy Tab 2 10.1- handed and 2) wearing a CyberGlove III 18-sensor system (Cyber-
inch touchscreen tablet computer (Samsung Group, Seoul, South Glove Systems LLC, San Jose, CA). The CyberGlove III is reported by
Korea). The touchscreen tablet measured 10.10 6.90 0.38 inches the manufacturer to have sensor resolution of <1 and sensor
and weighed 1.28 lb. A custom software application written for the repeatability (average variability between wearings) of 3 . The
Android platform displayed 6-cm square buttons for each gesture CyberGlove recorded the joint angles of nger and thumb joints at a
(Fig. 1). The 6-cm button size was chosen to represent a standard sampling rate of 30 Hz throughout all gestures. Specically, we
size slide or stretch commonly used for book reader or map compared joint angle data for the index nger distal interphalan-
touchscreen software applications. We used the same size button geal (DIP), proximal interphalangeal (PIP) and meta-
for all gestures to facilitate comparison among gestures. Partici- carpophalangeal (MCP) joints and the thumb interphalangeal (IP),
pants were seated with the touchscreen mobile computing device metacarpophalangeal (MCP), and carpometacarpal (CMC) joints.
placed in front of them in the landscape orientation at on a table of The CyberGlove sensor system was calibrated for each subject by
height 0.91 m. The long edge of the tablet computing device was having them rst lay their hand at on the table surface with the
placed parallel to the edge of the table at a distance of 0.1 m. Par- ngers together and the thumb against the hand. This position
ticipants were not allowed to adjust the position of the touchscreen established 0 of exion/extension of the nger and thumb joints.
tablet on the table. We also ensured that participants did not rest Additional calibration was then completed for the index nger and
any part of their hand or wrist on the tablet computer or table. The thumb joint angles as required by the CyberGlove software by
participants completed sets of 11 gestures, presented in random having the subject touch the tips of the nger and thumb together
order, for a total of 231 gestures. The gestures were: tapping with (the OK sign). All nger angles for the thumb and index nger were
the index nger, sliding in four orthogonal directions on the checked with a goniometer to ensure the joint angles were accurate
and that the thumb and index nger tips were touching. We then
had all subjects perform 90 of index nger MCP joint exion as
measured with a goniometer and compared the value to the
CyberGlove joint angle reading to determine accuracy. Calibration
was repeated if necessary to ensure accurate joint angle measure-
ments. The set of all gestures was completed twice, once bare-
handed and once with the subject wearing the CyberGlove. A
counter-balance design was used such that some participants
started with the bare-handed condition and others started with
wearing the CyberGlove. Subjects with hand length less than
17.0 cm were not able to actuate the gesture buttons while wearing
the CyberGlove joint angle sensors. Therefore, we successfully
measured nger joint angles for 14 of the 18 participants.
Statistical analysis of data including analysis of variance
(ANOVA) and post-hoc tests were completed using SPSS Statistics
software (IBM Inc., New York, NY). We performed mixed-effects
ANOVA, with participant as the between-subject factor and
gesture as the within-subject factor. To determine signicant dif-
ferences, we used Bonferroni and Tukeys post hoc tests for equal
Fig. 1. Experimental apparatus. Participants were seated with the tablet computing variances, and Games-Howell post hoc test for measurements that
device placed on the table in front of them. Each participant actuated 6-cm square demonstrated unequal variances or unequal number of successfully
buttons displayed with custom software for each gesture type while wearing the
CyberGlove sensor to measure nger joint angles (task completion times were
completed gestures.
measured during barehanded gesture completion). Arrows on the buttons illustrated
the direction of nger movement for each gesture.
178 D.S. Asakawa et al. / Applied Ergonomics 58 (2017) 176e181
3. Results more than PIP excursions during two-nger gestures. Across all
gestures, the DIP joint moved a maximum of 3 5 , and DIP joint
3.1. Index nger tapping demonstrated faster task completion times movement was not signicantly different among gestures. Because
than other gestures studied the index nger MCP joint exhibited larger excursions than the PIP
joints, we focus our comparison of gestures to MCP joint
Mean (standard deviation) task completion time for index movement.
nger tapping was 567 190 m (Fig. 2). Tapping with the index MCP joint excursions were different among gestures. The
nger had a signicantly faster task completion time than all other maximum index nger MCP joint exion, 66.3 8.6 , and greatest
gestures (p < 0.001). Therefore, the data support our performance excursion of the index nger MCP joint, 63.2 16.0 , were recorded
hypothesis as tapping was the quickest gesture to complete. for the two-nger stretch gesture (Table 1). Index nger MCP joint
excursion during stretch was 86%e177% greater than MCP joint
3.2. Two nger gestures had faster task completion times than excursion during single-nger gestures. Stretch had signicantly
single-nger sliding gestures greater MCP joint excursion when compared to all single-nger
gestures (p < 0.01; Table 1). For pinch, excursion of the MCP joint
The two-nger pinch gesture, with completion time of was signicantly greater than tap, slide right and slide up (p < 0.01;
765 277 ms, had signicantly faster completion times than all Table 1). However, stretch joint excursion was also signicantly
four directions of single-nger sliding gestures (p < 0.001; Fig. 2). different than pinch joint excursion. Thus, these data support the
Moreover, even though the two-nger stretch gesture was signi- hypothesis that two-nger gestures require greater joint excursion
cantly slower at 843 317 ms than two-nger pinch (p < 0.001), than single-nger gestures.
two-nger stretch was signicantly faster than slide left
(945 276 ms, p < 0.01), slide down, 948 257 ms, and slide up 3.4. Slide direction affected task completion time
(998 341 ms, p < 0.001). We did not observe signicant differ-
ences between two-nger stretch and one-nger slide right. Based Sliding to the right with a task completion time of 880 199 ms,
on the majority of comparisons, we reject the hypothesis that two- was signicantly faster than slide left, down, and slide up (p < 0.01;
nger gestures take longer to complete than single-nger sliding Fig. 2). Neither MCP nor PIP joint excursions were signicantly
gestures. different among the four directions of sliding (Table 1). Overall, our
data support the hypothesis that single-nger sliding gestures in
3.3. Gestures using two ngers involved greater nger joint different directions differ in completion time, but not in overall
excursions than single-nger gestures joint excursions.
Joint excursions measured for the index nger and thumb 4. Discussion
support the hypothesis that gestures using two ngers, such as
pinch and stretch, involve greater joint excursions than single The purpose of this study was to test the hypothesis that one-
nger gestures (Table 1). However, comparisons within gestures and two-nger touchscreen gestures differ in joint movement
showed differences in excursions among the three nger joints extent and completion time on the tablet surface. Specically, we
during gestures. For example, index nger MCP excursions were compared two-nger gestures, pinch and stretch, to one-nger
44%e148% more than PIP excursions during single-nger gestures tapping and sliding in four directions. This study contributes to
(Table 1). Similarly, index nger MCP excursions were 85%e234% the understanding of gestural touchscreen interactions by
Fig. 2. Average task completion time (ms) for barehanded gestures. Errors bars represent one standard deviation. Single-nger gestures were completed with the index nger. Two-
nger gestures were completed with the index nger and thumb. Post hoc testing results are denoted with letters for signicant difference: t for tap (p < 0.001), r for slide right
(p < 0.01), l for slide left (p < 0.01), d for slide down (p < 0.05), u for slide up (p < 0.05), and p for pinch (p < 0.001).
D.S. Asakawa et al. / Applied Ergonomics 58 (2017) 176e181 179
Table 1
Finger joint angles in degrees for each gesture averaged for 14 participants. Joint angle minimum (min) and maximum (max) exion/extension (F/E) value (and standard
deviation) and joint excursion (EXC) are reported for the index nger PIP, and MCP joints, and the thumb IP, MCP, and CMC joints. Abduction/adduction (AB/AD) of the thumb
CMC is also reported. Negative (positive) values indicate joint exion (extension). All joint angles are relative to the hand at position used for calibration. Signicant dif-
ferences for post hoc comparisons in index nger MCP joint excursions are indicated by superscripts.
task completion time than tapping (Kim and Song, 2014). However, technology is in its infancy (Dennerlein, 2015). Quantitative mea-
we found index nger tapping to have signicantly faster task sures of touchscreen performance and kinematics will contribute to
completion time than all other gestures studied. Our results conict the development of recommendations for best practices. Quanti-
with data reporting that less time was required to zoom a map tative joint angle data may guide touchscreen software designers to
using stretch and pinch gestures compared to tapping touch but- select gestures that reduce joint excursions. Also, task completion
tons for in-vehicle navigation systems (Kim and Song, 2014). The times could be used in design to aid selection of gestures that
conict between our ndings and those of Kim and Song (2014) minimize muscle activity and exposure to force. Quantitative
could in part be due to differences in gesture tasks, visual tasks, measures can thereby possibly be used to reduce the amount of
and the experimental set up between studies. For the in-vehicle strain on the musculoskeletal system.
touchscreen tasks used by Kim and Song (2014), the touchscreen Posture and muscle activity are important for predicting
was angled at 75 and was placed to the right of the subject at a musculoskeletal complaints and disorders (Dennerlein, 2015).
distance of 700 mm. In our study, the touchscreen was placed Quantitative measures of joint angles during touchscreen gestures
directly in front of the participant and the only visual and gesture allows us to evaluate gestures such as stretch that have greater
task was gesture completion. Moreover, all gestures were motion of the MCP joint and could possibly contribute to greater
completed with same size gesture buttons so that we could control wear on the joint. Software interfaces that require fewer pinch and
the gesture size and compare among like-size gestures. The set up stretch gestures or smaller pinch and stretch gestures may poten-
and task differed from the Kim and Song (2014) study in which tially reduce strain on the index nger and thumb joints. Gestures
participants had to look to the right, reach, and complete a series of such taps and slides may reduce musculoskeletal strain as the joint
gestures for which an average unit operation time per gesture was postures of the ngers are more neutral, avoiding excessive exion
calculated. Therefore, the specic tablet computing device place- or extension of the nger joints.
ment, gesture task, and different calculation of gesture completion Gestures that can be completed more quickly may increase ease
time may inuence two-ngered gesture completion time results. of use. For example, using taps could improve interface perfor-
Task completion times may be dependent on the distance mance relative to using sliding gestures, pinch or stretch. Based on
traveled by the ngertip when evaluating gestures for use in our results, we hypothesize that tapping could reduce joint excur-
touchscreen software applications (Kim and Song, 2014). For sion at the index nger MCP joint relative to pinch or stretch.
example, in this study tap was the fastest gesture to complete, as Tapping a single button could be used to zoom in or zoom out (Kim
expected. The tap gesture does not require the nger to travel a and Song, 2014). Other tasks such as page turning on a touchscreen
distance over the touchscreen as do sliding, pinch, and stretch book reader can also be accomplished with taps rather slides, and
gestures. Two-nger pinch gestures may have demonstrated could potentially reduce strain on the musculoskeletal system and
shorter task completion times compared to other gestures because provide greater ease of use.
of a shorter distance traveled to complete a pinch or stretch In conclusion, we found that one- and two-ngered gestures
compared to sliding gestures. The two-nger gestures were differed in several respects. Tapping with the index nger had
completed on the diagonal of the 6-cm square button; each nger signicantly faster task completion time than all other gestures.
traversed 4.2 cm to successfully complete the gesture task. In the However, participants could complete the two-ngered pinch
sliding gestures, the entire 6-cm length of the button was traversed gesture more quickly than index nger sliding, despite the greater
in each gesture. Hence, the distance the index nger and thumb joint excursion needed for two-ngered gestures. The two-nger
traveled for sliding was 30% greater than the distance traveled by stretch gesture demonstrated greater joint excursion at the MCP
each of the ngers for pinch and stretch. We chose the 6-cm button joint than all other gestures studied. In addition, comparison
size to represent a typical slide to turn a page on a book reader among the four directions of sliding gestures showed differences in
touchscreen software application. However, completion times for a task completion time. This study provides a rst step to comparing
6-cm square button may not represent real usage conditions in index nger tapping, sliding and two-nger gestures used to zoom
which distance traveled, task completion times for gestures and an image in or out. Quantitative information about touchscreen
factors such as posture are more variable (Dennerlein, 2015; Gold gestures could be useful for developing further understanding of
et al., 2012). The most proximal joints of the thumb and index risk factors for repetitive touchscreen use and potential strategies
nger experienced the greatest joint angle excursion during for reducing musculoskeletal injury. Also, the quantitative com-
gesturing. Pinch and stretch gestures involved the most motion of parisons of nger joint kinematics, and completion times for ges-
the thumb CMC and index nger MCP joint. Chronic wear of the tures could help aid in the creation of guidelines for best
thumb CMC joint has been linked to incidence of osteoarthritis touchscreen use practices and design of touchscreen software in-
(Jonsson et al., 2011; Ming et al., 2006; Ugbolue et al., 2005). Based terfaces that incorporate performance and joint kinematic consid-
simply on joint excursion measures, the pinch and stretch gestures erations to benet users.
could potentially increase exposure to motion and consequently
increase wear on proximal joints of the index nger and thumb Acknowledgements
compared to the single-nger gestures studied. Furthermore, the
nding that the index nger DIP joint exes a maximum of 3 This work is funded by the National Science Foundation
during the gestures studied along with the joint excursion com- (0964220). We gratefully thank Alejah Tabula for her work on this
parisons presented could help with hand surgery planning, such as project. We thank Cecil Lozano for her preliminary work that hel-
determining joint angle for thumb joint arthrodesis, for those who ped to shape our hypotheses.
use smartphones and touchscreens regularly (Yao et al., 2012).
This study contributes several ndings with possible practical
References
importance to touchscreen interface design. There is great vari-
ability of postures when using touchscreen mobile computing de- Berolo, S., Wells, R.P., Amick 3rd, B.C., 2011. Musculoskeletal symptoms among
vices (Ciccarelli et al., 2015, Gold et al., 2012). Moreover, there is mobile hand-held device users and their relationship to device use: a pre-
currently limited knowledge of important factors such as muscle liminary study in a Canadian university population. Appl. Ergon. 42, 371e378.
Buf, J.H., Sancho Bru, J.L., Crisco, J.J., Murray, W.M., 2014. Evaluation of hand motion
activity for even the most common postures (Kietrys et al., 2015). capture protocol using static computed tomography images: application to an
Therefore, developing best practices for mobile computing instrumented glove. J. Biomech. Eng. 136, 124501.
D.S. Asakawa et al. / Applied Ergonomics 58 (2017) 176e181 181
Ciccarelli, M., Chen, J.D., Vaz, S., Cordier, R., Falkmer, T., 2015. Managing childrens nninen, O., 2006. Excessive texting in pathophysiology of
Ming, Z., Pietikainen, S., Ha
postural risk when using mobile technology at home: challenges and strategies. rst carpometacarpal joint arthritis. Pathophysiology 13, 269e270.
Appl. Ergon. 51, 189e198. Nacher, V., Jaen, J., Navarro, E., Catala, A., Gonz
alez, P., 2015. Multi-touch gestures for
Dennerlein, J.T., 2015. The state of ergonomics for mobile computing technology. pre-kindergarten children. Int. J. Hum. Computer Stud. 73, 37e51.
Work 52, 269e277. Pereira, A., Miller, T., Huang, Y., Odell, D., Rempel, D., 2013. Holding a tablet com-
Duggan, M., Rainie, L., 2012. Cell Phone Activities Pew Research Centers Internet & puter with one hand: effect of tablet design features on biomechanics and
American Life Project, p. 20. subjective usability among users with small hands. Ergonomics 56 (9),
Fontana, L., Neel, S., Claise, J.-M., Ughetto, S., Catilina, P., 2007. Osteoarthritis of the 1363e1375.
thumb carpometacarpal joint in women and occupational risk factors: a case- Trudeau, M.B., Asakawa, D.S., Jindrich, D.L., Dennerlein, J.T., 2016. Two-handed grip
econtrol study. J. Hand Surg. 32, 459e465. on a mobile phone affords greater thumb motor performance, decreased vari-
Gehrmann, S.V., Tang, J., Li, Z.M., Goitz, R.J., Windolf, J., Kaufmann, R.A., 2010. Mo- ability, and a more extended thumb posture than a one-handed grip. Appl.
tion decit of the thumb in CMC joint arthritis. J. Hand Surg. Am. 35, Ergon. 52, 24e28.
1449e1453. Trudeau, M.B., Catalano, P.J., Jindrich, D.L., Dennerlein, J.T., 2013. Tablet keyboard
Gold, J.E., Driban, J.B., Thomas, N., Chakravarty, T., Channell, V., Komaroff, E., 2012. conguration affects performance, discomfort and task difculty for thumb
Postures, typing strategies, and gender differences in mobile device usage: an typing in a two-handed grip. PLoS One 8, e67525.
observational study. Appl. Ergon. 43, 408e412. Trudeau, M.B., Udtamadilok, T., Karlson, A.K., Dennerlein, J.T., 2011. Thumb motor
Gustafsson, E., Johnson, P.W., Hagberg, M., 2010. Thumb postures and physical loads performance varies by movement orientation, direction, and device size during
during mobile phone use e a comparison of young adults with and without single-handed mobile phone use. Hum. Factors J. Hum. Factors Ergon. Soc. 54,
musculoskeletal symptoms. J. Electromyogr. Kinesiol. 20, 127e135. 52e59.
Jonsson, P., Johnson, P.W., Hagberg, M., Forsman, M., 2011. Thumb joint movement Trudeau, M.B., Young, J.G., Jindrich, D.L., Dennerlein, J.T., 2012. Thumb motor per-
and muscular activity during mobile phone texting e a methodological study. formance varies with thumb and wrist posture during single-handed mobile
J. Electromyogr. Kinesiol. 21, 363e370. phone use. J. Biomech. 45, 2349e2354.
Kietrys, D.M., Gerg, M.J., Dropkin, J., Gold, J.E., 2015. Mobile input device type, Ugbolue, U.C., Hsu, W.-H., Goitz, R.J., Li, Z.-M., 2005. Tendon and nerve displacement
texting style and screen size inuence upper extremity and trapezius muscle at the wrist during nger movements. Clin. Biomech. 20, 50e56.
activity, and cervical posture while texting. Appl. Ergon. 50, 98e104. Yao, J., Park, M.J., Davis, D., Chang, J., 2012. Ideal position for thumb inter-
Kim, H., Song, H., 2014. Evaluation of the safety and usability of touch gestures in phanlangeal arthodesis in the era of smartphones and text communication.
operating in-vehicle information systems with visual occlusion. Appl. Ergon. 45, Orthopedics 35, 3.
789e798. Young, J.G., Trudeau, M.B., Odell, D., Marinelli, K., Dennerlein, J.T., 2013. Wrist and
Kim, J.H., Aulck, L., Bartha, M.C., Harper, C.A., Johnson, P.W., 2014. Differences in shoulder posture and muscle activity during touch-screen tablet use: effects of
typing forces, muscle activity, comfort, and typing performance among virtual, usage conguration, tablet type, and interacting hand. Work 45, 59e71.
notebook, and desktop keyboards. Appl. Ergon. 45, 1406e1413.