Natural Gestures for Multi-Touch Applications
Elizabeth Starnes Paul Varcholik
University of Central Florida
Orlando, Florida
lizzystarnes@gmail.com pvarchol@bespokesoftware.org
Charles Hughes
University of Central Florida
Orlando, Florida
ceh@eecs.ucf.edu
Abstract
Multi-touch interfaces are becoming com-
monplace on smart phones and mouse pads, high-
end monitors and a number of notebook comput-
ers. Unfortunately, most of these commodity sys-
tems have small form factors and support only
two simultaneous touches. Examples of larger,
true multi-touch devices more commonly occur
in research and demonstration settings. The focus
of this project is to investigate a range of multi-
touch gestures (static and dynamic). Observations
were made through a group of students as to what
gestures users find the most “natural” for zoom-
ing, rotating, etc. when interacting with picture
based multi-touch applications. Such data can
help programmers to consider gestures for future
programs based on the idea that users may come
from artistic backgrounds versus technical back-
grounds. Programmers should also consider the
size of the table used with their applications and
whether the majority of the users own handheld
multi-touch devices.
1 Introduction
The way people interact with multi-touch dis-
plays is through gestures. A gesture is a physical
motion that maps to an action in an application.
For example, if a person were to touch an icon to
open a file, the action would be the file opening
on the screen. The pinching gesture for zooming
on an iPhone or other handheld multi-touch de-
vices has become the norm for many people. For
most people who own these devices, when pre-
sented a multi-touch table, they may prefer this
gesture over another. But, the size of the table
University of Central Florida
Orlando, Florida
may change their choice of gestures. For those
who do not own/use any multi-touch device,
what would be their preferred gesture for zoom-
ing or resizing an image? What if the user is not
from a technical background? This paper investi-
gates which gesture would be the most natural
for zooming and rotating an image depending on
the hemispheric dominance of the user and
whether the user owns a multi-touch device.
Most research for gestures in multi-touch is very
specific. The scope of this research is broad
enough to help programmers decide which ges-
tures to include for future applications, especially
those running on non-Windows based GUI inter-
faces.
2 Previous Work
In the 1970’s, Myron Krueger pioneered research
in the areas for augment reality through Videop-
lace. It was a room where people could interact
with the walls by movement and see their sil-
houettes and other objects projected on the wall.
The idea of “pinching” or stretching an image by
a hand gesture was mainly based off his work
(Krueger, Gionfriddo and Hinrichsen 1985).
In the realm of multi-touch, most HCI re-
search pertaining to gestures is specific to certain
topics such as robotics (Micire, Desai et al 2009),
ShadowGuides (Freeman, Benko et al 2009),
hand shapes (Epps, Linchman and Wu 2006),
etc. For example, researchers at the University of
Massachusetts focused on natural gestures relat-
ing to robotic teams. Users were asked to com-
plete a series of tasks with no information on
which gestures to utilize. The users who
owned/used an iPhone tend to use the “pinching”
gesture that is so commonly known for zooming.
Users who played video games also used certain
gestures pertaining to mouse interaction. The
researchers mainly concluded that the well-
known mouse driven interfaces need to carry
over into the multi-touch domain (Micire, Desai
et al 2009).
Koskien et al. (2008) acquired applicable in-
formation on gestures by a user study in Finland.
Their focus was on power control rooms and the
eventual change to multi-touch systems instead
of analog or digital. Since it was such a unique
Figure 1 shows the prototype table for TACTUS. Pic-
study where users did not have tasks associated
ture from Bespoke Software.
with desktops, users did not typically chose ges-
tures specific to the mouse. Users tend to use the The FTIR design was based on and named
multi-touch gestures when there was a task that for the idea of frustrated total internal reflection
required a larger area to initiate. in the field of optics. A diagram of the FTIR
Other examples of research pertaining to design for multi-touch tables is shown in figure
gestures include entail shadows, shapes and user 2. When light travels from one medium (such as
defined gestures. Hilliges et al. (2009) investi- glass and water) there is no angle of refraction
gated depth to gestures by shadows. Their exam- and the light is totally reflected (Han 2005).
ple shows a person picking up a ball by merely
hovering over the table, not by touching the ta-
ble. Epps et al. (2009) explored hand shapes for
interacting with a table top. The majority of their
subjects used index finger-based shapes and the
set up happened to be Windows-oriented for
icons, etc..
Most of the research for gestures involved
specifically asking people what gestures they
prefer; an entire gesture set is created. This paper
mainly focuses on a few common actions as op-
Figure 2 displays the FTIR design proposed by Jeff
posed to an entire gesture set. These actions are
Han, image from (NUI Group Authors 2006).
also general and not geared to a specific genre of
multi-touch applications. In the user study, stu-
In the case of multi-touch tables, infrared
dents were asked to show on the table which ges-
light is projected inside the acrylic table top and
ture to use pertaining to an action such as zoom-
is reflected. When a person touches the top of the
ing or moving an object.
acrylic, some of the infrared light is transfer to
the fingertip and hence the total internal reflec-
3 Hardware tion is frustrated (NUI Group Authors 2006). A
good example of frustrated total internal reflec-
The user study was conducted on the prototype tion in everyday life is fingerprints on a glass of
for TACTUS, a multi-touch research testbed. water. An infrared camera can detect this scatter-
The picture of the table is seen below in figure 1 ing of light, which shows up as a blob. A blob is
(Varcholik, LaViolia and Nicholison 2009). representation of the contact point of the table
TACTUS uses the FTIR (Frustrated Total Inter- and a fingertip to the camera. Relatively simple
nal Reflection) design proposed by Jeff Han in vision-based algorithms can find the coordinates
2005 as an inexpensive way to create a multi- of these blobs based on camera images. User
touch table based on a simple principle of phys- applications, such as the picture manipulation
ics. one for this study, can then be built using the
data available from these algorithms.

4 User Study
Sixteen volunteers from the University of Central
Florida participated in a user study – six females
and ten males, between the ages of 20 and 29.
One volunteer was a graduate student and anoth-
er was graduated. Majors ranged from Digital
Media, Art, Engineering, Computer Science,
Music Education and Event Management. Three
of the participants were left-handed.
The students were placed in front of the
prototype for TACTUS and shown a display that
contained seven pictures. The display contained
no other icons and the GUI of Windows was not
fully shown, except for the toolbar at the bottom.
A screenshot of what the students saw is below
in Figure 3. The users were asked a series of
questions before interacting with the table. They
were asked if they have used or owned an iTouch
or a handheld multi-touch device before and
whether they have interacted with a large multi-
touch display such as a wall, table or computer
screen. 87.5% of users have used a handheld
multi-touch device before and 37.5% owned
such devices. Students were also asked if they
have interacted with a multi-touch device larger
than handheld, such as a monitor, table, wall, etc.
44% of the students described interactions with
larger types of multi-touch devices.
The students were then asked to show on
the multi-touch table the gestures they would use
to perform actions on the pictures shown. They
were asked to zoom in and out, rotate and move
the pictures. They were also asked to clear all the
pictures from their sight, which usually resulted
in a “flicking” motion.
Figure 3 displays the pictures the students were asked
to describe how to manipulate through various ges-
tures.
5 Results
A majority of students used two hands to per-
form the tasks of zooming and rotating images.
For moving or clearing the pictures, one hand
was preferred The students who owned or used a
hand held multi-touch device before stuck to us-
ing two fingers for most of the tasks of zooming
and rotating. This was most likely because they
were accustomed to their handheld multi-touch
devices, though only a fourth of those students
used the pinch gesture with one hand. One stu-
dent commented that, since the table was so
large, he would not use the pinch gesture with
one hand. If the table had been a bit smaller, he
said his first reaction would be to use the one-
handed pinch gesture to zoom.
5.1 Zooming an image
Students came up with typical and different types
of gestures to zoom on an image. One unique
gesture was the four point zoom in which stu-
dents used two fingers on each hand, specifical-
ly. These students commented that the table was
quite large and the use of multiple fingers on
each hand felt necessary to accomplish the task.
A few students also displayed a particular
gesture of double tapping to zoom. This idea
probably came from double clicking on images
for a usual computer interface with a mouse. One
student’s gesture for zooming was to press down
for a period of time (selection) and then to zoom
out they tapped the screen and moved their finger
downwards in a “scrolling” manner. The student
had never used a multi-touch device before and
preferred a button rather than doing a certain
gesture. Another student suggested a double tap
to zoom in and a single tap to zoom out on a
picture. Both of these students were from
artistic/creative disciplines. A more visual expla-
nation of this gesture and the other gestures are
shown in figure 4 below.
Figure 4 shows the gestures the students used to
zoom in and out on a picture. Illustrations reproduced
from Gestureworks.
5.2 Rotating an image
There was a variety of gestures to rotate a pic-
ture. Figure 5 shows the visual representation of
the gestures used for rotating an image. There
were a few forms of two point gestures, such as
the pivot and rotate. One finger would stay sta-
tionary on the picture while the other hand/finger
rotated around the pivot finger. There was no
feedback given for students to persuade them to
use this gesture, but it happened to be the gesture
recognized by the program, though that recogni-
tion was turned off while the students performed
their gestures. The other form of the two point
gestures (two hands) was moving both points
instead of keeping one as a pivot point. These
points would move in opposite directions and in
an arch motion.
The art students again came up with inter-
esting gestures such as using a lasso (one point in
a circular fashion) to rotate a picture. Such a ges-
ture has mainly been seen to group images or
select multiple images. This data was seen in the
study for controlling robot teams (Micire, Desai
et al 2009). Another student suggested tapping
on an image, then moving their finger in a way
like the lasso. The second gesture choice was
five points on one hand and merely rotating the
hand left or right depicting which way the picture
would rotate.
Figure 5 shows the students gestures for rotating an
image. Illustrations reproduced from Gestureworks.
5.3 Moving and Clearing images
The students were asked to show the gesture for
how to move a picture and typically they would
use one or a few points. Figure 6 below displays
the gestures described for moving and clearing
pictures. Surprisingly, some students used two
points specifically to move an image, though
they said up to five points should be recognized.
One student emulated a mouse by tapping to se-
lect an image and dragging it with one finger.
The gesture for clearing the images was al-
so asked and needed to be clarified as to whether
it pertained to one image as opposed to all of the
images shown. Also, the subjects were confused
as to where the pictures would end up when they
were cleared and they wanted a place to throw
them into instead of using just the screen as the
reference. Students also came up with a five
point flick, which was described either moving
left or right.
Figure 6 shows the students’ gestures for clearing and
moving pictures. Illustrations reproduced from Ges-
tureworks.
6 Picture Application
After the students were asked the questions for
the user study, they were allowed to interact with
the picture application that was created for the
project. The picture application was initially
supposed to be the user study in which students
would only interact with the program and their
actions observed. There were many issues with
the application regarding hardware and software.
The students were able to zoom with the pinch
gesture, two point/two hand gesture and a ten
point hand gesture (same idea as two point/two
hand but with five points for each hand). They
were also able to rotate pictures using the pivot
and rotate hand gesture. When the program was
having issues, subjects tended to stick with one
handed gestures to compensate for the difficulty
with the table. We obtained a lot of feedback that
will be useful in future experiments.
7 Discussion
The data found from the students showed that
programmers should consider certain demo-
graphics and whether the majority of users own a
multi-touch device. Users may come from differ-
ent disciplines such as an artistic background as
opposed to a technical one. Users may also have
never interacted with a handheld multi-touch de-
vice or have never owned one before. The stu-
dents from artistic backgrounds came up with
very unique gestures to perform typical tasks.
Also, the students that have not owned a hand-
held multi-touch may use variations of typical
gestures for the same tasks. The zoom gesture for
an iTouch, iPad, etc. did not prove to be the most
natural gesture on a large surface like a multi-
touch table. Students commented that if the sur-
face was smaller they would use the pinch ges-
ture, but instead an adaptation of using two
hands came out of the study because of the size
of the table.
7.1 Zooming
The students who did not own a multi-touch
device still used typical zooming gestures. The
graph in figure 7 shows the gestures used by
students who do not own a handheld multi-touch
device. Only 12.5% of students who do not own
multi-touch devices used the pinch gesture. Most
of these students expressed an adaptation of the
pinch by using the two point/two handed gesture.
They commented that, if the surface was smaller,
they would use the pinch gesture. The students
from academic disciplines such as Digital Media
or Art came up with the unique gestures of press-
ing down, single tap, etc. People from localities
that are more art-centered may have a natural
inclination to some creative gestures for tasks,
especially when manipulating pictures. Pro-
grammers may want to consider these demo-
graphics when considering where the multi-touch
display will be located.
Figure 7 shows the number of students who do not
own multi-touch devices and what gestures they ex-
pressed to rotate an image.
7.2 Rotating
Since most multi-touch devices use other means
to rotate images such as, for most handheld mul-
ti-touch devices, a user literally rotates the device
to rotate an image. This may be where the rota-
tion gestures come from, rather than being based
on some natural or normative gesture. 44% of
students showed a two point/two hand gesture to
rotate a picture that involved both points moving.
They specifically requested to rotate pictures at
the corners rather than the middle with this ges-
ture. Figure 8 shows a graphical representation of
the number of students who chose certain ges-
tures to perform the action. The second most
popular gesture was the five point gesture, fol-
lowed by the pivot gesture and lasso.
Figure 8 shows the number of students who expressed
certain rotation gestures.
7.3 Moving and clearing pictures
The most used gesture for moving was simply
using one point. This concept was most likely
intuitive because one point relates to a mouse
cursor simply moving across the screen. There
were interesting results for clearing pictures off
the screen. The usual gesture for doing such a
task is using a one point flick because of the
handheld multi-touch devices, but the majority of
students used five points. In particular, half of
the students who own a handheld multi-touch
device used five points to flick an image.
8 Conclusion
Students from the University of Central Florida
were asked what gestures they would use for cer-
tain tasks when faced with a multi-touch picture
manipulation application. The data found by this
study can potentially help programmers choose
appropriate gestures based on demographics such
as if the users don’t own multi-touch devices and
if the users are artistically inclined. Users with
technical backgrounds versus artistic background
will most likely interact with an application dif-
ferently.
Acknowledgments
Partial support for this work was provided by the
National Science Foundation Scholarships in
Science, Technology, Engineering and Mathe-
matics (S-STEM) program under award No.
0806931 and grant No. 0638977. Any opinions,
findings and conclusions or recommendation
expressed in this material are those of the authors
an do not necessarily reflect the views of the Na-
tional Science Foundation. Partial support for
this work was also provided by Workforce Cen-
tral Florida and the UCF Office of Research and
Commercialization.
References
1. BespokeSoftware,
http://www.bespokesoftware.org
2. Dang, C. T., Straub, M., and Andre, E.
(2009). Hand Distinction for Multi-Touch
Tabletop Interaction. In Proceedings of the ACM
international Conference on interactive Table-
tops and Surfaces (Banff, Alberta, Canada, No-
vember 23 - 25, 2009). ITS '09. ACM, New
York, NY, 101-108.
3. Epps, J., Lichman, S., and Wu, M. 2006. A
study of hand shape use in tabletop gesture inte-
raction. In CHI '06 Extended Abstracts on Hu-
man Factors in Computing Systems (Montréal,
Québec, Canada, April 22 - 27, 2006). CHI '06.
ACM, New York, NY, 748-753.
4. Freeman, D., Benko, H., Morris, M. R., and
Wigdor, D. 2009. ShadowGuides: visualizations
for in-situ learning of multi-touch and whole-
hand gestures. In Proceedings of the ACM inter-
national Conference on interactive Tabletops
and Surfaces (Banff, Alberta, Canada, November
23 - 25, 2009). ITS '09. ACM, New York, NY,
165-172.
5. Gestureworks, http://www.gestureworks.com
6. Han, J. Y. 2005. Low-cost multi-touch sens-
ing through frustrated total internal reflection. In
Proceedings of the 18th Annual ACM Symposium
on User interface Software and Technology
(Seattle, WA, USA, October 23 - 26, 2005).
UIST '05. ACM, New York, NY, 115-118.
7. Hilliges, O., Izadi, S., Wilson, A. D., Hodges,
S., Garcia-Mendoza, A., and Butz, A. 2009. Inte-
ractions in the air: adding further depth to inter-
active tabletops. In Proceedings of the 22nd An-
nual ACM Symposium on User interface Soft-
ware and Technology (Victoria, BC, Canada,
October 04 - 07, 2009). UIST '09. ACM, New
York, NY, 139-148.
8. Koskinen, H. M., Laarni, J. O., and Honka-
maa, P. M. 2008. Hands-on the process control:
users preferences and associations on hand
movements. In CHI '08 Extended Abstracts on
Human Factors in Computing Systems (Florence,
Italy, April 05 - 10, 2008). CHI '08. ACM, New
York, NY, 3063-3068.
9. Krueger, M. W., Gionfriddo, T., and Hinrich-
sen, K. 1985. VIDEOPLACE—an artificial reali-
ty. In Proceedings of the SIGCHI Conference on
Human Factors in Computing Systems (San
Francisco, California, United States). CHI '85.
ACM, New York, NY, 35-40.
10. Micire, M., Desai, M., Courtemanche, A.,
Tsui, K. M., and Yanco, H. A. 2009. Analysis of
natural gestures for controlling robot teams on
multi-touch tabletop surfaces. In Proceedings of
the ACM international Conference on interactive
Tabletops and Surfaces (Banff, Alberta, Canada,
November 23 - 25, 2009). ITS '09. ACM, New
York, NY, 41-48.
11. NUI Group Authors. (2006). Frustrated
Total Internal Reflection(FTIR). Retrieved May
26, 2010, from NUI Group Wiki:
http://wiki.nuigroup.com/
12. Varcholik, P., LaViola, J., Nicholson, D.
(2009). TACTUS: A Hardware and Software
Testbed for Research in Multi-Touch Interaction.
Proceedings of the 13th International Confe-
rence on Human-Computer Interaction (HCI
International 2009).