Using eye tracking to apply focus detection to an adaptive

game controller

Blind for review

ABSTRACT
In the gaming experience, players are used to intuitive interfaces
that allow them to jump straight to the entertainment. Modern
joysticks are physical hardware components with a fixed layout,
being the main interface for a large variety of games with different
control methods and needs that will be played by users that also
present different ergonomic needs and preferences. This work
proposes a new interface, based on a touchscreen device. We
present a gamepad concept capable of adapting itself dynamically
to a user according to its touch and attention focus patterns, trying
to avoid errors and provide a more comfortable experience. We
also present the results of our usability tests, with both objective
and subjective evaluations and the discussion about our findings.

CCS Concepts
Human-centered computing Human computer interaction
(HCI) Interaction devices Touch screens
Human-centered computing Ubiquitous and mobile
computing Ubiquitous and mobile devices Mobile
devices

Keywords
Adaptive interfaces; adaptive game control; game input; eye
tracking; focus detection.

1. INTRODUCTION
Video games are one of the main entertainment fields nowadays.
They are composed of many elements, such as gameplay, audio,
graphics and narrative. When these factors are well performed and
combined, the game may produce engagement and immersion to
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. To copy
otherwise, or republish, to post on servers or to redistribute to lists,
requires prior specific permission and/or a fee.
SAC16, April 4-8, 2016, Pisa, Italy.
Copyright 2016 ACM 978-1-4503-3739-7/16/04$15.00.

http://dx.doi.org/xx.xxxx/xxxxxxx.xxxxxxx

the players. Nowadays a special attention has being given to the

game interaction, which is an important element for the
implementation of the gameplay [7] [3] and rules [11]. Due to this
fact, the controller and the control scheme must match user
expectations or even exceed them [12]. To achieve and enhance
the game interaction, new ways of inputs and devices has been
proposed [ANONYMOUS].
These inputs must be intuitive, provide a deep interaction, where
sometimes the user can map the action in the real world to the
game. A successfully control scheme must improve the gaming
experience. On the other hand, a game that does not have an
intuitive control and makes the user uneasy when playing may
decrease the overall entertainment process.
To provide a better user experience, we proposed in past works
the use of an adaptive controller using a mobile device to create a
dynamic touchscreen interface. Our controller can present any
layout, allowing game designer to project not only their games,
but the own joystick used to play it. Currently, games are played
using physical joysticks that present a generic button
configuration and layout, with a fixed amount and position for its
buttons. Our work aims to remove this constraint, allowing each
game to use a controller layout that best fits its needs. In
following works we proposed the dynamic improvement of the
custom interface based on machine learning techniques and data
captured from the users touches on the screen. In this work we
improve our proposal including attention focus elements based on
eye tracking approaches in order to best fit the ergonomic needs
and the current moment in the game.

2. RELATED WORK
This work presents the use of a mobile device as an adaptive
controller to games. Actually there are many different kinds of
controllers. To name few of them, we can mention gamepads,
keyboards, steering wheel and mobile devices. The study in [4]
compared the usability, user experience, functionality and the
design of some controllers but they did not compare mobile
devices as inputs neither adaptive devices. Mobile phones have
specific hardware (camera, accelerometer, GPS, Bluetooth and so
on) with lots of them being different from the ones found in
traditional game platforms, like video games and PCs. For this

reason, these devices bring new forms of user interaction,

however with the drawback of lacking tactile feedback [6].
Adaptive user interfaces are interactive software systems that
improve its ability to interact with a user based on partial
experience with that user [8]. Some of them will be detailed in
this section to provide a useful background for this work.
Mobile phone touch screen devices are very common. Most
devices that have such features possess few buttons, with almost
all input made by touch. Hence, mobile touchscreen games must
be designed to accept most of their input by touch that will be
used in games in a similar way to mouse clicks on a regular
computer allowing developers to make different types of mobile
games based on virtual buttons. Consequently, the screen can
draw buttons and use it to simulate button input.
In previous work [ANONYMOUS] the authors developed a
controller that adapt itself according to the users touch press. The
statistical analysis showed strong evidences that the adaptive
controller improved the accuracy of the users.
This work introduces new heuristics for controller adaptation
based on eye tracking that recognizes where the user is looking
and then decides if the adaptation is necessary. The use of eye
tracking to estimate users gaze location is present in several
works, such as [9]. It also has being used on games, mainly as a
control method, a technique used on [14] and to evaluate the
users focus in gaming interfaces [15]. In our case, we will track
the users focus to dynamically evaluate the best parameters for
our adaptive interface.

providing the best experience for the individual. Besides that, the
game itself is not constant: the challenges, level design and even
control options will change as the player progresses on it. To
create a dynamic interface that follows this process, our adaptive
controller constantly improves its interface to better fit both the
player and the current moment in the game. Adaptations may be
triggered by different adaptation causes, such as the context of the
interaction, the experience of the user, or user behavior [2]. The
use of a touchscreen device instead of a physical hardware, like
traditional controller, allows a single device to not only provide
custom interfaces for each game, but to change its own layout so
that it can respond to new requirements in the interaction.
In [ANONYMOUS], the authors developed agents that monitor
the player usage during the gameplay experience, tracking the
users touches and buttons interactions. Based on this data and
using machine learning approaches, the system dynamically
changed and adapted the buttons, in order to better fit the specific
player. In this work we propose a novel approach for the adaption
using eye tracking to perform better adaptations. In our
experiment we will focus in changes to the buttons position and
size, but other changes such as shape and even the way to interact
with a button can be altered, since our controller allows any kind
of interface element, including dragging into the sensible screen.
The communication between the mobile app on the smart device
and the PC that is running the game is performed via network,
using the traditional TCP protocol. Both devices must be in the
same network and an application on the device that is running the
game (PC or console) will be responsible for receiving the
commands from the controller and translate it to local keyboard
events that perform actions on the game. In this work we also
capture eye tracking data and send it to the controller.

4. CONTROLLER ADAPTATIONS BASED

ON THE USER BEHAVIOR
4.1 Adaptation Based on the Users Touches
All user touches on the screen are tracked and kept in an internal
database, which will use the 10 most recent touched points and
the amount of presses on each button for evaluating and perform
adaptations to improve the user experience. For our tests, we
manipulated buttons size and positions.

Figure 1: A user playing a game with the adaptive controller.

3. PROPOSED INTERFACE
Our adaptive controller consists in a mobile application for a
smart device (in our case, an Android smartphone or tablet) that
presents a customizable graphic interface, specifically built to
attend the games needs. The controls are presented in the screen
and the user interacts with it by touching buttons to perform
actions in a game running on a regular PC or game console.
Figure 1 shows the prototype controller in action.
However, the interaction with the controller is not a static
experience. Each user has personal ergonomic needs and
preferences, so that a generic controller is not capable of

The size adaptation is based on the following heuristic: The

amount of touches is stored for each button and the controller
creates an ordered list, where the first button is the one that was
pressed more times while the last one will be the least used
button. One third of the most used buttons will have their size
increased while one third of the less used buttons will decrease in
size. We limited in our tests the button size increase to a
maximum of two times its original size, while the lower bound
was established to its own initial size. This list is constantly
updated, so if a button stops being used frequently, it will
naturally move down the list and decrease in size.
For the position adaptation, it was used the K-means clustering
algorithm [13], an unsupervised machine learning method. The Kmeans receives a set of points and try to cluster them in K classes,
each one containing the corresponding points and a centroid. This
centroid will be the most important result, since it represents the
center of all touches in a class. During the interaction, the user
performs correct touches in the buttons and incorrect touches in

the surrounding areas. With K-means, these touches will be

grouped and the centroid will represent a position that allows the
button to better represent the area where the user is actually trying
to press. Each button will be paired with the closest centroid
found (if a centroid is close to it) and its position will be changed
so that the buttons center matches the centroid.
All these changes are made gradually, with each button changing
position or size by a maximum of constant defined pixels per
iteration. The adaptations also follow some rules and constraints.
Each time a button has to change its size or position, the
controller will verify if this change would make the button invade
another buttons area (an intersection). In positive case, the
change will not be made. However, it is important to note that the
button will grow or move as much as it can without intersecting
with a neighbor and that, if future layout changes moves the
neighbor out of the way, the desired change will now continue.
To better analyze the effect that our improved controller can have
on the general user experience, we used an eye tracking algorithm
to collect information about focus during the gameplay
experience. This data is used as another input to adaptation as will
be better explained in the next section.

4.2 Adaptation Based on Focus Detection

Gaming aims to be an immersive experience. In our study, we will
consider that the ideal controller should not distract the player
from the game and must avoid breaking this immersion. As our
controller relies on a touchscreen and the lack of physical
feedback can make the user miss buttons, a possible immersion
break can be observed. If the user cannot feel the position of the
button, he will have to look to the controller to determine visually
the buttons position. While the K-means algorithm is used to
optimize the buttons layout and correct the interface to avoid
errors, they still are a guess. With this situation on mind, we used
an eye tracking algorithm presented in [16] and modified it to
determine if and when the user may be losing the focus on the
game and looking back to the controller.
These events will be used to allow the interface to perform a
dynamic self-evaluation. In previous user tests [ANONYMOUS],
the authors observed that the user interface would change
constantly even if the users play style was not changing. The
feedback received from the users pointed that sometimes the
adaptation was too aggressive. This resulted in some cases where
an interface already close to the optimal just kept changing,
sometimes resulting in less adequate configurations. Another
interesting observation was that the users would look to the
controller when they couldnt find the buttons they were trying to
press. As the controller uses a touch interface, the lack of physical
feedback, something always present on regular joysticks, results in
a case where the only possible way to locate a button is to look at
the controller. But as the adaptive interface results in an increased
precision, the need to visually check the position for a button can
be reduced when the interface reaches a configuration closer to
the optimal layout for a user.
We decided to track the users attention focus to provide a new
source of data for our adaptations. We used the computers
webcam to track the pupils of the user and calculate if he is
looking to the screen or to the controller, using [16] method for
eye tracking. Our prototype controller would adjust the
aggressiveness of its adaptation according to the users focus. If

the user looks constantly to the controller, the speed of change for
the size and position of the buttons will be progressively
increased. If the interface stabilizes and the user stops looking at
the controller for more than 10 seconds, the controller will slowly
decrease the speed of the adaptation, stabilizing the interface and
performing way less changes to its layout.
We change the speed of the adaptation using two parameters: the
maximum change of a buttons size and position per iteration (the
algorithm is executed 2 times per second) and the amount of
points passed to the K-means clusterization algorithm. The first
parameter, when increased, makes the controller apply the
changes faster, changing its layout almost immediately, while a
lower value will result in slower changes. The last parameter,
when decreased, results in less points being passed to the
clusterization algorithm, resulting in an adaptation focused in the
more recent interaction patterns, being able to change its layout
more dramatically to answer to differences in the users behavior.
When this parameter is increased, the controller will base its
suggestions in long term characteristics of the gameplay section
and will be more conservative when performing changes.
We expect that this approach helps to avoid cases where the
interface keeps adapting itself after finding an optimal
configuration, performing unnecessary changes that can be
detrimental to the users experience.

5. USABILITY TESTS
In order to evaluate properly our proposal adaptation and the
interaction with the final user, we conducted a usability test,
observing some parameters given by the controller and the eye
tracking algorithm. To realize this evaluation, the tests were
divided in two stages: the pilot and the final user tests. The pilot
was the preliminary test and it worked to set the parameters to the
final test, determining the best configuration possible that would
fit to the game and the adaptation. As the controller do not need to
be identical to the physical one, the pilot test gave insights to
define his design.

5.1 Participants and Apparatus

The evaluation tested two different controllers, that apparently
resemble visually but the core algorithm is different. When the
user looks to the screen, the first impression is that both
controllers are equal. But, one controller is adaptive and the other
not. It is important to note that while the user plays, the adaptive
controller changes completely including the visual.
The functionality of both is the same, so each button corresponds
to the same action regardless of version. The participant must test
both controller and he does not know which version is being
tested at the moment.
The group of testers was composed by 8 users, with age that
varies between 21 and 51. Our group consisted of 3 women and 5
men. The whole group tested both controller versions with one
game, Streets of Rage. The users level of experience and profile
varied from experienced gamers to casual players and people that
usually do not play games at all, covering a wide range of possible
player profiles. In which play for more than ten years and still
play.

The experiment was conducted on an Asus Nexus 7 tablet (2012

version) running the Android OS 5.1.1. The capacitive
touchscreen has a diagonal of 7.0 inches with an aspect ratio of
16:9 and a resolution of 1280 x 800 pixels.

5.2 Procedure and Study Design

The first task that every user did was fill the User Profile
Questionnaire to understand the group that was testing. The
evaluation session was limited in 1 minute to adjust the focus loss
parameter, 1 minute of training, 5 minutes of playing with the
adaptive controller and 5 minutes of playing with the nonadaptive controller. To adjust the focus loss, we asked to the user
to look to the screen and then with the mobile leaning on the
belly, look to it. This little task performed more than once let us to
know the movement that qualifies the loss of focus. The
subsequent minute is a free to play, that allow the user to
familiarize with the control scheme and the game. We also
explained the game objective and how to play it. After that,
started the real test. Half of the users started with the non-adaptive
version and the other half started with the adaptive, to prevent the
learning related to play a game and we do not want that every user
tested the same version before the other. After the user had tested
the adaptive version, he was asked to fill the SUS (System
Usability Scale) questionnaire [3]. And after the session ended he
was asked to fill the Satisfaction questionnaire that compares both
controllers version.

Table 1: Precision per user, for both controllers

Graphics
User

Top
KA (%)

In-between
NA (%)

89.7

78.8

96.0

83.0

93.6

81.2

92.0

83.9

99.1

93.7

98.3

86.3

87.6

78.6

84.0

72.2

Average

92.5

82.2

Standard
deviation

5.3

6.3

6. RESULTS
The results were evaluated by analyzing each users precision,
that is, the percentage of touches on the screen that accurately hit
a button and performed an in-game action. All comparisons were
between the non-adaptive version (NA) and the K-means adaptive
version (KA), for each user and for the average of the entire
group. The results were compared using the Wilcoxon signedrank test, non-parametric statistical hypothesis test, which in this
case is used to compare if the difference between the results for
both controllers is significant. To execute this test, we set the
significance level to 0.05 and the two-tailed hypothesis, resulting
in a p-value that represents the difference. If the p-value is lower
than the significance level, the difference is significant.
To validate the improvements of our new adaptation approach,
using K-means and reacting to focus losses in real-time, we
compared the precision for KA and NA, using all users. Table 1
shows the precisions for all users, while figure 2 shows the
average for all players combined in a single plot displaying its
variation over time. With the Wilcoxon test we found a p-value of
0.007813 that is much lower than 0.05, which means that the
difference between the precision with both controllers were
significantly different. As the KA controller averaged a precision
of 92.5% while the NA controller achieved 82.2%, resulting in an
average precision 10.3% higher on the adaptive controller, we
have a strong indicator that the controller improved the users
precision, avoiding errors that could lead to frustration. The
standard deviation is 5.3 for the KA controller and 6.3 for the NA
controller, showing consistent results where all users display a
sizable improvement. In figure 3, we present the initial layout for
the adaptive controller, which is also the standard interface for the
non-adaptive version. In the same figure we can also see the final
configuration achieved by the adaptation based on touch input
and focus detection for one of our volunteers.

Figure 2: The average success rate for the KA controller (red)

and the NA controller (blue) over time for all users.

After playing the game, each user should fill the SUS
questionnaire [3] and the average of all users was 84.69 and the
standard deviation was 11.76. As [1], our SUS score of adaptive
controller version suggests that our controller had from good to
excellent acceptability. Comparing both versions, 75% of
participants preferred the adaptive version and believed that
played better. We asked the users about which controller version
they think was gazed more frequently during gameplay. Four out
of eight said that the non-adaptive was gazed more frequently, 2
said the adaptive was the most gazed and other 2 answered that
they did not detect any difference at all. The results in the

subjective evaluation have shown that a final analysis requires a

new test that totalizes the amount of gazes targeted to the
controller by the users.

controller can have on variables such as frustration, engagement

or excitement.

8. REFERENCES
[1] Bangor, A., Kortum, P., and Miller, J. Determining what
individual SUS scores mean: Adding an adjective rating
scale. Journal of usability studies, 4(3), 114-123, 2009.
[2] Bezold, M., and Minker, W.: Adaptive multimodal
interactive systems. Springer, Boston, 2011.
[3] Brooke, J. SUS-A quick and dirty usability scale. Usability
evaluation in industry, 189(194), 4-7.Chicago, 1996.
[4] Brown, M., Kehoe, A., Kirakowski, J., and Pitt, I. (2010).
Beyond the gamepad: HCI and game controller design and
evaluation. In Evaluating User Experience in Games (pp.
209-219). Springer London.
[5] [ANONYMOUS], details omitted due to double-blind
reviewing.
[6] Joselli,M. ,Silva Junior, J. R. S., Zamith, M., Clua, E., and
Soluri,E. A content adaptation architecture for games. In:
SBGames. SBC, 2012.
[7] Koster, R.: Theory of fun for game design. OReilly Media
Inc., Sebastopol, 2013.
[8] Langley, P. Machine learning for adaptive user interfaces. In:
Brewka, G.,Habel, C., Nebel, B. (eds.) KI 1997. LNCS, vol.
1303. Springer, Heidelberg, 1997.

Figure 3: The initial controller layout (upper side) and the

final configuration achieved by the adaptive controller for a
user (lower side)

7. CONCLUSION AND FUTURE WORKS

Many people avoid start playing games. One of the reasons is the
need to interact with complex devices with large amounts of
buttons and combinations, an issue capable of pushing them away
from video games.
In this work we provided an experience where each user can have
its own personalized game controller, automatically adapted to its
ergonomic and personal preferences. Our test results showed that
the adaptive controller can increase the users precision, leading
to less errors and a more comfortable interface. Additionally, the
subjective evaluation also demonstrated good acceptability by the
users.
After proposing a new controller that can be adapted to each user
behavior, our intention in a future work is to include new tools in
the proposed work that can be used by the game interface designer
to devise his own initial interface with adequate amount and
layout of the buttons. In this new paradigm, both the game and the
machine learning algorithms would work together to personalize
the interface. In this case, we would have a game controller for
each user and for each game, resulting in unique combinations.
Measuring the user experience is also an area that must be
explored. With this in mind, we would like to use an EEG
(electroencephalography) headset to evaluate the users emotions
during gameplay, trying to determine the exact impact that a game

[9] Majaranta, P., and Bulling, A. Eye tracking and eye-based

humancomputer interaction. In Advances in Physiological
Computing (pp. 39-65). Springer London, 2014.
[10] [ANONYMOUS], details omitted due to double-blind
reviewing.
[11] Salen, K., and Zimmerman, E. Rules of Play: Game Design
Fundamentals. Cambridge, Massachusetts: The MIT Press.
ISBN 978-0-262-24045-1. Game play is the formalized
interaction that occurs when players follow the rules of a
game and experience its system through play, 2004.
[12] Schell, J.: The Art of Game Design: A book of lenses. CRC
Press, Boca Raton, 2008.
[13] Smola, A., and Vishwanathan, S.: Introduction to Machine
Learning. Cambridge University, UK, 3234 (2008)
[14] Sundstedt, V. Gazing at games: using eye tracking to control
virtual characters. ACM SIGGRAPH 2010 Courses. ACM,
2010.
[15] Sundstedt, V., Bernhard, M., Stavrakis, E., Reinhard, E., and
Wimmer, M. Visual attention and gaze behavior in games:
An object-based approach. In Game analytics (pp. 543-583).
Springer London, 2013.
[16] Timm, F., and Barth, E. (2011, March). Accurate Eye Centre
Localization by Means of Gradients. In VISAPP (pp. 125130).
[17] [ANONYMOUS], details omitted due to double-blind
reviewing.
[18] [ANONYMOUS], details omitted due to double-blind
reviewing.