A Methodology for the Calculation of Usability

Economics Applied to Voice Activated Services

Juan Carlos Luengo Patrocinio
Teléfonica Móviles–Universidad de Alcalá
Distrito C, Edf. Este 3, Pl. 3ª
Ronda Comunicación s/n, 28050 MADRID
juancarlos.luengopatrocinio@telefonica.es
Sebastián Sánchez Prieto
Universidad de Alcalá, Esc. Politécnica
Carretera Madrid-Barcelona Km. 33.6
28871 Alcalá de Henares, MADRID
ssp@aut.uah.es
Daniel Tapias Merino
Telefónica Móviles
Distrito C, Edf. Sur 1, Pl. 8
Ronda Comunicación s/n, 28050 MADRID
daniel.tapiasmerino@telefonica.es

Pedro Concejero Juan José Rodríguez

Teléfonica Investigación y Desarrollo Teléfonica Investigación y Desarrollo
Emilio Vargas 6, 28043 MADRID Emilio Vargas 6, 28043 MADRID
pedro.concejero@tid.es

ABSTRACT 2. VOICE ACTIVATED SERVICES

In this paper we present a first approach to the methodology we
have developed to calculate the economics of voice activated
services. This methodology must be validated with the
experiences ongoing in definition.
Keywords
Usability, Vocal Access, Mobile Services, Economics
Area
Usability, Vocal Access, Mobile Services, Economics
1. INTRODUCTION
Usability has been widely recognized as one key aspect in the
design of all kind of elements. Even when this concept has been
deeply connected to technical issues, in fact usability must be
applied to all fields. In every field of our daily life, usability is
something we talk about, mostly when we miss it. Nobody talks
about an easy to use tool, but when you have difficulties
dealing with a concrete device, immediately the question “who
designed this?” arises.
And when reducing the area to Operators and more
technological services, there is an important deviation in the
usability community, to mainly consider this matter when
talking about web pages and software development. Even when
these areas were the first used to evaluate and determine the
importance of the usability, there are a lot of other areas in
which deeper work is required, in order to clarify the important
role that usability plays in the success of a concrete service.
On the other hand, modern companies need a clear and simple
“cost model”, to understand the return on investment (not only
the ROI as financial indicator) of each euro that is used inside
the company. This is why it is important to have a clear
perception of the return that usability investment gives to the
companies. And the word investment is not randomly selected,
since it is important to understand that usability is going to have
a return, and that it is not another expenditure to afford. The
works about Usability ROI began several years ago, but as
mentioned before, mainly focused on web usability [1].
In this paper, we focus on creating a methodology to allow an
economical evaluation of the benefits of usability.
The Telefónica Group has invested more than 20 years in
research and development of speech technology and in its use
in real services over the telephone. During all this time, the
voice activated services have experienced an important
evolution. And, if in the past the design of voice user interfaces
(VUIs) was often driven by the need to overcome the
limitations of the underlying technology, currently, since the
technology has improved, more effective and user friendly
designs have become the focus of the design process.
Therefore, usability in speech applications is not simply a
matter of the core technology, but mostly of tools and skills in
developing applications, i.e.: ease of use is the result of
systematic usability activities carried out through the project
lifecycle.
This methodology of service development applied to speech
applications together with our current state-of-the-art speech
technology has made it possible to implement more than 3,000
speech recognition and text to speech conversion telephone
channels that are providing our customers with different
services worldwide. The voice portal (404) and the information
service (11818) with a vocabulary of about 10,000 words are
good examples of services of this kind in Spain.
2.1 The Design of Voice Activated Services
Thus it is important to have a clear idea of the requirements that
this kind of services have, in order to properly design them.
Considering the public information about this item [2], as more
confidential information, related to the works carried out inside
Telefónica Móviles España to improve its Voice Portal Service
[3], there are several general recommendations that must be
considered when developing the service, in order to get an
appropriate result in the usability of these services.
From these two previously mentioned references, we could
extract the different aspects that must be considered when
designing speech interfaces. It is not the objective of this paper
to cover in detail all the questions that should be carefully
studied before implementing the interface, but we think it is
important to put them on the table, since this will give skeptics
a closer perception of the amount of work required to properly
define an interface. As mentioned before, and as we will
mention several times during this paper, usability is something
we talk about when we miss it. Thus the main points, that are
here briefly presented, are widely revised in [2],
 1) Physical properties of Speech Machines. Several
parameters have to be controlled and properly defined
before a vocal application is ready to be offered to the
public. The loudness and noise (including the volume),
the sound quality of spoken output (covering selected sex
for the output voice), timing values (almost for every
response a timing value has to be defined to avoid
incorrect behaviour of interaction), proportion values of
speech and silence, rhythm and rate.
2) Adequate selection of the wording. The voice, person,
tense and mood are important, together with an
appropriate selection of the “opening greeting”
3) Prompts. The design of the prompt, the type of prompt
used, and the way they are included in the general flow of
speech application.
4) Feedback. The types of feedback (trying to avoid literal
feedback, replace apology and blame with prompts) and
the way this feedback is included in the application flow
are other points to consider.
In addition to this, and as presented in [4], Telefónica Móviles
España is working very hard in evaluating how important is the
vocal access to mobile services, and which is the best way for
developing these vocal interfaces. For that study, the Yavoy
(commercial name in Telefónica Móviles España for the Ring
Back Tones service) was selected, and a pilot was carried out in
order to clarify these points.
But, as it is clear in other areas too and was mentioned in [4],
usability is not the only parameter to fine-tunning. There are
other actions that we must carry out in order to encourage
customers to use services. If we take as an example the ring
back tones service, there are some other important things to
increase the purchasing of tones by the users:
1. Update regularly the list of available ring back tones,
(musical hits, jokes or monologues).
2. Launch marketing campaigns to make the service well
known among the potential users. Even when this is a
clear example of a “viral service”, good marketing
campaigns always have influence on the services usage.
That is, an excellent interface is not enough to boost the service
if the content is not interesting and/or potential users do not
know the existence of the service. However, assuming that all
these conditions are granted, usability is essential.
3. GENERAL ECONOMICAL
CONSIDERATIONS ABOUT USABILITY
Nowadays, almost every aspect of our daily life can be
quantified and has an economical definition. Even when this is
a very arguable assessment, the truth is that this is the way
almost every sector works. Moving into the business arena, this
statement is even more important. And this need leads us to the
objective of understanding all the economical implications that
usability has.
As a basic premise when talking about usability economics, and
even when everybody has this idea in mind, we think important
to clearly state the following: “Usability will always have a
positive impact in the Company Balance Sheet”. But the
activities around this statement go through clearly identify
which are those impacts, which degree of importance have and,
and as the main objective of our work, try to quantify them.
In the following sections of this paper we make a first approach
to it. Initially, giving a classification of these impacts and, after
that, explaining how these impacts could be quantified.
3.1 Qualifying the different effects of
usability
Our initial classification considers different levels of
importance in the positive impact that usability generates.
Depending on the type of service when speech application is
used, different concepts should be considered, as explained
when defining those degrees.
First Degree
Here there is a clear difference in the impact to consider,
depending on the type of service considered (same
differentiation is presented in [2]). If we are talking about Cost
Saving Applications (CSA), in this case the impact to evaluate
is the reduction of the cost of those concrete activities by using
this new system. But if talking about Value Added
Applications (VAA), direct increment in the usage of the
service can be measured, generating new revenues. In this
second case, a new differentiation has to be made, this time
considering the tariff scheme defined. And this is something
that can change with time during the lifetime of a service, but is
much more difficult (not to say impossible) that a VAA
becomes a CSA.
Second Degree
At this level two concepts must be included: Churn Reduction
and Innovation Perception or Technological Leadership.
The first concept is widely used and, even today, is quantified
in most operators, considering the cost that getting a new client
has, and the even bigger cost that recapturing an old gone client
has. All these features and interfaces, that make simpler the
usage of the services, and that allow the customer easily obtain
the information required, obviously act as an “exit wall”. The
second concept is included here as a second degree effect, since
our experience shows that this perception is very important
when talking about technology related products, even when
some people consider it as a third degree effect (in fact, as part
of this third degree concept, as we will see when talking about
the following degree).
Third Degree
In this level Branding Perception or Corporate Reputation
is included. This parameter is always affected by every action
and decision that a Company takes. But the effect of these
decisions are normally considered for bad, not including the
positive effect that these kind of new features have in the brand.
A first approach and more information about this concept can
be found in [7], where differente methodologies to evaluate and
quantify the Corporate Reputation are presented. For this
approach, we have selected the RepTrack™ as the model to
use, since it is the world’s first standardized and integrated tool
for tracking corporate reputation internationally, and it uses
dimensions statistically independent.
Basicly, it is a model that considers the following parameters
for the evaluation: Products/Services, Innovation, Workplace,
Governance, Citizenship, Leadership, Performance. Each one
of this parameter has a weight, being a general assumption, that
could be probably slightly modified depending on the sector
and concrete company, the following: Product/Services - 33%,
Innovation – 10%, Workplace – 17%, Governance - 15%,
Citizenship – 10%, Leadership – 10%, Performance – 7%
And if these new feature is properly explained to general
public, this third degree effect could become a second degree
effect. Another less probable effect is the generation of interest
in possible new customers that begin to consider the possibility
of becoming our customers when knowing about this service.
Obviously this case is more related to marketing campaigns,
but usability plays an important role, since it can allow a “trial”
user to easily get into the service, pushing him/her to become a
customer.
At this level we can consider too the concept presented in [6] as
Internal Social ROI that means the importance that, for
Usability Group, a clear recognition of the importance of their
work by their Company colleagues has. This obviously will
improve the functioning of the group and, in the same sense,
will improve the results of internal satisfaction surveys, but its
influence is too much difficult to evaluate.
Normally, when defining the return that a concrete usability
investment has, only First Degree effects are considered. Being
a CSA or a VAA only matters when considering if the money is
saved o earned. But we consider important to move into Second
and Third Degrees, since they have the positive impact
mentioned, and thus it could be quantified, being this probably
the most difficult part of this work.
3.2 Quantifying the previously qualified
aspects
There is an important statement, included in [5], “adding
speech recognition capabilities to an existing touch-tone IVR
platform increase usage by 20% to 60%.” And this is
considering that the sentence only says “adding speech
recognition capabilities”, and not “adding adequately designed
speech recognition capabilities”. Probably an incorrect design
will increase usage by 20% and a correct design will increase
usage by 60% (this is something not mentioned in the study,
just an idea). Thus, only because of this statement usability in
voice activated services should be considered as a major
investment.
But, as it was previously mentioned, this statement only covers
what we called First Degree effects. And is the quantification of
previously indicated Second and Third Degree effects the most
recurrent activity that usability managers have to afford when
explaining their team activities to Top Management.
When talking about “quantification”, it is necessary to generate
a model and/or a mathematical formula to know how to
translate into concrete return numbers the results obtained using
the methodology that is going to be explained in the next
section. There are different ways of quantifying SD and TD
effects (FD effects have a clearer way of quantification), and
the one selected should fit with the general perception that the
company has of these cases.
4. METHODOLOGY TO EVALUATE
The work in progress in Telefónica Móviles España and
Universidad de Alcalá has the target of generating an
economical model, or at least and as a first step, a methodology
that could allow usability and/or designers teams to evaluate the
impact that usability improvements have in the services.
And if this model or methodology could be applied to several
services, we finally could get a range of values, and used when
estimating in advance economical importance of usability.
Initially, we propose a methodology, and work is required to
obtain the mathematical model, which first approach is
presented in this paper.
4.1 First Definition of Methodology
Here we indicate the steps required to properly obtain the
required information, following the information presented in [8]
about the way the Usability Lab has to be designed, and in [9]
and [10]. Another point about the evaluation of these features is
that is very difficult to obtain valuable data from Focus Groups
that have not used the service. This premise is considered when
defining the methodology.
4.1.1 Step1: Define new service
Define a service pilot considering the usability comments made
during design process (new service). Maintain too a pilot of the
service as it was originally defined (old service).
4.1.2 Step2: Create Parallel Groups
With same profile members, and offer “old service” to Group A
and “new service” to Group B (Note: if talking about the
improvements created for an existing and commercially
launched service, then information about the “old service” can
be obtained from normal users).
The creation of these groups could follow the scheme of a
Focus Group in the beginning, and only if required a deeper
analisys could be considered, considering more groups with
more members each one.
4.1.3 Step3: Make tests
It is necessary to make two kind of tests; Oriented Tests
(identifying several concrete tasks that must be covered by the
user) and General Tests (not defining concrete tasks, but
indicating that it is necessary to use the service a minimum time
per day/week/month).
Once we have listed the main requirements to be considered in
the creation of the questionnaire, then the next steps should be
covered.
Step 3.1: For Oriented Tests, create a questionnaire to fill out
just when finish the test. This could be formed by “closed-
ended” questions, that provide respondants with a list of the
possible answers.
Step 3.2: For General Tests, create a questionnaire to fill out
whenever the customer wants. In this case, a combination of
“closed-ended” and “open-ended” questions could be used. The
“open-ended” questions allows the respondant the possibility to
explain his perception of the service using his own words.
Of course the evaluation of these “open” tests is much more
costly (in time and resources), but we think it is very useful to
confront the responses with a previously created list of “key
words” that will allow us to extract the perception of the user
about the service and, by far, of the Company. This list could
become two lists, one with possitive terms and other with
negative ones, and the number of coincidences will give
information about the position of the user about the service.
Step 3.3: If possible, that is, whenever these Tests (Oriented or
General) are carried out in the Company premises, record and
save their reactions when using the service in the Usability Lab,
to study them properly. Encourage the people selected in the
groups to talk loud about any thought, idea or question they
consider necessary to mention about the process. Probably at
the beginning of the test, users will not be totally open to this,
 Definition of the task
Oriented Test and
Creation of the Group General Test
Execution of the task
Recording of
the fullfilment
Fullfilment of the questionnaire
Questionnaire to
Obtain information
Figure 1.- Graphical representation of the procedure to follow
but as soon as they go into the test they will relax and make
more useful observations.
In this case, it is necessary an adequate definition of the
questionnaires to fullfil. As indicated in [11], a wrong
definition of the questionnaires will give totally deviated
information, and talking about economical concrete questions
this is not an option. Presented in [11] too, there are some
general rules that must be followed to assure the questions
defined obtention of enough informati
ƒ Assure that each question ask only a single question. Avoid
double-barrelled questions that would give uninterpretable
answers.
ƒ Give all the alternatives for each “closed-ended” question. If
the answer is biased for the use of some “positive or
negative” words, this could favor one option and affect the
results.
ƒ For some inconvenient questions, that could obtain an answer
not socially acceptable, it could be necessary to “counterbias”
the question, that is to write the question in such a way that
respondants find easy to answer properly.
ƒ It is necessary to avoid words with “vague” meaning, or
“slang” words that could have different or offensive meaning.
Figure 1 gives a graphical representation of the proccess,
considering the steps previously mentioned.
Now all this information has to be evaluated and processed, and
when comparing the results between “old service” and “new
service”, probably enough information about the quantification
of usability improvements can be obtained.
In order to properly do that, it is necessary to use a
measurement scale. There are three commonly used category
scale formats, as presented in [11].
ƒ Semantic differential scales. A sequence of unlabeled
categories between two bipolar adjectives.
Recording of
the execution
Final Report
ƒ Stapel scale. Only one single adjective and a range of values
from –X to +X.
ƒ Likert scale. A number of statements are developed, to
measure different aspects that might influence the
respondent’s overall attitude toward the service.
These scales are generally used, but it is quite clear that it is
necessary to asses scale validity and reliability for the concrete
case of use of the scale. And thus the scale accuracy refers to
the degree to which a measurement scale is free of both bias
and random error. Our main preference is the usage of the
Likert scale, but the other scales mentioned could be used with
good and interesting results too.
4.2 How the Methodology fits with the
effects listed.
Once the list of FD, SD and TD to consider is defined, and after
explaining how the quantification of those effects can be made.
it is necessary to identify how the required information for the
quantification can be extracted from the activities covered by
the methodology defined.
After a detailed study of the diferent sources that could be used,
and considering how important and realiable these sources are,
Table 1 shows which are the sources that have to be considered
when quantifying. We have to diferentiate between Primary
Sources (PS), those mandatory to obtain valuable information
about that concrete Degree, and Secondary Sources (SS) those
interesting to add granularity and information to the result, but
not mandatory.
From the following table we see that, in general, all the
different Degrees could be affected by information obtained
from all the different Steps, but the fact is that some are
primary sources (PS) and some are secondary sources (SS).

Step/Degree First Second Third
3.1 PS PS --
3.2 PS SS PS
3.3 SS SS SS
Table 1.- Sources of information and where they can be used
4.3 First Approach to the Final Model.
After identifying these relationships, now it is time to define
which is the way to combine all the information to obtain a
clear and simple model to finally obtain a clear measure of the
ROI obtained from usability decissions and activities.
This activity is probably the most delicate, and thus the less
evident, part of the work, since the translation between obtained
general data and final economical results can have a lot of
different ways to do it. Our first approach, that have to be
confirmed and confronted with several real situations to be
totally accepted, is based in the experience.
Thus, considering these limitations, our first approach is
covered by the following formula, which has two versions,
depending on if we are talking about CSA or about VAA :
ROI = ∆E(VAA) + ChR * VChP + IP * VIPP + BP * VBPP
Where
∆E(VAA): Earning increment obtained in one year due to the
improvements in service usability.
ChR: Churn Reduction; VChP: Value of Churn Point.
IP: Innovation Perception; VIPP: Value of Innovation
Perception Point.
BP: Branding Perception; VBPP: Value of Branding Perception
Point.
This formula changes when considering ∆S(CSA) instead of
∆E(VAA), being ∆S(CSA): Saving increment obtained in one
year due to the improvements in service usability, but the rest
of the parameters would be the same.
The information obtained from the questionnaires mentioned in
the previous points is the base of the calculation of ChR, IP and
BP. The way the final value is obtained will depend on the
information obtained, but a general model is being defined in
the proccess of this work.
The value of VChP, VIPP and VBPP is something that each
company has to define for each case and moment, since it is
clear that the value of one point of churn could be different
depending on the market share, general situation and other
factors that could vary in time.
5. CONCLUSIONS AND FUTURE WORK
The importance of usability in the design and development of
products and services has grown in the last years. In the case of
voice activated services, usability becomes crucial when the
application reaches a minimum level of complexity.
Additionally, as we have shown, making services easy-to-use,
require the inclusion of technical and usability considerations in
the design and development process.
This paper shows which are the effects that usability has in the
Company Economics, and defines a methodology that must be
used to properly obtain the information of the benefits obtained
with an adequate design of the services.
This methodology should lead us to a model that allows
designers a quick evaluation of usability benefits. The first
approach of this final model has been presented, but it is
necessary to make the translation from the information obtained
from the questionnaires to the values required in the formula
presented.
The other pending work is to use this model in different real
cases, and then make a model assessment and confirm its value.
Short term future work must cover in deep this quantitative
approach, making different studies following the proposed
methodology and evaluating the information. From all this
information, a concrete model should be extracted and applied
in other different applications.
6. REFERENCES
[1] J. Nielsen, S. Gilutz, “Usability Return on Investment”,
Norman Nielsen Group,
http://www.NNgroup.com/reports/roi (January 2003)
[2] Bruce Balentine, David P. Morgan, “How to Build a
Speech Recognition Application; A Style Guide for
Telephony Dialogues”, Second Edition, EIG Press (2001)
[3] “Design Recommendations for Telefónica Móviles Voice
Portal”, EIG and Infospeech, December 2003.
[4] J.C. Luengo, C. Lázaro, D. Tapias, P. Concejero, J.J.
Rodríguez, “Usability Considerations in the Design and
Development of Voice Activated Services”. ICIN 2006
(May-June. 2006), Bourdeaux (France)
[5] Donna M. Fluss, “The Practical Guide to Speech
Recognition. Using Speech Recognition to Decrease Cost
and Increase Revenue” DMG Consulting (2007)
[6] Randolph G. Bias, Deborah J. Mayhew, “Cost-Justifying
Usability: An Update for the Internet Age”, 2nd Edition,
Elsevier (2005)
[7] Cees B.M. van Riel, Charles J. Fombrun, “Essentials of
Corporate Communication”, Routledge, (2007)
[8] J.J. Rodríguez, P. Concejero, S. Diego, J.A. Collado, D.
Tapias, A.J. Sánchez, “Laboratorio de Usabilidad de
Telefónica Móviles España”. Boletín de Factores
Humanos de Telefónica I+D No 27 (Agosto 2005)
[9] Joseph S. Dumas, Janice C. Redish, “A Practical Guide to
Usability Testing”. Revised Edition, Intellect Books,
(1999)
[10] Jeffrey Rubin, “Handbook of Usability Testing: How to
Plan, Design and Conduct Effective Tests”. John Wiley &
Sons, (1994)
[11] Melvin Crask, Richard J. Fox, Roy G. Stout, “Marketing
Research: Principles and Applications”, Prentice Hall,
(1995)