410 IEEE TRANSACTIONS ON AFFECTIVE COMPUTING, VOL. 3, NO.

4, OCTOBER-DECEMBER 2012

The Good Our Field Can Hope to Do,

the Harm It Should Avoid
Roddy Cowie, Member, IEEE

Abstract—This paper tries to achieve a balanced view of the ethical issues raised by emotion-oriented technology as it is, rather than
as it might be imagined. A high proportion of applications seem ethically neutral. Uses in entertainment and allied areas do no great
harm or good. Empowering professions may do either, but regulatory systems already exist. Ethically positive aspirations involve
mitigating problems that already exist by supporting humans in emotion-related judgments, by replacing technology that treats people
in dehumanized and/or demeaning ways, and by improving access for groups who struggle with existing interfaces. Emotion-oriented
computing may also contribute to revaluing human faculties other than pure intellect. Many potential negatives apply to technology as a
whole. Concerns specifically related to emotion involve creating a lie, by simulate emotions that the systems do not have, or promoting
mechanistic conceptions of emotion. Intermediate issues arise where more general problems could be exacerbated—helping systems
to sway human choices or encouraging humans to choose virtual worlds rather than reality. “SIIF” systems (semi-intelligent information
filters) are particularly problematic. These use simplified rules to make judgments about people that are complex, and have potentially
serious consequences. The picture is one of balances to recognize and negotiate, not uniform good or evil.

Index Terms—Ethics, emotion, affective computing

1 INTRODUCTION

P EOPLEspend a huge part of their lives interfacing with

computers, knowingly or unknowingly. Correspond-
ingly, when research devises new ways for people to
interface with computers, there may be substantial implica-
tions for what Arnold [1] memorably called our “weal or
woe.” Affective computing envisages that kind of shift, and
for that reason, there is a strong case for thinking through
its likely effects on our weal and woe before it is launched
on a large scale. That is a thoroughly interdisciplinary
challenge because it involves coming to terms with several
different kinds of issue: what the technology can achieve,
the characteristics of humans that determine what new
technologies will mean to them, and the moral and ethical
frameworks relevant to evaluating the interactions. The aim
of this paper is to provide a framework for addressing the
new issues that gives due weight to these different
perspectives.
It would be natural to assume that there was already a
considerable literature which did exactly that. But although
a good deal has been written about ethics and affective
computing, much of it seems to deal with rather a different
problem—the desirability in principle of things that we can
imagine in an abstract sense, but that there is no obvious
prospect of building in the foreseeable future, if ever. That
is a fascinating subject, but it should not be confused with
the more immediate one that this paper aims to address.
. The author is with the School of Psychology, Queen’s University Belfast,
Northern Ireland, United Kingdom. E-mail: r.cowie@qub.ac.uk.
Manuscript received 1 May 2011; accepted 19 Dec. 2012; published online 20
Dec. 2012.
Recommended for acceptance by A. Beavers.
For information on obtaining reprints of this article, please send e-mail to:
toac@computer.org, and reference IEEECS Log Number
TAFFCSI-2011-05-0037.
Digital Object Identifier no. 10.1109/T-AFFC.2012.40.
1949-3045/12/$31.00 ß 2012 IEEE
This paper engages with ethical questions at two levels.
At a generic level, it is concerned with suspicions that the
whole discipline of affective computing is ethically tainted
—a technology too far. It argues that there is no need for
generic condemnation of the technology that we actually
have, or can reasonably imagine. That does not mean that it
takes an anodyne view, though. The second level that the
paper considers involves distinguishing between some
application areas that present no ethical problems of note,
and others that do. It does not dispute that some areas do
pose problems. The outstanding examples arise not because
technology endows machines with dangerous powers, but
because there is a risk of applying technology to tasks that it
is not very good at—not because the tower of Babel may
infringe on Heaven, but because a building as high as that,
made with clay bricks, is likely not to be fit for purpose.
2 CORE JUDGMENTS
The outlook that the paper offers depends on judgments
about issues that are open to debate—the subject matter of
the technology, what it has achieved, and the relevant kind
of moral framework. There is no question of resolving the
debates here, but neither should they be glossed over. This
section sets out summaries of the positions that are taken so
that readers can judge whether they regard them as
reasonable. The fact that the audience is interdisciplinary
means that most readers will find at least some of the
material too familiar to be worth mentioning.
2.1 The Natural Subject Matter of the Discipline
There is real difficulty in identifying the natural subject
matter of what is sometimes called affective computing,
sometimes emotion-oriented computing. “Affect” and
“emotion” are the only words that could reasonably be
used to identify it, but they set traps [2].
Published by the IEEE Computer Society
COWIE: THE GOOD OUR FIELD CAN HOPE TO DO, THE HARM IT SHOULD AVOID
Historically, and in many contemporary writers, “affect”
means pure, irreducibly subjective feeling—for instance, the
glow of positiveness and energy that a happy person feels,
or the negativeness and lethargy at the core of depression
[3], [4]. It is easy for nonexperts to assume that the goal of a
discipline called “affective computing” must be to build
computers which have that kind of subjective experience.
Alternatively, they might assume that the goal was to build
machines capable of peering into those quintessentially
private layers of experience. Somebody who believed that
developments like that were under way might reasonably
feel that they raised deep and disturbing ethical questions.
The view taken here is that neither of those images
actually captures what the discipline generally does—and
correspondingly, although the ethical questions associated
with them may be fascinating, they are not the ones that are
practically pressing.
Similarly, the word “emotion” has a sense in which it
refers specifically to brief, intense episodes evoked by
dramatic situations, and substantial figures in the field
prefer to use the word only in that sense [5]. It is easy for
nonexperts to assume that the goal of a discipline called
“emotion-oriented computing” must be to build computers
which intervene in that kind of situation. If so, they might
reasonably protest that interventions like that are notor-
iously difficult even for a sensitive human; and accordingly,
emotions like that are a part of life that computers should be
kept well away from. Again, the view taken here is that that
is not what the discipline is about, at least not for the most
part; and correspondingly, the associated ethical issues are
not as prominent as the image suggests.
The obvious subject matter for the discipline is more
mundane. Most of human life is emotionally colored, to at
least some extent [6]. People feel something about the other
people and things that they are dealing with or discussing;
they give signs of the way they feel, they expect other
people to register them to at least some extent, and the
feelings influence their perception, judgments, and deci-
sions in a variety of ways. The term “pervasive emotion”
has been used to refer to these colorings, signs, and
cognitive patterns, which are an integral part of everyday
life [6].
Looking at standard sources such as the proceedings of
ACII conferences [7], [8] and the HUMAINE handbook on
emotion-oriented systems [9], it seems fair to say that the
core research problem is to engage in a limited way with the
emotional coloring that ordinarily pervades human inter-
action. That involves recognizing signs (not necessarily the
ones that humans use), knowing how to generate appro-
priate signs, and being able to take some account of their
implications. Clearly there are ethical questions associated
with these tasks. However, they are not primarily about
grand principles: They are about the uses that can ethically
be made of techniques which may tread on sensitive
ground, and which at present almost always fall far short
of the levels of competence that humans take for granted.
On that understanding, effective ethical discussion depends
on being aware of the actual state of the art.
411
2.2 The State of the Art
It is a difficulty that the field lacks authoritative reviews of
the state of the art. This section provides a brief (and
necessarily personal) assessment. It emphasizes that the
techniques we have, or can reasonably envisage, are very
limited. That is no criticism of the research that has been
done. It is more appropriate to say that it underlines how
sophisticated human emotional competence is—a point that
is taken up later.
A substantial part of the field is concerned with
communication of emotion. Within that, probably the most
developed area involves recognition of emotion from
nonverbal signs—voice (as opposed to words spoken),
facial expression, and gesture, singly or in combination [10],
[11], [12]. Near-perfect recognition rates have been reported
for all these modalities, but they are for artificial displays.
For naturalistic displays of emotion, recognition rarely
attempts to do more than assign episodes to one of half a
dozen classes, and the success rates that it achieves are
usually modest. Recognition is also context-bound—a
system that recognizes emotion effectively in one kind of
phone call will perform poorly in another [13].
The idea of recognizing emotion from physiological
signs has captured the public imagination, but it is subject
to a major problem, which is that the variables affected by
emotion are also affected by many other things—physical
movement, mental effort, temperature, etc. An interesting
range of discriminations can be achieved when all of these
variables are controlled (e.g., the person is sitting still in a
regulated environment), but that is not a common applica-
tion scenario [14].
One might also expect that emotion recognition would
become easy when a person was speaking and the words
could be taken into account. In fact, the standard task with
text is to recognize whether the sentiment it is expressing is
positive, negative, or neutral, and that is difficult and
uncertain [15].
Cutting across these areas is the challenge of represent-
ing everyday emotional states in a way that relates to
detectable signs [16], [17]. A key reason is that automatic
techniques can rarely assign a precise description of an
emotional state with any reliability. Typical solutions
involve assigning a “cover class” such as “negative” [18]
or a few numbers indicating how well each of a small
number of descriptors applies. The descriptors may specify
emotion-related states [19] or dimensions [20].
On the output side, synthesis sets out to convey
distinctions which are also coarse, but (at least until
recently) more often based on “basic emotion” categories
[21]. It can generate displays that people can identify
reliably, but they rarely look or sound natural. Work with
robots has added a different slant by exploring forms of
expression that are not meant to be human (often animal-
like) [22].
It is important for later arguments that although the
problems sketched above are, so to speak, on the periphery
of emotion—dealing with external signs rather than internal
states—they are nevertheless extremely difficult, and have
absorbed a large part of the effort in affective computing.
Correspondingly, a core part of evaluating the technology is
412 IEEE TRANSACTIONS ON AFFECTIVE COMPUTING,
gauging what can be done with rudimentary ability to
detect and generate signals related to emotion in various
modalities.
Efforts to move beyond that into central issues remain
very fragmentary. Philosophers have evoked the rich
network of connections and dispositions that go with being
in a given emotional state [23]. Attempts to model that kind
of network are at an early stage [24]. Without progress on
that problem, it is very difficult for a system to respond
with any sophistication to a user’s emotional state. Even
choosing when to smile, frown, or nod in a routine
interaction is a major challenge [25].
2.3 The Ethical Framework
When technologists debate ethics, they are often drawn to
one of two positions. One equates ethics with an explicit
code that has legal or quasi-legal force. The other assumes
that the correct way to justify particular ethical conclusions
is by derivation from fundamental philosophical principles
[26]. This paper steers away from both, and adopts a
position recommended by philosophers in the HUMAINE
network [27].
Focusing on ethics in the sense of an explicit code is
unsatisfactory for two reasons. First, it is unwise to invest
effort in developing technologies that are likely to be
prohibited as soon as codemakers realize that they are
possible. Second, codes tend to focus on the negative and
are not good at distinguishing between activities that are
merely harmless and those that there is a case for under-
taking because they do good. There are good reasons for
affective computing to consider issues in both of those
areas, which means looking beyond ethics in the narrow
sense. In that context, the fact that “ethics” connotes codes
and proscriptions sometimes means that the word sits
uncomfortably, and “morality” is used instead.
Turning to fundamental principles, it is well known that
the major positions (Kantian, Utilitarian, etc.) do not
translate easily into practical ethical guidance, partly
because there are conflicts between them. For that reason,
philosophers have developed systems that are closer to
applied questions. Doring et al. have argued that one of
these, Principalism, provides a suitable framework for
emotion-oriented technology [27], and this paper follows
their recommendation. That means options are evaluated in
terms of beneficence (in Arnold’s terms, whether they
promote weal), malificience (whether they inflict woe),
autonomy, and equity.
Principalism highlights those values because their
relevance to ethical judgments is widely accepted. How-
ever, there are other ethical stances which often seem to be
implicit in critiques of emotion-oriented computing, but
which are by no means universally shared. One is a
disposition to value activities that meet indisputable needs
and to be critical of others. The other is a disposition to
value things and activities that are natural and to mistrust
artificiality. For brevity, they can be called needs-oriented
and nature-oriented ethics respectively. Both have long
roots, but neither is by any means universally accepted. At
least some objections to affective computing seem to be
grounded at least partly in these personal stances rather
than in more universally accepted values.
VOL. 3, NO. 4, OCTOBER-DECEMBER 2012
On the other side, an important claim that is made in
favor of emotion-oriented computing also seems to stand
outside Principalism. It hinges on the belief that valuing
different aspects of humanity appropriately has moral
value, and that emotion-oriented computing contributes to
revaluing emotion. The belief is consistent with the effort
that philosophers from Plato onward have invested into
considering how we should evaluate emotion. If computing
can contribute to the debate, it is making a morally
significant contribution.
2.4 The Structure of the Paper
The discussion that follows contains four main sections.
They deal with areas that seem to be essentially neutral
from a moral point of view; with the positive moral goals
that we can realistically hope to serve by working on
emotion-oriented computing; with the negative moral
consequences we should be seriously concerned about;
and finally, with areas that seem to be mixed or
controversial. After considering these, the paper revisits
some of the more dramatic stances that are taken elsewhere,
and considers how they relate to the more pragmatic
assessment that has been outlined.
In many ways, the division is not particularly satisfying.
It does not yield the kind of neat grid where each major
section has similar subheadings, and so on. There are
overlaps and gray areas. However, other divisions were
explored, and they work even less well. If they are neat,
they leave out research areas and problems that really
should be considered; and if they cover the ground, they are
structurally unsatisfying.
The problem of finding a satisfying structure may
actually be informative, insofar as it signals the inherent
messiness of ethical judgments in real cases. The same
technology may be ethically neutral when it is used to do
one thing, ethically positive when it is used to do another,
and have ethically undesirable effects when it is used in
some other ways. That kind of complexity is part of the
reason for believing that serious ethical discussion needs to
engage with specific technologies and specific applications.
3 MORALLY NEUTRAL APPLICATIONS
Looking across the applications of emotion-oriented com-
puting as it currently stands, it seems reasonable to say that
the great majority of them do not need a great deal of
attention because they appear to be morally neutral or all
but morally neutral. They can be grouped into two main
categories.
3.1 Frivolous Applications
Probably the largest single category of applications for
emotion-oriented technology can fairly be called frivolous.
The new technology allows people to have experiences that
are positive, but of no great significance. Very probably,
there will be other means that could have produced equally
positive experiences in the same circumstances; and so the
effect is to increase the range of ways in which people can
induce mildly positive experiences in themselves and
others, not to affect the sum total of human weal or woe.
COWIE: THE GOOD OUR FIELD CAN HOPE TO DO, THE HARM IT SHOULD AVOID
It is hard to argue that innovations like that will affect the
moral status of the human race one way or the other.
Arguably it has the positive effects of directing human
ingenuity into harmless streams and creating employment,
neither of which is to be sneered at.
There is a great range of “frivolous” options. Electronic
postcards designed to convey the sender’s feelings have
become so common as to be hardly worth a mention. Given
the synthesis technologies mentioned above, they could
easily be made far more elaborate, and that will presumably
become more standard. Chatbots are now an established
feature of the internet, and they are routinely programmed
to express a degree of emotion [28]. Computer games have
included characters that express emotion since the Sims
over a decade ago. Ambitious new developments have been
widely canvassed [29], though it is not clear how successful
they have been [30].
A different strand of research involves attaching in-
dicators of emotion to electronic messages. Smileys are
ubiquitous. Subtler technologies enhance messages with
colours and movements that convey their moods [31].
Related techniques have produced diaries designed to let
users record (and reflect on) their emotional state through-
out the day [32].
A distinctive set of emotion-oriented technologies has
developed around music. They are very varied. At one
extreme, there are technologies that allow performers to use
emotion-related signals to shape performance [33]. At the
other, there are efforts to offer iPod users music that suits
their mood by monitoring signals that reflect their emo-
tional state [34].
It is hard to argue that any of these applications adds
much either to human weal or to human woe. Nor do the
technologies trespass into areas that we might think should
be reserved for humans; in general, they are relatively
straightforward. The products are, so to speak, ethically
lightweight, and ethicists should not make much of them.
The obvious ground for disagreeing relates to one of the
stances mentioned earlier. According to a needs-oriented
ethic, technology ought to focus on weightier matters, and
the applications considered here are objectionable simply
because they are frivolous. It seems fair to regard that as an
ethic that people are entitled to adopt for themselves, but
not one that should be enforced on others. For most people,
promoting mild happiness is an unexceptionable occupa-
tion, and the emotion-related technologies that contribute to
it are not a problem.
3.2 Resourcing Professions
It might at first sight be more surprising to suggest that a
second group of applications should be considered broadly
neutral. They involve providing various professions with
resources that they can use to do things that they might do
in other ways, but that would be more expensive, less
flexible, harder to adapt to local cultures, and so on. It
seems fair to suggest that if that is what an application does,
it is not ethically problematic.
There is a large overlap between the technologies used in
that way and the technologies used for entertainment.
Chatbots increasingly feature on websites designed for
practical use rather than entertainment. The aim is generally
413
simple: to make the website more interesting to a user, and
to create a generally positive atmosphere.
It is not totally clear yet how successful they are:
Methods of evaluation are still under development [28].
However, an established company, Cantoche, offers web-
marketing avatars that play roles such as brand ambassa-
dors, interactive assistants, and dialog agents for customer
care. Its website claims that 90 percent of clients prefer sites
with avatars, and clients can expect a 30 percent increase in
customer satisfaction with the website experience [35].
A variation on the theme includes one of affective
computing’s most famous citizens, Max [36]. Max is a
museum guide, and as such he is a successor to the tape
recordings that guided earlier generations round their
museums. Like chatbots on a website, he carries out a
standard function with some added flair.
Allied to the museum, educators have long used simula-
tions to bring histories, other societies, and so on to life.
Emotion has been integral to their simulation of events like
the great battles of history. As digital simulations have come
into play, it is appropriate that they should simulate emotion
too, as was done in a recent simulation of Pompeii [37].
Simulation, particularly interactive simulation, is also
widely used in training, and has been since long before
computers were invented. Here too, it is wholly natural that
digital simulations should incorporate emotion into simu-
lated scenarios—for instance, incorporating the emotions of
the trainer and the patient into a scenario where the trainee
is giving an injection [38].
There are two sides to the reason for doubting that
resourcing professions in these various ways needs very
much ethical debate. The first is that in this area, nothing
that the affective computing community can realistically
expect to do will change very much. The impact is not like
the technologies that have transformed conception and
death. Technology may be able to make teachers marginally
more effective with marginally fewer resources. That is
positive, but it is not earth-shattering, and not worth
launching moral crusades about. The second side is that
professions are already regulated. If technological develop-
ments did pass uncomfortable power into their hands,
society would already have levers to claw it back. In areas
like that, there is no ethical story—only the ongoing revision
of codes that already keep the professions in tune with
public values.
3.3 Coda
Currently, many applications, perhaps most, fall into the
categories considered here. From an ethical point of view,
the issues are minor—small pleasures, small improvements,
minor risks, minor adjustments to practice.
There are issues to be considered. Chatbots can convey
hurtful messages as well as benign ones. It is hard to see
major issues involving autonomy, but there could be
problems of equity if, for instance, avatars disproportio-
nately represented particular ethnic types or personalities.
Issues like that are not specific to the new media, though:
They are well covered by existing regulations.
414 IEEE TRANSACTIONS ON AFFECTIVE COMPUTING,
4 MORAL POSITIVES
From long experience in affective computing, it seems that
moral aims often play a part in attracting people to the area.
They tend to regard the enterprise as distinctively humane
—a point reflected in the name of one of the influential
projects in the area, HUMAINE. Some of their activities
may be less beneficial than they hope, but others do seem to
be relatively straightforward moral positives, where it
seems reasonably clear that computing can do good by
engaging with emotion. This section summarizes activities
in that category under four headings.
4.1 Understanding Ourselves
Contemporary research on emotion has been much con-
cerned with questions about the nature of humanity that
Plato would recognize. They hinge on the relationship
between intellect and other facets of human life, of which
emotions are a salient example.
There is a long tradition of regarding intellect as the true
essence of humanity, to which other attributes—including
emotion—should be subordinated, if they cannot be
eradicated altogether. Augustine, for example, believed
that our intellects could enter Heaven, but our emotions
would not: “they...belong to this life, not to the life we hope
for in the future” [39]. Descartes famously regarded the
thinking part of him as his essential self [40]. Academic
psychology in the second half of the 20th century placed
huge emphasis on cognition, and for most of that time it
treated emotion as marginal.
In reaction, thinkers have protested that other attributes,
particularly emotion, are an essential part of our humanity.
Hume famously pronounced that “reason is, and ought only
to be the slave of the passions” [41]; and the Romantics
extolled passion as the center of both art and human
relationships. The late 20th century saw a distinctive turn in
the dialogue, hinging on the idea that emotion is not simply a
swirl of feelings overlaid on the systems that allow us to
function. On the contrary, it is a compendium of routines that
solve massively complex problems and that are essential to
normal functioning. The case was made in various ways:
through neuroscientific studies suggesting that damage to
areas associated with emotion seriously impaired the
individual’s ability to function [42]; by showing that emo-
tion-related individual differences predict practical success
[43]; by showing that people in appropriate emotional states
could perform various kinds of task more effectively [44]; by
reaffirming the link between emotion and moral judgment
[23]; and so on.
That debate has a clear moral component. It is a moral
problem if the dominant intellectual tradition belittles parts
of human nature that are actually integral to effective
action, wellbeing, and moral functioning. Correspondingly,
if computing can contribute to righting the balance, it is a
morally significant activity.
It seems fair to claim that research on emotion-oriented
computing makes distinctive inputs to the ongoing re-
evaluation of emotion. Two levels are involved. First,
artificial systems that have no emotional competences
whatsoever demonstrate painfully clearly that those com-
petences matter. Second, attempting to build systems that
VOL. 3, NO. 4, OCTOBER-DECEMBER 2012
have even minimal emotional competence underscores the
level of respect due to human abilities that we usually take
for granted.
Computing provides an abundance of evidence that
emotion-related competences are practically important, and
their absence is a significant drawback. There are examples
throughout this paper, but it is worth identifying a few key
cases here. Emotional distress is a highly functional signal,
and it is a serious limitation if an automatic answer system
cannot recognize it [46]. Similarly, both learning support
and interview systems are substantially more effective if
they can use the emotional channels that normally operate
alongside exchange of propositions to achieve rapport
between two parties [47]. When rapport breaks down,
emotion-related behaviors such as laughter are key to
defusing the situation [48].
Evidence for the sophistication of apparently simple
competences is even more pervasive. The introduction
stressed how difficult it has proven to simulate the most
peripheral aspects of emotional competence, perceptual and
productive. From the standpoint of someone trying to
match it, human ability to grasp how the other party feels,
and to plan accordingly, is awe-inspiring.
Allied to these is a shift in people’s views of humans as a
whole. Humans have learned to compare ourselves un-
favorably with computers: They are wholly rational, free
from emotion, and we are neither. Affective computing
turns that round, and invites people to realize that the
deficiency is on the machines’ side, and the rational path is
to make them, to the very limited extent that we can or want
to, more like us.
In these ways, and others related to them, it seems fair to
claim that emotion-oriented computing can make a real
contribution to rebalancing the ways that human beings
understand themselves. That, as suggested earlier, is part of
the good that the field can hope to do.
4.2 Humanizing Electronic Communication
There are links between the last area and another where it
seems hard to deny that there is clear moral good to be
done. It is already very common for electronic systems to
interact with a person roughly as another person might, and
it is all too common for the interaction to be bad. Websites
and answering services bully, patronize, demean, confuse,
infuriate, and simply don’t work in ways that cause people
real problems. They also make statements that it is hard to
regard as anything other than a lie—for example, “your call
matters to us” repeated at regular intervals, apparently
indefinitely.
The problem cuts in two directions. An emotionally
insensitive website, for instance, does not simply inflict
unpleasant experiences on users. It can also be understood
as representing the company or individual who owns it,
and systems can easily give an impression of their owners
that are very far from reflecting their actual values and
feelings. During the writing of this paragraph, an interac-
tion with a utility company provided an example. A
recorded message ended with “thank you,” but the
intonation rose at the end, and the “you” was both clipped
short and stressed. The impression was summary dismissal,
and it did not leave a good feeling toward the company.
COWIE: THE GOOD OUR FIELD CAN HOPE TO DO, THE HARM IT SHOULD AVOID
The point behind these examples is that technology is
already used in ways that impact on people’s emotions,
whether the designers intend it to or not. Given that it is
used in those ways, it certainly seems fair to say that if
emotion-oriented technologies can make malinteraction less
common, then they are doing good. It is not far fetched to
go further and say that the designers have a moral
obligation to consider the impact, and to seek out ways to
ensure that it is not needlessly unpleasant.
There are abundant examples of the way that emotion-
oriented technology does or could contribute to preventing
needless unpleasantness. One which has already been
mentioned involves detecting when a person is distressed,
and a human being should be alerted [49]. An interestingly
subtle example comes from the technology of in-car satellite
navigation systems [50]. Nass et al. realized that because of
the way directions were given, complying infringed on the
driver’s self-respect. Changing the voice used to deliver
them meant that the device could be much more effective.
The example of the utility answering phone points to a
related set of possibilities which is potentially quite large.
When messages are recorded by humans, it is quite difficult
to control the emotional coloring with any precision. The
best contemporary text to speech systems convey emotional
coloring quite effectively [51], and using them rather than
conventional recording technology would make it relatively
straightforward to generate a more emotionally appropriate
“thank you.”
All of these cases involve relatively simple kinds of
electronic communication that are already part of people’s
lives, where it is not difficult to improve the way emotional
aspects of communication are dealt with. It hardly seems
controversial to regard improvement in that kind of
interaction as an ethical positive. Matters are not altogether
simple, though. There are cases where it is difficult to
separate appropriate emotional coloring from a role that
one might want to argue machines should not take—for
example, in the case of artificial companions or teachers.
Those are taken up later.
4.3 Broadening Access
It is a recognized issue that technologies which facilitate
access for most people reduce it for others. The classical
example is the telephone, which accentuated the isolation of
people with hearing difficulties because it became the norm
for social interaction to use a medium where the commu-
nication channels that they had (sign, lipreading, writing,
and facial expression) could not be used to supplement the
one that they lacked [52].
Since the late 20th century, computer-based systems
have become the default way of accessing more and more
services—flights, taxis, shopping, news, in-home entertain-
ment, and so on. As with deafness and the telephone, the
corollary is that those who have difficulty with the systems,
and the interfaces that they use, are excluded. As a result,
the Principalist criterion of equity comes into play. Ensuring
that those groups are not excluded becomes an ethical issue.
The link between emotion and inclusion is not con-
clusive, but there are several reasons for taking it seriously.
At the most general level, it seems reasonable a priori to
assume that the more natural the interface, the more widely
415
usable it will be; and conversely, the more people have to
adjust their communicative style to use an interface, the
fewer people will be able to make the adjustment.
Expressing emotion certainly is part of human beings’
natural communicative repertoire, and that alone suggests
integrating emotion into interfaces could help to make them
more natural and more accessible. More specifically,
emotional colouring is an integral part of “really natural
language” [53], that is, language as people spontaneously
use it among themselves. Hence, one might expect its
absence to be one of the barriers to developing “really
natural language” interfaces, which would seem to offer a
very considerable broadening of access. That is quite a
strong expectation in at least some areas: It has long been
recognized that absence of affective coloring is one of the
factors that makes synthesized speech unpalatable to
human listeners [54].
More specific evidence exists in a few areas. Limited
uptake of technology among older adults is recognized as
an issue. Affect may well be a specific issue because age
brings changes in the way cognition and emotion interact—
for example, emotion-related information is used to
compensate for cognitive decline, and emotion regulation
is prioritized [55]. These appear to be reflected in direct
studies of interactions between older adults and technology.
Medeiros et al. [55, p. 33] hypothesize that the dominant
level of response to technology changes with age: “sensory
for youth, cognitive for younger adults and affective for
older adults.”
Last but not least, it is very widely recognized that
“computer anxiety” is a major factor in the uptake of
computing technology. It is known to affect different
groups and types of people differentially [57], [58]. It seems
eminently reasonable to believe that their disadvantage
would be reduced if we could develop interfaces that were
less demanding, and that had some ability to respond to
anxiety if they did provoke it.
Improvements in access are not something that can be
guaranteed, but ethics does not depend on guarantees. It is
reasonable to think that developing more natural interfaces
with a degree of emotional competence as an integral part
would address inequities that currently exist. Given that, it
is hard to deny that the goal of making that kind of interface
a reality is an eminently moral one.
4.4 Additional Resources to Make Judgments
A contrasting category includes the work that brought our
group into emotion-oriented computing many years ago
[59]. It involves supporting people who are already making
judgments that involve emotion, which have substantial
implications for the person about whom the judgment is
made, by providing them with new kinds of instrumenta-
tion. In the most straightforward cases, the process involves
automatic identification of features that an expert observer
would use to make the judgment, but that someone who is
not an expert observer might not register reliably.
Our work involved psychiatric issues, and specifically
the diagnosis of flat affect. Flat affect is a feature of some
kinds of schizophrenia, and it is indicated by the absence of
signs of affective expression. The most prominent signs are
in the face and the voice. There are large individual
416 IEEE TRANSACTIONS ON AFFECTIVE COMPUTING,
differences in people’s ability to make analytic judgments
about vocal patterns: Some people have “a good ear,” others
do not. There is no guarantee that psychiatrists will be in
the first category. Hence, instrumentation designed to
supplement their judgment would appear to be a useful
resource. The task is not simply a matter of signal
processing because it is necessary to find the relevant
features (i.e. features that characterize voices which expert
judges regard as showing flat affect), and to establish
reliable ways of extracting them. Early efforts encountered
problems on both counts, but the general area has
developed since. For example, recent studies on the related
problem of vocal signs of depression have shown that the
durations of particular speech sounds relate to particular
aspects of depression more closely than more global (and
more easily recovered) speech features do [60].
Similar work is being done with facial expression. So, for
instance, depressed patients show reduced frequency of
Duchenne smiles, and less overall facial animation [61].
These are variables that it is now possible to measure
automatically so that clinicians can be given the relevant
information.
It is worth stressing that what is being considered here is
not allowing automatic systems to diagnose, for instance,
flat affect or depression: It is using them to give clinicians
information about patterns that are relevant to their
diagnosis. Unlike a diagnosis, the identification of patterns
is concerned with physical events that are directly
observable; and confidence in it could theoretically ap-
proach 100 percent, though in practice various kinds of
noise will usually mean that the level is lower. In logical
terms, the task is classification, not abduction (reasoning to
the most probable cause or explanation). The distinction can
be blurred in practice, for instance if the machine output
uses descriptors that invite interpretation (e.g. tense or
trembling voice), or if the user treats theoretically objective
descriptors as a coy way of specifying a mental state. That
kind of blurring does raise ethical issues, but it is discussed
later. It is also worth mentioning that human beings in
general are very bad at identifying objective patterns in the
face and voice, partly because they are so strongly disposed
to jump directly to what they believe lies behind the
patterns. For example, even phoneticians trained to code
voice quality achieve quite low rates of agreement beyond
coarse-grained description [62].
Interestingly, this is one of the areas where specialist
research is being translated directly into systems that can be
used more widely. Work by Picard on somatic signs of
emotion and El Kaliouby on facial signs form the basis of
products that have recently been released [63], for use in the
first instance by teams who are studying emotion-related
phenomena and want to reduce their reliance on impres-
sionistic judgments. Of course, any instrumentation can be
abused, but providing instrumentation that is valid if it is
used appropriately is hard to classify as anything but a
moral good.
4.5 Therapeutic Applications
This section partly overlaps with the last, but it brings
different issues to the fore. Emotion-oriented technologies
have various therapeutic uses that it is difficult to deny are
VOL. 3, NO. 4, OCTOBER-DECEMBER 2012
ethically positive, not so much because of their logical status
as because of their usefulness.
One example which seems relatively clear is pain
detection. (Pain is not an emotion, but in the traditional
use of the term, it is an archetypal affect.) Ingrained social
norms inhibit patients from describing pain objectively even
in situations where that is what the clinician needs, for
instance to assess when movement in a damaged shoulder
becomes painful. It has been shown that certain facial
actions are characteristic of pain [61]. They can now be
detected automatically, and automatic pain detectors con-
structed on that basis can indicate quite precisely when pain
occurs [64], [65].
A very different example comes from “cybertherapies,”
where the reality is that two humans are interacting, but
affective technology is used to present one —the “therapist”
—as an avatar [66]. The technology is a device that allows
socially anxious individuals to achieve self-disclosure that
they would find difficult when sensory evidence made it
impossible to ignore the fact that the other party was
another human being. No deception is involved: The
anxious individual knows what is happening. Nevertheless,
interacting with the avatar avoids problems that would
obstruct other forms of interaction. If that presentational
device allows particular groups of people to deal with
problems that would resist other forms of therapy, it is
difficult to see why other people should doubt that it is
ethically positive.
5 MORAL NEGATIVES
This section contains three parts, one dealing with negatives
that include emotion-oriented computing as part of a much
wider case against technology or computing; the second
dealing with negatives that are very specifically related to
emotion; and the third a middle ground, where the issues
can arise when emotion is not involved, but are particularly
likely to arise when it is.
5.1 General
It is useful to start with the general issues, mainly to clear
the air. It seems reasonably clear that some arguments
against emotion-oriented computing are really particular
applications of arguments that stand or fall on much more
general grounds. If that is the case, it simply confuses the
issue to talk as if the problems lay with emotion-oriented
computing in particular when in fact the arguments apply
equally well to a wide range of technologies.
The most general of all the positions is the one that was
described earlier as nature-oriented. Critical arguments
often seem to be grounded, explicitly or implicitly, in
Rousseau-like suspicion of any deviation from the state of
nature. A computer behaving like a person is certainly not
natural, and given that general perspective, it must there-
fore be a cause for concern. But if that is the premise, it
should be brought out and debated explicitly. It is difficult
to believe that an honest debate would reveal many people
willing to divest themselves of everything in their sur-
roundings that was artificial.
A second position is related, but narrower. It is
concerned is not with technology in general, but with
COWIE: THE GOOD OUR FIELD CAN HOPE TO DO, THE HARM IT SHOULD AVOID
technology that blurs the line between humans and
machines. Ideas like that are often raised in debates, but
on inspection, are difficult to express in a defensible way.
One form of the argument is saying that artifacts which
are too human-like are doomed to fall into the famous
uncanny valley [67]. If that were true, it would seem to be a
practical problem, not an ethical one—much as it is a
practical problem that there are certain smells that humans
dislike (presumably nobody thinks that plastics with an
unpleasant smell are an ethical issue). That aside, it is not at
all clear that uncanniness is associated with a valley
dividing acceptably human-like things from everything
else [68]. The problem may be more to do with things that
mix features of the living and the dead.
Another argument claims that the way to meet human
needs is to design things that fit human requirements, not
things that mimic humans [69]. Again, that seems to be a
practical issue, not an ethical one. It is also not obvious that
there is an opposition: Why not do both? The logic would be
clearer if it were being argued that it is God’s place to create
human beings, and it is sacrilege when humans create
human-like things. Presumably, though, that is not an
argument that many people in computing would want to
pursue. One is left feeling that there is something in this area
that bothers people, but that it is difficult to know what it is.
Beyond all that, it is not obvious that emotion-oriented
computing has to involve artifacts that are particularly
human-like. A serious web page designer ought to take
careful account of the way the page affects a user’s
emotions. That does not mean it has to have a smiling face
and a sweet voice. For example, offering to send an error
message when the application crashes is one of the better
ways of acknowledging the user’s emotions. So even if the
objection were clear, it would only apply to some parts of
emotion-oriented computing.
A third position is easier to understand. It is to do with
creating slaves. It has been pointed out that our society uses
computers much as the Romans used Greek slaves—except,
of course, that the Romans required the slaves to speak
Latin, not versa. The argument touches an uncomfortable
intuition that something about human beings likes being
able to turn to an agent with amazing abilities, and say “do
this” and it is done, “find that” and it is found, “terminate”
and it terminates. It is very obvious why society should
worry about encouraging the part of us that likes to do that.
It does seem right to worry about creating systems that
encourage that attitude, and it is not obvious what to do
about it. On the other hand, it applies whether or not the
systems have emotional capabilities.
One particular variant of that problem does have
particular ties to emotion, though. People do take particular
pleasure in forcing another agent to take part in sex or to
suffer pain. Nobody should seriously doubt that there could
be a market in that area for intelligent systems with
emotional capacities—as objects for sex and sadism. There
may be grounds to worry about creating systems that
encourage that side of humanity.
There is one other general negative that is perhaps
harder to discount than any of the others. It is about
concentration of resource. That kind of moral question is
417
quite salient when the resource is devoted to making
computer games marginally more entertaining. It is less
worrying when the resource is being concentrated for a
purpose that seems morally good in and of itself, such as
letting everyone in our society access the resources that
computing makes available. However, the question is not at
all specific to emotion-oriented computing. It applies
equally to improving processor speed or designing more
alluring cars.
These are the general moral negatives that seem to recur.
It is clearly reasonable to be concerned about creating
mindsets that accept slavery, and about concentrating
resources on already rich sections of humanity. However,
these things do not make emotion-oriented computing a
worse area to work in than most others. Society provides a
host of ways for people to exercise power in unsavory ways,
and to concentrate resources. Confronting those moral
problems means confronting society, not confronting emo-
tion-oriented computing.
5.2 Emotion-Specific
At the opposite extreme, there are moral negatives that
seem to be very specific to emotion-oriented computing.
Perhaps surprisingly, though, it is difficult to find many.
The first moral negative is that there is a peculiar kind of
lie at the heart of the enterprise. A beaming Duchenne smile
says, “I am really happy”; a trembling voice says, “I am
afraid.” But if the signs are being given by an agent, there is
no happiness or fear inside the box, just symbols being
connected slightly differently or parameters being adjusted.
The whole process is about creating a lie—the appearance
of emotion without the reality.
That argument can be understood in various different
ways, some of which carry much more weight than others.
As might be expected, the bias here is to focus on the
concrete issues.
The most abstract form of the argument is open to
several rebuttals. People do not generally regard it as
deception when artifacts convey an impression of some-
thing that is not there. It takes a philosopher to dwell on the
idea that a picture constitutes a lie. In the specific case of
emotion, it is not at all obvious that people in general care
very much about the relationship between emotion-related
signs and the irreducibly private experiences of the agents
who give them. They would consider it a lie if an agent’s
overt signals were seriously at odds with its private
intentions. It is not at all clear that they would consider it
a lie if they discovered that a colleague had spent the past
decade unable to experience subjective feelings of emotion,
but had continued to give signals that accurately reflected
his/her evaluations and intentions.
Arguments like these undercut a strong kind of objection
to affective computing, based on concern that it necessarily
entails deception. However, they do not refute the objection
that affective computing invites particular kinds of decep-
tion, which are particularly distasteful. That kind of
argument can be cast at two levels.
The more benign level argues that the ability to
manipulate emotional signs at will brings with it a
particularly heavy moral obligation. People know how to
discount words because they know that words can be used
418 IEEE TRANSACTIONS ON AFFECTIVE COMPUTING,
to lie to them. However, the agents that they know—i.e.,
humans—do not have the same freedom to generate
dishonest emotion-related signals. As a result, people
cannot help treating emotion-related signals as trustworthy
in a way that words are not. Therefore, if an agent does use
emotion-related signals dishonestly, people are particularly
vulnerable to it. For that reason, there is a special need to
ensure that artificial agents do not generate misleading
emotion-related signals.
The less benign level accepts the argument in the last
paragraph, but extends it pessimistically. The extension is
that the way people read emotion-related signals depends
on the sender having particular, peculiarly human combi-
nations of connections and dispositions. Therefore, what is
required for an agent to give signals that will not grievously
mislead a human is not just that the designer tries sincerely
to avoid deception: It is that the designer somehow
manages to endow the agent with the whole network of
connections and dispositions that humans will presuppose
when they try to interpret the signals.
From a pragmatic point of view, it seems fair to think
that the truth is somewhere between the last two argu-
ments, and its impact is qualified by practical considera-
tions. It is hard to doubt that affective computing will open
up opportunities for new kinds of deception: It will be
technically possible for agents to give misleading signals,
which uncritical people will be overwhelmingly disposed to
accept. Some avatars will be confidence tricksters and some
people will be conned.
It is another matter entirely to assume that the deception
will be any less manageable than older forms. One reason
for optimism is human flexibility: People do learn to deal
with the emotion-related signals offered by species with
significantly different kinds of emotional organization. A
second has been repeated over and over: There is no
foreseeable prospect of avatars matching the deceptive
capabilities of a human being. The third is that humans are
capable of policing new technologies. One of the funda-
mental reason for writing about ethics and emotion-
oriented computing is precisely to anticipate where policing
may be required. Deception carried by emotional signs is
one of the areas where policing is most obvioussly needed.
But the fact that an activity needs to be policed does not
mean that it should be banned.
The second specific moral negative is effectively a mirror
image of a positive claim that was made earlier.
Emotion-oriented computing can give concrete expres-
sion to the sense that emotion matters. However, it can also
give concrete expression to the sense that emotion is
fundamentally mundane—a hardwired coarse communica-
tion system. It can encourage people to see a smile not as a
thing of wonder and richness, but as a particular kind of
token in a communicative exchange.
It is not an issue that should be dismissed, but there is
good reason to doubt that the problem is particularly grave.
Comparable dehumanizing metaphors have been part of
cognitive psychology for decades—attention is a filter,
memory is a filing cabinet, and so on. However, it is not
obvious that people who have been exposed to metaphors
of that kind in the course of psychology training suffer
particular ill effects.
VOL. 3, NO. 4, OCTOBER-DECEMBER 2012
5.3 Problems Exacerbated by Emotion
The previous section focused on concerns that are specific
to emotion. However, there are many cases where the issue
is that research on emotion could contribute to a more
general problem. Key examples are sketched here, working
from the issues where the risks seem least clear to those
where there seems most.
5.3.1 Surrogacy
A widely cited concern was dramatized in Isaac Asimov’s
description of the planet Solaria [70]. People there ceased to
interact with other human beings: They only interacted with
robots or via electronic communications. Now, Asimov
could walk down the street and see people virtually
indifferent to their real surroundings—listening to music
on iPods or talking on mobile phones, looking up when
satnavs tell them where to turn at the next corner.
There is no doubt that immersion in a virtual world is
happening. It is reasonable to worry about developments
that could encourage people to move deeper into it. What is
not clear is whether new types of emotion-oriented
technology are likely to exacerbate the problem. Current
techniques do not allow us to create virtual worlds that are
emotionally rich enough to compete with reality, and it is
hard to imagine that changing in the the foreseeable future,
if ever. Refining old technologies (such as toys, music, films,
TV, and action games), and making access to them easier,
seems a more believable route to withdrawal from human-
to-human contact.
5.3.2 Thought Control
It is clear that information technology has the potential to
produce formidable instruments of thought control. People
can be trained and taught very effectively using old
computer-based learning techniques or new virtual reality
systems. They can be shown things that happened long ago
or never happened. They can be saturated with advertise-
ments tuned to their individual style on nicely calculated
schedules. It is hard to doubt that adding emotional
sophistication enhances the potential of these technologies
to convince and to sway people, though it is not clear how
much difference it makes.
Some of the most obviously benevolent pieces of
emotion-oriented computing have been in this area. An
excellent example is FearNot, the program that tries to teach
children how to deal with bullying [71]. Good teaching
always has appealed effectively to emotion, and enhancing
that tradition technologically has obvious potential for
good. However, there is equally obvious potential for ill.
Schools can teach pupils to hate minorities as easily as to
stand up to bullies. Candidates with great resources can
finance campaigns that slip telling emotional messages into
web pages day in and day out.
The scenario being condidered here is very like one that
was considered earlier and classified as unproblematic. The
difference is that the earlier scenario assumed both that
there was professional regulation and that it was ethically
sound. Clearly, though, regulation may break down, or be
corrupted, and that changes things.
COWIE: THE GOOD OUR FIELD CAN HOPE TO DO, THE HARM IT SHOULD AVOID
The underlying issue here is a general one. It is hard to
think of an exception to the rule that technologies which
empower humans may be used for ill as well as for good.
The question is whether the threat of emotion-oriented
technologies falling into manipulative hands has particu-
larly worrying features.
There is no easy way to answer the question. That is
partly because the real potential of emotion-oriented
technology is so unclear, and partly because of uncertainties
surrounding the question of how illegitimate control may
be established and resisted. Among other things, if experts
cannot be trusted to resist illegitimate control of technolo-
gies, it is not clear how much is gained if they refuse to
develop them under more benign regimes. The issues
deserve a clear-headed debate. It impinges on emotion-
oriented technology, but it is not restricted to it.
5.3.3 Misrepresenting Scientific Understanding
It is assumed in this paper that there are moral issues
surrounding the way we understand or misunderstand
humanity. If so, misrepresenting the depth of our under-
standing is a moral issue. The issue affects technology in
general because demonstrations can eaily give a profoundly
misleading impression of the extent to which their creators
understand some aspect of humanity.
The archetypal example was Weizenbaum’s ELIZA [72],
which actually depended on simple tricks, but which is
easily portrayed to a naive user as evidence of a funda-
mental breakthrough in understanding human intelligence
in general, and language in particular. It is revealing that
Weizenbaum himself reacted by making strong public
statements about the shallowness of AI [73].
Emotion particularly lends itself to ELIZA-like demon-
strations. The toy industry testifies that it takes no intellectual
understanding of emotion to make things that evoke it.
Nevertheless, if a device engages emotional responses, it is
difficult not to feel that it vindicates the creator’s claim to
understand something deep about emotion.
The result is that scientists have to strike a careful
balance. It is perfectly legitimate to engage an audience by
showing robots that flutter their eyelashes [74] or appear to
nuzzle up to a human [75]. But it is an ethical problem if
they use the spontaneous response to invite false evalua-
tions of the understanding behind them.
5.3.4 SIIFs
The last category is the one that raises the clearest concerns.
It links back to an earlier example. It seems thoroughly
ethical to provide a psychiatrist with printouts that show
what a patient’s voice is actually doing. It would be entirely
different if the computer monitored the voice, and sent
word to the police that the speaker should be taken to a
secure mental hospital.
The example illustrates a general class of systems that
does need to be taken seriously. They have been called
semi-intelligent information filters, or SIIFs for short [76].
The important point about SIIFs is that they try to interpret
data and draw conclusions that have serious implications
for the people concerned.
The area raises a multitude of concerns. The point is not
that machines are making decisions that humans should be
419
making—it is that machines are making decisions that they
may not be very good at; and the consequences of a wrong
decision are serious. One of the reasons why a judgment is
likely to have serious consequences is if it has moral
implications.
Emotions are not the only kind of case where that kind of
issue is likely to arise, but they are an important one. It is
reasonable to object if machines are left to make judgments
about whether people were fighting or behaving suspi-
ciously, and so on. Emotions belong in a similar bracket—
areas where machines should not be trusted to make
decisions for the foreseeable future.
That is a technical judgment. It is grounded in long
experience of research on machine perception, which gives
a sense of what the algorithms that we can write at present
can and can’t be expected to do. Unfortunately, not many
people do understand what can and can’t be expected from
machine perception. That is part of the problem. People are
not educated to judge how much weight should be put on a
readout from a machine that says, “she was distressed/he
was angry”—and so on.
From a nonexpert’s point of view, SIIFs are very
attractive. There is long-standing interest in monitoring
the states of drivers and pilots [77]. Retailers would
welcome a machine that detected whether staff were
behaving in a friendly way toward customers, and educa-
tion authorities would love a machine that detected
whether staff were showing inappropriate attraction to
children. Unfortunately, that would almost certainly mean
people losing their jobs because they had a gruff voice even
if they were actually friendly, or a breathy voice which had
nothing to do with their sexual orientation.
It is worth separating three levels of concern in this area.
One is a general concern about too much surveillance. Like
so much else, that is a wider debate, not specifically for
emotion-oriented computing. The second is a technical
concern about competence. There the emotion-oriented
computing community has a very specific input to make.
The key is adequate assessment of what a given machine
will or will not achieve. That takes genuine knowledge of
the area, both the computational side and the emotional
side. SIIFs need to be certified as fit for purpose by people
who know what they are doing.
The third level follows from that. It is extremely difficult
to communicate to a nonexpert what an emotion-oriented
system can and cannot achieve. The problem is particularly
acute with SIIFs, where there will usually be an accessible
way of describing what the system does—it detects anger,
or attraction, and so on. It is difficult for nonexperts given
that kind of description to avoid feeling that they under-
stand well enough what the system does. In reality, though,
no system that exists or can be foreseen does anything like
what a nonexpert infers from an accessible description. It is
likely to detect, with doubtful reliability, some of the
patterns that are associated with a particular expression of a
particular state in a particular situation; and if they are
present, to infer that the state is present without testing the
plausibility of the conclusion against the multitude of
contextual and personal considerations that a human would
bring into play.
420 IEEE TRANSACTIONS ON AFFECTIVE COMPUTING,
It may well be that in some situations, it is in a user’s
interest to deploy a system like that despite its limitations.
However, the user cannot make the decision unless he or
she genuinely understands what the limitations are. Similar
points apply to others who may be drawn in, for example
police and courts. Understanding the limitations depends
on experts accepting responsibility not only for telling the
unvarnished truth about their creations, but also for finding
ways to convey it to potential users who may be deeply
reluctant to go beyond what seems to them a perfectly
adequate grasp.
There is a strong case for saying that after other avenues
have been mapped, the single most difficult ethical issue is
truth. People find it singularly difficult to state, or to absorb,
what the devices do because the vocabulary that comes to
mind applies to systems of an entirely different order. If
they respond by conspiring to give and to accept descrip-
tions that misrepresent the reality, then there is a double
ethical problem: a problem of honesty and a problem of
unwanted consequences.
6 ISSUES FOR ETHICAL REGULATION
The discussion so far has prioritized the question of
whether emotion-oriented technologies as they exist now,
and as they can realistically be envisaged, are intrinsically
pernicious. The broad conclusion is that they are not: There
is good that they can do, and the harm that they might do
can be forestalled. Clearly that does not end the matter,
though. It indicates that there is a need to consider the kind
of action that is needed to forestall harm.
A key part of the argument is that existing kinds of
regulation either cover a large proportion of the things that
can be done or could easily be extended to cover them. That
applies to resources in teaching, training, advertising,
communication, and so on. Although regulation tends to
lag behind new technologies, it tends to catch up, not least
because reputable users want regulation [78]. The same
applies to participants in research needed to develop
systems, where standard ethical committeees are being
brought into play [26].
There are issues that are nonstandard, though, and
where real care is needed to avoid problems. Following on
from the previous section, the single most difficult issue is
misinterpretation. Central to the problem is the principle of
“pars pro toto.” It is difficult for people to avoid inferring
that if a system shows fragments of behavior that are
strikingly human, it has other characteristics that would be
associated with those fragments in a human. The problem is
exacerbated by the fact that most people do not have the
technical background to understand what they observe in
any other way.
That problem is a major factor in dealing with several
difficulties that were mentioned above. One is ethical use of
SIIF systems. If people are considering installing a system to
detect distress, they need to know not only that it succeeds
in 95 percent of cases in a trial, but also that, unlike a
human, faced with an unforeseen form of distress, it will
very probably misclassify the situation completely. Another
involves signaling emotion. There is an immediate appeal
about generating signs that convey warmth or concern, but
it becomes highly dubious if users take them to mean
VOL. 3, NO. 4, OCTOBER-DECEMBER 2012
everything that they would mean if a human being gave
them. If people choose to have endearing artificial compa-
nions, the provider needs to find ways of ensuring that their
endearing features do not lead people into relying on them
in inappropriate ways. The same point goes for public
understanding of science: The way the human mind works
makes it all too easy for a superficial demonstration to
suggest profound understanding. It seems quite possible in
principle to find ways of dealing with the peculiar ethical
problems that arise in these areas, but real work is needed
to establish them.
There are strong links between that and another
immediate challenge. It might be called lieutenancy—the
extent to which an authority figure can or should invest part
of that authority in a machine—a virtual teacher, diet coach,
therapist, etc. The question here is how to draw the line
between acceptable delegation and delegation that either
infringes on the recipient’s autonomy, or abrogates the
authority figure’s responsibility.
There are intriguing general discussions about autonomy
[79]. The point here, though, is more straightforward. It is
that this is an area where there is need of regulation (formal
or informal) to ensure that potentially useful applications
are not abused. There is no obvious reason to doubt that
workable regulation can be established. Some elements
seem reasonably obvious, such as establishing that a
professional cannot legally delegate responsibility to an
automatic system; and that would provide an incentive for
professionals to confront the problem noted above, estab-
lishing exactly what a system does.
It would be dishonest not to mention applications that
would be widely seen as morally repugnant, but that
history suggests it would be extremely difficult to regulate.
It has been argued that sex robots are inevitable [80], and
examples have already been reported [81]. The case is as an
exception that proves an important rule. It seems unlikely
that many other applications will have the kind of appeal
that would lead potential users to defy reasonably framed
agreements on legitimate and illegitimate uses.
7 Dark Imaginings
Looking at the emotion-related technologies that currently
exist, and that can realistically be envisaged, it is hard to
argue that rational, well-informed people would regard
them as threatening. However, that does not alter the fact
that substantial numbers of people do find the idea of
emotion-related technology disturbing. Considering how to
react to that is an ethical problem in itself: What status has
widespread disquiet with a technology that seems to have
real (if undramatic) potential for good, and to pose few risks
that cannot be controlled?
If the overall view that has been sketched in this paper is
accepted, then part of the answer would seem to be simple.
The field will not be able to do the good that it has the
potential to do if it is dogged by public suspicion of the
systems it makes. The suspicion would be reduced to
reasonable and limited wariness if the public understood
the systems for what they are, rather than believing that
they are, or might easily become, quasi-living creatures
with disturbing powers. On that argument, there is an
COWIE: THE GOOD OUR FIELD CAN HOPE TO DO, THE HARM IT SHOULD AVOID
ethical burden on people in the field to portray the systems
for what they are, and to avoid the kind of inflation that
science, and science journalism, usually regard as legitimate
sales talk. Conversely, talking up the risk that the technol-
ogy will produce unnatural monsters is not a harmless
intellectual sport: It is unethical because it makes it less
likely that good will be done.
It may be that however clearly the reality is portrayed,
people will be unable to resist the draw of “pars pro toto”
and therefore will be unable to reach a modus vivendi with
emotion-oriented systems—too prone to misunderstand
them for comfort or safety. If so, then enterprise has to be
abandoned. However, the proliferation of harmless exam-
ples that is already occurring (discussed in Sections 3 and 4)
suggests that that outcome is avoidable.
8 CONCLUSION
Debate in practical ethics is an interplay between specifics
and generalities. Principles are articulated in the light of
what seems right in particular cases, and judgments about
particular cases are measured against more general princi-
ples. As Beauchamp and Childress put it, “Progressive
specification often must occur to handle the variety of
problems that arise, gradually reducing the dilemmas and
conflicts that abstract principles lack sufficient content to
resolve” [82].
Until recently, ethical debate about emotion and comput-
ing has been unavoidably skewed toward generalities: The
shape of the specifics has simply not been clear enough to
support very much argument. However, research has
accumulated to the point where arguments from specifics
can make a more substantial input. This paper has tried to
contribute to that kind of development. It is to be expected
that it will seem unbalanced to those whose arguments
prioritize generalities: After all, the aim is to redress a skew
toward debates that say rather little about actual systems and
projects and the specific issues that they raise, for good or ill.
This paper offers a way of developing arguments from
specifics that is undoubtedly crude, but that seems a natural
starting point: to sort activities and aspirations into those
that potentially do good, those that potentially do harm,
and those that seem unlikely to do either; and to look for
steps that might help to maximize the good and minimize
the harm. It is to be expected that others will feel that key
examples have been left out and will question key
judgments, but if the framework allows that kind of
argument to develop, that is all to the good.
Bringing specifics to the fore is partly about achieving a
logical balance, but only partly. Part of the motivation is an
impression that arguments from principles tend not to
reflect a fundamental feature of the field, which is the depth
of good intention behind it. Specific projects do aim to
humanize computing, promote access, support delicate
decisions, and facilitate therapy; and there is a widespread
sense that the discipline makes a distinctive contribution to
the long project of bringing the emotional side of human
nature in from the cold. To somebody whose daily work is
oriented toward goals like those, it seems wrong that people
should try to reach ethical judgments on it without
understanding that those are the goals. It may be that the
good intentions are the kind that pave the road to Hell, but
421
it would be sad indeed if that conclusion were allowed to
take hold without the public having understood what in
reality was being done.
ACKNOWLEDGMENTS
The author wishes to thank Peter Goldie for insightful
discussions of the topic. This paper draws on work that was
supported by several grants from the European Commu-
nity: HUMAINE, SEMAINE, and SSPnet.
REFERENCES
[1] M.B. Arnold, Emotion and Personality: Vol 2. Physiological Aspects.
Columbia Univ. Press, 1960.
[2] R. Cowie, N. Sussman, and A. Ben-Ze’ev, “Emotions: Concepts
and Definitions,” Emotion-Oriented Systems: The Humaine Hand-
book, P. Petta, C. Pelachaud, and R. Cowie eds., pp. 9-30, Springer-
Verlag, 2011.
[3] J. Russell and L. Barrett-Feldman, “Core Affect, Prototypical
Emotional Episodes, and Other Things Called Emotion: Dissecting
the Elephant,” J. Personal and Social Psychology, vol. 76, pp. 805-819,
1999.
[4] J. Panksepp, “At the Interface of the Affective, Behavioral, and
Cognitive Neurosciences: Decoding the Emotional Feelings of the
Brain,” Brain Cognition, vol. 52, pp. 4-14, 2003.
[5] K.R. Scherer, “What Are Emotions? How Can They Be Mea-
sured?” Social Science Informatics, vol. 44, no. 4, pp. 695-729, 2005.
[6] R. Cowie, “Describing the Forms of Emotional Colouring That
Pervade Everyday Life,” Oxford Handbook of Philosophy of Emotion,
P. Goldie,ed., pp. 63-94, Oxford Univ. Press, 2010.
[7] Affective Computing and Intelligent Interaction, A. Paiva, R. Prada,
and R. Picard, eds., Springer-Verlag, 2007.
[8] Proc. IEEE Third Int’l Conf. Affective Computing and Intelligent
Interaction, J. Cohn, A. Nijholt, and M. Pantic, eds., vol. 1, 2009.
[9] Emotion-Oriented Systems: The Humaine Handbook, P. Petta,
C. Pelachaud, and R. Cowie, eds. Springer-Verlag, 2011.
[10] A. Batliner, B. Schuller, D. Seppi, S. Steidl, L. Devillers, L.
Vidrascu, T. Vogt, V. Aharonson, and N. Amir, “The Automatic
Recognition of Emotions in Speech,” Emotion-Oriented Systems: The
Humaine Handbook, P. Petta, C. Pelachaud, and R. Cowie, eds.,
pp. 71-99, Springer-Verlag, 2011.
[11] G. Castellano, S.D. Villalba, and A. Camurri, “Recognising
Human Emotions from Body Movement and Gesture Dy-
namics,” Affective Computing and Intelligent Interaction, pp. 71-
82, 2007.
[12] Z. Zeng, M. Pantic, G.I. Roisman, and T.S. Huang, “A Survey of
Affect Recognition Methods: Audio, Visual, and Spontaneous
Expressions,” IEEE Transa. Pattern Analysis and Machine Intelli-
gence, vol. 31, no. 1, pp. 39-58, Jan. 2009.
[13] M. Brendel, R. Zacharelli, B. Schuller, and L. Devillers, “Towards
Measuring Similarity between Emotional Corpora,” Proc. LREC
Workshop Emotion and Affect, 2010.
[14] R.B. Knapp, J. Kim, and E. André, “Physiological Signals and
Their Use in Augmenting Emotion Recognition for Human-
Machine Interaction,” Emotion-Oriented Systems: The Humaine
Handbook, P. Petta, C. Pelachaud, and R. Cowie, eds., pp. 133-
159, Springer-Verlag, 2011.
[15] M. Shaikh, H. Prendinger, and I. Mitsuru, “Assessing Sentiment of
Text by Semantic Dependency and Contextual Valence Analysis,”
Affective Computing and Intelligent Interaction, pp. 191-202, 2007.
[16] R. Cowie and R. Cornelius, “Describing the Emotional States That
Are Expressed in Speech,” Speech Comm., pp. 5-32, 2003.
[17] M. Schröder, H. Pirker, M. Lamolle, F. Burkhardt, C. Peter, and E.
Zovato, “Representing Emotions and Related States in Technolo-
gical Systems,” Emotion-Oriented Systems: The Humaine Handbook,
P. Petta, C. Pelachaud, and R. Cowie, eds., pp. 369-387, Springer-
Verlag, 2011.
[18] B. Schuller, S. Steidl, A. Batliner, F. Burkhardt, L. Devillers, C.
Müller, and S. Narayanan, “The INTERSPEECH 2010 Paralinguis-
tic Challenge,” Proc. Interspeech 2010, pp. 2794-2797, 2010.
[19] R. El Kaliouby and P. Robinson, “Real-Time Inference of Complex
Mental States from Facial Expression and Head Gestures,” Proc.
IEEE Int’l Conf. Computer Vision and Pattern Recognition, vol. 3,
pp. 181-200, 2004.
422 IEEE TRANSACTIONS ON AFFECTIVE COMPUTING,
[20] M. Wöllmer, F. Eyben, S. Reiter, B. Schuller, C. Cox, E. Douglas-
Cowie, and R. Cowie, “Abandoning Emotion Classes—Towards
Continuous Emotion Recognition with Modelling of Long-Range
Dependencies,” Proc. Interspeech 2008, pp. 597- 600, 2009.
[21] J.-C. Martin et al., “Coordinating the Generation of Signs in
Multiple Modalities in an Affective Agent,” Emotion-Oriented
Systems: The Humaine Handbook, P. Petta, C. Pelachaud, and
R. Cowie, eds., pp. 349-367, Springer-Verlag, 2011.
[22] http://www.parorobots.com/, 2012.
[23] P. Goldie, The Emotions: A Philosophical Exploration. Clarendon
Press, 2000.
[24] F. de Rosis, C. Castelfranchi, P. Goldie, and V. Carofiglio,
“Cognitive Evaluations and Intuitive Appraisals: Can Emotion
Models Handle Them Both?” Emotion-Oriented Systems: The
Humaine Handbook, P. Petta, C. Pelachaud, and R. Cowie eds.,
pp. 459-481, Springer-Verlag, 2011.
[25] L.D. Riek, P.C. Paul, and P. Robinson, “When My Robot Smiles at
Me: Enabling Human-Robot Rapport via Real-Time Head Gesture
Mimicry,” J. Multimodal User Interfaces, vol. 3, pp. 99-108, 2010.
doi: 10.1007/s12193-009-0028-2.
[26] I. Sneddon, P. Goldie, and P. Petta, “Ethics in Emotion-Oriented
Systems: The Challenges for an Ethics Committee,” Emotion-
Oriented Systems: The Humaine Handbook, P. Petta, C. Pelachaud,
and R. Cowie eds., pp. 756-767, Springer-Verlag, 2011.
[27] S. Döring, P. Goldie, and S. McGuinness, “Principalism: A Method
for the Ethics of Emotion-Oriented Machines,” Emotion-Oriented
Systems: The Humaine Handbook, P. Petta, C. Pelachaud, and
R. Cowie eds., pp. 713-724, Springer-Verlag, 2011.
[28] S. Slater, R. Moreton, and K. Buckley, “Emotional Agents as
Software Interfaces,” Proc. Workshop Emotion in HCI—Designing for
People, pp. 38-43, 2010.
[29] http://www.guardian.co.uk/technology/2010/sep/19/3d-
games-xbox-playstation, 2012.
[30] http://en.wikipedia.org/wiki/Milo_and_Kate, 2012.
[31] P. Sundstrom, A. Stahl, and K. Hook, “In Situ Informants
Exploring an Emotional Mobile Messaging System in Their
Everyday Practice,” Int’l J. Human Computer Studies, vol. 65,
pp. 388-403, 2007.
[32] K. Höök, “Affective Loop Experiences: Designing for Interactional
Embodiment,” Philosphical Trans. Royal Soc. B, vol. 364, pp. 3585-
3595, 2009.
[33] R.B. Knapp, J. Jaimovich, and N. Coghlan, “Measurement of
Motion and Emotion during Musical Performance,” Proc. Third
IEEE Int’l Conf. Affective Computing and Intelligent Interaction,
pp. 1735-739, 2009.
[34] M.D. van der Zwaag and J.H.D.M. Westerink, “Deploying Music
Characteristics for an Affective Music Player,” Proc. Third IEEE
Int’l Conf. Affective Computing and Intelligent Interaction, vol. 1,
pp. 459-465, 2009.
[35] http://www.cantoche.com/en~AvatarGallery.html, 2012.
[36] S. Kopp and I. Wachsmuth, “Synthesizing Multimodal Utterances
for Conversational Agents,” J. Computer Animation of Virtual
Worlds, vol. 15, pp. 39-52, 2004.
[37] G. Papagiannakis, S. Schertenleib, B. O’Kennedy, M. Arevalo-
Poizat, N. Magnenat-Thalmann, A. Stoddart, and D. Thalmann,
“Mixing Virtual and Real Scenes in the Site of Ancient Pompeii,”
J. Computer Animation of Virtual Worlds, vol. 16, pp. 11-24, 2005.
[38] D. Heylen, A. Nijholt, and R. op den Akker, “Affect in Tutoring
Dialogues,” Applied Artificial Intelligence, vol. 19, pp. 287-311, 2005.
[39] Augustine,, City of God, H. Bettenson, trans. Penguin Classics,
1984.
[40] Descartes: Philosophical Writings. A selection translated and edited
by E. Anscombe and P.T. Geach, with introduction by A.K.
Nelson, 1954.
[41] David Hume: A Treatise of Human Nature: Volume 1, D.F. Norton
and F.J. Norton, eds. Oxford Univ. Press, 2007.
[42] H. Damasio, T. Grabowski, R. Frank, A. Galaburda, and A.R.
Damasio, “The Return of Phineas Gage: Clues about the Brain for
the Skull of a Famous Patient,” Science, vol. 264, pp. 1102-1105,
1994.
[43] V. Dulewicz and M. Higgs, “Emotional Intelligence—A Review
and evaluation Study,” J. Managerial Psychology, vol. 15, pp. 341-
372, 2000.
[44] A.M. Isen, “Positive Affect,” Handbook of Cognition and Emotion,
T. Dalgleish and M. Power, eds., pp. 521-539, 1999.
[45] J. Decety and W.J. Ickes, The Social Neuroscience of Empathy. MIT
Press, 2009.
VOL. 3, NO. 4, OCTOBER-DECEMBER 2012
[46] L. Devillers, L. Vidrascu, and L. Lamel, “Challenges in Real-Life
Emotion Annotation and Machine Learning Based Detection,”
J. Neural Networks, vol. 18, pp. 407-422, 2005.
[47] J. Cassell, “Modelling Rapport in Embodied Conversational
Agents,” Proc. Interspeech 2008, pp. 18-19, 2008.
[48] K. Murata, “Laughter for Defusing Tension: Examples from
Business Meetings in Japanese and in English,” New Frontiers in
Artificial Intelligence, pp. 294-305, Springer, 2009.
[49] L. Devillers, I. Vasilescu, and L. Lamel, “Emotion Detection in
Task-Oriented Dialog Corpus,” Proc. IEEE Int’l Conf. Multimedia
and Expo, vol. III, pp. 549-552, 2003.
[50] C. Nass, L. Takayama, and S. Brave, “Socializing Consistency:
From Technical Homogeneity to Human Epitome,” Advances in
Management Information Systems, vol. 6, pp. 373-391, 2006.
[51] O. Turk and M. Schroeder, “Evaluation of Expressive Speech
Synthesis with Voice Conversion and Copy Resynthesis Techni-
ques,” IEEE Trans. Audio, Speech, and Language Processing, vol. 18,
pp. 965-973, 2010.
[52] Keating and Mirus, deaf phones, 2003.
[53] R. Cowie and M. Schroeder, “Piecing Together the Emotion
Jigsaw,” Machine Learning for Multimodal Interaction, S. Bengio and
H. Bourlard, eds., pp. 305-317, Springer Verlag, 2005.
[54] M. Bulut, S.S. Narayanan, and A.K. Syrdal, “Expressive Speech
Synthesis Using a Concatenative Synthesizer,” Proc. Int’l Conf.
Spoken Language Processing, pp. 1265-1268, 2002.
[55] L.L. Carstensen and J.A. Mikels, “At the Intersection of Emotion
and Cognition: Aging and the Positivity Effect,” Current Directions
in Psychological Science, vol. 14, pp. 117-121, 2005.
[56] A.C.B. Medeiros, N. Crilly, and P.J. Clarkson, “Affective Response
to ICT Products in Old Age,” Proc. 2008 Int’l Workshop Emotion in
Human Computer Interaction—Designing for People, pp. 32-37, 2010.
[57] A.R. Korukonda, “Differences That Do Matter: A Dialectic
Analysis of Individual Characteristics and Personality Dimensions
Contributing to Computer Anxiety,” Computers in Human Behavior,
vol. 23, pp. 1921-1942, 2007.
[58] G. Conti-Ramsden, K. Durkin, and A.J. Walker, “Computer
Anxiety: A Comparison of Adolescents with and without a
History of Specific Language Impairment,” Computers and Educa-
tion, vol. 54, pp. 136-145, 2010.
[59] S. McGilloway, R. Cowie, and E. Douglas-Cowie, “Prosodic Signs
of Emotion in Speech: Preliminary Results from a New Technique
for Automatic Statistical Analysis,” Proc. 13th Int’l Conf. Phonetic
Sciences, vol. 1, pp. 250-253, 1995.
[60] A. Trevino, T. Quatieri, and N. Malyska, “Phonologically-Based
Biomarkers for Major Depressive Disorder,” EURASIP J. Advances
in Signal Processing, to appear.
[61] J.F. Cohn, “Advances in Behavioral Science Using Automated
Facial Image Analysis and Synthesis,” IEEE Signal Processing
Magazine, vol. 128, pp. 128-133, 2010.
[62] A.L. Webb, P.N. Carding, I.J. Deary, K. MacKenzie, N. Steen, and
J.A. Wilson, “The Reliability of Three Perceptual Evaluation Scales
for Dysphonia,” European Archives of Oto-Rhino-Laryngology,
vol. 261, pp. 429-434, 2004.
[63] http://www.affectiva.com/, 2012.
[64] P. Lucey, J.F. Cohn, S. Lucey, S. Sridharan, and K. Prkachin,
“Automatically Detecting Pain Using Facial Actions,” Proc. Third
IEEE Int’l Conf. Affective Computing and Intelligent Interaction, vol. 1,
pp. 12-18, 2009.
[65] A.B. Ashraf, S. Lucey, J.F. Cohn, T. Chen, K.M. Prkachin, and P.
Solomon, “The Painful Face: Pain Expression Recognition Using
Active Appearance Models,” Image Visual Computing, vol. 27,
no. 12, pp. 1788-1796, 2009.
[66] S.-H. Kang and J. Gratch, “Virtual Humans Elicit Socially Anxious
Interactants’ Verbal Self-Disclosure,” Computer Animation of
Virtual Worlds, 2010.
[67] M. Mori, “The Uncanny Valley,” Energy, vol. 7, pp. 33-35, 1970.
[68] D. Hanson, A. Olney et al., “Upending the Uncanny Valley,” Proc.
20th Nat’l Conf. Artificial Intelligence, pp. 1728-1729, 2005.
[69] W. Gaver, “Designing for Emotion (among Other Things),”
Philosophical Trans. Royal Soc. B, vol. 364, pp. 3597-3604, 2009.
[70] I. Asimov, The Naked Sun. Collins, 1993.
[71] L. Hall, M. Vala, M. Hall, M. Webster, S. Woods, A. Gordon, R.
Aylett, “FearNot’s Appearance: Reflecting Children’s Expecta-
tions and Perspectives,” Proc. Int’l Conf. Intelligent Virtual Agents,
pp. 407-419, 2006.
COWIE: THE GOOD OUR FIELD CAN HOPE TO DO, THE HARM IT SHOULD AVOID 423

[72] J. Weizenbaum, “ELIZA—A Computer Program for the Study of Roddy Cowie graduated in philosophy and
Natural Language Communication between Man and Machine,” psychology, and his PhD was on machine vision.
Comm. ACM, vol. 9, no. 1, pp. 36-45, 1966. He is a professor of psychology at Queen’s,
[73] J. Weizenbaum, Computer Power and Human Reason: From Judgment Belfast. His research has used computational
to Calculation. Freeman & Co., 1976. methods to study a range of complex perceptual
[74] C. Breazel, “Role of Expressive Behaviour for Robots That Learn phenomena—perceiving pictures, the experi-
from People,” Philosohhical Trans. Royal Soc. B, vol. 364, pp. 3527- ence of deafness, what speech conveys about
3538, 2009. the speaker, and, in a series of EC projects, the
[75] A. Hiolle, L. Canamero, and A.J. Blanchard, “Learning to Interact perception of emotion, where he has developed
with the Caretaker: A Developmental Approach,” Affective methods of measuring perceived emotion and
Computing and Intelligent Interaction, pp. 422-433, Springer-Verlag, inducing emotionally coloured interactions. Key outputs include special
2007. editions of Speech Communication (2003) and Neural Networks (2005),
[76] P. Goldie, S. Döring, and R. Cowie, “The Ethical Distinctiveness of and the HUMAINE Handbook on Emotion-Oriented Systems (2011). He
Emotion-Oriented Technology: Four Long-Term Issues,” Emotion- is a member of the IEEE.
Oriented Systems: The Humaine Handbook, P. Petta, C. Pelachaud,
and R. Cowie, eds., pp. 725-733, Springer-Verlag, 2011.
[77] Hadfield and Marks, “This Is Your Captain Dozing,” New
. For more information on this or any other computing topic,
Scientist, vol. 1682, p. 267, 2000.
please visit our Digital Library at www.computer.org/publications/dlib.
[78] http://www.telegraph.co.uk/finance/newsbysector/mediatech-
nologyandtelecoms/digital-media/6539402/Google-to-fund-
ASAs-regulation-of-web-advertising.html, 2012.
[79] H. Bauman and S. Doring, “Emotion-Oriented Systems and the
Autonomy of Persons,” Emotion-Oriented Systems: The Humaine
Handbook, P. Petta, C. Pelachaud, and R. Cowie, eds., pp. 735-752,
Springer-Verlag, 2011.
[80] D. Levy, Love and Sex with Robots: The Evolution of Human-Robot
Relationships. Harper Collins, 2007.
[81] http://www.huffingtonpost.com/2010/01/10/roxxxy-sex-robot-
photo-wo_n_417976.html, 2012.
[82] T.L. Beauchamp and J.F. Childress, Principles of Biomedical Ethics,
fifth ed. Oxford Univ. Press, 2001.