IEEE TRANSACTIONS ON AUTOMATION SCIENCE AND ENGINEERING, VOL. 5, NO.
1, JANUARY 2008
Short Papers
A Steward Robot for Human-Friendly Human-Machine
Interaction in a Smart House Environment
Kwang-Hyun Park, Hyong-Euk Lee, Youngmin Kim, and
Z. Zenn Bien
Abstract—The independence of people who need help with daily activi-
ties will become of vital importance to all societies in the future. This paper
addresses the problem of controlling the assistive home environment and
emphasizes human-friendly human-machine interactions in an approach
designed to achieve independence. To provide residents with an accessible,
convenient, and cost-effective environment for independent living, we in-
troduce a new service robot, categorized as a steward robot, as an interme-
diate agent between residents and their complex smart house environment.
The learning capability and emotional interaction of the robot can make it
more human-friendly in various tasks. A learning system enables the robot
to provide customized services by accumulating knowledge of the user’s be-
havioral patterns in daily activities. An emotional interaction system gen-
erates facial expressions to communicate with the user in a human-friendly
manner. We have developed two types of a steward robot: a software type,
which can be used everywhere via personal computing devices such as a
PDA and a cellular phone, and a hardware type, which provides tangible
services with physical interaction via two robotic arms and a mobile base.
Note to Practitioners—The trend of an aging population and of a non-
decreasing proportion of people with disabilities could lead to a shortage
of caregivers, which will become a serious problem in the near future. To
assist the elderly and people with disabilities, a smart house can be a solu-
tion for improving quality-of-life and coping with their demands. However,
the increase in the number and complexity of assistive systems to enhance
available services in a smart house could lead to a decrease in convenience,
accessibility, and cost-effectiveness, because it is very difficult to control
all the devices using a large number of corresponding human-machine in-
terfaces. To resolve this problem, this paper introduces the concept of a
steward robot that helps the user in the smart house environment to op-
erate all the systems easily. The steward robot can reduce the user’s cog-
nitive load during operation because it commands and controls all subsys-
tems in lieu of the user and provides proactive services based on the user’s
behavioral patterns.
Index Terms—Emotional interaction, human-robot interaction, learning
of behavioral patterns, smart house, steward robot.
I. INTRODUCTION
The focus of advanced research studies in designing automation sys-
tems has significantly shifted over the past decade from production and
Manuscript received March 15, 2006; revised December 2, 2006 and May
30, 2007. This paper was recommended for publication by Associate Editor
C. Nugent and Editor D. Meldrum upon evaluation of the reviewers’ comments.
This work was supported in part by the Science Research Center/Engineering
Research Center (SRC/ERC) Program of the Ministry of Science and Tech-
nology/Korea Science and Engineering Foundation (MOST/KOSEF) under
Grant R11-1999-008.
K.-H. Park, H.-E. Lee, and Z. Z. Bien are with the Department of Elec-
trical Engineering and Computer Science, Korea Advanced Institute of Sci-
ence and Technology, Daejeon 305-701, Korea (e-mail: akaii@ee.kaist.ac.kr;
helee@ctrsys.kaist.ac.kr; zbien@ee.kaist.ac.kr).
Y. Kim is with the Department of Mechanical Engineering, Korea Ad-
vanced Institute of Science and Technology, Daejeon 305-701, Korea (e-mail:
kimym@robot.kaist.ac.kr).
Color versions of one or more of the figures in this paper are available online
at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/TASE.2007.911674
1545-5955/$25.00 © 2007 IEEE
Authorized licensed use limited to: K.S. Institute of Technology. Downloaded on January 30, 2010 at 23:56 from IEEE Xplore. Restrictions apply.
21
manufacturing to the service sectors [3]. Introducing various home au-
tomation systems, most simple and repetitive tasks are executed by
computers or microprocessors. We change the channel or volume of
a TV set with a simple click of a remote controller, whereas early TV
was manually operated. These kinds of changes have been realized in
almost all electronic devices and home appliances. We can control our
home’s temperature remotely by telephone and use several automated
kitchen devices such as a coffee maker and a toaster. As modern tech-
nology comes into our living environment more closely, our desire to
lead a more convenient and safe life is increasing. In some cases, we
may wish to control home appliances in a simple and natural way, or
just by thought.
The initial design of home automation has been recently expanded
to a “smart house” concept, which incorporates advanced sensors and
various automation facilities to make the existing houses more comfort-
able, convenient, and safe [21]. One of the very important user groups
of smart houses in the future is believed to be the elderly and/or people
with disabilities, because the smart house has the potential to provide
them with independence in mobility, manipulation, and human-ma-
chine communication, as well as improved health and living conditions
[2]. It is instructive to note that the world is now paying close attention
to the trend of the aging population and the nondecreasing proportion
of people with disabilities, in projections of the future [10], [14]. There-
fore, a shortage of caregivers for nursing the elderly is expected to be a
serious problem in the near future. As a care-giving aid, a smart house
can be an alternative to assist the elderly and people with disabilities in
improving their quality-of-life and carrying out certain daily activities
for them because of their limitations in doing the work on their own.
The effect of a smart house on its residents greatly depends on the
list of automatic systems and the functionality of the devices that build
the home environment. However, the increase in the number and com-
plexity of subsystems, designed to enhance available services in a smart
house, could lead to a decrease in convenience, accessibility, and cost-
effectiveness because of the difficulty of controlling all the devices/sys-
tems where they use a large number of corresponding human-machine
interfaces. The user must learn the complicated procedures and com-
mands for all the machines, which requires a cognitive load during op-
eration. For example, for a going-out task in our smart house (Intelli-
gent Sweet Home) [4], [19], [20], the user initiates the task by using a
voice command and/or a hand gesture. An intelligent bed assists and
supports the user to change position in bed. A robotic hoist moves to
the bed and lifts the user. An intelligent wheelchair also moves to dock
with the robotic hoist, and then the robotic hoist lowers the user onto
the wheelchair. Finally, the wheelchair navigates to go out of the house.
In this task, the user must command each step and remember the whole
sequence of commands.
As an alternative, we propose the concept of a “steward robot,” which
can help the resident in the smart house environment to operate all the
subsystems easily. A “steward” means someone employed in a large
household or estate to manage domestic concerns such as the supervi-
sion of servants, collection of rents, and keeping of accounts [1]. We
apply the same concept to a robotic agent in the smart house to per-
form a set of specific tasks for the resident, i.e., the steward robot can
be an intermediate agent between the complex home environment and
the end-user to enhance usability and convenience [11].
In this paper, we introduce two types of steward robot. A software-
type robot is represented by a virtual 3-D avatar and can be accessed
22 IEEE TRANSACTIONS ON AUTOMATION SCIENCE AND ENGINEERING, VOL. 5, NO. 1, JANUARY 2008
everywhere using personal computing devices such as a PDA and a
cellular phone when a wired/wireless communication network is avail-
able. A hardware-type robot has been developed to provide active ser-
vices such as delivering a meal or bringing an object with physical in-
teraction by using two robotic arms and a mobile platform. Both types
of robot have an intelligent processing module with learning and emo-
tional interaction capability. Learning capability enables the robot to
provide customized and proactive services depending on the user’s
preferences and living behavioral patterns. The learning system col-
lects environmental sensory information and the command history of
the user to the target subsystems. Then, it reveals the empirical pat-
terns from an incrementally drawn set of behaviors and appropriately
controls the target devices based on the obtained knowledge [11]. In
addition, if the user assigns a task name for a complicated sequence of
commands or behaviors, the steward robot can be commanded later by
the given name of a task such as “going-out.” For more human-friendly
interaction, the robots generate and express their own emotional state
to the user.
This paper is organized as follows. In Section II, the design phi-
losophy is briefly described. The structure and functions of the soft-
ware-type robot are explained in Section III, and the realization of a
hardware-type robot is presented in Section IV. Conclusions follow in
Section V.
II. DESIGN PHILOSOPHY
From the functional point-of-view, a smart house can be considered
as a large-scale system that includes various assistive subsystems car-
rying out their functions to provide the resident with comfortable living
conditions, independence, and happiness [2]. Therefore, the way the
resident interacts with the home-installed systems is a significant as-
pect in the design of a smart house, especially for the elderly and people
with disabilities.
The steward robot has been designed based on Leifer’s design laws
for service robots [13] and on the idea that the technology and solutions
for the service robot should be as human-friendly as possible. That is,
the robots should possess a high level of intelligence in their control,
actions, and interaction with the users, thus offering users a high level of
comfort and functionality. When the user commands a predefined task
in the smart house, the steward robot generates a sequence of subtasks,
distributes subtasks to subsystems, and supervises the actions of each
subsystem to synchronize subtasks. This reduces the user’s cognitive
load and becomes more human-friendly.
It is usually difficult to model and handle the interaction between
human beings and machines because of the variability of the user’s
status and the uncertainty of the environment [3]. Therefore, the infor-
mation acquired in interaction, which includes the user’s intention and
physical, emotional, and mental states, should be incorporated into the
system to make it more human-friendly. The steward robot should be
able to provide personalized services autonomously, in advance of the
user’s command thanks to its learning capability, and reflect the user’s
degree of satisfaction with its knowledge database of the user’s living
behavioral patterns. It also has to be a more human-friendly interactive
agent with its own emotional expression.
III. THE SOFTWARE-TYPE STEWARD ROBOT
A. Structure
Fig. 1 shows a block diagram of the software-type robot and the In-
telligent Sweet Home. All home-installed systems are linked via a net-
work for common control and information exchange. The user can di-
Authorized licensed use limited to: K.S. Institute of Technology. Downloaded on January 30, 2010 at 23:56 from IEEE Xplore. Restrictions apply.
Fig. 1. Block diagram of the software-type steward robot and the intelligent
sweet home.
rectly control subsystems in the smart house using various human-ma-
chine interfaces; furthermore, the user can command the steward robot
to control subsystems indirectly by voice. The steward robot collects
the user’s commands to the subsystems and the user’s status informa-
tion from health monitoring and environmental sensing devices, which
are used in the learning module to recognize living behavioral patterns
and provide proactive services. Facial expression of a 3-D avatar is gen-
erated in the emotional interaction module according to the service of
the robot and in the context of the interaction.
B. Learning Module
During the past decade, a number of application platforms of smart
houses have been developed to deal with the learning problem of living
behavioral patterns. The adaptive house developed at the University
of Colorado is based on an optimal heating/lighting control scheme
to minimize energy and discomfort cost using environmental sensory
information and an artificial neural network (ANN) [15]. iDorm at the
University of Essex was intended to control various home appliances
automatically, based on the resident’s lifestyle using a fuzzy rule-based
learning technique [6], [7].
In this paper, we focus on the understandable representation of
knowledge of behavioral patterns, to combine the obtained knowledge
with the linguistic expression of the user’s preference. In this sense, the
fuzzy logic-based approach of iDorm is more relevant to our system,
because an ANN expresses the accumulated knowledge by numerical
weights between neurons, whereas fuzzy logic generates a knowledge
base on the way humans think with fuzzy terms and association rules
from uncertain data. However, the learning system in iDorm requires
a predefined fuzzy partition of input space, which can lead to lack
of autonomy. Moreover, the improperly divided input space usually
degenerates the performance of the system, and we have no guidelines
on how to divide the input space.
To relieve this limitation and construct efficient fuzzy partitions, we
propose an algorithm called Iterative Fuzzy Clustering with Super-
vision (IFCS) [12]. Meaningful fuzzy partitions with a corresponding
rule base are obtained gradually through an iterative process of selec-
tive fuzzy clustering with supervisory guidance based on cluster-pure-
ness and class-separability. To describe the IFCS algorithm in more
detail, we first define cluster-pureness and class-separability.
IEEE TRANSACTIONS ON AUTOMATION SCIENCE AND ENGINEERING, VOL. 5, NO. 1, JANUARY 2008 23

TABLE I
CLASSIFICATION RESULTS FOR LAMP CONTROL

Let xjk be the k th data with a class label j; 1  j  K , and nj be

the number of data with the class label j . The cluster-pureness index
Pi of the cluster i is defined by

n
maxj2f1;2;...;Kg ijk f (ijk 0 )
Pi = k=1
K n
ijk f (ijk 0 )
j =1 k=1

where ijk is the membership value of xjk for the cluster i and f (1) is
a function satisfying
1; z  0
f (z ) = 0; z < 0 : Fig. 2. Generation of eight emotion words based on the OCC model.

Let mi (j ) and 6i (j ) be the mean vector and the covariance matrix,

respectively, of data satisfying ijk  , k = 1; 2; . . . ; nj . Then, the
class-separability Si of the cluster i is defined by

Si = max
a6=b
Sepi (a; b)Ni (a)Ni (b)
where
n
Ni (j ) = f f (ijk 0 ) 0 2
k=1 Fig. 3. Balanced sentiment relations in Heider’s balance theory.

Sepi (a; b) = 1
1 + e04:4( (a;b)00:5)
  01
i (a; b) = 1 (mi (a) 0 mi (b))T 6i (a) + 6i (b)
8 2
2 (mi (a) 0 mi (b))
6 (a)+6 (b) 2
+ 21 ln 2
: tree-structured model forms branches according to three main dimen-
j6i (a)j j6i (b)j
The learning algorithm is as follows.
1) We first perform initial clustering using the fuzzy c-means (FCM)
algorithm with the cluster validity vXB proposed by Xie and Beni
[23], and find the first local minimum to obtain the number of
clusters c with high reliability and high calculation speed.
2) We calculate the cluster-pureness Pi for each cluster i, and find
i3 , which corresponds to the smallest value of Pi and indicates
the most mixed cluster containing data with different class labels.
3) For cluster i3 , we calculate the class-separability Si .
4) If Si is larger than a threshold value Smin , we perform a reclus-
tering process using the FCM with the cluster validity vXB for the
data in the cluster i3 .
5) Using the newly obtained clusters, we generate a fuzzy rule base
and get the recognition rate as a performance measure. If the
recognition rate is not increased by a given value P , the proce-
dure is terminated. Otherwise, the procedure is repeated from 2).
To show the effectiveness of the proposed algorithm, we compared
the recognition rates with conventional methods such as a fuzzy
c-means algorithm and Wang and Mendel’s method [22] applying the
algorithms to the on/off control patterns of a lamp in a house and using
seven features including internal/external temperature, illumination,
and time. Table I shows that the proposed IFCS algorithm obtains a
comparatively high success rate with a small number of rules.
C. Emotional Interaction Module
The emotional interaction module of the steward robot is based on
the OCC emotion model [18], which handles interaction-related emo-
tion words assigned from observation of human interaction. The upper
part of Fig. 2 describes the structure of the OCC emotion model. The
sions: consequences of events, actions of the robot, and aspects of the
user. Using this model, Ortony addressed 22 emotion words related to
real interaction [16], [18] and we selected four pairs of two opposite
emotion words: like/dislike, pride/shame, hope/fear, and joy/distress,
considering the services and interactions between the steward robot and
the user in the smart house.
Each emotion word is assigned to the event in executing service
tasks, as shown in Table II. When the robot is called by the user or
recognizes other individuals, it generates a “like” or “dislike” emo-
tion according to the sentiment relation between the user and the robot.
When the user commands or the learning module initiates a service
task, the robot expresses “pride” or “shame” based on the self-evalua-
tion of praiseworthiness for the execution of the service. If the service
task is completed successfully, the robot expresses “hope”; otherwise, it
expresses “fear.” The robot also generates “joy” or “distress” according
to the user’s response or feedback. These emotion words are expressed
as facial expressions of a 3-D avatar, as shown in Fig. 1.
To describe the sentiment relation in human-robot interaction, we
adopt Heider’s balance theory [5], [8] which discusses the relation
among individuals based on sentiment. The sentiment relations are bal-
anced between two people when they like or dislike each other, and if
one has different sentiment, the relations become imbalanced. In case
of multiple individuals, the sentiment relations are balanced if the al-
gebraic multiplication of signs in the relations has a positive value, as
shown in Fig. 3. Based on the predefined interpersonal relation of the

Authorized licensed use limited to: K.S. Institute of Technology. Downloaded on January 30, 2010 at 23:56 from IEEE Xplore. Restrictions apply.
24 IEEE TRANSACTIONS ON AUTOMATION SCIENCE AND ENGINEERING, VOL. 5, NO. 1, JANUARY 2008
TABLE II
EMOTION WORDS OF THE STEWARD ROBOT
Fig. 4. Appearance design of hardware-type steward robot.
Fig. 5. Structure of the hardware-type steward robot.
user, the steward robot chooses an emotional word, “like” or “dislike,”
to make the balance [9].
IV. THE HARDWARE-TYPE STEWARD ROBOT
A. Design
For a human-friendly appearance, we adopted the Sa-sang consti-
tution theory [17], which is the basis of Korean oriental medicine, as
shown in Fig. 4. In the theory, we categorize humans into four groups
(tae-yang-in, tae-eum-in, so-yang-in, and so-eum-in) according to
appearance. Among these groups, we selected the so-eum-in for the
steward robot’s appearance because of the corresponding personality
of so-eum-in (prudent, obedient, adorable, and friendly). Finally, we
selected orange as its color, because it represents a joyful, lively, and
cheerful feeling.
B. System Specification
Fig. 5 shows the structure of the hardware-type steward robot, which
can provide active services using two robotic arms with grippers and a
mobile base; it can also control home appliances by accessing the home
network server in the smart house. For the external communication with
the home network server, a TCP/IP-based communication protocol has
Authorized licensed use limited to: K.S. Institute of Technology. Downloaded on January 30, 2010 at 23:56 from IEEE Xplore. Restrictions apply.
Fig. 6. Emotional interaction of the hardware-type steward robot.
been defined. We also defined a CAN-based communication protocol
with 8 bytes of data for internal communication between the modules
of the steward robot.
The hardware specification of the robot is as follows [11].
• Two DOFs for head motion.
• 2 2 (6 DOFs robotic arm + 1 DOF gripper).
• Two wheels-driven mobile platform + 1 omni castor.
• Ultrasonic sensor for obstacle detection.
• Cricket board for localization.
• DC-DC converting/recharging system.
• 600 mm 2 600 mm 2 1100 mm.
• Two CPUs + one microprocessor.
• Wireless LAN for external communication.
• CAN for internal communication.
C. Description of Functions
The functions of the software-type robot, such as learning and emo-
tional interaction capabilities, are implemented in the head part of the
hardware-type robot. The learning module performs the management
of tasks. For example, the steward robot controls subsystems in the
smart house to support preparing a meal and to provide a going-out ser-
vice. Based on the obtained fuzzy rule base from the user’s behavioral
patterns, it controls home appliances such as an air conditioner, lamp,
and TV, and also recommends the user’s favorite TV channels. Emo-
tional expressions are displayed using animation of the eyes and prede-
fined motions of the head and arms, while performing service tasks, as
shown in Fig. 6. Localization of the mobile platform is achieved using
three selected beacons from among all beacons attached to the ceiling
of the smart house for global path generation. Ultrasonic sensors, at-
tached on the body of the robot, are also used for online obstacle de-
tection. For power supply, a small DC-DC converter has been installed
in each subsystem, and a recharging system with an equalizer has been
developed for efficient recharging of the battery.
V. CONCLUSION
In this paper, we proposed a concept of a steward robot to appropri-
ately provide the elderly and disabled people with various services in
a smart house. To apply the concept to a robotic system, we studied
the learning of the user’s behavioral patterns and emotional interaction
with the user, which are essential for the steward robot. To achieve the
learning capability, we developed an algorithm called Iterative Fuzzy
Clustering with Supervision (IFCS), and we adopted the OCC model
and Heider’s balance theory for emotional interaction. However, the
investigation of subjective measures, such as the degree of satisfaction
from the end-users, will have to be carried out in further work.
IEEE TRANSACTIONS ON AUTOMATION SCIENCE AND ENGINEERING, VOL. 5, NO. 1, JANUARY 2008
ACKNOWLEDGMENT
The authors would like to acknowledge various forms of help in
developing the steward robot from Prof. B. K. Kim, Prof. J.-H. Oh,
Prof. M.-J. Youn, Prof. B. H. Lee, Prof. K.-W. Chung, and their
students.
REFERENCES
[1] Merrian-Webster online dictionary. [Online]. Available: http://www.
m-w.com
[2] Z. Z. Bien, J.-W. Jung, K.-H. Park, and F. Wong, “Human-friendly
man-machine interaction in smart home,” Int. J. Human-Friendly Wel-
fare Robotic Syst., vol. 6, no. 3, pp. 31–38, 2005.
[3] Z. Z. Bien, K.-H. Park, J.-W. Jung, and J.-H. Do, “Intention reading
is essential in human-friendly interfaces for the elderly and the hand-
icapped,” IEEE Trans. Ind. Electron., vol. 52, no. 6, pp. 1500–1505,
Dec. 2005.
[4] Z. Z. Bien, K.-H. Park, D.-J. Kim, and J.-W. Jung, “Welfare-oriented
service robotic systems: Intelligent sweet home & KARES II,” Adv. Re-
habilitation Robotics: Human-Friendly Technol. Movement Assistance
Restoration For People With Disabilities, , ser. Lecture Notes Control
Inf. Sci., vol. 306, pp. 57–94, Sep. 2004.
[5] D. Cartwright and F. Harrary, “Structural balance: A generalization of
Heider’s theory,” Psychol. Rev., vol. 63, pp. 277–293, 1956.
[6] H. Duman, V. Callaghan, H. Hagras, M. Colley, G. Clarke, Z. Bien, H.
Lee, and Y. Kim, “A multi-agent pervasive computing architecture for
geographically dispersed care environments,” in Proc. 10th Int. Fuzzy
Syst. Assoc. World Congr. (IFSA 2003), Istanbul, Turkey, 2003.
[7] H. Hagras, V. Callaghan, G. Clarke, M. Colley, A. Pounds-Cornish, A.
Holmes, and H. Duman, “Incremental synchronous learning for em-
bedded agents operating in ubiquitous computing environments,” in
Soft Comput. Agents, 2002, pp. 25–55.
[8] D. Khanafiah and H. Situngkir, “Social balance theory,” Dept. Comput.
Sociology, Bandung Fe Institute, Bandung, Indonesia, Tech. Rep.
WPN2004BFI, 2004.
[9] Y. Kim, H.-E. Lee, and Z. Bien, “A study on a new emotion factor, af-
fective relation, for emotional intelligence of a sociable robot in human-
friendly interaction environment,” in Proc. 2005 Int. Conf. Human-
Computer Interaction (HCII 2005), 2005.
[10] “Statistics of population,” Korea National Statistical Office, Korea.
[11] H.-E. Lee, Y.-M. Kim, K.-H. Park, and Z. Z. Bien, “Development of
a steward robot for intelligent sweet home,” Int. J. Human-Friendly
Welfare Robotic Syst., vol. 6, no. 4, pp. 57–64, 2005.
[12] H.-E. Lee, K.-H. Park, and Z. Bien, “Iterative fuzzy clustering algo-
rithm with supervision to construct probabilistic fuzzy rule base from
numerical data,” IEEE Trans. Fuzzy Syst., 2007.
[13] L. Leifer, G. Toye, and M. Van der Loos, “Tele-service-robot: Inte-
gratingthe socio-technical framework of human service through the
internet-world-wide-web,” in Proc. Int. Workshopon Biorobotics:
Human-Robot Symbiosis, Tsukuba, Japan, 1995.
[14] “Report of statistical bureau,” The Management and Coordination
Agency, Japan.
[15] M. C. Mozer, “The neural network house: An environment that adapts
to its inhabitants,” in Proc. Amer. Assoc. Artif. Intell. Spring Symp.
Intell. Environ., Menlo Park, CA, 1998, pp. 110–114.
[16] D. A. Norman, A. Ortony, and D. M. Russell, “Affect and machine
design: Lessons for the development of autonomous machines,” IBM
Syst. J., vol. 42, no. 1, pp. 38–44, 2003.
[17] The official site of the Association of Korean Oriental Medicine. [On-
line]. Available: http://www.akom.org/eng/theory/t1_2.htm
[18] A. Ortony, G. L. Clore, and A. Collins, The Cognitive Structure of Emo-
tions. Cambridge, U.K.: Cambridge Univ. Press, 1988.
[19] K.-H. Park, Z. Bien, J.-J. Lee, B. K. Kim, J.-T. Lim, J.-O. Kim, H.
Lee, D. H.Stefanov, D.-J. Kim, J.-W. Jung, J.-H. Do, K.-H. Seo, C.
H. Kim, W.-G. Song, and W.-J. Lee, “Robotic smart house to assist
people with movement disabilities,” Autonomous Robots, vol. 22, no.
2, pp. 183–198, 2007.
[20] K.-H. Park and Z. Z. Bien, M. Mokhtari, Ed., “Intelligent sweet home
for assisting the elderly and the handicapped,” in Independent Living
For Persons Disabilities Elderly People, Amsterdam, The Netherlands,
2003, vol. 12, pp. 151–158.
1545-5955/$25.00 © 2007 IEEE
Authorized licensed use limited to: K.S. Institute of Technology. Downloaded on January 30, 2010 at 23:56 from IEEE Xplore. Restrictions apply.
25
[21] D. H. Stefanov, Z. Bien, and W.-C. Bang, “The smart house for older
persons and persons with physical disabilities: Structure, technology
arrangements, and perspectives,” IEEE Trans. Neural Syst. Rehabil.
Eng., vol. 12, pp. 228–250, 2004.
[22] L. X. Wang and J. M. Mendel, “Generating fuzzy rules by learning from
examples,” IEEE Syst., Man, Cybern. C, vol. 22, no. 6, pp. 1414–1427,
Nov. 1992.
[23] X. L. Xie and G. Beni, “A validity measure for fuzzy clustering,” IEEE
Trans. Pattern Anal. Mach. Intell., vol. 13, no. 8, pp. 841–847, Aug.
1991.
Robust Environmental Sound Recognition
for Home Automation
Jia-Ching Wang, Hsiao-Ping Lee, Jhing-Fa Wang, and Cai-Bei Lin
Abstract—This work presents a robust environmental sound recognition
system for home automation. Specific home automation services can be ac-
tivated based on identified sound classes. Additionally, when the sound cat-
egory is human speech, such speech can be recognized for detecting human
intentions as in conventional research on home automation. To attain this
ambitious goal, this study uses two key techniques: signal-to-noise ratio-
aware subspace-based signal enhancement and sound recognition with in-
dependent component analysis mel-frequency cepstral coefficients and a
frame-based multiclass support vector machines, respectively. Simulations
and an experiment in a real-world environment are given to illustrate the
performance of the proposed robust sound recognition system.
Note to Practitioners—The sound recognition system comprises of
signal enhancement module and sound classifier module. To receive better
acoustic signals in a room, a multimicrophone setting is preferred. An
acoustic activity detection algorithm is essential for detecting a salient
sound waveform. The detected salient sound waveform is fed into the sound
recognition system. In the signal enhancement module, an environmental
noise segment is the prior information that is required to calculate the
estimator’s gain. In the sound classifier module, the input sound waveform
is segmented into separate frames to extract the frame-based acoustic
features.
Index Terms—Home automation, independent component analysis
(ICA), mel-frequency cepstral coefficients (MFCCs), signal enhancement,
sound recognition, support vector machines (SVMs), wavelet transform.
I. INTRODUCTION
Rapid developments in electronic circuitry and signal processing
techniques have facilitated intelligent home automation. Home au-
tomation can be found in applications such as security and safety,
heating, environmental, lighting, appliance control, and energy
management [1]–[5]. These services are activated by humans, envi-
ronmental conditions, or both. This study used acoustical information
to activate home automation services. Acoustic information in the
home environment generally includes human speech and a wide
Manuscript received March 21, 2006; revised November 29, 2006 and June
13, 2007. This paper was recommended for publication by Associate Editor
C. Nugent and Editor D. Meldrum upon evaluation of the reviewers’ comments.
The authors are with the Department of Electrical Engineering, National
Cheng Kung University, Tainan City 701, Taiwan, R.O.C. (e-mail:
wjc@icwang.ee.ncku.edu.tw; saralee@hotmail.com; wangjf@csie.ncku.edu.
tw; abfly@pchome.com.tw).
Color versions of one or more of the figures in this paper are available online
at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/TASE.2007.911680