IEEE TRANSACTIONS ON CONSUMER ELECTRONICS, VOL. 64, NO.
2, MAY 2018
SensePods: A ZigBee-Based Tangible
Smart Home Interface
Waqqas M. Khan and Imran A. Zualkernan , Member, IEEE
Abstract—Low-cost sensors and ubiquitous wireless
networking is enabling novel ways in which homeowners
can interact with their smart homes. Many complementary
approaches like using voice commands, direct interaction
by using touch or weight, or by using body gestures are
emerging. This paper shows the design and implementation of
a novel low-power, low-cost, hand-held wireless device called
a SensePod. SensePods can be used by a consumer to interact
with a smart home using simple gestures like rubbing, taping or
rolling the device on any home surface like a dining table. The
device is only 4.5 cm long, forms an ad-hoc wireless network
using the ZigBee protocol, and can be easily interfaced to
existing home management systems using a universal serial
bus port. The gestures in each device can be programmed to
control various objects of a smart home like smart curtains,
for example. Hidden Markov models were used to train the
device to recognize various gestures. The device was tested with
a variety of gestures and has a recognition rate of over 99.7%,
and a response time of less than two milliseconds.
Index Terms—Ad-hoc wireless networks, hidden Markov
models, home automation, smart homes, user interfaces, ZigBee.
I. I NTRODUCTION
VAILABILITY of low-cost sensors and ubiquity of
A wireless networking technologies is changing the land-
scape of how consumers interact with their smart homes. There
are at least three ways in which consumers will interact with
their smart homes in the near future; 1) tangibly interacting
with objects in the home through touch or by sitting on, for
example, 2) by communicating with objects within the home
using voice commands, and 3) by using body gestures.
First, sensors are increasingly being embedded within
the smart home and in objects within a home to enable
tangible and physical interfaces. The use of embedded sen-
sors allows for a tangible interaction where objects in a home
detect a consumer’s intent through movement, touch, force,
weight, etc. For example, “smart furniture” today utilizes
force sensors and accelerometers, wireless mesh networking
(e.g., IEEE 802.14.15), and conductive materials [1] that
Manuscript received January 31, 2018; accepted May 23, 2018. Date
of publication June 7, 2018; date of current version June 28, 2018.
This work was supported by the American University of Sharjah, UAE.
(Corresponding author: Imran A. Zualkernan.)
W. M. Khan is with the Field Support Department, Aker Solutions,
Abu Dhabi 108931, UAE (e-mail: auswkhan@gmail.com).
I. A. Zualkernan is with the Department of Computer Science and
Engineering, American University of Sharjah, Sharjah 26666, UAE (e-mail:
izualkernan@aus.edu).
Color versions of one or more of the figures in this paper are available
online at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/TCE.2018.2844729
1558-4127 c 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
145
enable a consumer to physically and wirelessly interact with
their furniture.
Second, voice recognition technology has come a long way,
and Web-based services augmented with voice-recognition
devices are commercially available today from multiple ven-
dors that typically allow a homeowner to control Wi-Fi-based
control modules connected to various home appliances using
wireless networking. Recently, a voice-based control applica-
tion using a mobile phone as the primary interface to the
smart home has been designed [2]. However, one limitation
of the voice-based approach is that the required quality of
audio input necessitates expensive audio filtering for larger
and complex spaces. In addition, the homeowner needs to be
close enough to a voice-recognition device, to have multiple
voice-recognition devices around the home, or use expen-
sive microphone arrays [3] to get full coverage of the home.
Finally, some voice services today require the input device to
be connected to the Internet.
A third way to interact with a smart home is by using
gestures. Systems using gestures can be divided into two
categories; visual and sensor-based. For example, some com-
mercial gaming consoles provide camera-based hardware that
can track joint positions of a homeowner. Joint positions can
then be used to recognize hand or body gestures that can even-
tually be used to control appliances or actuators in the smart
home. Cheap RGB-D cameras have also been used for detect-
ing and tracking humans in smart homes [4]. Similarly, deep
neural networks can be used to identify hand-gestures from
a video [5]. However, a disadvantage of visual interfaces is
that a person has to be in view of the camera system, and
this is impractical in general because such a visual system
will require a controller and camera in every primary space
of the home. Alternatively, sensor-based approaches to gesture
recognition typically employ low-cost sensors like accelerom-
eters, gyroscopes or magnetometers. These sensors can either
be ambient, based in the environment, or on-body. The on-
body sensors are either wearable or embedded in a hand-held
device and can be used to detect gestures and activities of
a homeowner [6]. One advantage of sensor-based approaches
to interacting with a smart home is that, unlike vision-based
systems, the sensor-based approaches do not rely on lighting
conditions etc., or the availability of cameras in a particular
location of the smart home.
The picture that emerges is that future smart homes will
support a multi-modal interaction involving a host of com-
plementary devices that may include tangible interaction with
the smart home through a physical connection with objects
146 IEEE TRANSACTIONS ON CONSUMER ELECTRONICS, VOL. 64, NO. 2, MAY 2018

in the home like smart furniture or other smart surfaces. II. BACKGROUND
Secondly, the homeowner may interact with their home using A. ZigBee for Smart Homes
voice commands using built-in voice-systems or through their
Using ZigBee-based remote controllers [9], [10] for a smart
smart phones. Thirdly, homeowners may interact with the
home, and ZigBee wireless network for home automation
home via gestures recognized through a variety of sensors like
was proposed many years ago [11]. Smart home energy man-
accelerometers or cameras.
agement systems based on the ZigBee protocol have also
While smart phones and smart network-enabled watches are
been implemented [12]. ZigBee has also been proposed for
good candidates for implementing gesture-based smart home
home-monitoring [13]. ZigBee is pervasive in commercial
controls, they can be augmented with specialized limited-
home automation systems today. Most such systems support
functionality, low-cost, low-power wireless devices that allow
smart hubs that deploy heterogeneous wireless communica-
a home owner to interact with their smart home using simple
tions including ZigBee, Wi-Fi, and ZWave, etc. ZigBee is
gestures. The primary motivation for using such specialized
typically used to communicate between switches or appli-
devices is convenience and low complexity. For example, if
ances, while Wi-Fi is used to connect to the Internet, or with
one wants to turn up water temperature in their bath tub, one
hand-held mobile phones and laptops. Many current home
could fetch their mobile phone or a smart watch (assuming
automation system support ZigBee-enabled movement, vibra-
it is waterproof), open the smart phone app, access the smart
tion, orientation, leak, and temperature sensors. For example,
tub controller, and raise water temperature using a button or
Byun et al. [14] describe a smart LED Lighting system
a slider, or by using a voice command. Alternatively, one could
that communicates over the ZigBee network. In addition,
simply pick up a small wireless device already located near the
ZigBee-controlled electrical outlets are also available. For
bathtub and tap the device against a wall a few times to raise
smart homes, ZigBee has also been integrated with other home
the temperature. The simple functionality of tapping, rubbing
management protocols like DALI [15].
or rolling the device against a surface also has an advantage
Since RSSI from ZigBee networks has been used to estimate
that younger children and older adults can use such devices
a person’s location in the home [16], a smart home manage-
with ease [7].
ment system can also determine the indoor location of the
This paper presents the design and implementation of such
ZigBee device based on signals from ZigBee enabled outlets
a hand-held small wireless device called a SensePod that
and light switches or other ZigBee-based devices. This capa-
allows a consumer to use simple gestures to interact with their
bility can be used by the smart home to also determine the
smart home. Unlike camera-based gesture systems, this device
context of use of a ZigBee device; if the device is being used
is held in one’s hand, and the user can control various appli-
from the bathroom, living room or the kitchen, for example.
ances and actuators in the smart home by using simple gestures
ZigBee has also been used extensively in specialized
like tapping and rolling this device on any surface. The simpli-
smart applications that cater to the elderly. For example,
fied functionality of the device should make it feasible even for
Clarke et al. [17] have implemented a remote monitoring
individuals who are not able to speak, or have visual impair-
system for frail and elderly based on the ZigBee health
ment; they only need to be able to hold, tap and roll this
care profile. ZigBee-based devices using tri-axial accelerom-
device on any surface of the home. The proposed device uses
eters have also been used to detect falls in the elderly [18].
the ZigBee wireless protocol that is often used in smart homes
Similarly, Chen et al. [19] present a reliable ZigBee-based
and is supported in many existing home management systems
monitoring system that not only detects falls, but also reports
and smart home devices and appliances [8].
the position of a person and their EGC signals. Binary sensors
While some hand-held commercial video game-controllers
on ZigBee network have also been used to detect early onset
and 3D television remote controls also support gesture recog-
of dementia by detecting forgetting behaviors [20]. Finally,
nition, these controllers use proprietary protocols, and are
Tung et al. [21] have implemented a ZigBee gateway to
specifically designed to work with either the game con-
remotely monitor patients.
sole or the television. The unique contribution of the design
In summary, ZigBee remains one of the prevalent protocols
of the SensePod is that it uses an open ad-hoc wireless
for smart homes and indoor spaces, and many applications
networking standard (i.e., ZigBee), and is low-cost, power
catering to the needs of homeowners have been implemented
efficient, extendible, and can support mesh-based scenarios
based on this protocol.
involving multiple controllers. The aforementioned gesture-
based commercially available controllers are limited to using
one-to-one connectivity using either Bluetooth or Wi-Fi. In B. Gesture Recognition
addition, unlike proprietary devices, due to open standards- A number of techniques have been used to build ges-
based design, SensePods can easily be integrated with existing ture classifiers based on 3D accelerometer data. For example,
smart home systems. Kela et al. [22] presented a gesture control device that wire-
The rest of the paper is organized as follows. A short sur- lessly interacts with a primary controller using TCP/IP, and
vey of background work is presented next. This is followed found that users primarily relied on simple 2-D gestures that
by an explanation of the basic concept behind the proposed mostly consisted of moving the device up or down in opposite
system, and a description of the hardware and software design directions at various angles. A program using Hidden Markov
of the proposed device. Evaluation of the proposed system is Machines (HMM) successfully classified these 2-D gestures.
presented next, followed by a conclusion. Wang and Li [23] used Dynamic Time Warping (DTW) to
KHAN AND ZUALKERNAN: SensePods: ZIGBEE-BASED TANGIBLE SMART HOME INTERFACE
Fig. 1. SensePods for smart homes forming an ad-hoc ZigBee network.
construct a mobile-phone based gesture-recognition system.
Similarly, Xie and Cao [24] used a pen-based device in con-
junction with a Feed Forward Neural Network (FFNN) and
Similarity Matching (SM) to identify gestures where the
data was transmitted to the computer using a wired USB
interface. Most of the aforementioned techniques for ges-
ture recognition had similar results in terms of accuracy and
precision.
While DTW has been used extensively for gesture recog-
nition, [25] has shown that with increasing size of the
training data DTW tends to get worse while HMM’s
performance becomes better with more data. HMMs have
also been successfully used with very high dimensioned
joint data from camera-based sensors [26]. Accelerometer-
based devices carried in various pockets have successfully
been used to classify activities like resting, walking, bicy-
cling etc. [27]. A similar approach using accelerometer
to detect activities in an assisted living context has been
proposed by [28].
III. P ROPOSED S YSTEM
In order to help the consumers interact with smart homes,
this paper proposes the use of a small wireless, hand-held tan-
gible cylindrical device called a SensePod. As Fig. 1 shows
these low-cost devices can be placed in various living spaces
like the bedroom, the living room and even in the bathroom. In
order to communicate with the smart home, the homeowner
employs gestures like tapping and rolling the SensePod on
any surface. For example, to raise the curtain, the homeowner
can simply tap the curtain twice. Each SensePod can be pro-
grammed to control a specific appliance or object or a group
of objects within a space. For example, the same device can
be used to control multiple curtains in the living room. The
devices form an ad-hoc wireless ZigBee network and commu-
nicate with the smart home controller through a coordinating
device connected to the home controller via a Universal Serial
Bus (USB) interface. The coordinating device forwards an
appropriate code (e.g., “raise curtain 234”) for object/appliance
to the smart home system that takes an appropriate action
either through a wired (e.g., X10) or wireless interfaces (e.g.,
Wi-Fi, ZWave, ZigBee etc.). SensePods can be easily inte-
grated with any smart home system because only a USB
147
interface is required. In addition, SensePods do not require
frequent charging because they are based on the low-power
ZigBee protocol. Finally, SensePods use the 2.5 GHz wireless
frequency, which makes them particularly suited for a home
environment.
IV. D ESIGN AND I MPLEMENTATION
The primary criteria for design of SensePods was simplicity
and an appropriately low physical, processing, and memory
footprint. The design and implementation of hardware and
software is described below.
A. Hardware
The hardware for the SensePod was designed around
three chips available in Quad-flat No Lead (QFN) pack-
aging; a ZigBee Network Controller (ZNC), a three-axis
accelerometer, and a USB bridge. The ZNC incorporates a low
power 2.4 GHz radio frequency transceiver and an 8-bit micro-
controller unit (MCU) into a single LGA package. The 8-bit
onboard MCU contains 60K of flash and 18KB of RAM and
supports the 802.15.4 MAC and ZigBee network stacks. This
MCU allows the communications stack and the application
code to reside in the same memory. The RF transceiver is an
802.15.4 Standard compliant radio, includes a low noise ampli-
fier, and supports full spread-spectrum encoding and decoding.
The nominal power for the RF transceiver is 1mW. The wire-
less transceiver uses a low-cost chip antenna. The transceiver
was tested to have an effective range of 50 meters. The
chip has an operating voltage range of 2V to 3.4V with on-
chip voltage regulators, and can tolerate a −40◦ C to +85◦ C
temperature range. The system also uses a low power, three-
axis analog MEMS accelerometer. This accelerometer has
a maximum resolution of ±6g. Finally, the USB Bridge is
a single-chip USB to UART switch with an integrated USB
transceiver, integrated clock, 1024-Byte EEPROM and an
on-chip voltage regulator.
Each SensePod has two Micro-USB ports that can be uti-
lized for programming and interfacing with additional external
devices or a home management system. An additional, I/O
Port interface is used to provide four surface mount LEDs,
four pad switches, and a reset pad switch for each SensePod.
The LED’s can be optionally programmed to provide addi-
tional information like low battery warnings, etc. Additional
sensors like an ambient light sensor, for example, can also be
interfaced through the I/O Port to sense when the light is off
and send off a beeping LED signal to allow a homeowner to
locate the device. Similarly, temperature and gas sensors can
be added via the I/O interface to detect unusual events like
the heat or air-conditioning breaking down, or to detect and
report a fire.
A two-layer PCB board design resulted in overall dimen-
sions of 45x20 mm for the main SensePod board. Fig. 2 shows
a prototype SensePod being held in a typical hand. As the fig-
ure shows, the prototype is powered with 3 AAA rechargeable
batteries that can be recharged using one of the two onboard
USB ports.
148
Fig. 2. A prototype SensePod showing the main board.
Fig. 3. Overview of the software design for SensePods.
B. Software Design
Software design of the system is based on the approach
shown in Fig. 3. As Fig. 3 shows, the device continuously
collects the 3-axis accelerometer data from the MEMS sensors
and applies filtering to smooth the data. This step is followed
by collapsing the data from three dimensions to a single-
dimensional encoding. The single dimensional encoded data is
then fed into specially trained Hidden Markov Models (HMM)
for classifying the encoded data into one of the known ges-
tures. A recognized gesture is transmitted over the ZigBee
network along with additional information like device identi-
fier etc., and the process continues. The details of each step
are described next.
1) Accelerometer Data Collection and Filtering: The accel-
eration data from the onboard 3-axis accelerometer on
a SensePod is collected and a smoothing filter is applied.
Several different types of filtering techniques have been
used for accelerometer data in the past. For example,
Krassnig et al. [29] used a elliptic IIR low pass for removing
jittering noise, and an elliptic IIR high pass filter for removing
gravity component from accelerometer data.
For simplicity of implementation and efficiency, a moving
average filter was used for each axis as show in (1).
y(n) = (x(n) + x(n − 1) + . . . x(n − M + 1))/M (1)
IEEE TRANSACTIONS ON CONSUMER ELECTRONICS, VOL. 64, NO. 2, MAY 2018
Fig. 4. Selecting the order of moving average filter.
The frequency response a simple moving average filter, where
M represents the size of the moving window, is given by (2)
 
H f = sin(π fM)/Msin(π f ) (2)
Frequency response of M = 5, 10 and 20 are shown in
Fig. 4. The pass band was based on using a −3dB filter band-
width. For the lowest filter order (M = 5), the pass band
(X: 9 Y: 0.708) is shown in red in Fig. 4. Based on accelerome-
ter data sampling from the various gestures at 66.66 Hz, a pass
band of 0-6 Hz was considered to be the most appropriate for
M = 5, and hence the moving window size was chosen to be
5. For higher filter orders (i.e., M = 10 and M = 20), the pass
bands are 0-4 Hz and 0-2 Hz respectively, and will attenuate
the gesture frequencies lying between 0-6 Hz. Fig. 5 shows
a sample of filtered three-axis accelerometer data collected
for one gesture called the rolling gesture. In this gesture, the
user rolls the device along its longer axis on any flat surface.
Fig. 5 shows the filtered acceleration profile for each of the
three accelerometer axes. As Fig. 5 shows, a typical simple
gesture lasts for less than 3 seconds. This raw data is fed into
a feature extraction process as shown next.
2) Feature Extraction: Various feature extraction tech-
niques have been used with accelerometer data. For
example, Orlandi and Bocchi [30] used autoregressive-
coefficients (AR) for feature segmentation to identify the
presence of fatigue using accelerometer data. Similarly,
Rubaiyeat et al. [31] used auto-regressive-coefficients in
conjunction with Linear Discriminant Analysis (LDA) to
reduce the dimensionality of the activity data based on a sin-
gle accelerometer. Other methods such as mean, variance,
correlation, FFT energy and frequency-domain entropy were
used by Sun et al. [32] to identify human activity based on
mobile-phone sensors.
Following a feature extraction approach similar
to Yang [33], K-means algorithm was used in SensePods to
reduce the dimensionality of the data from the 3-axis to
a single-dimensional space. K-means is used to cluster
the three dimensional accelerometer data into k clusters.
The number of clusters chosen (i.e., k) is also called the
KHAN AND ZUALKERNAN: SensePods: ZIGBEE-BASED TANGIBLE SMART HOME INTERFACE
Fig. 5. Filtered accelerometer data for one gesture.
Fig. 6. Code-book encoded data one gesture (k = 8).
size of a codebook and represents an important design
decision. Prior research has used various codebook sizes.
For example, Kela et al. [22] used k = 4 for coding two
dimensional accelerometer data for gesture recognition.
Similarly, Schlömer et al. [34] proposed k = 8 for coding
three-dimensional accelerometer data. Consequently, code-
book sizes of k = 8, 10 and 14 were tested for accelerometer
data retrieved from SensePods, and k with the best results
was chosen. A sample codebook based on a codebook size
of k = 8 is shown in Fig. 6. As Fig. 6 shows, the original
three-dimensional acceleration data shown in Fig. 5 has been
encoded into transitions between eight symbols (i.e., 1 to
8) in time. Instead of three continuous streams of real num-
bers, the streaming data now consists of single-dimensioned
observations where each observations represents a discrete
level between 1 and 8.
3) Gesture Identification Using HMM: A number of differ-
ent methods including neural networks (e.g., [29] and [31]),
149
Dynamic Time Warping (DTW) (e.g., [35] and [36]) and
Hidden Markov Machines (HMM) (e.g., [22], [37], and [38])
have been used for gesture identification from accelerome-
ter data. The SensePod used Hidden Markov Models (HMM)
to construct a decision algorithm for identifying each gesture
based on the coded timed data from the previous step. While
DTW and neural networks are both promising approaches,
HMM was chosen primarily because once the HMM is trained,
the classification is linear and computationally efficient on
a small microcontroller as opposed to, for example, DTW that
requires dynamic programming and has quadratic time and
space complexity.
An HMM (λ) is defined by a number of states N and three
sets of probability distributions.
1. The initiate state distribution π = {πi } is given by (3)
where N is equal to the number states.
πi = P{qi = i}, 1 ≤ i ≤ N (3)
2. State transition probabilities distribution A = {aij } given
by (4) where the state at time t is denoted as qt .
aij = P{qt+1 = j | qt = i}, 1 ≤ i ≤ N (4)
3. Output probability distribution B = {bj (k)} given by (5)
where vk denotes the kth observation symbol in the
alphabet indicating the level in the codebook, and ot
is the current parameter vector.
bj (k) = P{ot = vk | qt = j}, 1 ≤ j ≤ N, 1 ≤ k ≤ M (5)
A single HMM given by λ = (A, B, π ) was trained for each
gesture based on the respective codebook derived from the
3D gesture data. The Baum Welch algorithm [39] was used
for training such that given the observation sequence O =
O1 , O2 , . . . , OT from the one-dimensional codebook, where
each Oi is from 1 to k, the model parameters maximized the
probability P(O|λ).
Since both ergodic and left-to-right HMM models gave sim-
ilar results in the past [34], and the number of states in the
HMM did not seem to have a significant effect on the ges-
ture recognition results in similar applications [40], a five state
ergodic HMMs was used for training a single HMM for each
gesture. Each trained λ∗ = (A, B, π ) was implemented in
embedded C language code and required 0.4 KB of storage
per HMM on the SensePod platform.
Forward algorithm [39] was implemented for each trained
HMM to recognize an observation sequence. Given observa-
tion sequence O = O1 , O2 , . . . , OT from the one-dimensional
codebook, and λ∗ = (A, B, π ), a trained model, the forward
algorithm computes the probability P(O|λ∗ ). While direct
computation has exponential computational complexity, the
forward algorithms solves the problem by using αt (i), the
forward variable, and reduces computational complexity to lin-
ear. The forward algorithm recurs on consists of three steps
given below.
1. Initialization of the forward variable from the trained
model λ∗ and the codebook sequence O given in (6).
α1 (j) = a0 bj (o1 ), 1 ≤ j ≤ N (6)
150
2. Recursion to find the forward variable from the trained
model λ∗ and the codebook sequence O given in (7).

αt (j) = αt−1 (i)aij bj (ot ); 1 ≤ j ≤ N, 1 ≤ t ≤ T (7)
3. Termination to find the probability that the given code-
book observation sequence O is generated from the
trained model λ∗ given in (8).

P{Oλ∗} = αT (qF ) = αT (i)aiF (8)
In summary, data from the on-board accelerometer on
a SensePod is continuously collected at 66 Hz, filtered using
a moving average filter, and collapsed to a single dimensional
codebook. The single dimensional encoded data is then fed
to multiple HMMs trained using the Baum Welch algorithms,
and forward algorithm is used to calculate the probability of
matching each HMM; the HMM with the highest probability
is chosen as the recognized gesture. In the case of a tie, the
consumer is asked to repeat the gesture.
V. E VALUATION
Once the hardware and software were implemented, the
platform was evaluated on an initial set of gestures that would
be easy and convenient for a typical consumer to comprehend
and use. The stress was on the simplicity of the interaction
to ensure that the consumer can convey their message using
minimum movements.
A. Gesture Selection
Gestures that did not require significant cognitive or motor
control were selected for an initial evaluation. In specific,
rolling, rubbing and tapping gestures were chosen. Rolling
involves a user rolling the device on any horizontal surface.
The rubbing gesture consists of the user rubbing the device
against a horizontal surface in one of two directions; along
the device’s orientation and across it. Finally, various types of
tapping was considered; the user could tap the device against
a surface any number of times. For the initial evaluation, one
rolling gesture (roll), two rubbing gestures (roll-horizontal,
roll-vertical), and five tapping (tap-1 through tap-5) gestures
each indicating the number of taps were implemented.
B. Data Collection and HMM Training
For each of the eight gestures selected, thirty representative
samples of the gesture were collected by using a SensePod
and having it store and transmit the training data wirelessly
to a ZigBee coordinator. The data was collected at 66 Hz,
and a typical gesture lasted about 2.5 to 4 seconds resulting
in up to 350 3-d accelerometer data points per gesture. The
collected data was then used train eight HMMs; one HMM
for recognizing each gesture selected earlier (e.g., rolling).
After all the HMMs were trained, Monte-Carlo cross-
validation [41] was used for evaluation. Monte-Carlo valida-
tion splits the data randomly between training and test sets.
This method is more pessimistic, and hence provides a more
realistic estimation of accuracy and performance. Since the
codebook size was one of the design parameter, the eight
IEEE TRANSACTIONS ON CONSUMER ELECTRONICS, VOL. 64, NO. 2, MAY 2018
Fig. 7. Accuracy verses the size of training sub-set for various codebooks.
HMMs were trained for each of the three codebook sizes of
k = 8, 12 and 14 to determine the best codebook size. Each
HMM corresponded to the recognition of a single gesture like
tapping, etc.
C. Results
The implemented gestures were evaluated for the accuracy
with which the various gestures are selected and the memory
and response time requirements.
1) Classification Accuracy: Fig. 7 shows the results based
on randomly selecting 12 training points each from each of the
8 gestures, and testing the trained HMM’s on the remaining
data points. As the Figure shows, best results were obtained
with a codebook size of k = 14 that achieved an overall
accuracy of 99.7%. It is clear from Fig. 7 that increasing the
codebook size and the size of the training set also increased
the performance.
Fig. 8 shows a sample Receiver Operating Curves (ROC)
for the eight gestures. An ROC plots the false positive rate
against the true positive rate. The shapes of the curves clearly
shows that the HMM identified their corresponding gestures in
a robust fashion and the result is close to the ideal performance
curve.
2) Memory and Speed: The forward algorithm used for
classifying a gesture was implemented in the C programming
language and occupied 1 KB of memory after compilation.
The total memory requirements for the software were 18 KB
for the ZigBee networking stack, 1 KB for the forward algo-
rithm, and 1.2 KB for the K-means algorithm, resulting in
a total memory requirement of 20.2 KB. In addition, 0.4 KB
was required for each gesture implemented. Implementing
eight different gestures, therefore, required a total of 3.2KB
+ 20.2KB = 23.4 KB of memory, which was well within
64 KB of Flash memory available on the SensePod hardware.
This also implies that many additional gestures can be added
to the device with ease.
The implemented software on average took 1.65 millisec-
onds to execute the filtering/K-means algorithm, and 0.09 mil-
liseconds to execute the forward algorithm for a window
of 30 successive accelerometer data points. This resulted in
a very acceptable response rate of less than 2 milliseconds for
recognizing a particular gesture.
KHAN AND ZUALKERNAN: SensePods: ZIGBEE-BASED TANGIBLE SMART HOME INTERFACE
Fig. 8. ROC Curves for the best classifier (k = 14).
Based on the high classification accuracy, small memory
footprint and fast speed of execution, the results strongly
suggest that the design is feasible.
VI. C ONCLUSION
This paper has presented the design and implementation of
a unique consumer device called a SensePod that will help
consumers communicate with their smart homes using simple
physical gestures like tapping or rolling. The proposed device
is based on the ZigBee protocol which is one of the indus-
try standards for smart homes. The proposed system can be
seamlessly integrated with any smart home system that sup-
ports a USB interface. Another advantage of the system is
the low-power consumption and low cost; each device can be
manufactured for less than $10 per device which is signifi-
cantly less than the cost of many ZigBee-enabled electrical
outlets available today. This also implies that a homeowner
can easily afford tens of such devices for their home.
Future extensions include adding the use of RSSI to auto-
matically determine the location of each device in the home.
In addition, since ZigBee supports mesh networking, multiple
SensePods can be programmed to enact complex interactions
as well. For example, a large number of smart home commands
can be generated by using two SensePods as drum sticks, and
by issuing a sequence of tapping instructions to form various
drumming patterns.
In summary, SensePods represent a unique and convenient
way for a home-owner to interact with their smart home.
The device does not replace but complements voice or smart
phone-based interfaces to smart homes.
R EFERENCES
[1] A. L. Bleda, F. J. Fernández-Luque, A. Rosa, J. Zapata, and R. Maestre,
“Smart sensory furniture based on WSN for ambient assisted living,”
IEEE Sensors J., vol. 17, no. 17, pp. 5626–5636, Sep. 2017.
[2] H. Basanta, Y.-P. Huang, and T.-T. Lee, “Using voice and gesture to
control living space for the elderly people,” in Proc. Int. Conf. Syst. Sci.
Eng. (ICSSE), 2017, pp. 20–23.
[3] Y. Sasaki, S. Kagami, H. Mizoguchi, and T. Enomoto, “A predefined
command recognition system using a ceiling microphone array in noisy
housing environments,” in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst.,
2008, pp. 2178–2184.
151
[4] J. Han, E. J. Pauwels, P. M. de Zeeuw, and P. H. N. De With, “Employing
a RGB-D sensor for real-time tracking of humans across multiple re-
entries in a smart environment,” IEEE Trans. Consum. Electron., vol. 58,
no. 2, pp. 255–263, May 2012.
[5] P. Bao, A. I. Maqueda, C. R. Del-Blanco, and N. García, “Tiny hand ges-
ture recognition without localization via a deep convolutional network,”
IEEE Trans. Consum. Electron., vol. 63, no. 3, pp. 251–257, Aug. 2017.
[6] O. D. Lara and M. A. Labrador, “A survey on human activity recognition
using wearable sensors,” IEEE Commun. Surveys Tuts., vol. 15, no. 3,
pp. 1192–1209, 3rd Quart., 2013.
[7] L. Liu, E. Stroulia, I. Nikolaidis, A. Miguel-Cruz, and A. R. Rincon,
“Smart homes and home health monitoring technologies for older
adults: A systematic review,” Int. J. Med. Informat., vol. 91, pp. 44–59,
Jul. 2016.
[8] M. Kuzlu, M. Pipattanasomporn, and S. Rahman, “Review of commu-
nication technologies for smart homes/building applications,” in Proc.
IEEE Innov. Smart Grid Technol. Asia (ISGT ASIA), 2015, pp. 1–6.
[9] I.-K. Hwang, D.-S. Lee, and J.-W. Baek, “Home network configur-
ing scheme for all electric appliances using ZigBee-based integrated
remote controller,” IEEE Trans. Consum. Electron., vol. 55, no. 3,
pp. 1300–1307, Aug. 2009.
[10] J. Han, C.-S. Choi, and I. Lee, “More efficient home energy management
system based on ZigBee communication and infrared remote controls,”
IEEE Trans. Consum. Electron., vol. 57, no. 1, pp. 85–89, Feb. 2011.
[11] K. Gill, S.-H. Yang, F. Yao, and X. Lu, “A ZigBee-based home
automation system,” IEEE Trans. Consum. Electron., vol. 55, no. 2,
pp. 422–430, May 2009.
[12] D.-M. Han and J.-H. Lim, “Design and implementation of smart home
energy management systems based on ZigBee,” IEEE Trans. Consum.
Electron., vol. 56, no. 3, pp. 1417–1425, Aug. 2010.
[13] I. A. Zualkernan, A.-R. Al-Ali, M. A. Jabbar, I. Zabalawi, and
A. Wasfy, “InfoPods: ZigBee-based remote information monitoring
devices for smart-homes,” IEEE Trans. Consum. Electron., vol. 55, no. 3,
pp. 1221–1226, Oct. 2009.
[14] J. Byun, I. Hong, B. Lee, and S. Park, “Intelligent household LED light-
ing system considering energy efficiency and user satisfaction,” IEEE
Trans. Consum. Electron., vol. 59, no. 1, pp. 70–76, Feb. 2013.
[15] F. J. Bellido-Outeirino, J. M. Flores-Arias, F. Domingo-Perez,
A. Gil-De-Castro, and A. Moreno-Munoz, “Building lighting automa-
tion through the integration of DALI with wireless sensor networks,”
IEEE Trans. Consum. Electron., vol. 58, no. 1, pp. 47–52, Feb. 2012.
[16] B. Mrazovac, M. Z. Bjelica, D. Kukolj, B. M. Todorovic, and
D. Samardzija, “A human detection method for residential smart
energy systems based on ZigBee RSSI changes,” IEEE Trans. Consum.
Electron., vol. 58, no. 3, pp. 819–824, Aug. 2012.
[17] M. Clarke, J. de Folter, V. Verma, and H. Gokalp, “Interoperable end-to-
end remote patient monitoring platform based on IEEE 11073 PHD and
ZigBee health care profile,” IEEE Trans. Biomed. Eng., vol. 65, no. 5,
pp. 1014–1025, May 2018.
[18] J. Wang, Z. Zhang, B. Li, S. Lee, and R. S. Sherratt, “An enhanced fall
detection system for elderly person monitoring using consumer home
networks,” IEEE Trans. Consum. Electron., vol. 60, no. 1, pp. 23–29,
Feb. 2014.
[19] S. K. Chen et al., “A reliable transmission protocol for ZigBee-based
wireless patient monitoring,” IEEE Trans. Inf. Technol. Biomed., vol. 16,
no. 1, pp. 6–16, Jan. 2012.
[20] T.-H. Tan, M. Gochoo, F.-R. Jean, S.-C. Huang, and S.-Y. Kuo, “Front-
door event classification algorithm for elderly people living alone in
smart house using wireless binary sensors,” IEEE Access, vol. 5,
pp. 10734–10743, 2017.
[21] H. Y. Tung, K. F. Tsang, H. C. Tung, K. T. Chui, and H. R. Chi, “The
design of dual radio ZigBee homecare gateway for remote patient mon-
itoring,” IEEE Trans. Consum. Electron., vol. 59, no. 4, pp. 756–764,
Nov. 2013.
[22] J. Kela et al., “Accelerometer-based gesture control for a design envi-
ronment,” Pers. Ubiquitous Comput., vol. 10, no. 5, pp. 285–299,
Jul. 2006.
[23] H. Wang and Z. Li, “Accelerometer-based gesture recognition using
dynamic time warping and sparse representation,” Multimedia Tools
Appl., vol. 75, no. 14, pp. 8637–8655, Jul. 2016.
[24] R. Xie and J. Cao, “Accelerometer-based hand gesture recognition by
neural network and similarity matching,” IEEE Sensors J., vol. 16,
no. 11, pp. 4537–4545, Jun. 2016.
[25] A. D. Calin, “Gesture recognition on Kinect time series data using
dynamic time warping and hidden Markov models,” in Proc. 18th Int.
Symp. Symbolic Numeric Algorithms Sci. Comput. (SYNASC), 2016,
pp. 264–271.
152
[26] I.-J. Ding and Y.-J. Chang, “HMM with improved feature extraction-
based feature parameters for identity recognition of gesture command
operators by using a sensed Kinect-data stream,” Neurocomputing,
vol. 262, pp. 108–119, Nov. 2017.
[27] A. M. Khan, Y.-K. Lee, S. Lee, and T.-S. Kim, “Accelerometer’s position
independent physical activity recognition system for long-term activity
monitoring in the elderly,” Med. Biol. Eng. Comput., vol. 48, no. 12,
pp. 1271–1279, Dec. 2010.
[28] X. Fan et al., “Activity recognition as a service for smart home: Ambient
assisted living application via sensing home,” in Proc. IEEE Int. Conf.
AI Mobile Services (AIMS), 2017, pp. 54–61.
[29] G. Krassnig, D. Tantinger, C. Hofmann, T. Wittenberg, and M. Struck,
“User-friendly system for recognition of activities with an accelerom-
eter,” in Proc. 4th Int. Conf. Pervasive Comput. Technol. Healthcare
(Pervasive Health), Munich, Germany, 2010, pp. 1–8.
[30] S. Orlandi and L. Bocchi, “Discrimination of fatigue in walk-
ing patterns,” in Proc. IFMBE, vol. 25. Munich, Germany, 2009,
pp. 1275–1278.
[31] H. A. Rubaiyeat, T.-S. Kim, and M. K. Hasan, “Real-time recognition of
daily human activities using a single tri-axial accelerometer,” in Proc. 5th
Int. Conf. Embedded Multimedia Comput. (EMC), 2010, pp. 1–5.
[32] L. Sun, D. Zhang, B. Li, B. Guo, and S. Li, Activity Recognition
on an Accelerometer Embedded Mobile Phone With Varying Positions
and Orientations [LNCS 6406 (Including Subseries Lecture Notes in
Artificial Intelligence and Lecture Notes in Bioinformatics)]. Heidelberg,
Germany: Springer, 2010, pp. 548–562.
[33] J. Yang, “Toward Physical Activity Diary: Motion Recognition Using
Simple Acceleration Features with Mobile Phones,” in Proc. 1st Int.
Workshop Interact. Multimedia Consum. Electron., Beijing, China, 2009,
pp. 1–10.
[34] T. Schlömer, B. Poppinga, N. Henze, and S. Boll, “Gesture recogni-
tion with a Wii controller,” in Proc. 2nd Int. Conf. Tangible Embedded
Interact. (TEI), Bonn, Germany, 2008, pp. 11–14.
[35] J. Liu, L. Zhong, J. Wickramasuriya, and V. Vasudevan, “uWave:
Accelerometer-based personalized gesture recognition and its applica-
tions,” Pervasive Mobile Comput., vol. 5, no. 6, pp. 657–675, 2009.
[36] K. Altun and B. Barshan, Human Activity Recognition Using
Inertial/Magnetic Sensor Units [LNCS 6219 (Including Subseries
Lecture Notes in Artificial Intelligence and Lecture Notes in
Bioinformatics)], Heidelberg, Germany: Springer, 2010,pp. 38–51.
[37] X. Zhang et al., “Hand gesture recognition and virtual game control
based on 3D accelerometer and EMG sensors,” in Proc. Int. Conf. Intell.
User Interfaces (IUI), 2009, pp. 401–405.
[38] J. Mäntyjärvi, J. Kela, P. Korpipää, and S. Kallio, “Enabling fast and
effortless customisation in accelerometer based gesture interaction,” in
Proc. 3rd Int. Conf. Mobile Ubiquitous Multimedia, College Park, MD,
USA, 2004, pp. 25–31.
IEEE TRANSACTIONS ON CONSUMER ELECTRONICS, VOL. 64, NO. 2, MAY 2018
[39] L. R. Rabiner, “A tutorial on hidden Markov models and selected appli-
cations in speech recognition,” Proc. IEEE, vol. 77, no. 2, pp. 257–286,
Feb. 1989.
[40] V. Mantyla, Discrete Hidden Markov Models With Application to
Isolated User-Dependent Hand Gesture Recognition, VTT Publications
449, Tech. Res. Center Finland, Espoo, Finland, 2001.
[41] Q.-S. Xu and Y.-Z. Liang, “Monte–Carlo cross validation,”
Chemometrics Intell. Lab. Syst., vol. 56, no. 1, pp. 1–11, Apr. 2001.
Waqqas M. Khan received the B.S. degree
in mechatronics from Air University, Islamabad,
Pakistan, in 2007, and the M.S. degree in mecha-
tronics from the American University of Sharjah,
Sharjah, UAE, in 2015.
He was a Hardware Design Engineer with
Knowledge Platform Inc., Singapore, from 2008 to
2010. From 2010 to 2012, he was a Research
Assistant with the American University of Sharjah.
He joined Aker Solutions, UAE, as a Field Engineer,
in 2012. His research interests are in sensing, instru-
mentation, and modeling and simulation of non-linear control systems.
Imran A. Zualkernan (M’03) received the B.S.
(High Distinction) and Ph.D. degrees in com-
puter science from the University of Minnesota,
Minneapolis, MN, USA, in 1983 and 1991, respec-
tively.
From 1984 to 1990, he was a Research Assistant
with the Knowledge Systems Laboratory, University
of Minnesota. He was an Assistant Professor
with the Computer and Electrical Engineering
Department, Pennsylvania State University, College
Park, from 1992 to 1995. He was a Principal Design
Engineer with AMCS Inc., Chanhassen, MN, USA, from 1995 to 1998.
He was the Chief Executive Officer with Askari Information Systems from
1998 to 2000, and the Chief Technology Officer for Knowledge Platform, Inc.,
Singapore, from 2000 to 2003. In 2003, he joined the American University of
Sharjah, UAE, where he is currently an Associate Professor of computer sci-
ence and engineering. He has authored over 170 articles. His research interests
include consumer systems, sensor-based Internet applications, Internet of
Things (IoT), and application of big data to IoT.