Hindawi

Applied Computational Intelligence and So Computing

Volume 2018, Article ID 5434897, 15 pages
https://doi.org/10.1155/2018/5434897

Research Article
Post-Fall Intelligence Supporting Fall Severity Diagnosis Using
Kinect Sensor

Bunthit Watanapa ,1 Orasa Patsadu ,2 Piyapat Dajpratham,3 and Chakarida Nukoolkit1

1
School of Information Technology, King Mongkut’s University of Technology Thonburi, Bangkok, Thailand
2
Faculty of Science and Technology, Rajamangala University of Technology Krungthep, Bangkok, Thailand
3
Faculty of Medicine, Siriraj Hospital, Mahidol University, Bangkok, Thailand

Correspondence should be addressed to Bunthit Watanapa; bunthit@sit.kmutt.ac.th

Received 20 October 2017; Revised 26 January 2018; Accepted 12 March 2018; Published 3 June 2018

Academic Editor: Xiaohui Yuan

Copyright © 2018 Bunthit Watanapa et al. This is an open access article distributed under the Creative Commons Attribution
License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly
cited.

This paper proposes a fall severity analytic and post-fall intelligence system with three interdependent modules. Module I is the
analysis of fall severity based on factors extracted in the phases of during and after fall which include innovative measures of the
sequence of body impact, level of impact, and duration of motionlessness. Module II is a timely autonomic notification to relevant
persons with context-dependent fall severity alert via electronic communication channels (e.g., smartphone, tablet, or smart TV
set). Lastly, Module III is the diagnostic support for caregivers and doctors to have information for making a well-informed decision
of first aid or postcure with the chronologically traceable intelligence of information and knowledge found in Modules I and II. The
system shall be beneficial to caregivers or doctors, in giving first aid/diagnosis/treatment to the subject, especially, in cases where
the subject has lost consciousness and is unable to respond.

1. Introduction estimation based on velocity and kinetic energy as a surrogate

Falls are a major cause of fatal injury, especially for elderly
people. Falls for elderly persons can adversely affect their
health status and quality of life. They may be a cause of
morbidity and mortality, particularly for those who are
suffering from dementia or Alzheimer’s disease because they
are stricken with forgetfulness, confusion, impaired decision-
making ability, and delayed responses when asking for assis-
tance [1]. This is still true at present, even though there
exists a revolution of automatic fall detection systems based
on various approaches, for example, acoustic and ambient
sensor-based, kinematic sensor-based, and computer vision
and Natural User Interface (NUI) sensor-based approaches
[2]. A common limitation of them is no provision of timely
and traceable incident information to physicians for making
fall diagnosis which could lead to proper treatment and
even to support first aid which is given by caregivers. Such
a limitation becomes serious if the subject lives alone and
is unable to respond or is conscious but cannot recall
the incident details. Lately, Patsadu [3] and Patsadu et al.
[4] proposed fall motion detection with fall severity level
for seriousness of injury on three areas of the body: head, hip,
and knee. However, this work also has a limitation because
the fall severity alone is not enough to support diagnosis by
physicians for further treatment.
In response to this challenge, this paper proposes an
intelligent system that encompasses the framework of fall
detection in our previous work [3, 4] so that ongoing
diagnosis for severity of fall impact on important organs
can be systematically supported for immediate first aid or
further treatment. Three main parts of post-fall intelligence
are proposed to ensure threefold contributions: (1) insightful
and reliable analytic information about fall sequence and level
of impact on important body joints based on the monitored
data gathered via the discreet Kinect; (2) ability to accurately
make a prompt notification of a fall to the persons in charge
by a supportive and manageable dashboard; (3) ability to
perform an online analytical process on the chronological
data of a fall to support diagnosis in the later-treatment stage.
The first part of the proposed system is the selection of
appropriate fall severity factors as proposed by a domain
expert, a physician of medicine in rehabilitation. These
2 Applied Computational Intelligence and Soft Computing
factors include derived expert opinion rules of sequence of
body-joint fall on impact, velocity on impact, kinetic energy
on impact, and duration of any motionless state after a fall.
The second part is notification sent after a fall to convey
incident details via electronic communication channels such
as a smartphone, tablet, or smart TV set. Notification is a
crucial early step to take in response to a fall. So, the system
can notify and help reduce injury even though the caregiver
or relevant persons are not with the elderly during or after a
fall. The last part is the provided information of fall severity
determination by the persons in charge (e.g., caregivers and
physicians). This insightful information helps the personnel
to give proper assistance and diagnosis intelligence for
prescribing treatments based on injury severity of the three
selected risk areas of the body: head, hip, and knee. These
three parts are crucial for successful processing of post-fall
intelligence in any smart home system.
The organization of this paper is as follows: Section 2
presents related works; Section 3 describes the methodology
of our proposed system; Section 4 shows the experimental
results and discussions; Section 5 presents the demonstration;
finally, a conclusion and future work directions are presented
in Section 6.
2. Related Works
Intelligent systems have been developed to collect and analyze
data based on experience, security, and connectivity for
decision-makers [5]. For the domain of healthcare, there are
various intelligent systems such as medical diagnosis, robot
control, remote sensing, and real-time monitoring.
In the case of falling with the elderly, there are many
attempts to provide protection against falls and send timely
information to relevant persons to help get a fallen elder out
of danger in time. According to the fall detection techniques
used, there are several areas of research for fall detection and
notification systems using Kinect. K. C. Lee and Y. V. Lee [6]
and Mundher and Zhong [7] have created a fall detection
system with message notification that uses a cell phone to
send a Short Message Service (SMS) message to the caregiver.
Kawatsu et al. [8] also proposed a fall detection system which
resembles the one of Mundher and Zhong [7]. When a fall
has occurred, the system sends a warning message via email
and Multimedia Messaging Service (MMS). Moreover, Rantz
et al. [9] proposed real-time alerts of actual falls that are
sent to clinicians or caregivers via mobile devices. Pathak and
Bhosale [10] presented a method for fall detection based on
body-joint positions of human subjects. After a fall detection
is encountered, an alert is sent to caregiver by using a
SIM900A GSM modem. In addition, Stone and Skubic [11]
proposed a method for a real-time fall alert with embedded
depth video clip based on hyperlink. When a fall occurs,
notification is sent to facility staff members via email.
Once a fall event is detected, Gagana and Vani [12]
proposed that a serious fall and consequent injury may lead
to the risk of death and “post-fall syndrome.” Fenton [13]
reported that body-joint falls impact not only vary but also
result in different levels of severity. Therefore, the system
needs to analyze body-joint positions being impacted based
on fatal impact. This work proposed a method for fall
detection by analyzing tracked key body joints of subject
using a depth-camera. There are several researches to detect
body joints position in the human body using a Kinect
based on depth image [14, 15]. Amongst important body
positions, head position was reported as the first one that
could suffer the most impact. Head position is also a suitable
monitored position to resolve occlusion problem in fall
detection as reported in the work of Bian et al. [16]. In
addition, elderly persons over the age of 70 have a high risk
of traumatic brain injury-related hospitalization and death
due to falls [17]. The causes of traumatic brain injury come
from prolonged unconsciousness, as well as the severity of
symptoms. Unfortunately, elderly persons have the most risk
of hip fractures because they frequently use hip hit on the
floor, although they have a low speed during the fall [18].
The third rank of injury from falls is knee bone fracture [19].
Finally, a fall may result in hand bone fractures [20].
Generally, the severity of injury can be evaluated based
on key influencing factors, for example, the height of the fall,
post-fall velocity, or acceleration of the impacted position and
kinetic energy of the fall [4, 21–23].
To our knowledge, the integrated post-fall intelligent
system has never been explicitly defined to provide the
probable post-fall intelligence of accidental falls that may
happen during daily routines. Such an information system
would be valuable not only for first aid but also for supporting
decisions about subsequent care. Unfortunately, most of the
existing models have been designed for medical healthcare
staff members based on paper form per incident to inquire
about fall information (i.e., fall history, activity of daily
life, congenital diseases, and side-effects of medicine) from
patients or caregivers [24–27].
It is essential for the model of post-fall intelligence to
take into account the integrated fall history and crucial
information using online information without the intention
to replace the specialist’s judgment that both warns and
supports caregivers to take actions on first aid or assists
specialists in deciding on subsequent care.
Next, we discuss in detail the design methodology for
systematically determining post-fall intelligence.
3. Proposed System
The proposed post-fall intelligence aims at providing timely
information to support immediate decisions of caregivers
who give first aid to a fallen person or providing traceable
chronological information to physicians for later diagnosis
or treatment. Figure 1 shows the architectural design of our
proposed system integrated with the fall detection system
as proposed in Patsadu et al. [4]. When setting up an
experiment, we used a Kinect to track the motion of a
human in an indoor environment. The Kinect was set up
approximately 1 meter above the floor to cover a room which
has dimensions of approximately 5 × 7 m. (see Figure 2).
Kinect generated a video stream of 15 body-joint positions (𝑋,
𝑌, 𝑍) with a resolution of 640 × 480 at a rate of 30 frames
Applied Computational Intelligence and Soft Computing 3

Contact person

Response
App. + ＄B Server

USB Port Fall Detected Physician

(15 body joint positions data)

Activities
Notification + Caregiver
Message

Velocity on impact Kinetic energy on

Fall impact
family
ADL
Fall severity level Estimation

Figure 1: Architecture of proposed system.

(a) RGB of monitoring of a subject (b) Depth image of monitoring of a subject

Figure 2: Setting for proposed system within a home.

per second (fps) which was extracted using the OpenNI
depth metadata process [28]. The severity analytic and post-
fall intelligence works hand-in-hand with the fall detection
system. When a fall occurs, the data from the fall detection
system will be instantly delivered to the intelligence server
for analyzing incident information (e.g., expert opinion rules
of body-joint fall impact, velocity on impact, kinetic energy
on impact, duration of motionless state after a fall, and a
fall video clip recorded during a fall). In a few seconds, a
notification message packed with the mentioned analytic data
will be sent to assigned relevant persons (e.g., caregivers,
family, and physicians) via a set of assignable communication
channels such as a smartphone, tablet, or smart TV set. Once
the message is received, the responsible persons can click a
button on the notification dashboard to take charge.
our previous work [4], the post-fall intelligence to support fall
severity diagnosis on three key body joints, head, hip, and
knee, will be proceeded. The process of post-fall intelligence
is comprised of three main steps, namely, the analysis of fall
severity factors, notification, and fall severity determination.
3.1.1. Investigating Fall Severity Factors to Support Fall Severity
Diagnosis. The severity factors in this research are coinvented
by a domain expert in rehabilitative medicine to ensure the
pragmatically effective support of fall severity diagnosis. The
four fall severity factors are defined as the expert opinion
rules of body-joint fall impact, duration of motionless state
after a fall, velocity on impact, and kinetic energy on impact.
The detail is presented next.

(1) Expert Opinion Rules of Body-Joint Fall Impact. This factor

3.1. Integrated System of Fall Severity Analytic and Post-Fall analyzes body-joint positions being impacted to examine the
Intelligence. Once a fall is detected using the algorithm of characteristics of a fall and any consequent injury. Normally,
4 Applied Computational Intelligence and Soft Computing
Table 1: Rules for examination of prolonged duration of motionlessness.
First position (hit
Rules
floor)
1 head
hip
2 or
knee
Head
Yes No
Major Hip
Severity
Yes No
Major Knee
Severity
Yes No
Moderate Minor
Severity Severity
Figure 3: Body-joint fall impact divided into severity level.
certain body-joint position fractures can frequently occur
when falling, especially, the hips, knees, hands, and so on [29].
Fractures, especially hip fractures, can cause a disability and
have a high mortality rate. Head trauma is also frequently
found. The most dangerous events occur when the head
or hip is the first body part that impacts the floor or
obtrusive objects. Empirically, the rule of considered body-
joint positions being impacted was agreed by a domain expert
in rehabilitation medicine as shown in Figure 3. The proposed
rule is divided into two stages: examination of the body-
joint position that firsts hit the floor and examination of
consequent severity level. The stages are shown as follows.
(1) The first stage considers the body-joint position that
first hits the floor and the velocity of the movement to identify
the threshold value and studied the patterns of the velocity
of the movement of each body part in a huge amount of fall
data. The threshold value was trained using 1,320 fall video
clips of eight subjects randomly selected from a total of 1,650
fall video clips as described in our previous work [4]. Each
subject performed a simulated fall with different types of
actions and different speeds. The result shows that the suitable
threshold value is equal to 0.03 m/s. So, if the velocity of the
considered body-joint position is less than or equal to the
indicated threshold, that body-joint position is hitting the
floor for the time being.
(2) The second stage is to identify the severity level
acquired in the first stage. The severity level is considered
Duration that subjects
remained Output
motionlessness
>30 seconds
Prolonged
motionlessness/did
not receive assistance
>5 minutes
from the body-joint positions being impacted. In the medical
expert’s opinion, the head or hip position is most impacted
after a fall. For the knee position and hand, the severity level
is moderate or minor impact, respectively.
Additionally, more than one body-joint position has the
possibility of hitting the floor. Therefore, the system will
also report the ordering sequence of the body-joint positions
impacted, for instance, “knee → hand → head”, “hip → hand
→ head”, or else.
(2) Duration of Motionless State after a Fall. This post-
fall factor is used to examine any subsequent prolonged
motionless status after a fall which may lead to hypothermia,
dehydration, or bronchopneumonia [30]. The post-fall phase
is defined as the duration that starts when a fall is detected.
Figures 4(a)–4(d) conceptualize three phases of fall and the
probability of fast recovery or prolonged motionless in the
3rd phase of post-fall. This suggests two subphases to be
considered: the recovery phase and the motionless phase. For
the recovery phase, the subject is able to recover within a
few seconds due to minor injury (see Figure 4(b)). For the
motionless phase, the subject may be unable to get up again
without assistance due to severe injury or unconsciousness
(see Figures 4(c)-4(d)).
The process to detect the duration of having prolonged
motionlessness can be divided into three stages as follows.
(1) Examination of the body-joint position that first hits
the floor is conducted by using the body-joint fall impact
algorithm as described in Section 3.1.1(1).
(2) Examination of duration of subject’s motionlessness:
to identify the threshold value, the patterns of the velocity
of the movement of each body part were studied in a huge
amount of fall data. The data set was trained similar to
what is shown in Section 3.1.1(1). Empirically, the suitable
threshold value is equal to 0.08 m/s. Indeed, if the velocity
of the movement of each body part is less than or equal to
the indicated threshold, it means that the subjects remained
motionless or did not receive assistance.
(3) Examination of prolonged duration of motionlessness
is based on the medical expert’s opinions. The knowledge
elicited from the medical expert was transformed into the
following rules (see Table 1).
Additionally, in some cases where more than one body-
joint position hitting the floor at the same time, we examined
the duration of the motionlessness for the body-joint position
Applied Computational Intelligence and Soft Computing 5

fall detected
fall detected

Time series head/hip/knee position data

Time series head/hip/knee position data
2.5
2.5
2
2

1.5 1.5

1 1

0.5 0.5

0 0
2 3 4 1 5 1 2 3 4 5
Time (sec) Time (sec)
Pre-fall phase Transition Post-fall phase Pre-fall phase Transition Recovery
(P1) phase (P2) (P3) (P1) phase (P2) phase

(a) General framework of three phases based on the seg- (b) Recovery phase
mented boundary
fall detected

2.5
Time series head position data

1.5

0.5

0
1 2 3 4 5 6 7 8 9 10 32 33 34 35
Pre-fall Transition Time (sec)
motionless duration prolonged motionless duration
phase (P1) phase (P2)

(c) Motionless duration based on the head hitting the floor

fall detected
Time series hip/knee position data

2.5

1.5

0.5

0
….. ….. ….. ….. …... ….. ….. ….. 5 6 7 8

Pre-fall Transition Time (sec)

motionless duration prolonged motionless duration
phase (P1) phase (P2)

(d) Motionless duration based on hip/knee hitting the floor

Figure 4: Motionless duration detection after a fall.

6 Applied Computational Intelligence and Soft Computing
4.0
3.5
3.0
V (m/s) 2.5
2.0
1.5
1.0
0.5
Head Hip
(a) Velocity level
500
400
Ek (Joules)
300
200
100
Head Hip
(b) Kinetic energy level
Figure 5: Fall severity criterion [4].
with the highest level of impact using body-joint fall impact
algorithm (see Section 3.1.1(1)).
Finally, the remaining two fall severity factors (velocity
on impact and kinetic energy on impact) are proposed in our
previous work [4]. In order to describe velocity on impact and
kinetic energy on impact of a consequent fall, the scale for
fall severity estimation developed in our previous work (see
Figure 5), called Fall Severity Injury Score (FSIS) to classify
severity level of falls, is applied. The data set was trained to
create a model similar to what is shown in Section 3.1.1(1).
3.1.2. Notification. Once a fall is detected and fall severity
factors are computed, the system that provides message
notifications to users will be automatically activated. It
provides the updated status of the fall incident on a dashboard
graphical user interface (see Figure 6). Generally, most people
have electronic mobile devices such as smartphones and
tablets. To make the system friendly and affordable, we
developed a notification system that relies on heterogeneous
communication channels such as a smartphone, tablet, or
smart TV on an Android platform, which supports Android
version 2.3 and later. The connection is through a hierarchical
Level 4
Most Severity
Level 3
High Severity
Level 2
Moderate Severity
Level 1
Minor Severity
Knee
Level 4
Most Severity
Level 3
High Severity
Level 2
Moderate Severity
Level 1
Minor Severity
Knee
network using the push notification message concept via
Transmission Control Protocol/Internet Protocol (TCP/IP)
networking through Wi-Fi. Figure 7 shows the process flow
for notification. Possible types of notification message are
summarized in Table 2.
Figure 6(a) shows the monitor screen of the proposed
system. The display is divided into 4 parts: online monitoring
and captured images that present the current state of the
subject (video clip), instant notifications and user interaction,
patient information, and fall information.
For the first part, the monitoring system is divided into
two parts: online monitoring and captured images recorded
during the fall. The first part is to record a stream of data
representing time sequential frames of fifteen body-joint
positions obtained from the Kinect for fall detection and is
analyzed for fall severity factors in real time. The second part
captures video of images occurring during the fall from the
start of the fall until the end of the fall when subjects are laying
on the floor by resampling every 5 pictures in 1 second, which
are sent to the relevant persons (e.g., caregivers, family, and
physicians). Video is planned to be available in 2 modes: Red,
Green, and Blue (RGB) and depth image (see Figure 6(a)).
The user can select either RGB or depth image. By default, the
Applied Computational Intelligence and Soft Computing 7

(a) Main GUI for message notification via computer

(b) Main GUI for message notification via smartphone

Figure 6: Continued.
8 Applied Computational Intelligence and Soft Computing

(c) Summary report of fall history

Figure 6: GUI for message notification.

Table 2: Types of message notification.

Number Message Receiver Devices

#R1,
Fall status, fall severity factors (e.g., velocity on impact, Caregiver and relevant
#R2, Computer, smartphone,
kinetic energy on impact, and body-joint fall impact persons
#R3 tablet, or smart TV
sequences), fall video clip recorded during the fall
#R4 physicians

system was set up with depth image. Depth image is beneficial
for subjects to preserve their privacy. However, in the medical
expert’s opinion, RGB is more beneficial for physicians to
understand the circumstances of a fall (e.g., place, time of day,
and subject’s activity before the fall) and retrieval data to show
fall mechanics. These fall mechanics can provide benefits in
two aspects: extrinsic factors and intrinsic factors [31]. The
first factor is beneficial for physicians to know environmental
hazards which is cause of fall such as obstruction of furniture,
wet floor, different level of floor, and house ladder. The second
factor allows physicians to know about risky body conditions
such as unstable joints, muscle weakness, visual problem,
congenital disease, and drug side-effects. So, the physician
can use these data to support decisions about physical therapy
of body-joint fall on impact. In addition, the physician can
suggest proper preventive exercise routine to reduce risk of
severe injury in future falls.
Secondly, the system triggers an alarm and displays a
message notification on several devices (see Figures 6(a)
and 6(b)). This is very important and provides timely fall
information that can support human judgment. Nevertheless,
sometimes caregivers or relevant persons do not respond
to notifications within the specified timeframe. The system
records complete incident information that is useful for
quality improvement, risk management, and peer review.
Thirdly, patient information is retrieved for viewing on a
monitor screen to confirm personal identity and to be sure of
obtaining the right information.
Lastly, fall information is divided into two parts: a fall
event and summary report of a fall detection. The first part
shows a fall event, which consists of different time between
the current fall detected and previous falls detected, velocity
on impact, kinetic energy on impact, body-joint fall impact
sequences, duration of motionlessness after a fall, and a fall
video clip captured during the fall. Note that different colors
represent different response states: red for waiting response
state, orange for not response state, and green for responded
state. The second part is to report consequent fall detection
and activity detection, which consists of the summary report
of frequency of both fall detection and activity detection and
frequency of body-joint position that is high impact acquired
in the first part.
Additionally, we provide a fall history to further support
fall severity diagnosis (see Figure 6(c)). The physicians can
select a specific time period for which they want to view fall
information such as week, month, or year. Fall information is
ranked by time, fall severity level, and so on. The report shows
fall information consisting of fall time stamp, fall direction,
body-joint fall impact sequences, motionlessness duration,
different colors representing different levels of fall severity,
velocity, and kinetic energy: Level 1 = Green, Level 2 = Yellow,
Level 3 = Orange, and Level 4 = Red, acknowledge time,
action taken time, and fall video clip.
As shown in Figure 7, the message notification works in
such a way that if a fall is detected, the notification system
will be activated. The first notification with message #R1 will
Applied Computational Intelligence and Soft Computing 9

Kinect Subject Caregiver

System Physician

15 body joint positions data

Track motion of (150 frames/5 seconds)
subject
Detect a fall

No
Yes (fall detected)

Calculate fall severity

factors (incident
information)

(Send message R1#) Normal soun＞ +

warning
Check if level of severity No message
is high (e.g. hip or head (Minor
first impact and severity/mode
prolonged motionless) rate severity)

Loud soun＞ +
Yes
emergency
message
(High
severity/Most
severity)

Check if subject is able

to get up or received
immediate assistance.

Yes

Check if caregiver is No
response to system.

No

Yes
Check if subject is able
to get up or received
assistance.

10 seconds Yes Confirm

interval polling assistance
No
Check if message is
senＮ > 3 times. Show incident information
(i) Body-joint fall impact
(ii) Velocity on impact
(Send message R2#/R3#) No Yes (Send message R4#/Rn#) (iii) Kinetic energy on impact
(iv) Duration of motionless state after
a fall
Record incident
information

Figure 7: Process flow for message notification.

10 Applied Computational Intelligence and Soft Computing

F1
(Body-joint fall impact)

Caregivers
F2
(Velocity on impact)
First aid decision and
F3 diagnosis support
(Kinetic energy on
impact) Relevant persons

F4
(Duration of
motionlessness state Physicians
after a fall)

Fall video clip

Figure 8: The modular structure for supporting decisions in first aid and diagnosis by caregivers and physicians.

be instantly sent to the caregiver and relevant persons. By a

default setting of 10-second polling, messages #R2 and #R3
will be sent in sequential order to notify the caregiver and
relatives if there is no sign of assistance or recovery. In the
next polling onwards, if there is still no sign of improvement,
message #R4 or above (#R𝑛, 𝑛 = number of the polling
rounds) will be automatically sent and brought up to a
physician. The design of multinotification is to mitigate the
effect of false positives, and the interval polling is to minimize
the risk of injury or fatality.
The notification messages are divided into two types: nor-
mal warning message and emergency message. The message
type and interval polling of notification are based on the
severity level. Once the system detected that the subject is
able to get up or receive immediate assistance, the system Figure 9: Experiment setup for post-fall intelligence system.
will record the incident information, and the application will
resume its initial state. Note that the system allows users
to subjectively select the interval polling of notification and especially useful for cases of subjects who are not able to
number of notification messages. Appropriate settings should respond due to a loss of consciousness or for cases of subjects
agree with the medical expert’s opinion. who are able to get up but also frequently fall.

3.1.3. Fall Severity Intelligence Support. Once a caregiver (or
relevant persons or physician) obtains a notification message,
the ability to trace the chronological event and severity level
of the fall to decide on aid (or to gain support for diagnosis)
can proceed by hand. The number of fall severity factors
given to support the judgment of caregivers, relevant persons,
and physicians is four factors (e.g., expert opinion rules of
body-joint fall impact (F1), velocity on impact (F2), kinetic
energy on impact (F3), duration of motionless state after a fall
(F4), and fall video clip recorded during the fall) as shown
in Figure 8. In our system, caregivers and relevant persons
use these factors to support strategic decisions about giving
first aid and taking the fall subject to see a physician. Also,
the physicians can utilize these factors to trace abnormalities
of the subject’s body condition that caused the fall and
to support the diagnosis process for later treatment. It is
4. Experimental Results and Discussion
In this section, we explain the design of the experiment and
show experimental results from post-fall intelligence.
4.1. Design of Experiment. In our experiment, we established
an indoor environment setting with a Kinect to track the
movement of sample subjects as seen in Figure 9. The setting
is with some limitations, for example, a limited area of 5
× 7 m. in closed room and only young healthy adults were
in these simulated falls which may not optimally robust to
represent the condition of frail older adults. In real use, a
physician may suggest some adjustment to fall monitoring
system threshold values to be specific to elderly subjects. We
performed preliminary experiments to select representative
sample subjects for testing both accuracy and reliability
Applied Computational Intelligence and Soft Computing 11

t − 5 (s) t − 3 (s) t − 1 (s) t (s)

consecutive frames in torso joint

Euclidean distance of each two

t − 5 (s) t − 3 (s) t − 1 (s) t (s)

1 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 100 105 110 115 120 125 130

Pre-fall phase Transition phase Post-fall phase

(a) The scenario of demonstration for fall detection with 5-second duration

Time Kinect Subject Caregiver

System Physician
(sec)

15 body joint positions data

5 sec Track motion of (150 frames/5 seconds)
subject

Detect a fall
1 sec
No
Yes (fall detected)

Calculate fall severity factors

(i) Body-joint fall impact

(ii) Velocity on impact
(iii) Kinetic energy on impact
(iv) Duration of motionless state Loud soun＞ +
after a fall
(v) Fall video clip occurring emergency
during the fall message
(Most severity)
10 seconds
interval polling

The caregiver does not

31 sec response to system

(Send message R2#/R3#) The subject is unable

to get up

Unable get up

Check if message is
senＮ ≥ 3 times.

No

Yes
Show incident
(Send message R4#) information
(i) Body-joint fall impact
(i) Body-joint fall impact
(ii) Velocity on impact
(iii) Kinetic energy on impact
(ii) Velocity on impact
(iv) Duration of motionless state after a fall
(iii) Kinetic energy on
(v) Fall video clip occurring during the fall impact
(iv) Duration of motionless
0.5 sec state after a fall
(v) Fall video clip occurring
during the fall

Record incident
information

(b) The derived factors and results severity for the case study

Figure 10: Continued.

12
(c) Monitor screen for the case study
Figure 10: The case study for fall detection and fall severity level estimation.
of the proposed system. The instruction with approval of
the Institutional Review Board (IRB) of King Mongkut’s
University of Technology Thonburi was explained to the
subjects for understanding the purpose of this data collection
and experimental guideline as mentioned in Section 5. Every
subject signed the consent form. We compared three factors:
(1) gender, (2) body weight, and (3) height. Based on the
results of our preliminary experiment, the alternatives of 18
possible cases (e.g., (thin, short), (medium, medium), and
(fat, tall)) of sample subjects can be chosen. However, we
found that some cases are not significantly different. To cope
with such a varied sample of subjects for evaluation, there
are 6 subjects (age 30 ± 8 years, body weight 75 ± 35 kg,
and height 165 ± 15 cm with an equal number of males and
females of various weights and heights). The fall monitoring
system was tested with different types of falls (such as falling
forward, falling backward, falling to the right, and falling
to the left). We also evaluated the ability of the system to
detect Activities of Daily Living (ADLs) including standing,
Applied Computational Intelligence and Soft Computing
sitting, lying down, and walking on a variety of seat types
such as sofas, chairs with a backrest, and stools. The subjects
performed all activities on safety mats.
4.2. Results and Discussion. To gain insightful experimental
results, we conducted an experiment evaluating the fall
severity level based on the results of fall classification (22 fall
video clips from 24 simulated fall video clips). All of the 22 fall
video clips consist of various situations of fall such as forward
fall, backward fall, and left/right fall. The classification results
are illustrated as a set of confusion matrix shown in Table 3.
As shown in Table 3, all 22 detected falls were further
processed to classify the levels of severity, and an accuracy of
95.45% was reported. There was only one error (Fault Nega-
tive (FN)). The case was Level 2, but the system misidentified
it as Level 1. This kind of error delays aid and could lead to
even more severe injuries. This scenario was a backward fall.
The person tried to sit down to attenuate the impact; hence,
the velocity and kinetic energy of the fall were unintentionally
Applied Computational Intelligence and Soft Computing
Table 3: Confusion matrix of fall severity level estimation using
velocity and kinetic energy.
Prediction
Actual
Level 1 Level 2 Level 3 Level 4
Level 1 12 0 0 0 In a simulated case of highest severity, if the subject
Level 2 1 6 0 0
Level 3 0 0 5 0
Level 4 0 0 0 0
softened. Moreover, to gain further insight into the effect
of the case of FN, the two cases of undetected falls were
considered. It was found that, in both cases, the severity was
classified as Level 1, but the resulting velocity and kinetic
energy were very low, which was similar to those of normal
activities. There was almost no severity involved in either
case. This means that the falls would not have any effect on the
body, and this helped to assure us of the effectiveness of the
proposed system. However, such data could still be utilized to
raise awareness of abnormality of the falls that actually took
place to yield support to further diagnoses and treatments.
5. Case Demonstration to Show Effectiveness
of the Post-Fall Support System
In this section, we demonstrate how the proposed system will
systematically and effectively work on a specifically situated
case. Our established environment setting of demonstration
is as defined in Section 4.1. In this demonstration, the subject
was a female (age 31 years old, body mass of 50 kg., and
height of 159 cm.). The plot of the demonstration was to let
our subject simulate a severe case (highest severity level) of
accidental slip, which caused the hip hit to the floor with great
force. After the fall, the subject acted as if she could not move
for longer than 30 seconds. The first 5 seconds of the fall event
was recorded and is shown in Figure 10(a).
How the system instantly responds to the situation of a
fall to support responsible personnel assisting the subject is as
illustrated in Figures 10(b)-10(c). As shown in Figure 10(b), in
a second, the intelligence system can complete the analysis of
data streamed from Kinect and learn that subject had fallen
with the hip position hitting the floor. Then one second later,
the system could identify the severity and impact sequence
of the fall (1st hit = hip (level 4 severity), 2nd hit = hand
(level 2), 3rd hit = knee (level 1), and 4th hit = head (level
2)), and it instantly notified relevant persons (both caregiver
and family) of the occurrence of the fall via the preassigned
electronic channels. The notification came in a multimedia
format of sound alert, fall video clip, and important analytic
data (online and offline records) which was adaptively shown
on a well-organized dashboard graphical user interface (GUI)
on relevant persons’ devices as seen in Figure 10(c). The
system also monitored the response of any notified person
and routinely repeated notification in cases in which no one
acknowledged the event. Any progress, for example, a person
pressing the buttons of “acknowledge” or “action taken,”
resulted in the displays of all persons being automatically
13
updated with traceable records. In addition, a person can
recheck directly with physician by pressing the buttons of
“raise up” to confirm fall severity consequence regardless of
whether the subject is able to get up or has received immediate
assistance.
could not get up for quite a long time, or beyond the 10-
second interval polling between the notifications, the system
notified the caregiver and relevant persons 3 times (#R1–#R3
messages) before elevating the severity level and automati-
cally alerting the physician (#R4) or above (#R𝑛 message).
However, if the subject self-recovered or got first aid from
the caregiver or relevant persons before #R4 activated, the
incident was recorded and the application went back to its
initial state.
In addition to the first two contributions ((i) severity
and impacting sequence of fall analysis and (ii) systematical
notification and first aid support) of our post-fall intelligence,
next we demonstrated the third contribution of diagnosis
support for further treatment. The latter contribution is
helpful to the physicians, especially, if the subject was unable
to respond due to loss of consciousness or was conscious but
unable to recall the incident details. In Figure 6(c), the current
and historical records of falls (packed with the analytical
results as illustrated above) plus the recorded collaborative
actions during the notification and first aid support can be
retrieved/sorted by impacted body parts or by time period.
Overall, the demonstration shows that the framework of
our post-fall intelligence system enables effective post-fall
analytic and diagnostic support to caregivers and physicians
when taking care of the fall subject for instant first aid and
later treatment.
Finally, in the future work, we have planned to conduct
more detail formal interview with caregivers and rehabili-
tation physicians based on questionnaires. In this study, we
have only informally demonstrated the system and inter-
viewed six caregivers and a rehabilitation physician to ask
for their overview feedback on the direction of future system
usability study. The caregivers strongly agreed that our post-
fall intelligence can provide valuable incident information
because the system contains recorded incident information
of both current fall information and fall history by week,
month, or year in all cases even though the caregiver or
relevant persons may not be with the elderly during a fall
or the fall may have been forgotten by the subject. The
physician was also satisfied and showed the intention to
use the system because the data that were derived from the
system could support the diagnosis. The physician could
focus the diagnosis on body-joint position of the injury to
ensure prompt treatment. Therefore, based on the findings,
it could be concluded that physicians will be able to quickly
and accurately diagnose the subject, particularly during an
emergency situation in which the subject requires immediate
attention and assistance.
6. Conclusions and Future Works
We proposed a post-fall support system, designed to sys-
tematically connect with unobtrusive monitoring of falls
14
(Kinect or depth-camera based system). The post-fall support
system provides a three-part contribution. The first is to
investigate fall severity factors and the impact sequence of
body parts based on expert opinion rules. The second is to
send a fall message to the caregivers and relevant persons
via electronic channels, for example, a smartphone, tablet, or
smart TV set, and provide a systematic collaborative platform
for all relevant persons. The last is to provide online analytic
information that supports the diagnostic process of caregivers
and physicians for giving instant assistance to the fall victim
and making a decision on further medical treatment.
In the experiment, the results obtained in the fall severity
level classification prove that our system yields a satisfactory
performance of 95.45% accuracy. The post-fall demonstration
illustrates an extreme case of post-fall intelligence and gives
insight into how the proposed system is beneficial to the diag-
nostic and posttreatment decision process for the relevant
persons (e.g., caregivers, family, and physicians), especially in
cases in which incident information is otherwise unavailable.
The caregiver or relevant person is timely notified and
supported to make an informed decision to take care of
the fall subject. This can help to effectively reduce risks of
injury and fatality as well as enhance the quality of life of fall
subjects. Last but not least, the ability to provide timely and
traceable incident information for supporting the physicians’
diagnosis for treatment can make their work faster, more
accurate, and more effective, particularly for subjects who
have fallen down and are unconscious and unable to respond
or are conscious but cannot recall the incident details.
However, this work still has some limitations. Firstly, a
single Kinect camera is with limited viewing angle. This might
lead to inaccurate body-joint positions in some situations,
for example, viewing angle not exposing both sides of the
body, part of the body being occluded, or sunlight effect.
Secondly, the system was tested by younger healthy adults
and only under the controlled environment. These concerns
shade some challenging issues for future work. Some modules
should be added, for example, environmental assessment,
gait assessment, multiple Kinects, and integrative model with
alternative sensors (smartphone based on gyroscopes [32,
33]). This should enhance the analytic information provided
to physicians when conducting diagnosis for causes of falls
and their treatment. Another idea is to extend the scope
of involvement to include more medical healthcare staff
members (e.g., physicians, nurses, or other medical staff) for
better collaboration.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Acknowledgments
This work was supported by Rajamangala University of Tech-
nology Krungthep. The authors thank students and staffs of
the School of Information Technology (SIT), King Mongkut’s
University of Technology Thonburi, for their invaluable
assistance in setting up the experimental environment for the
capturing sessions.
Applied Computational Intelligence and Soft Computing
References
[1] M. White, J. Segal, and M. Smith, “Understanding dementia,”
http://www.helpguide.org/articles/alzheimers-dementia/un-
derstanding-dementia.htm.
[2] O. Patsadu, C. Nukoolkit, and B. Watanapa, “Survey of Smart
Technologies for Fall Motion Detection: Techniques, Algo-
rithms and Tools,” Communications in Computer and Informa-
tion Science, vol. 344, pp. 137–147, 2012.
[3] O. Patsadu, Video Mining for Fall Motion Detection Using
Kinect And Hybrid Classification Methods [Dissertation], King
Mongkut’s University of Technology Thonburi, 2016.
[4] O. Patsadu, B. Watanapa, P. Dajpratham, and C. Nukoolkit,
“Fall motion detection with fall severity level estimation by
mining kinect 3D data stream,” The International Arab Journal
of Information Technology, vol. 15, no. 3, 2017, In press.
[5] P. Langley and J. E. Laird, “Artificial intelligence and intelligent
systems,” http://www.isle.org/∼langley/papers/systems.fellows
06.pdf.
[6] K. C. Lee and Y. V. Lee, “Fall detection system based on
Kinect sensor using novel detection and posture recognition
algorithm,” in Proceedings of the 11th International Conference
on Smart Homes and Health Telematics, pp. 238–244, Singapore,
2013.
[7] Z. A. Mundher and J. Zhong, “Real-time fall detection system in
elderly care using mobile robot and Kinect sensor,” International
Journal of Materials Mechanics and Manufacturing, vol. 2, no. 2,
pp. 133–138, 2014.
[8] C. Kawatsu, J. Li, and C. J. Chung, “Development of a fall
detection system with microsoft kinect,” Advances in Intelligent
Systems and Computing, vol. 208, pp. 623–630, 2012.
[9] M. Rantz, M. Skubic, C. Abbott et al., “Automated in-home fall
risk assessment and detection sensor system for elders,” The
Gerontologist, vol. 55, pp. S78–S87, 2015.
[10] D. Pathak and V. K. Bhosale, “Fall detection for elderly people in
homes using Kinect sensor,” International Journal of Innovative
Research in Computer and Communication Engineering, vol. 5,
no. 2, pp. 1468–1474, 2017.
[11] E. E. Stone and M. Skubic, “Testing real-time in-home fall
alerts with embedded depth video hyperlink,” in Proceedings of
the 12th International Conference on Smart Homes and Health
Telematics, pp. 41–48, Ames, IA, USA, 2014.
[12] P. Gagana and S. Vani, “Fall detection methods for the safe and
independent living of elderly people and patients,” International
Journal of Advanced Research in Computer and Communication
Engineering, vol. 4, no. 3, pp. 128–132, 2015.
[13] W. Fenton, “Introducing a post-fall assessment algorithm into
a community rehabilitation hospital for older adults.,” Nursing
Older People, vol. 20, no. 10, pp. 36–39, 2008.
[14] L. Kong, X. Yuan, and A. M. Maharjan, “A hybrid framework
for automatic joint detection of human poses in depth frames,”
Pattern Recognition, vol. 77, pp. 216–225, 2018.
[15] X. Yuan, L. Kong, D. Feng, and Z. Wei, “Automatic feature
point detection and tracking of human actions in time-of-flight
videos,” IEEE/CAA Journal of Automatica Sinica, vol. 4, no. 4,
pp. 677–685, 2017.
[16] Z.-P. Bian, J. Hou, L.-P. Chau, and N. Magnenat-Thalmann, “Fall
detection based on body part tracking using a depth camera,”
IEEE Journal of Biomedical and Health Informatics, vol. 19, no.
2, pp. 430–439, 2015.
[17] “Traumatic Brain Injury,” http://www.alz.org/dementia/trau-
matic-brain-injury-head-trauma-symptoms.asp.
Applied Computational Intelligence and Soft Computing 15

[18] P. Vestergaard, L. Rejnmark, and L. Mosekilde, “Increased

mortality in patients with a hip fracture-effect of pre-morbid
conditions and post-fracture complications,” Osteoporosis Inter-
national, vol. 18, no. 12, pp. 1583–1593, 2007.
[19] T. Kottmeier, “Fractures of the knee,” http://orthoinfo.aaos.org/
topic.cfm?topic=A00526.
[20] B. C. Hoynak, “Carpal bone injuries presentation,” http://emed-
icine.medscape.com/article/97565-clinical.
[21] B. Bowers, J. D. Lloyd, W. Lee, G. Powell-Cope, and A. Baptiste,
“Biomechanical evaluation of injury severity associated with
patient falls from bed,” Rehabilitation Nursing, vol. 33, no. 6, pp.
253–259, 2008.
[22] C. Murthy, S. Harish, and Y. Chandra, “The study of pattern
of injuries in fatal cases of fall from height,” Journal of Medical
Sciences, vol. 5, no. 1, pp. 45–52, 2012.
[23] B. W. Schulz, W. E. Lee III, and J. D. Lloyd, “Estimation,
simulation, and experimentation of a fall from bed,” Journal
of Rehabilitation Research and Development , vol. 45, no. 8, pp.
1227–1236, 2008.
[24] Falls Management, “Post fall assessment tool,” http://www.whca
.org/files/2013/04/Falls management post fall assessment tool
.pdf.
[25] Department of Health, Post-Fall Management Guidelines in
WA Healthcare Settings, Perth: Health Strategy and Networks,
Department of Health, Western Australia, 2015.
[26] M. Laskowski, “A prototype agent based model and machine
learning hybrid system for healthcare decision support,” Inter-
national Journal of E-Health and Medical Communications, vol.
2, no. 4, pp. 67–90, 2011.
[27] W. Rueangsirarak, A. S. Atkins, B. Sharp, N. Chakpitak, K.
Meksamoot, and P. Pothongsunun, “Clustering the clusters -
Knowledge enhancing tool for diagnosing elderly falling risk,”
International Journal of Healthcare Technology and Manage-
ment, vol. 14, no. 1-2, pp. 39–60, 2013.
[28] “OpenNI,” https://github.com/OpenNI/OpenNI.
[29] National Institute on Aging, “Health information,” https://www
.nia.nih.gov.
[30] L. Z. Rubenstein and K. R. Josephson, “The epidemiology of falls
and syncope,” Clinics in Geriatric Medicine, vol. 18, no. 2, pp. 141–
158, 2002.
[31] Institute of Medicine (US) Division of Health Promotion and
Disease Prevention, “Falls in older persons: risk factors and
prevention,” in The Second Fifty Years: Promoting Health and
Preventing Disability, Institute of Medicine (US) Division of
Health Promotion and Disease Prevention, Washington, DC,
USA, 1992.
[32] K. Aminian, B. Najafi, C. Büla, P.-F. Leyvraz, and P. Robert,
“Spatio-temporal parameters of gait measured by an ambula-
tory system using miniature gyroscopes,” Journal of Biomechan-
ics, vol. 35, no. 5, pp. 689–699, 2002.
[33] K. Aminian, C. Trevisan, B. Najafi et al., “Evaluation of an
ambulatory system for gait analysis in hip osteoarthritis and
after total hip replacement,” Gait & Posture, vol. 20, no. 1, pp.
102–107, 2004.
 Advances in
Multimedia
Applied
Computational
Intelligence and Soft
Computing
The Scientific
Engineering
Journal of
Mathematical Problems
Hindawi
World Journal
Hindawi Publishing Corporation
in Engineering
Hindawi Hindawi Hindawi
www.hindawi.com Volume 2018 http://www.hindawi.com
www.hindawi.com Volume 2018
2013 www.hindawi.com Volume 2018 www.hindawi.com Volume 2018 www.hindawi.com Volume 2018

Modelling &  Advances in 

Simulation  Artificial
in Engineering Intelligence
Hindawi Hindawi
www.hindawi.com Volume 2018 www.hindawi.com Volume 2018

Advances in

International Journal of
Fuzzy
Reconfigurable Submit your manuscripts at Systems
Computing www.hindawi.com

Hindawi
www.hindawi.com Volume 2018 Hindawi Volume 2018
www.hindawi.com

Journal of

Computer Networks
and Communications
Advances in International Journal of
Scientific Human-Computer Engineering Advances in
Programming
Hindawi
Interaction
Hindawi
Mathematics
Hindawi
Civil Engineering
Hindawi
Hindawi
www.hindawi.com Volume 2018
www.hindawi.com Volume 2018 www.hindawi.com Volume 2018 www.hindawi.com Volume 2018 www.hindawi.com Volume 2018

International Journal of

Biomedical Imaging

International Journal of Journal of

Journal of Computer Games Electrical and Computer Computational Intelligence
Robotics
Hindawi
Technology
Hindawi Hindawi
Engineering
Hindawi
and Neuroscience
Hindawi
www.hindawi.com Volume 2018 www.hindawi.com Volume 2018 www.hindawi.com Volume 2018 www.hindawi.com Volume 2018 www.hindawi.com Volume 2018