Eyes blink detection using Haar Cascade

Subtitle as needed (paper subtitle)

Nguyen Duy Quang Nguyen Dinh Nguyen

Faculty of Electrical & Electronics Engineering Faculty of Electrical & Electronics Engineering
Bach Khoa University Bach Khoa University
Ho Chi Minh City, Vietnam Ho Chi Minh City, Vietnam
nguyenduyquang06@gmail.com nguyen0096@gmail.com
 Abstract By checking
eyes is opened or closed, we
can easily detect if a driver is
sleeping or count the number
of blinking eyes.
Furthermore, it can be used
in healthcare to help doctors
cure patients - who has
problems with eyes like Dry
Eye - as a tool to measure
parameters related to blink.
By using Haar Cascade and
image processing, eyes blink
detections algorithm not only
easily implements but also is
strong enough to use in
practice, specially we only
need a cheap webcam and a
mid-range laptop or PC to
run.
Keywords Face
detection; Eyes detection;
Haar Cascade; Eyes blink
detection; warning avoid
sleeping, Dry Eyes diagnose
system automatically.
I. INTRODUCTION
According to a recent
survey, the number of
portable devices is increasing
unbelievable. In 2014, this
number get over the number
of people in the world.
Therefore, peoples will
spend more times with
computer and portable
devices. Consequently, many
diseases are appeared
regularly, specially that are
problems with eyes like Dry
Eyes because people tend to
reduce blink rate on screen
time. This will make the rate
of unconscious eye blinking
while looking at screen
within close distance reduce
notably and make eyes tired.
Our mission aim to create
a blinking detection program
to measure parameters
related to blink in real-time.
By this way, it will make
patients increase or decrease
their blink rate and type of
blink to avoid Dry Eyes
efficiently.
Identify applicable sponsor/s here. If no sponsors, delete this text box (sponsors).
According to a report of
the General Statistics Office,
in the first seven months of
2015, there were 12,910
traffic accidents and an
average of 61 accidents per
day. And nearly 70% of
traffic accidents occur
between 12 and 24 hours,
according to the Traffic
Police Department. This is
the period when the driver of
the vehicle is subjected to the
psychological effects of
fatigue. The difference in
temperature, light between
day and night (especially for
means of passenger
transport, goods ...). Addition
to, based on report(1), Dr.
Telfilo Lee Chiong says that
the lack of sleep is one of the
major causes of traffic
accidents in the world.
Estimate that 10-15% of the
car accidents are related to
lack of sleep. Specially, in
Vietnam, the driver is usually
work more than 20 hours per
day. According to AAA
Foundation, the accident rate
will be 11,5 times compared
to people sleeping 7 hours or
more per day. So, people
need a convenient way to
avoid accident from lack of
sleep. Eyes blink detection
come as a handy way to
solve this problem. By using
this program, we can alert
the driver in the event of
eyes closed more than 3
seconds, we hope that by this
way, it let the death rate
cause the lack of sleep
decrease significantly.
Furthermore, we hope
that this blinking detection
system will be available on
portable devices as a smart
feature to help people answer
the question how to have a
good general healthy body
and change the habit using
portable devices.
II. METHODOLOGY image and the
A. Object detection using rotated integral image
Haar-like feature based from an original
cascaded classifier.
image :
First, confirm that you
have the correct template for
your paper size. This
template has been tailored
for output on the US-letter
paper size. If you are using
A4-sized paper, please close
this file and download the
file MSW_A4_format.
The main idea of object
detection is to use available
classifier which is trained
before with images of a
particular object(2). The
classifier scans whole image
to fit a region of pixels at
different sizes. The resultant
classifier includes multiple
simpler classifiers of 3
types: (a) edge features, (b)
line features and (c) center- Fig. 2 (a) Upright Summed Area
surround features as shown Table and (b) Rotated Summed
in figure below Area Table; calculation scheme of
the pixel sum of SAT (c) and
RSAT(d)
B. Detect face(2) and eye
with OpenCV Cascade
Classifier.
Image preprocessing is
required in order to make
detection effortless. These
steps are (a) downsampling
and (b) contrast
Fig. 1 Types of rectangular
enhancement.
features
Because the algorithm
The algorithm applies all
will scan whole image to fit
of these simple classifiers to
Haar features. If the image is
the region of interest.
too large, it would take more
These simple rectangular
time to finish. Therefore, we
features include upright
have to downsampling the
features and tilted features
image.
are respectively calculated
by using integral image and
rotated integral image also
known as Summed Area
Table (SAT) and Rotated
Summed Area Table Fig. 2 Shrinking the image then
(RSAT). These equations calculte the face Rect on the
original size image
and figure below show how
For face and eye detect
to determine the integral
we will have 2 variables:
face_scale and eye_scale.
They are ratio between
detect size and original size
of image. We recorded the
minimum eye_scale that
detect both eye smoothly
with different video
resolution at 15cm and 20cm
in the table below:
Table 1 Best minimum eye_scale
Resolution Distance Original
1920x1080 15cm 531x177
20cm 480x160
1280x720 15cm 345x115
20cm 315x105
480x320 15cm 147x49
20cm 135x45
Original and detect
columns are sizes of eye
region before and after
shrinking. Original eye
region is calculated from
face Rect by determining
relatively eye position on
face. From the table, we can
see that eye can be easily
detected if detect region is
about 165x55 pixels. We
designed an automatic mode
that define best eye_scale
according to detected face
Rect.
Then, we equalize the
histogram to differ the bright
and dark area(3). Beside
equalizing histogram for
entire image, we process a
separate equalizing by
splitting the image into left
and right halves of the face.
If we simply applied
histogram equalization on
the left half and then again
on the right half, we would
see a very distinct edge in
the middle because the
average brightness is likely
to be different on the left and
the right side, so to remove
this edge, we will apply the
two histogram equalizations
gradually from the left-or
right-hand side towards the
center and mix it with a
whole-face histogram
equalization, so that the far
left-hand side will use the
left histogram equalization,
the far right-hand side will
use the right histogram
equalization, and the center
will use a smooth mix of left
or right value and the whole-
face equalized value.
Fig. 3 Separate histogram
equalizing
This process will remove
unbalance lighting on each
side of face.
In conclusion, the process is
demonstrated as the chart in
Figure 4.
Fig. 4 Algorithm chart
III. RESULT AND DISCUSSION
A. Specification
Base on algorithm above,
we create an application that
using eyes blink detection to
provide eyes blink counter
tool with specification
below:
Table 2 Specification in real-time
Parameter
Distance operation range:
Fig. 5 Open eye detection with high
Optimal operating distance: brightness
Frame size
Frame per second average
Frame per second maximum
Frame per second minimum
Environment requirement
Brightness operation range
Optimal operating brightness
Video Source
These parameters are
measured using the
following environment:
Table 3 Environment parameters
Parameter Fig. 6 Close eye detection with
CPU high brightness
Video Graphic Card
Ram For 2 figures above, the
OS test environment is a full
Brightness
light room about 200 lux.
Webcam
This is also the ideal
environment for this
application. The ability of
B. Experiment
eyes blink detection is
We carried out the test
maximized. The average of
with several environments
FPS will be about 20 FPS
for our application.
and 18 FPS when dropped.

Fig. 7 Open eye detection with
medium brightness
For figure above, the test
environment is a half light
room about 100 lux. FPS
will be dropped to about 17
FPS.
Fig. 8 Open eye detection with low
brightness
Fig. 9 Close eye detection with low
brightness
For figure 8 and 9 above, the
test environment is no room
light but laptop screen light
with brightness. As we
calculated, brightness is
about 40 lux. FPS will be
dropped to about 14 FPS.
Fig. 10 Open eye detection with
low brightness
Fig. 11 Close eye detection with
low brightness
For figure 10 and 11
above, the test environment
is no room light but laptop
screen light with
brightness and sunlight
splashing into. FPS is
calculated about 14.
For 480p (640x480)
video, FPS is calculated
about 25 and it would be 11
if video has Full-HD
(1080p) resolution. Video
Full-HD make FPS decrease
but the ability of detection
more accurately
IV. ADDITIONAL FEATURES
A. Authors and Affiliations
The template is designed
so that author affiliations are
not repeated each time for
multiple authors of the same
affiliation. Please keep your
affiliations as succinct as
possible (for example, do not
differentiate among
departments of the same
organization). This template
was designed for two
affiliations.
As mentioned, the
program supports source
video from real-time camera
or video. The external
interface supports real-time
observation, with an FPS
counter (Frame Per Second).
When eyes are tired, eyes
tend to focus on longer than
usual. During the
examination, with this
automated system, it helps to
alert patients to open their
eyes so that the diagnosis
and examination can be
more accurate. Or simply to
warn employees or drivers
not to fall asleep. Thus, to
avoid accidental
occupational accidents or
accidents to others
unfortunately. We easily set
the time it takes to warn of
this situation.
Fig. 10 Enable/disable sleep or
wake alarm
Besides, the proverbial
"blink of an eye" lasts only a
tenth of a second, but that is
all the time needed to clear
away dust particles and
spread lubricating fluids
across the eyeball. Every
time you blink, your eyelids
spread a cocktail of oils and
mucous secretions across the
surface of the eye to keep
your globes from drying out.
Blinking also keeps eyes
safe from potentially
damaging stimuli, such as
bright lights and foreign
bodies like dust. That is why
we also add alert function
when the eye has not been
blinked for too long to avoid
eye disease as much as
possible.
To optimize the
performance and
functionality of this
application or just serves
personal purposes, we allow
to adjusted detect scale
parameters manually to scale
up or scale down the region
of interest. If there is no
special purpose, we
recommend using scale eyes
1.2 for low resolution
webcam (about 640 x 480
pixel), using auto-scale for
larger resolution like Full-
HD or HD camera to
optimize performance. At the
top-right of this application,
FPS (beside the ability to
detect) will also be a
parameter to evaluate
adjustments.
B. Diagnosis of eye
condition
Not only an application
that propose eyes blink
counter, but also diagnostic
tool for eyes disease,
especially dry eye.
We enable analyze
parameters of an eyes
examination. Based on the
normal eyes samples and
scientific studies on the eye,
we analyze parameters of an
eyes examination. Therefore,
we can give an advice about
eyes condition of the patient.
We use the mean of lid-
contact rate and the mean
duration of lid closure/blink
as parameters. The mean
lid-closure time per
type of full closure was
divided by the
subjects total
observation time, and
then standardized per
minute. Because based
on the research in the
paper(4), we realize that
these parameters differ
noticeably between normal
eyes and dry eyes.
For each parameter, we
classify them into 3 different
smaller groups depending on
the duration of eyes closure.
Specifically, we have 3
groups as follows: all full lid
closure (> 0 second in
duration), extended lid
closures (> 0.1 second in
duration) and superextended
lid closures (> 0.5 second in
duration).
Table 4 Sample of normal eyes
Parameter
All full lid closure (> 0 0.7%
second in duration)
Extended lid closure (> 0.4%
0.1 second in duration)
Superextended lid closure 0.2%
(> 0 second in duration)
Table 5 Sample of normal eyes
Parameter Value (second)
All full lid closure (> 0 0.08
second in duration)
Extended lid closure (> 0.575
0.1 second in duration)
Superextended lid closure 1.589
(> 0 second in duration)
In total we have six
parameters to measure,
corresponding to six
conditions that need to be
checked. Then we split up
into 3 cases that the eye can
encountered. That is Non
Dry Eyes when only meet
below 2 conditions, Dry
Eyes Potential- when meet
from 2 4 conditions, and
the rest is Dry Eyes.
For users, it is possible to
directly enter the above
parameters of the patient or
use the record function to
record auto values and
diagnose disease in real-
time.
Below is a result of eyes
examination after 3 hours
work with computer in a 110
seconds eyes examination:
Table 6 Personal information
Beginning
time Beginning date
7:39:36 25/06/2017
Name Age
Object001 21 continuously, the second
tester feels little eyestrain,
this is the result of an eye
Table 7 Operation at some point of
time examination using real-time
to diagnose in about 1
minute:
Value Time
Measurement parameter
(ms)
Contact rate all
Contact rate extended
Contact rate superextended
68441 Mean duration all
Mean duration extended
Mean duration superextended
All lid closure blink count
Extended lid closure count
Superextended lid closure count
Time
Measurement parameter
(ms) Fig. 11 Using parameters real-time
to diagnose
Contact rate all
Contact rate extended All of personal
information,
Contact rate superextended
date and time,
measured parameters,
85707 Mean duration all detailed information during
the examination and results
Mean duration extendedwill be stored in a datasheet.
Mean duration superextended
All lid closure blink count REFERENCES
Extended lid closure count
Superextended lid closure[1]countTimothy I. Morgenthaler,
Teoflo Lee-Chiong, Cathy
Alessi et al. (2007) Practice
Parameters for the Clinical
Evaluation and Treatment of
Table 8 Result of measured Circadian Rhythm Sleep
information Disorders
[2] Dr. John Fernandez, Phillip
Ian Wilson (2006) Facial
Contact rate Value Mean duration
feature detection using Haar
Classifiers
All full lid closure 6.70228 All full lid closure
(> 0 second in [3] in
(> 0 second Daniel Llis Baggio, Shervin
duration) duration) Emami, David Milln Escriv,
Extended lid closure 5.54223 Extended lidKhvedchenia
closure Ievgen, Naureen
(> 0.1 second in (> 0.1 secondMahmood,
in Jason Saragih, Roy
duration) duration) Shilkrot et al. (2012)
Mastering OpenCV with
Superextended lid 2.18019 Superextended lid
closure (> 0 second
Practical Computer Vision
closure (> 0 second
Projects.
in duration) in duration)
[4] Dr. John Fernandez, Phillip
Ian Wilson (2006) Facial
feature detection using Haar
Result of this test is: Dry Classifiers
Eyes Potential. [5] George W Ousler 3rd, Mark B
Abelson, Patrick r Johnston et
al. (2014) Blink patterns and
Another test, after lid-contact times in dry-eye
watching movie 4 hours and normal subjects
[6] Video Capture using OpenCV
with C++ at
https://www.codeproject.com/
Articles/741055/Video-
Capture-using-OpenCV-with-
VCplusplus.
[7] Computer Vision The
Integral Image at
https://computersciencesource
.wordpress.com
[8]