Proceedings of the Second International Conference on Knowledge and Smart Technologies 2010 (July, 24-25, 2010)

A Survey of Face Occlusion Detection for Visual Surveillance System

Theekapun Charoenpong Department of Electrical Engineering, Faculty of Engineering, Srinakharinwirot University, Nakornnayok, Thailand, 26120, Email: theekapun@gmail.com

Abstract
In the present, chance of crime or terrorism occurring is increased. Thief or suspects usually cover their face with helmet, sunglasses or mask to blind their identity while they are committing a crime. The person covering their face should not be allowed to enter the security area. Video surveillance system is widely used in security areas as bank, government area or shopping mall, however, such systems cannot detect the thief or suspect automatically. To prevent the crime, an intelligent algorithm for detecting thief or suspects whom cover their face in security area is necessary. This paper presents an overview of face occlusion detection algorithms; it highlights the main algorithm components, some research challenges and limitations of existing algorithms in case it is implemented to real-world applications. Key Words: Occluded face, face detection, video surveillance system occlusion

1. Introduction
Video surveillance system is widely used in security areas as bank, government area, shipping mall or public areas [1, 2]. Thief usually covers their face with helmet, sunglasses or mask to blind their identity while they are committing a crime. The person covering their face should not be allowed to enter the security area. To reduce the chance of crime happening, an intelligent algorithm which is able to detect the person covering their face in the security area is necessary. The surveillance system should perform effectively and automatically for non-static background and when there are many people in the image. Occluded face detection algorithms are presented in the past decade [2, 3, 4]. However, the existing research does not perform effectively when the background is changed and there is more than one person in the image. To improve the performance of the existing research, it is important to study the

limitations of the existing research before developing a new algorithm. The goal of this paper is to present an overview of existing literatures on face occluded detection; it is focus on the performance of method to automation, the variant of objects that person is worn and the surrounding variation. The existing research of face occlusion detection use two types of data: still image and image sequence. Comparing with still image, the image sequence contains more information than still image, for example, detecting moving object as human by using image sequence is more effective than using still image. Several methods of face occlusion detection by using still image and image sequence have been proposed so far. Sang M.Y. and et al. [5] use Support Vector Machines (SVM) for detecting the person occluded his face while using the Automated Teller Machines (ATMs). This method rejects the occluded face that wears sunglasses or mask. The features of the normal faces and the occluded face are used to train the system. By integrating Principal Component Analysis (PCA) and SVM, the detected face is classified that it is normal face or occluded face. The accuracies are 95.5% and 98.8% for detecting sunglasses and mark, respectively. By using SVM, global features of face are influenced easily by noise or occlusion. K. Hotta [6] overcomes this problem by applying local kernels into SVM. The robustness for occlusion detection is improved. Hottas method is tested by facts wearing sunglasses or scarf. T. Kurita and et al. [7] improve the robustness of occlusion in face recognition and detection. A neural network architecture which integrates an auto-associative neural network into a simple classifier is proposed. A partial of face is harbored by rectangular and wearing sunglasses. The original face image is recalled from a partial occluded face image. The occluded region is then detected. The original face can be reconstructed by replacing the occluded regions with the recall pixels. Therefore, the integrated network can classify occluded face robustly. J. Kim and et al. [4] propose an example-based learning approach for detecting a

Proceedings of the Second International Conference on Knowledge and Smart Technologies 2010 (July, 24-25, 2010)

Figure 1. Example of techniques which are used in each process partially occluded face in an image captured by a camera of ATM in a bank. The face wearing mask, sunglasses and muffler are detected. SVM is used to classify the occluded face. By using this method, face representation is robust to illumination variance. J. Chen and et al. [8] propose the method for detecting face occluded by other objects. AdaBoost-based face detector is trained by using Haar-like feature for classification. Face region is divided into multiple patches. Each weak classifier is mapped into patches. By verifying face patch, the threshold is then determined. The threshold is decided for a face candidate by assigning the patches with the different weights to vote whether the input sub-window is a face. D.T. Liu and at el. [9] presented an automatic algorithm for detecting face wearing helmet, mask, sunglasses and hat. This method is developed for ATM surveillance. Image is provided by the surveillance camera installed in nearly all ATMs. Moving object is identified and segmented from background. The moving edge is then captured using Sobel operator and subtraction of two continual frames. A Straight Line Fitting algorithm is used to merge the splitting blobs. Face region is defined by the elliptical head tracker. The face occlusion is classified by using skin area ration. For the viewpoint of implementing the research to real-world applications, the existing methods have to detect the occluded face automatically even though the object the person worn is varied. This paper is organized as follows: main architectural components are described in section 2. In section 3, we propose some challenges of existing method for implementation and summarize the overall of paper in last section.

2. Main Architectural Components

This section describes definition of face occlusion, generic architecture and finally, general description.

2.1. Definition of Face Occlusion

Definition of the partially occluded face is defined by S.M Yoon and et al. [5]. For higher performance of novel method, not only partially occluded face can be detected but the whole head occlusion should be also detected. Generally, the feature of face used for people and machine recognition is eyes, mouth, eyebrows and nose. It is more difficult to identify ones face if one component or whole of head is covered with an obstacle. In this paper, face occlusion is defined that the face is not able to be recognized or classified because of being covered by an obstacle such as sunglasses, mask, helmet, hat or muffler while the face without an obstacle which is able to identify is normal face. To prevent a crime, a system warning when the person with an obstacle accesses security areas as bank, government area, shipping mall or public areas is necessary. As controlling the access of security zone, the crime can be prevented.

Proceedings of the Second International Conference on Knowledge and Smart Technologies 2010 (July, 24-25, 2010)

2.2. Main Architectural Components

Main process of face occlusion detection consists of image acquisition, pre-processing, feature extraction, classification and labeling as shown by the sequence blocks in Fig. 1. Table 1 presents a small sample of method for face occlusion detection. 1) Image Acquisition: there are two types of data used in the existing methods, one is 2D still image and another is image sequence as shown in Fig. 1. The 2D still image is used in several methods [5, 6, 7, 8]. These methods are proposed based on the assumption that the face is captured from the frontal viewpoint. For image sequence, it is captured from the video camera with setting of back ATM. The methods of J. Kim [4] and D.T. Lin [9] are able to detect the person whom access to use of ATM. 2) Pre-processing: existing research using still image are developed based on the assumption that the face data has been available as collected in the AR face database. Therefore, pre-processing often tasks the form of face segmentation, normalization, including with moving objection detection for the research using image sequence [4, 9]. Face segmentation is concerned with the face region identification. The method of S.M. Yoon [5] segments the face region by using Gabor wavelet and SVM and size of face is then normalized based on the eye position. J. Chen [8] divides the face region into multiple patches. Each patch is then classified. The moving object detection is often implemented for defining the position of person in an image sequence. Face region is then defined. Different technique with mentioned above is also used such as technique of T. Kurita [7]. T. Kurita method uses an auto-associative neural network for recalling the original face. The original face pixel is used for replacing the original pixel into the occluded region. 3) Feature extraction: feature extraction converts pixel data into a higher-level representation of eye, mouse, nose, eyebrow, face components or partial face occlusion. Feature extraction generally reduces the information which is redundant in order to improve the performance of classification. J. Kim [4] represents the full face with the gradient map in scale space. The face image is separated into two parts upper part and lower parts. Principle Component Analysis (PCA) is used to find a low dimensional subspace. One benefit of redundant information reduction is to make the computational time faster. PCA is also used in the method of S.M. Yoon [5] for extracting the feature of normal face and partially occluded face that wears the sunglasses. D.T. Lin [9] calculates the skin color ratio from skin color on face region for occluded face classification.

4) Classification: the character of the face feature is analyzed by classifiers. In the existing method, the unknown data is classified into normal face and occluded face. SVM is most popular for using as classifier. Standard database is also important for experiment. The performance of the proposed methods is compared with other research by using a standard database. The most popular usage is the AR Face database [10, 11]. In each existing method [4, 5, 6, 7, 8, 9], normal face and occluded face are used to train the system. The obstacle covering the face is different. All of that are sunglasses, muffler, hat, helmet and mask.

3. Some Challenges for Implementation

Although many research of face occlusion detection have been proposed, several drawbacks are still a challenge. As the automatic surveillance system is needed for detecting the thief or suspect who commits a crime, the problem that impacts the efficiency of current methods and challenge to the further research is mentioned in this section. 1) Viewpoint of head: viewpoint of head means that direction of the camera to a face or head when capturing data. Most of research, especially, using still image, the face image is captured from frontal viewpoint. However, in the real-world applications, the data is captured from several viewpoints of face. A method which performs effectively for the data captured from several viewpoints of face is necessary. 2) Illumination condition: as illumination for capturing the data is stable, light condition impacts the information of the image. Silhouette may occur when an image is taken against a light background or there is shadow on a part of face. In these cases, the method cannot perform effectively, especially the research using the skin color for classification [9]. Therefore, the system (in ideal) should be robust for illumination changes. 3) Variance of objects covering face: in each current research, the object used for covering the face in experiment is different. All of objects used in current experiment are sunglasses, mask, helmet, hat and muffler. Therefore, the intelligent algorithm which is able to detect several objects is necessary. 4) Complex image: in some situations, the face may be overlapped by other objects or persons. It is more difficult to detect the face. This process should be flexible enough for complex image. The controlling of image conditions is detrimental to the wide applications of automatic face occlusion detection because real-world applications require operational flexibility. However, existing method was still done based on different assumptions, for instance, the input image is frontal face viewpoint or

Proceedings of the Second International Conference on Knowledge and Smart Technologies 2010 (July, 24-25, 2010)

Table 1 Some samples of face occlusion detection method

References J. Kim and et. al. [4] Image Acquisition Image sequence which is provided by a camera of Automated Teller Machine (ATM). Database: SAIT Database [4] and Purdue Universitys AR Database [10, 11] Pre-processing Representing the face by a gradient map method for more robustness of the proposed method to illumination change. Feature extraction Detecting the face region by Support vector machines and then separating the face image which is represented in the gradient map in scale space into two parts: one is upper part and another is lower part. Principle Component Analysis (PCA) method is used to fine a low dimension of data. Classification Object covering a face: mask, muffler and sunglasses. Classes: Normal face and Occluded Face. Classifier: Support vector machines Classification performance summarization Data set: using 636 frontal face images from Purdue Universitys AR Database and 1500 images from SAIT Database. Criterion: detection rate and false alarm rate. System performance: By using Purdue Universitys AR Database, detection rate and false alarm rate of upper part are 97.28% and 1.2%, respectively. For lower part, detection rate and false alarm rate are 97.96% and 0.41%, respectively. By using SAIT Database, detection rate and false alarm rate of upper part are 95.50% and 1.08%, respectively. For lower part, detection rate and false alarm rate are 96.20% and 2.76%, respectively. Data set: 370 images wearing sunglasses and 390 images wearing the mask and using Purdue Universitys AR Database. Criterion: detection ratio and false accept ratio. System performance: 98% for the detection ratio of the normal face and 0% for false accept ratio. Data set: Frontal face image with occlusion from Purdue Universitys AR Database and shadow on face from Yale Face Database [13]. Criterion: false positive rate and true positive rate. Classes: Normal face and Occluded Face. Classifier: Support vector machines with local kernels Object covering a face: sunglasses. Classes: Normal face and nonface. Classifier: network auto-associative System performance: The performance of the proposed method using local kernel is compared with global Gaussian and global polynomial kernel. The proposed method gives the better result than the others. Data set: 250 face images, 800 non face images, Occluded face from Purdue Universitys AR Database [10, 11] and normal face from MIT face database [12]. System performance: the performance is evaluated by using recognition rate. The proposed method shown that the robustness against occlusion is improved.

S.M. Yoon and et. al. [5]

Still image Database: Purdue University Face Database [10, 11]

Segmenting the face region by using Gabor wavelet and SVM and then normalizing the size of face based on the eye position.

Using PCA for extracting the feature of normal face and partially occluded face that wears the sunglasses.

Object covering a face: mask, muffler and sunglasses. Classes: Normal face and Occluded Face. Classifier: Support vector machines Object covering a face: muffler, sunglasses, shadow on the face.

K. [6]

Hotta

Still image Database: Purdue University Face Database [10, 11] for detecting the occluded face and Yale Face Database [13] for detecting the shadow on a face. Still image. Database: Purdue University Face Database [10, 11] for occluded face and MIT face database [12] for non-face detection.

Defining the face regions by using the position of eyes, noise and mouth.

Using Gabor filters to extract local appearance of the face.

T. Kurita and et. al. [7]

Defining the face regions by using the simple nearest neighbor classifier and normalizing the size of the face regions to 30x28 pixels.

Using the auto-associate network to detect the occluded regions.

Proceedings of the Second International Conference on Knowledge and Smart Technologies 2010 (July, 24-25, 2010)

a value of some parameter as skin is fixed. Therefore, if current researches are implemented to real-world application, it may have some errors in some situations.

4. Conclusions
Thief or suspects usually cover their face with helmet, sunglasses or mask to blind their identity while they are committing a crime. An automatic system of face occlusion detection is needed. In this paper, we study limitations of current research in case of implemented to real-world applications. Main architectural components of existing methods consist of image acquisition, pre-processing, feature extraction, classification and labeling. Some challenges for implementing the existing research to real-world applications are listed as follows: 1) Viewpoint of head, 2) the variance of the illumination conditions, 3) Variance of objects covering face and 4) The performance of system for complex image.

[9] D.T. Lin and M.J. Liu, Face Occlusion Detection for Automated Teller Machine Surveillance, PSIVT 2006, pp. 641-651, 2006. [10] A.M. Martinez and R. Benavente, The AR Face Database, CVC Technical Report 24, 1998. [11] A.M. Martinez, Recognition of Partially Occluded and/or Imprecisely Localized Faces using a Probabilistic Approach, CVPR, Vol. 1, pp. 712-717, 2000. [12] MIT face database, ftp://whitechapel.media.mit.edu/pub/images/ [13] A.S. Georghiades, P.N. Belhumeur, and D.J. Kriegman, From few to many: Illumination cone models for recognition under variable lighting and pose, IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 23, no.6, pp. 643-660, 2001.

5. References
[1] B. Wu, H. AI and R. LIU, Glasses Detection by Boosting Simple Wavelet Features, Proceedings of the 17th Int. Conf. on Pattern Recognition ICPR 2004, V. 1, pp. 292-295, 2004. [2] Lin D.T. and Liu M.J, Face Occlusion Detection for Automated Teller Machine Surveillance, Advances in Image and Video Technology, Springer Berlin/Heidelberg, 2006. [3] Wen CY., Chiu SH., Liaw JJ. and Lu C., The safety helmet detection for ATMs surveillance system via the modified Hough transform, Security Technology, 2003. Proceeding. IEEE 37th Annual 2003 International Carnahan Conference, pp. 364-369, October, 2003. [4] J. Kim, Y. Sung, S.M. Yoon and B.G. Park, A New Video Surveillance System Employing Occluded Face Detection, Proc. of the 18th int. conf. on Innovations in Applied Artificial Intelligence, Italy, pp. 65-68, 2005. [5] S. M. Yoon and S. C. Kee, Detection of Partially Occluded Face using Support Vector Machines, IAPR Workshop on Machine Vision Applications, Japan, pp. 546-549, 2002. [6] K. Hotta, A Robust Face Detector Under Partial Occlusion, Int. Conf. on Image Processing, 2004 (ICIP '04), pp. 597-600, 2004. [7] T. Kurita, M. Pic and T. Takahashi, Recognition and Detection of Occluded Faces by a Neural Network Classifier with Recursive Data Reconstruction, Proc. of the IEEE Conf. on Advanced Video and Signal Based Surveillance (AVSS03), pp. 53-58, 2003. [8] J. Chen, S. Shan, Sh. Yang, X. Chen and W. Gao, Modification of the AdaBoost-based Detector for Partially Occluded Faces, The 18th Int. Conf. on Pattern Recognition (ICPR06), Hong Kong, pp. 516 519, 2006.