Vision based forest smoke detection using analyzing of temporal patterns of smoke and their probability models

SunJae Ham a, Byoung-Chul Ko*a, Jae-Yeal Nama Dept. of Computer Engineering, Keimyung University, 1000 Sindang-dong, Dalseo-gu, Daegu, 704-701, Korea
ABSTRACT

In general, since smoke appears before flames, smoke detection is particularly important for early fire detection systems. To detect fire-smoke using video camera is a difficult work because main characteristics of a smoke are uncertain, vague, constant patterns of shape and color. Thus, this paper proposes a new fire-smoke detection method, especially forest smoke using analyzing of temporal patterns of smoke and Fuzzy Finite Automata (FFA). To consider the smoke characteristics over time, the temporal patterns of intensity entropy, wavelet energy and motion orientation have been used for generating, multivariate probability density functions (PDFs) are applied Fuzzy Finite Automata (FFA) for smoke verification. The proposed FFA consist of a set of fuzzy states (VH, H, L, VL), and a transition mapping that describes what event can occur at which state and resulting new state. For smoke verification, FFA is most appropriate method in case variables are time-dependent and uncertain. The proposed algorithm is successfully applied to various fire-smoke videos and shows a better detection performance. Keywords: forest-smoke detection, probability density function, fuzzy finite automata, state, event

1. INTRODUCTION
The increasing occurrence of large-scale forest fires across the world has created an urgent need for research on automatic forest-fire smoke detection as an early warning system. Most current fire alarm systems are based on infrared sensors, optical sensors, or ion sensors that depend on certain characteristics of fire, such as smoke, heat, or radiation. However, in the case of forest fires, such sensors cannot detect fire or smoke due to the extended distance between the sensors and the burning point. Therefore, many studies [1-4] have investigated the use of CCD cameras to overcome the drawbacks of traditional sensors and provide dependable fire detection results. Fire detection can be divided into two research categories: flame detection and smoke detection. Smoke detection is particularly important for early warning systems, as smoke usually occurs before flames, plus smoke can be detected at significant distances due to its diffusion characteristic. Nonetheless, despite the clear importance of smoke detection, there have only been a few studies in this area, where the main smoke detection methods can be summarized as follows. Chen et al.[1] proposed a chromaticity-based static decision rule and diffusion-based dynamic characteristic decision rule to detect smoke. While the chromatic decision rule deduces the grayish color of smoke, the dynamic decision rule analyzes the spreading attributes of smoke. Treyin et al.[2] used a Hidden Markov Model (HMM) to mimic the temporal behavior of smoke using the periodic behavior of smoke boundaries. In addition, the boundaries of smoke regions are represented in a wavelet domain, and the high frequency nature of the boundaries of smoke regions used as a basis for modeling. Yuan et al.[3] used motion information using the upward behavior of smoke. Thus, a candidate region is declared a smoke region if the ratio of the upward directions against other directions is over a pre-defined threshold.

Image Processing: Machine Vision Applications IV, edited by David Fofi, Philip R. Bingham, Proc. of SPIE-IS&T Electronic Imaging, SPIE Vol. 7877, 78770A 2011 SPIE-IS&T CCC code: 0277-786X/11/$18 doi: 10.1117/12.871995 Proc. of SPIE-IS&T/ Vol. 7877 78770A-1

Vezzani et al.[4] presented the architecture of ViSOR, a built-in surveillance ontology, which integrates many concepts, such as automatic smoke detection and the detection of people and moving objects. For smoke detection, the energy of candidate smoke regions is analyzed using the wavelet coefficients and the temporal evolution evaluated based on the color properties. Finally, a Bayesian approach is used to determine whether a foreground object is smoke. Unlike normal smoke, forest-fire smoke has a relatively slow spreading speed in relation to long-range fire-detection CCD surveillance cameras. However, most previous studies have only focused on indoor and outdoor smoke with shortrange CCD cameras. Therefore, a robust forest fire-smoke detection algorithm is needed, regarding the long distance between the smoke and the CCD camera. Accordingly, this paper presents a new smoke detection method for forest fire-smoke, based on a few characteristics of smoke and Fuzzy Finite Automata (FFA) to consider vague and uncertain information of smoke. First, moving regions are detected by analyzing the frame difference between two consecutive key frames, and candidate smoke regions identified by applying a smoke color model. Since a smoke region generally has a similar color, simple texture, and upward motion, the statistical features of these three visual features are extracted and probabilistic density functions (PDFs) are generated for the proposed FFA and smoke verification.

2. CANDIDATE SMOKE DETECTION

One of the main characteristics of smoke is a constant change of shape due to the airflow caused by wind or burning material. Thus, candidate smoke regions are initially detected using a simple background subtraction model. This process is essential for improving the smoke detection performance and reducing the detection time. 2.1 Moving block detection using key-frame differencing Unlike indoor smoke, forest-fire smoke has a relatively slow spreading speed, as the surveillance cameras for detecting smoke are installed at extended distances, as shown in Figures 1-(a). Therefore, a general frame difference cannot detect moving regions in the case of a forest fire-smoke. Thus, to overcome this problem, key-frames are selected from a video sequence whenever the frame differencing is over a certain threshold ( 1 ). First, for real-time processing, the image sequences are divided into MxN blocks and all subsequent procedures are applied to these block units. In this research, the images are divided into 32x24 blocks according to the ratio of MPEG standardization. Using the new Kframe[k+1] and previous key-frame Kframe[k],a block( Bb ) at position b in the current key-frame is declared as a moving block using the following formula:

> 2 1 ( x , y )b | Kframe[k ]( x, y ) Kframe[ k + 1]( x, y ) | Bb 0Otherwise (k 1)

(1)

D
Fig. 1. Detection results for moving blocks when key-frame differencing is performed. (a) input image and (b) result of moving blocks detected by key-frame difference

E

Proc. of SPIE-IS&T/ Vol. 7877 78770A-2

2.2 2.2 Detection of smoke block using Gaussian Mixture Color Model After detecting the moving blocks, the non-smoke colored blocks need to be filtered out, which is an essential process to reduce the computational time for smoke verification. Therefore, in this research, based on the training data used, it was found that the color of smoke is distributed in a specific area of Saturation (S) and Brightness (B) color without Hue in HSV color space. Thus, using this color distribution, a gaussian mixture color model (GMCM) was generated and trained via expectation & maximization (EM) [5] using S and B color. A GMCM with k Gaussian distributions estimated based on the fire-smoke data used in this study, where k was set at 2, which produced the best results during experimentation. From the GMCM, the likelihood of a moving block being the color of smoke is obtained by summing the weighted Gaussians, and if the result is over a pre-defined threshold, the moving block is declared a fire-smoke block. In case of images containing smoke region, almost smoke regions are detected correctly using Gaussian single model, but it is hardly distinguish non-smoke blocks which have similar color with smoke caused by light reflection. However, the proposed mixture color model has less false positives than single method depicted. After detecting the candidate smoke blocks, the image is scanned to group the blocks into clusters based on a block connectivity. Thus, all the blocks in a connected cluster are connected to each other. Once all the clusters have been determined as N clusters, some clusters are removed when the number of blocks is less than three.

3. FOREST-SMOKE VERIFICATION USING FUZZY FINITE AUTOMATA

Although the detection of candidate smoke blocks using the key-frame differencing and GMCM was excellent in videos containing smoke, non-smoke blocks were also detected due to the difficulty of distinguishing smoke-colored objects. Therefore, an additional process is necessary to remove these false candidate flame blocks. In contrast to previous heuristic methods, this study applies a FFA to verify whether or not the candidate blocks are real flames. 3.1 The feature of Fuzzy Finite Automata There is an increasing interest in using fuzzy logic in many new areas because it is a very efficient method for handling imprecision which is an intrinsic property of many systems. Especially, FFA fits best in applications where the variables are continuous and/or mathematical models are difficult to define or they do not exist. FFA combines the capabilities of automata with fuzzy logic. Therefore it not only provides a systemic approach to handle uncertainty in computational systems, but also is able to handle continuous spaces [6]. In the case of finite automata, the state is represented graphically by nodes, while the transition is represented by arcs between the nodes, which depend on certain input strings, as shown in Figure 2.

Proc. of SPIE-IS&T/ Vol. 7877 78770A-3

Fig. 2. Proposed FFA for forest-fire smoke verification. The circular nodes represent a state, while the transition from the current state to the next state is shown by an arrow with an input symbol (event). Since the shape of forest-fire smoke changes continuously with the process of time, the states of previous frames influence the state of the current time frame. Therefore, for forest-smoke verification, FFA is most appropriate method in case variables are time-dependent and uncertain. ~ A FFA F for flame verification is a 6-tuple denoted as: ~ (2) F = {Q, , , R, Z , } Where, Q = {VH, H, L, VL} is a finite set of states as VeryHigh(VH), High(H), Low(L), VeryLow(VL). is a finite set of input symbols (events), = {a11 K a13 , a21 K a24 , a31 K a34 , a42 K a44 } as shown in Figure 2 and R = {VH} is a set of initial state. We assume the membership value (mv) of VH is 1 at the beginning (i.e t 0 (VH ) = 1.0 ). A fuzzy transition function = Q Q (0,1] is used to map a current state into next state upon an input symbol, attributing a value in the fuzzy interval (0,1] to the next state. Z is a finite set of output symbols (labels), Z = {accept ( Smoke), reject ( Non Smoke)} . A symbol : Q Z is the output function for mapping a state to the output set. Since FFA permits partial membership of states [6] and all events can occur simultaneously, as shown in Figure 2, an event in fuzzy automata can take the system to more than one state with different degrees. The transition values (tvs) connected to a state are estimated from the probability values of the PDFs for VH, H, L, and VL, respectively, and estimated using equation (3) with the input feature value.
a11 a12 = a11 a11 / 4 a13 = a12 a11 / 4 0 a = a a / 4 a a a a / 4 a a a / 4 = = 21 22 22 22 23 22 22 24 23 22 a31 = a32 a33 / 4 a32 = a33 a33 / 4 a33 a34 = a33 a33 / 4 0 a42 = a43 a44 / 4 a43 = a44 a44 / 4 a44

(3)

As shown in equation (3), all the tvs, except for the self transition values a11 , a22 , a33 , a44 , are estimated by predefined rules based on the distance from the self transition event. Also, since three features are used, four self transition values are estimated for each state at time t using a linear combination of the PDFs for VH, H, L, and VL.

Proc. of SPIE-IS&T/ Vol. 7877 78770A-4

a11 = w1 p I (x | VH) + w2 pW (x | VH) + w3 pM (x | VH) a22 = w1 p I (x | H) + w2 pW (x | H) + w3 pM (x | H) a33 = w1 p I (x | L) + w2 pW (x | L) + w3 pM (x | L) a44 = w1 p I (x | VL) + w2 pW (x | VL) + w3 pM (x | VL)
where p I (x | ) , pW (x | ) , and p M (x | ) are the probabilities estimated from the three PDFs for the Intensity, Wavelet energy, and Motion orientation at time t for input vector x. Since the proposed FFA include simultaneous transitions to the active state q j at time t + 1, the mv of the current state
t (qi ) and corresponding tvs (weights) are linearly combined to assign the mv of the current state to the next state. A

(4)

single output mv can be assigned as the mv for a particular state at time t+1 ( t +1 (q j ) ) using the following maximum multi-membership resolution [7].

u t +1 ( q j ) = Max [ F ( t ( qi ), ( qi , a k , q j ))]
i =1 to n

(5)

where n is the number of simultaneous transitions from state qi to state q j . Each tv, (qi , ak , q j ) from qi to q j with input ak , together with the mv of the corresponding predecessor qi , is processed using the membership assignment ~ by using product operation F, the function F, resulting in a new mv. After calculating the next state candidate vector q next state and its mv are estimated using the Max operation. When the next state is only one of two states VH and H, it is declared as forest-fire smoke. 3.2 Probability density functions (PDFs) for state transition value Forest-fire smoke regions generally have a higher luminance with a lower frequency, as well as an upward motion with irregular fluctuation patterns in the time domain. Thus, to consider the above characteristics over time, the average of the intensity, wavelet energy, and motion orientation are used to generate probability density functions for the state transition value.

(a)

(b)

(c)

Fig. 3. Two-dimensional PDFs of average and skewness of (a) intensity and (b) wavelet energy and PDFs of UOR and motion consistency of (c) motion

Proc. of SPIE-IS&T/ Vol. 7877 78770A-5

First, to measure the visual business of a cluster, the intensity value and entropy are measured in the case of training smoke data. Since a smoke region has a relatively higher intensity value and little contrast, the PDFs of smoke exhibits a high intensity value and low entropy as shown in Figure 3-(a). Furthermore, as high frequency information is not sensitive to lighting changes or more prominent features, to discriminate a forest-fire smoke from a smoke-colored object, the wavelet energy of each block is computed using a linear combination of high frequency coefficients after a Daubechies wavelet transform. Since a forest-smoke region has continuously regular coefficient values, the PDFs for a smoke have relatively lower wavelet energy, lower skewness of the wavelet energy than a background, plus the PDFs for a smoke show a distinctive graph (VH, H) when compared to those for a non-smoke (VL, L), as shown in Figure 3-(b). Since a forest-smoke region has continually drift upwards and a relatively consistent motion vector than the PDFs for a background. Therefore, the upward orientation ratio and the motion consistency are estimated using Ko et als idea [8]. The PDFs for a smoke (VH, H) also show a distinctive graph when compared to those for a non-flame (VL L), as shown in Figure 3-(c). The four classes are VH, H, L, and VL with each feature vector x=[x1, x2], where p(x | VH) and p(x | H) are the PDFs for a smoke (VH and H, respectively), and p(x | L) and p(x | VL) are the PDFs for a non-smoke (L and VL, respectively).

4. EXPERIMENTAL RESULTS
To evaluate the performance of the proposed algorithm, Treyins algorithm[2], which performs well among existing algorithms, was compared with the proposed method. Due to the lack of public availability of video sequences containing smoke, as used in previous research, the performance tests used the same test videos containing smoke and smoke-colored objects as used in Treyins research, which included four forest-fire smoke videos and two non forestfire videos. The frame rate of the video data varied from 15Hz to 30Hz and the size of the input images was 320 by 240 pixels. Movies 1, 2, 3 and 4 were forest-smoke videos, while Movies 5 and 6 were non forest-smoke videos.

Fig. 4. The comparative results between proposed method and Treyins method [2].

As seen in comparative results Figure 4, the proposed approach outperformed Treyins method with an average true positive rate (ATPR) of 92.8% compared to 55.8% and average missing rate (AMR) of 0% compared to 32.4%, with the exception of an average false positive rate (AFPR) of 7.2% compared to 11.8%.

Proc. of SPIE-IS&T/ Vol. 7877 78770A-6

5. CONCLUSION
Most vision-based forest-fire smoke detection methods using CCD cameras have problems with frequent false alarms. Thus, to overcome these limitations, this paper proposed a forest-fire smoke detection algorithm using FFA and spatialtemporal visual features. Using the key-frame difference and GMCM, candidate smoke blocks are extracted more accurately. For smoke verification, three two-dimensional PDFs estimated from the average and entropy of the intensity, the average and skewness of wavelet, and motion information are then applied to the FFA in contrast to previous heuristic methods. Experimental results showed that the proposed approach was more robust to moving smoke-colored objects such as cloud and fog and produced a better performance compared to previous research.

ACKNOWLEDGEMENTS
This work was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education, Science and Technology (MEST) (2010-0007230)

REFERENCES
[1] T. Chen, Y. Yin, S. Huang and Y. Yen, "The Smoke Detection for Early Fire-Alarming System Base on Video Processing.", Intelligent Information Hiding and Multimedia Signal, 427-430 (2006). [2] B.U. Treyin, Y. Dedeoglu. and A. E. Cetin, "Contour based smoke detection in video using wavelets.", 14th European Signal Processing Conference EUSIPCO, 1-5 ( 2006). [3] F. Yuan, "A fast accumulative motion orientation model based on integral image for video smoke detection", Pattern Recognition Letter 29, 925-932 (2008). [4] R. Vezzani, S. Calderara, P. Piccinini, R. Cucchiara,"Smoke detection in video surveillance: the use of ViSOR (video surveillance on-line repository)", Proceedings of the 2008 international conference on Content, 289-298 (2008). [5] L. Xu, MI. Jordan, On Convergence Properties of the EM Algorithm for Gaussian Mixtures, Neural Computation (8-1), 129-151 (1996). [6] M. Doostfatemeh and S. C. Kremer, New directions in fuzzy automata, International Journal of Approximate Reasoning(38), 175-214 (2005). [7] F. Lin, H. Ying, "Modeling and control of fuzzy discrete event systems," IEEETrans.On SMC-B(32),.408-415(2002). [8] S.J. Ham, B.C. Ko, J.Y. Nam, Fire-Flame Detection based on Fuzzy Finite Automation, International Conference on Pattern Recognition, 3919-3922 (2010).

Proc. of SPIE-IS&T/ Vol. 7877 78770A-7