IIT KANPUR

Video Segmentation using Optical Flow Motion Vectors

Term Paper for EE 608 Digital Video Processing

Kushagra Singhal-Y9227304, Yogesh K Soniwal-Y9227672

2012-2013

ACKNOWLEDGEMENT
We would like to thank Professor Sumana Gupta for giving us an opportunity to work on such a project for our course. Also we would like to thank our friends and everyone else who helped us in bringing this project from initialization to finalization. Thank You All

INDEX 1. Abstract..............................................................................................................3 2. Introduction........................................................................................................3 3. Method used.......................................................................................................3 4. The Algorithm....................................................................................................4 5. Simulation Results..............................................................................................4 6. Understanding Developed..................................................................................5 7. Conclusion..........................................................................................................6 8. References...........................................................................................................6

ABSTRACT
In this term paper we have tried to implement the work done in [1]. We have first obtained motion vector between two video frames using Horn-Schunck optical flow method. Then we have segmented the moving objects using motion vector. We have also segmented the objects when they are occluded.

smooth. Also as the neighbouring points have the same energy, optical flow is varying smoothly. Because of smoothness constraint we can assumes to be isotropic as follows
1 , = ( 2 + 2 ) 2

INTRODUCTION
Moving video object segmentation is an important technique used in video surveillance. In this term paper we have used motion vectors for segmenting video objects. We have obtained motion vectors using optical flow. Optical flow is the distribution of apparent velocities of movement of brightness pattern in an image. Optical flow can be used to obtain motion of pixels of videos frames. Optical flow methods calculate the motion between two image frames taken at time t and t+t. For image intensity I(x,y,t) if we move x, y, t in spatiotemporal direction then image constraint equation can be given as
, , = ( + , + , + )

From calculus of variation following two PDEs can be obtained

+ + = 0 + + = 0

Using Gauss-Seidel relaxation suggested by Horn-Schunck motion vector for kth iteration can be written as
+1 = +1 + + + + + + = + +

Where k is iteration number, and are averages of neighboring points of u and v respectively. Images of optical flow can be obtained using iterative method as follows
(0) +1, + (1) +1, + (0) +1, +1 + (1) +1, +1 ( 0 8 (0) , +1 + (1) , +1 + (0) +1, +1 + (1) +1, +1 ( 0 = 8 =
,

+ 1 + 1

+ 0 + 0

, +1 + +1, +

1 1

, +1 ) +1, )

From Taylor series expansion on RHS of above equation following can be established
+ , + , + , , + + +

And time difference is

= ((1) , + (1) +1, + (1) , +1 + (1) , +1 ) ( 0 8
,

+ 0

+1,

+ 0

, +

, )

and can be obtained from following relation

= 1, + ,1 + +1, + , +1 2 1, + , 1 + +1, + , +1 2 + 1, 1 + 1, +1 + +1, 1 + +1, +1 4 1, 1 + 1, +1 + +1, 1 + +1, +1 4

From these two relations we can conclude that

+ + =0 + + = 0

Where u and v are x and y components of velocity or optical flow of , , and , , are derivative of image at location (x,y,t). Above equation has 2 unknowns u and v, hence cannot be solved directly. There are various methods to solve this optical flow equation viz. phase correlation, Lucas Kanade, BuxtonBuxton, BlackJepson method and Horn Schunck method. We have used HornSchunck method which optimizes a functional based on residuals from the brightness constancy constraint.

Horn-Schunck algorithm is summarized in figure 1.

METHOD USED Horn-Schunck Technique

The energy functional used in Horn-Schunck method is
=
2 (( + + ) + ( 2

+ 2 ))

Where ispartial derivative of f with respect to x and = 2 + 2 . Horn-Schunck method assumes that the grey values are constant across frames because the displacement between the frames is small and the flow field has to be globally

Figure 1 Horn-Schunck Algorithm

THE ALGORITHM
The overall algorithm is as follows: 1. INPUT: Two frames of video signal 2. INITIALIZATION: Initialize optical flow vector as zero 3. Estimate spatio-temporal derivative 4. FOR i=1 to ITER I. II. Compute local averages of flow vector Compute flow vectors constrained by its local average and the optical flow constraints

5. End FOR 6. Plot the flow vector diagram. 7. Segment the moving video objects using the optical flow motion vectors
Figure 2(b) Optical Flow Motion Vectors

SIMULATION RESULTS
Simulations were performed using MATLAB. Two cases were considered. First the case with no occlusion is simulated for two sets of frames. Then the case of occlusion is also considered for one set of frames. Occlusion corresponds to the motion estimation where an object is present in one frame but not in the other frame or when an object hides certain features in a frame. The steps followed in the simulations are same as that given in the algorithm described above. i) Without Occlusion Here the simulation is performed for two sets of frames as shown in figure 2(a). It contains three moving objects represented by three female walkers in the corridor. Figure 2(b) shows the results for the optical flow estimation using the Horn-Schunck method. We have shown motion vectors for two objects only for clarity. As can be seen from the figure that vectors(arrows) have larger length for the moving objects. Using these motion vectors we segmented the image into moving objects as shown in figure 2(c). The black region corresponds to the stationary regions and the moving objects can be seen readily. There are some issues in the segmentation. We will address them in a later section. As can be readily seen, the three moving objects are identified after segmentation.

Figure 2(c) Segmented Moving Objects

Similar simulations were conducted for set-2 frames which consisted of four moving objects. The results are shown in figure 3. As can be seen the four objects are detected in segmentation.

Figure 2(a) Original Frames 1 and 2 respectively(set1)

Figure 3(a) Original Frames 1 and 2 respectively(set-2)

5
thing for us which we will take forward with us. Our MATLAB coding is also boosted due to this project. Also we learnt that rather than doing the segmentation on full image we should do it in facets to get good results. We used to think that image and video processing is a real tough task but thanks to this project our misconception is now cleared. We would really like to work on this project in some more detail if we get an opportunity in the future.

Figure 3(b) Optical Flow Motion Vectors Figure 4(a) Original Frames 1 and 2 respectively(occluded)

Figure 3(c) Segmented Moving Objects Figure 4(b) Optical Flow Motion Vectors

ii) With Occlusion Motion Estimation in presence of the occlusions is a difficult task as the information is lost. We have also done the simulations on an occluded set. Figure 4(a) shows two frames which are occluded as the man is missing in the second frame. The optical flow vectors are shown in figure 4(b) and segmentation results are shown in figure 4(c). The segmented results show some more areas as motion regions due to occlusions.

UNDERSTANDING DEVELOPED
There are a lot of things that we have learnt from this project. This is the first time we have worked on video signals. Hence the concepts that we learnt in lectures are applied for the first time on real video data. Working on Optical Flow is a new concept for us and we have learnt the Horn-Schunck method for solving the problem. Segmentation on images is also a new

Figure 4(c) Segmented Moving Objects(Occluded set)

CONCLUSIONS
In this term paper we have implemented the algorithm given in [1] for image segmentation using the optical flow motion vectors. We performed the simulation of the algorithm on three data sets. Two sets were used for the non-occluded frames and one for the occluded frames. We got good results for the former case and decent results for the latter case. Also we need to improve our segmentation approach as it can be seen in the results that the regions that are stationary are also segmented as moving regions. The reason may be the threshold value we used for segmentation. We have put our effort in this paper and although the results may not be very good but it was a great learning experience for us.

REFERENCES
[1] Jiang, M.; Crookes, D., "Video Object Motion Segmentation for Intelligent Visual Surveillance," Machine Vision and Image Processing Conference, 2007. IMVIP 2007. International , vol., no., pp.202,202, 5-7 Sept. 2007 [2] N. Vasconcelos, A. Lippman, Empirical Bayesian Motion Segmentation, IEEE Trans. Pattern Anal. and Machine Intell., Vol. 23, No. 2, Feb. 2001, p217; [3] T. Yu, Y. Zhang, Retrieval of video clips using global information, Electronics Letters, July 2001, Vol.37 No.14, p.895; [4] A. Mansouri and J. Konrad, Multiple Motion Segmentation With Level Sets, IEEE Trans. Image Processing, Vol. 12, No. 2, February 2003, p.201;