Image Processing - CH 10 - Image Segmentation

Image Segmentation Image Segmentation
Objective: extract attributes (objects) of interest from an image
Points, lines, regions, etc. Discontinuities and similarities Point and line detection
Common properties considered in segmentation:
Approaches considered:
Image Processing with Biomedical Applications ELEG-475/675 Prof. Barner
Edge linking Histogram, adaptive, etc.
Thresholding methods
Region growing and splitting

Prof. Barner, ECE Department, University of Delaware
Image Processing Image Segmentation
Detection of Discontinuities
Line Detection
Mask filtering approach: R = z + z + + z = z Isolated point detection: |R|T Example:

9 1 1 2 2 9 9 i =1 i i
X-ray image T=90% of max value Input, gradient, threshold output
Line detection masks

Detects lines one pixel wide Line orientation specific
Set orientations specific thresholds
Image Processing Image Segmentation Prof. Barner, ECE Department, University of Delaware
Second derivative based

Prof. Barner, ECE Department, University of Delaware 4
Line Detection Example
Edge Detection
Wire-bond mask for electronic circuit Application of 45 edge mask Result of thresholding
Concepts:

Edge local Boundary global Step Ramp Ideal edges are smoothed by optics, sampling, illumination conditions Inch thickness determined by transition region
Ideal edge:
Practical edge:

Image Processing Image Segmentation Prof. Barner, ECE Department, University of Delaware
Edge Example Noiseless Case
Edge Example Noisy Case

Gaussian noise corrupted edge Derivatives amplify noise
Ramp edge The first derivative:

Pulse Thick edges
Second derivative:

Spikes at onset and termination Zero crossing marks edge center

Even modest levels of noise severely degraded gradient-based edge detection Possible solution: noise smoothing prior to edge detection
Gradient Operators
Gradient Operators and Example
Two-dimensional gradient:
f Gx x f = = G y f y
1/ 2
Magnitude:
2 2 Gx f = mag ( f ) = + Gy
Direction (angle)
( x, y ) = tan 1
Gy Gx
Perpendicular to edge
f Gx + G y
Approximation:
Shown: mask realizations

Application: Horizontal, vertical and (additive) gradient

Gradient Example (I)
Gradient Example (II)
Preprocess image
Smooth detail textures Thicken edges Filter: 5x5 averaging filter
Extension to 45 gradients and their application

11
12
Laplacian of a Gaussian (LoG)
LoG Example
Angiogram example
Recall Laplacian:
2 f = 2 f 2 f + x 2 y 2
Sobel output shown for reference Threshold LoG Mark zero crossings
To obtain edges:

Edge detection limitations: Produces double edges, insensitive to edge direction, sensitive to noise Pre-smooth with Gaussian filter
h( r ) = e
r2 2 2
Numerous false (spaghetti) edges First derivative more widely used
Combined (linear) smoothing and derivative operations r2 2 2
r2 2 2
h(r ) = 4
LoG models certain aspects of the human visual system

13
14
Edge Linking
Edge Linking Example
Procedures often yield broken edges
Noise, illumination irregularities, etc.
Link neighboring segments based on predefined criteria
Example criteria:
Goal: license plate localization Shown: horizontal and vertical gradient images Linking criteria:

Strength of gradients
f ( x, y ) f ( x0 , y0 ) E
Gradient 25 Angle differences 15 Linked edges Search for license plate based on rectangle side ratios
Direction of gradients
( x, y ) ( x0 , y0 ) < A
Final result:

Applied over predefined search neighborhood
15
Hough Transform (I)
Hough Transform (II)

General approach:

Project feature into a parameter space Examples: lines, circles, etc. Defining parameters: slope and intercept Map lines into the single (slope, intercept) 2-tuple
Line case:

Line equation: yi=axi+b Parameter space:

Advantage: an infinite number of points get mapped to a single 2-tuple
Reverse operation for isolated (binary) points
Line case: a point is located on an infinite number of lines

Map to all (slope, intercept) 2-tuples corresponding to the infinite number of lines passing through the point Result: a curve in the (slope, intercept)
Fix xi and yi Line in parameter space: b=-xia+yi All lines (in parameter space) for points on a line in image space cross at a single point
Crossing point: common (slope, intercept)

17
18
Hough Transform (II)

Hough Transform Example (I)
Problem: infinite slopes Solution: normal representation of a line
Image space:
Five points Five curves
Parameter space:
x cos + y sin =
Procedure for binary images:

Subdivide , plane Map each point in the image plane to a curve in the , plane
Parameters space curve intersections:
Increment cells in subdivided , plane crossed by curve
Peak values in , plane correspond to lines in the image planes

Lines connecting points in image space

20
Hough Transform Example (II)

Collimation of X-Ray Images (I)

Edge localization example Input: infrared image Process:

Edge detection Hough transform Peak detection Map (lines) back to image space
Problem: identify region of exposure Problem: x-ray scattering smoothes edges Solution:

Enhance edges Detect edges Radon transform detected edges
Hough transform generalization
Generalizations to other shapes

Identify lines Map border (lines) back to image space

22
Collimation of X-Ray Images (II)
Collimation of X-Ray Images (III)
Radon transform:
xy
R ( , ) = f ( x, y ) ( x cos + y sin )dxdy
Examples:
Sample border
Noise free and noisy cases
Image and transform domain representations
Peaks in transform domain correspond to lines in image domain

Test case categories and the percentage of images in each category

23 Image Processing Image Segmentation Prof. Barner, ECE Department, University of Delaware 24
Collimation of X-Ray Images (IV)
Collimation of X-Ray Images (V)
Sobel edge detection results

Image Processing Image Segmentation Prof. Barner, ECE Department, University of Delaware 25
Radon transform of edge detection results

Collimation of X-Ray Images (VI)
Thresholding Approaches
Thresholding is appropriate when:
Objects and background have different intensities Multimodal distribution Globally Locally Adaptively
Threshold can be set:

Collimation results based on lines detected in the radon transform domain

Utilized multiple thresholds

Illumination Effects

K-Means Algorithm
Recall image model:

f ( x, y ) i ( x, y ) r ( x, y )
Clustering algorithm
Apply to spatial or multidimensional samples Apply to intensities (one-dimensional) to cluster histogram
Utilizing the log:

z ( x, y ) = ln f ( x, y ) = ln i ( x, y ) + ln r ( x, y ) = i '( x, y ) + r '( x, y )
Procedure:
1. 2. 3. 4.
Place K points in the space

Initial cluster centroids Set randomly or with a priori knowledge
If components independent:

Convolve distributions If i(x,y) is constant
Assign all points (pixel values) to the cluster defined by the closest centroid
Utilized appropriate distance metric (Euclidian, city block, etc.)
Density is a delta
Recalculate the positions (values) of the K centroids

For Euclidian distance, calculate the mean of all points in a specific cluster
Uneven illumination yields convolved, distorted distributions
No longer separable
Repeat Steps 2 and 3 until centroid movements areto below a fixed threshold
K-Means Clustering Example

10 9 8 7 6 5 4 3 2 1 0 0 1 2 3 4 5 6 7 8 9 10
10 9 8 7 6 5 4
Fingerprint Example

10 9 8 7 6 5
Assign each objects to most similar center
3 2 1 0 0 1 2 3 4 5 6 7 8 9 10
Update the cluster means
4 3 2 1 0 0 1 2 3 4 5 6 7 8 9 10
reassign
10 9 8
10 9 8 7 6
reassign
Input image: grayscale Histogram shows two modes Set threshold with Kmeans algorithm
K=2 Arbitrarily choose K object as initial cluster center
K=2
7 6 5 4 3 2 1
Java Demo
0 0 1 2 3 4 5 6 7 8 9 10
Update the cluster means
5 4 3 2 1 0 0 1 2 3 4 5 6 7 8 9 10
31
32
Mixture Model and the EM Algorithm (I)
Mixture Model and the EM Algorithm (II)
Statistical modeling
Relaxation of K-means
Assume samples are from a mixture of two Gaussians:

Y1 ~ N(1,21) Y2 ~ N(2,22) Y = (1-)Y1 + Y2 where {0,1} with Pr(=1)=
Objective: given N observed samples, determined all parameters = (,1,2) = (,1,21,2,22) Estimation technique: Maximum Likelihood
Log-likelihood function:
L ( ; Z) = log[(1 i ) p ( yi ) + i p 2 ( yi )]
i =1
1
The PDF of the samples is thus pY(y) = (1-)p (y) + p (y) 1 2

1
Direct maximization is difficult Solution: suppose we know the values of the is

L ( ; Z, ) = log[(1 i ) p ( yi ) + i p 2 ( yi )]
i =1
1
Log-likelihood function reduces to:

N
where p (y) is a Gaussian PDF with parameters =(,2)

Image Processing Image Segmentation 33 Image Processing Image Segmentation
34
Mixture Model and the EM Algorithm (III)
EM Algorithm for Two-Component Gaussian Mixture

1. 2.
2 1 , 1 2 , 2 , Make an initial guesses for the parameters 2 , Expectation Step: compute the responsibilities p ( yi ) i = , i = 1, 2, , N ) p ( yi ) + p ( yi ) (1
2 1 2
Given L ( ; Z, ) = log[(1 i ) p ( yi ) + i p 2 ( yi )]
Case 1:
i =1
ML estimates of 1 and 21 are the sample mean and variance for all samples with i = 0 ML estimates of 2 and 22 are the sample mean and variance for all samples with i = 1
3.
Case 2:
Maximization Step: compute the weighted means and variances

1 =
But is are unknown
Solution: proceed in an iterative fashion utilizing a the expected values of is ( ) = E ( i | , Z) = Pr( i = 1| , Z) These are hidden terms, referred to as responsibilities
(1 ) y (1 ) y =
N i =1 N i i =1 i
2 1 =
N i =1
1 ) 2 (1 i )( yi
N i =1
(1 i )
i =1 i N i =1
2 2
2 )2 ( yi i =1 i
N N i =1
Determined using soft assignments of all samples and the probability of a sample being from distribution #2
4.
= 1 i =1 i and the mixing probability N Integrate steps 2 and 3 until convergence

Three Mixture Example
Biomedical Example Clustering of Assay Data
Model samples as three Gaussian mixtures Initialize with guess Colors indicate probability of belonging to each parent distribution Shown:

Samples with probabilities Initial guess distributions Distribution variance contour Iterations 1-6 and final result (iteration 20)
Shown:

Observed data Sixth mixture model with contours Mixture distribution

Java Demo
37
38
Relation Between EM and K-Means
Adapted Thresholding (I)
In the EM (Baum-Welch) algorithm

2 |<| yi 1 | Utilized binary decisions: i = 1 if | yi 1 and 2 are unweighted means Then
Simple approach:

1 =

i =1 N
(1 i ) yi (1 i )
2 =
i =1
i =1 i N
yi
Partition image Check homogeneity in each partition
i =1 i
Example: test variance
Equivalent to K-means (K=2)
Trivially generalized to a larger number of partitions/mixtures
Mixture model gives soft (probability) cluster assignments
Generalizations
Segment nonhomogeneous partitions
Fuzzy C-Means

Example: K-means
Samples assigned to more than one cluster membership function Assignments are functions of distance More robust
Shown:

K-medoids use cluster median as central representative point
Global thresholding Adaptive thresholding

10
Adapted Thresholding (II)
Optimal Thresholding (I)
Enlargements of:
Considered two objects

Correctly segmented partition
Foreground/background Overlapping PDFs p( z) = P 1 p1 ( z ) + P 2 p2 ( z )
Overall (mixture) PDF:

where P1+ P2=1
Bimodal histogram
Incorrectly segmented partition
Probability of classifying Object 2 as Object 1:

E1 (T ) = p2 ( z )dz
T
(Nearly) uni-modal histogram
Probability of classifying Object 1 as Object 2:

E2 (T ) = p1 ( z )dz
T
Solution:
Finer partitioning
Overall probability of error:

E (T ) = P2 E1 (T ) + P 1 E2 (T )
41
42
Optimal Thresholding (II)
Cardiogram Example (I)

Input After Preprocessing
Solve for optimal threshold T:

T dE (T ) d = P2 p2 ( z )dz + P 1 T p1 ( z ) dz dT dT
Objective:
= P2 p2 (T ) P 1 p1 (T )
P2 p2 (T ) = P 1 p1 (T )
In the Gaussian case:

p( z ) =
P 1 2 1
( z 1 ) 2 212
P2 2 2
( z 2 )2
2 2 2
Automatically outline heart ventricle boundaries Utilizes contrast medium Intensity log mapping to counter exponential radioactive absorption effects Subtraction of base (noncontrast) image Image (frame) averaging to reduce noise Subdivide image Generate local histograms
Preprocessing:
2 Solution is in the form of the quadratic: AT + BT + C = 0

2 2 12 ) B = 2 ( 1 2 2 2 ln ( 2 P 2 12 + 2 12 2 2 C = 1 2 1 / 1P 2) 2 A = 12 2
Results may yield two thresholds If both distributions have a common variance, 2:
T=
Procedure:

1 + 2
2
P 1 ln 2 + 1 2 P 1
43
11
Cardiogram Example (II)

After Preprocessing Final Result
Region-Based Segmentation
Procedure (contd):
Fit (uni/bi-modal) Gaussian distributions to histograms For blocks with bimodal distributions:
Previous approaches utilized continuities and/or pixel value attributes (gray value)
They do not operate on or directly consider regions Let R be the entire image region Segment R into n subregions, R1, R2,, Rn, such that:
(a) Ri = R.
i =1 n
Region-based formulation:

Set adaptively determined threshold
(b) Ri is a connected region, i = 1, 2,..., n. (c) Ri R j = for all i and j , i j. (d) P ( Ri )=TRUE for i=1,2,...,n. (e) P ( Ri R j )=FALSE for i j.
Set boundaries by taking derivative of thresholded image

P(Ri) is a logical operator that defines the properties of the region
Example: P(Ri)=TRUE if the pixel values in Ri are from a predefined set

45
46
Region Growing
Region Growing Example (I)
Approach:
Objective:
Group pixels/subregions into larger subregions based on a set criteria

Segment x-ray image to identify weld failures All pixels having maximum (255) value Absolute gray value difference 65
Set seed points:
Criteria examples: gray level, texture, color, size, shape Multiple criteria: gray value and size, etc.
Iterative procedure

Region growing criteria:
How to set the seed regions, number of regions? How to set criteria? When to stop?
Set as difference between maximum value and first mode in histogram
Pixel is 8-connected to at least one pixel in the region

47
48
12
Region Growing Example (II)
Region Splitting and Merging
Shown results:

Alternative approach to seed regions:
Input Seed regions Results of region growing Region boundaries
Subdivide image into a set of arbitrary, disjoint regions
Merge and/or split the set of regions to satisfy region segmentation conditions
Value similarity, connectivity, etc.
Observations:
Quadtree method:
Histogram is not suited to strict thresholding Connectivity criteria critical to satisfactory result
Split into four disjoint quadrants any region Ri for which P(Ri)=FALSE Merged in the adjacent regions Ri and Rk for which P(RiRk)=TRUE Stop when no further merging or splitting is possible
49
Quadtree Split-Merge Example
Morphology
Homogeneity criteria:
P(Ri)=TRUE if |zi-mi|2i for at least 80% of the pixels in Ri

Shown: input, quadtree and threshold segmentations
Region mean: mi Region standard deviation: i
Mathematical Morphology focuses on extracting image components
Threshold set as midpoint between main histogram modes
Thresholding loses details
Useful in the representation and description of region shapes Examples: boundaries, skeletons and convex hulls Typically applied to binary images Multilevel extensions exist
Set operations

51
13
Basic Set Operations
Dilation
Let A and B be sets in Z2 Standard operations:

The dilation of A by B : ) A } A B = {z | (B Result:

z
Union Intersection Complement Difference Reflection
The set of all displacements, z, such that the reflection of B and A overlap by at least one element
z
= { | = b, for b B} B
for a A}
Rewrite dilation: A B = {z | ( B) A A}
Translation ( A) z = {c | c = a + z ,
Dilation structuring element: B

Dilation Example
Erosion
Dilation application:
Filling in gaps
Example:
Scanned text
The erosion of A by B : AB={z|(B)z A} Result: The set of all displacements, z, such that B, translated by z, is contained in A Note dilation and erosion are duels of each other with the respective complementation and reflection: (AB)c=Ac B
55
14
Erosion Example

Opening
Erosion application: Elimination of irrelevant detail
A function of detail (structuring element) size Erosion removes (and shrinks) details Postprocess through dilation: expands remaining details
Example: 13x13 structuring element removes small details

Opening smooths the contour of an object, breaks narrow isthmuses, and eliminates in protrusions
A B = ( A B) B The erosion of A by B, followed by dilation of the result by B
57
58
Closing
Opening and Closing Example (I)
Closing smooths the contour of an object, fuses narrow breaks and long thin gulfs, eliminate small holes, and fills gaps in the contour
Ai B = ( A B ) B
The dilation of A by B, followed by erosion of the result by B

) ( Ai B)c = ( Ac B
Opening and closing our duels of each other with respect to complementation and reflection
59
60
15
Opening and Closing Example (II)
Watershed Segmentation (I)
Fingerprint example Objective:
Methodology: topographical interpretation of image
Three types of points:

Points belonging to a regional minimum Points that drain to a common minimum point

Remove noise and connect broken lines Opening
A drop of water released at all such points flows downhill, reaching a common minimum Points referred to as catchment basin or watershed points Points referred to as divide or watershed lines
Points that can drain to more than one minimum point
Operations:
Eliminates noise Connects broken lines

Prof. Barner, ECE Department, University of Delaware 61 Image Processing Image Segmentation Prof. Barner, ECE Department, University of Delaware 62
Closing
Watershed Segmentation (II)
Watershed Segmentation (III)

Interpretation:
Punch a hole in each regional minimum Flood entire topography from below with rising water Build dams to prevent catchment basins from merging Once fully flooded, only dams remain
Let M1 and M2 denote the set of points and two regional minima Catchment basins associated with the two regional minima:
Cn-1(M1) and Cn-1(M2)

Defines a set of points (pixel locations) that drain to each of the minima Flooding stage denoted by: n-1
Catchment basin merging occurs at step n if:

Union of points: C[n-1]=Cn-1(M1) Cn-1(M2)
Dams define (closed) boundaries

C[n-1] has two connected components At step n there is a single, one connected component, q
63
16
Watershed Segmentation (IV)
Watershed Algorithm (I)

Dam construction:
Dilate Cn-1(M1) and Cn-1(M2) subject to:

1. 2.
The dilation is constrained to q Dilation is not performed on points that cause the sets to merge
Example:

The watershed algorithm is typically applied to a gradient image Regional minima (coordinates) in image g(x,y): M1, M2,, MR Set containing the coordinates of the samples in the catchment basin associated with Mi: C (Mi) Set of image points less than threshold n:
T [ n ] = {( s, t ) | g ( s, t ) < n}
Top: Cn-1(M1) and Cn-1(M2) Middle: q Bottom: dilation

First dilation (light gray) Second dilation (medium gray)

Flood the image, and mark all pixels < the flood plane g(x,y)=n Set containing (coordinates) of points in catchment basin C (Mi) that are < n
Cn ( M i ) = C ( M i ) T [ n ]
Condition 1 violations: some dilations outside q (not shown) Point satisfying both conditions constitute dams Marked with xs
Binary image (set) indicating if catchment basin points are > n (1) < n (0)
65
66
Watershed Algorithm (II)
Watershed Algorithm (III)

Union of flooded catchment basins: C [ n] = C (M ) Union of all catchment basins:

R n i i =1
Initialization: C[min+1] = T[min+1] Recursively determine C[n] from C[n-1]:

Set of connected components in T[n]: Q[n] Three possibilities for each qQ[n]:
1.
q C[n-1] is empty

C [ max + 1] = C ( M i )
i =1
A new minimum is encountered Incorporate q into C[n-1] to form C[n]: C[n]=q C[n-1] Connected component q lies within the catchment basin of the some regional minimum (q Cn-1(Mi)) Incorporate q into C[n-1] to form C[n]: C[n]=q C[n-1] A ridge separating two (or more) catchment basins is encountered A damn is built to prevent flooding across basins Dilate qC[n-1] with a 3x3 structuring element, restricting the dilation to q See previous description
Maximum image value: max Minimum image value: min Each connected component in C[n-1] is contained in exactly one connected component of T[n]
2.
q C[n-1] contains one connected component of C[n-1]

Note: C[n-1]C[n]T[n]
3.
q C[n-1] contains more than one connected component of C[n-1]

68
17
Watershed Segmentation Example
Over Segmentation
Diffuse object example Images shown:

Watershed advantages:

Closed boundaries Good edge localization Over segmentation
Watershed disadvantages:
Observation Gradient Watershed result Watershed result superimposed on observation
Excessive number of minima Preprocessing filtering Restrict minima by using markers Merged generated regions
Over segmentation solutions:

69
70
Over Segmentation Prefiltering

Over Segmentation Markers
Original image gradient contains excessive minima Prefiltering:

Use markers to define super minima

Median removes isolated minima Thresholding removes inconsequential background minima

Region that is surrounded by greater magnitude points Points in region form a connected component Points in the connected component have the same gray level value Light gray denotes markers Markers are the only allowable minima Each region contains a single marker and background Partition each region into foreground and background

Postprocess to remove background lines
Marked points shown on a smoothed image
Results of gradient definition
Apply watershed

Each region is considered in independent image Final results consist of boundaries around the foreground in each marker defined region
71
72
18

Image Processing - CH 10 - Image Segmentation

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Image Processing - CH 10 - Image Segmentation

Uploaded by

Copyright:

Available Formats

Image Segmentation Image Segmentation

Objective: extract attributes (objects) of interest from an image

Common properties considered in segmentation:

Image Processing with Biomedical Applications ELEG-475/675 Prof. Barner

Edge linking Histogram, adaptive, etc.

Region growing and splitting

Image Processing Image Segmentation

Mask filtering approach: R = z + z + + z = z Isolated point detection: |R|T Example:

X-ray image T=90% of max value Input, gradient, threshold output

Line detection masks

Detects lines one pixel wide Line orientation specific

Set orientations specific thresholds

Second derivative based

Image Processing Image Segmentation

Line Detection Example

Image Processing Image Segmentation

Edge Example Noiseless Case

Edge Example Noisy Case

Gaussian noise corrupted edge Derivatives amplify noise

Ramp edge The first derivative:

Pulse Thick edges

Spikes at onset and termination Zero crossing marks edge center

Image Processing Image Segmentation

Image Processing Image Segmentation

Gradient Operators and Example

Shown: mask realizations

Application: Horizontal, vertical and (additive) gradient

Image Processing Image Segmentation

Image Processing Image Segmentation

Gradient Example (I)

Gradient Example (II)

Smooth detail textures Thicken edges Filter: 5x5 averaging filter

Extension to 45 gradients and their application

Image Processing Image Segmentation

Prof. Barner, ECE Department, University of Delaware

Image Processing Image Segmentation

Laplacian of a Gaussian (LoG)

Numerous false (spaghetti) edges First derivative more widely used

Combined (linear) smoothing and derivative operations r2 2 2

LoG models certain aspects of the human visual system

Image Processing Image Segmentation

Prof. Barner, ECE Department, University of Delaware

Image Processing Image Segmentation

Edge Linking Example

Procedures often yield broken edges

Noise, illumination irregularities, etc.

Link neighboring segments based on predefined criteria

Applied over predefined search neighborhood

Image Processing Image Segmentation

Prof. Barner, ECE Department, University of Delaware

Image Processing Image Segmentation

Hough Transform (I)

Hough Transform (II)

Line equation: yi=axi+b Parameter space:

Advantage: an infinite number of points get mapped to a single 2-tuple

Reverse operation for isolated (binary) points

Line case: a point is located on an infinite number of lines

Crossing point: common (slope, intercept)

Image Processing Image Segmentation

Image Processing Image Segmentation

Hough Transform (II)

Hough Transform Example (I)

Problem: infinite slopes Solution: normal representation of a line