PATTERN CLASSIFICATION AND SCENE ANALYSIS

RICHARD O. DUDA PETER E. HART Stanford Research Institute, Menlo Park, California

A WILEY-INTERSCIENCE PUBLICATION

JOHN WILEY & SONS New York Chichester Brisbane Toronto Singapore

CONTENTS
Part I PATTERN CLASSIFICATION
1
1 2 4 5 6 7 10 10 13 15 16 17 17 20 20 22 22 23 24 26 27 30 31 32 34
xi

1 INTRODUCTION
1.1 1.2 1.3 1.4 1.5 1.6 2 2.1 2.2 2.3 2.4 2.5 Machine Perception An Example The Classification Model The Descriptive Approach Summary of the Book by Chapters Bibliographical Remarks BAYES DECISION THEORY Introduction Bayes Decision TheoryThe Continuous Case Two-Category Classification Minimum-Error-Rate Classification Classifiers, Discriminant Functions and Decision Surfaces 2.5.1 The Multicategory Case 2.5.2 The Two-Category Case Error Probabilities and Integrals The Normal Density 2.7.1 The Univariate Normal Density 2.7.2 The Multivariate Normal Density Discriminant Functions for the Normal Density 2.8.1 Case 1: 2< = a 2 / 2.8.2 Case 2: L, = 2 2.8.3 Case 3: 2 , Arbitrary Bayesian Decision TheoryThe Discrete Case Independent Binary Features Compound Bayes Decision Theory and Context

2.6 2.7

2.8

2.9 2.10 2.11

xii

CONTENTS 35 36 39

2.12 Remarks 2.13 Bibliographical and Historical Remarks Problems

3 PARAMETER ESTIMATION AND SUPERVISED LEARNING

3.1 Parameter Estimation and Supervised Learning 3.2 Maximum Likelihood Estimation 3.2.1 The General Principle 3.2.2 The Multivariate Normal Case: Unknown Mean 3.2.3 The General Multivariate Normal Case 3.3 The Bayes Classifier 3.3.1 The Class-Conditional Densities 3.3.2 The Parameter Distribution 3.4 Learning the Mean of a Normal Density 3.4.1 The Univariate Case: />([* | 3P) 3.4.2 The Univariate Case: p(x | 3T) 3.4.3 The Multivariate Case 3.5 General Bayesian Learning 3.6 Sufficient Statistics 3.7 Sufficient Statistics and the Exponential Family 3.8 Problems of Dimensionality 3.8.1 An Unexpected Problem 3.8.2 Estimating a Covariance Matrix 3.8.3 The Capacity of a Separating Plane 3.8.4 The Problem-Average Error Rate 3.9 Estimating the Error Rate 3.10 Bibliographical and Historical Remarks Problems 4 NONPARAMETRIC TECHNIQUES -

44
44 45 45 47 48 49 50 51 52 52 55 55 57 59 62 66 66 67 69 70 73 76 80 85 85 85 88 88 90 91 91 95

4.1 Introduction 4.2 Density Estimation 4.3 Parzen Windows 4.3.1 General Discussion 4.3.2 Convergence of the Mean 4.3.3 Convergence of the Variance 4.3.4 Two Examples 4.4 A>Nearest Neighbor Estimation

= 4.5 Estimation of A Posteriori Probabilities 4.6 The Nearest-Neighbor Rule 4.6.1 General Considerations 4.6.2 Convergence of the Nearest-Neighbor 4.6.3 Error Rate for the Nearest-Neighbor Rule 4.6.4 Error Bounds 4.7 The fc-Nearest-Neighbor Rule 4.8 Approximations by Series Expansions 4.9 Approximations for the Binary Case 4.9.1 The Rademacher-Walsh Expansion 4.9.2 The Bahadur-Lazarsfeld Expansion 4.9.3 The Chow Expansion 4.10 Fisher's Linear Discriminant 4.11 Multiple Discriminant Analysis 4.12 Bibliographical and Historical Remarks Problems

CONTENTS

xiii 97 98 98 99 100 101 103 105 108 108 111 113 114 118 121 126

5 LINEAR DISCRIMINANT FUNCTIONS

5.1 Introduction 5.2 Linear Discriminant Functions and Decision Surfaces 5.2.1 The Two-Category Case 5.2.2 The Multicategory Case 5.3 Generalized Linear Discriminant Functions 5.4 The Two-Category Linearly-Separable Case 5.4.1 Geometry and Terminology 5.4.2 Gradient Descent Procedures 5.5 Minimizing the Perceptron Criterion Function 5.5.1 The Perceptron Criterion Function 5.5.2 Convergence Proof for Single-Sample Correction 5.5.3 Some Direct Generalizations 5.6 Relaxation Procedures 5.6.1 The Descent Algorithm 5.6.2 Convergence Proof 5.7 Nonseparable Behavior 5.8 Minimum Squared Error Procedures 5.8.1 Minimum Squared Error and the Pseudoinverse 5.8.2 Relation to Fisher's Linear Discriminant 5.8.3 Asymptotic Approximation to an Optimal Discriminant 5.8.4 The Widrow-HofT Procedure 5.8.5 Stochastic Approximation Methods .

130
130 131 131 132 134 138 138 140 141 141 142 146 147 147 148 149 151 151 152 154 155 156

xiv

CONTENTS 159 159 161 163 163 166 166 167 168 169 172 174 174 176 177 179 186

5.9 The Ho-Kashyap Procedures 5.9.1 The Descent Procedure 5.9.2 Convergence Proof 5.9.3 Nonseparable Behavior 5.9.4 Some Related Procedures 5.10 Linear Programming Procedures 5.10.1 Linear Programming 5.10.2 The Linearly Separable Case 5.10.3 Minimizing the Perceptron Criterion Function 5.10.4 Remarks 5.11 The Method of Potential Functions 5.12 Multicategory Generalizations 5.12.1 Kesler's Construction 5.12.2 The Fixed-Increment Rule 5.12.3 Generalization for MSE Procedures 5.13 Bibliographical and Historical Remarks Problems

6 UNSUPERVISED LEARNING AND CLUSTERING

Introduction Mixture Densities and Identifiability Maximum Likelihood Estimates Application to Normal Mixtures 6.4.1 Case 1: Unknown Mean Vectors 6.4.2 An Example 6.4.3 Case 2: All Parameters Unknown 6.4.4 A Simple Approximate Procedure 6.5 Unsupervised Bayesian Learning 6.5.1 The Bayes Classifier 6.5.2 Learning the Parameter Vector 6.5.3 An Example 6.5.4 Decision-Directed Approximations 6.6 Data Description and Clustering 6.7 Similarity Measures 6.8 Criterion Functions for Clustering 6.8.1 The Sum-of-Squared-Error Criterion 6.8.2 Related Minimum Variance Criteria 6.8.3 Scattering Criteria 6.8.3.1 The Scatter Matrices 6.1 6.2 6.3 6.4

189
189 190 192 193 194 195 198 201 203 203 204 207 210 211 213 217 217 219 221 221

CONTENTS 6.8.3.2 The Trace Criterion 6.8.3.3 The Determinant Criterion 6.8.3.4 Invariant Criteria Iterative Optimization Hierarchical Clustering 6.10.1 Definitions 6.10.2 Agglomerative Hierarchical Clustering 6.10.2.1 The Nearest-Neighbor Algorithm 6.10.2.2 The Furthest-Neighbor Algorithm 6.10.2.3 Compromises 6.10.3 Stepwise-Optimal Hierarchical Clustering 6.10.4 Hierarchical Clustering and Induced Metrics Graph Theoretic Methods The Problem of Validity Low-Dimensional Representations and Multidimensional Scaling Clustering and Dimensionality Reduction Bibliographical and Historical Remarks Problems

xv 222 222 223 225 228 228 230 233 233 235 235 236 237 239 243 246 248 256

6.9 6.10

6.11 6.12 6.13 6.14 6.15

Part II SCENE ANALYSIS

7 REPRESENTATION AND INITIAL SIMPLIFICATIONS
7.1 7.2 7.3 7.4 7.5 Introduction Representations Spatial Differentiation Spatial Smoothing Template Matching 7.5.1 Template MatchingMetric Interpretation 7.5.2 Template MatchingStatistical Interpretation 7.6 Region Analysis 7.6.1 Basic Concepts 7.6.2 Extensions 7.7 Contour Following 7.8 Bibliographical and Historical Remarks Problems

263

263 264 267 272 276 276 282 284 284 288 290 293 297

xvi

CONTENTS

8 THE SPATIAL FREQUENCY DOMAIN

8.1 8.2 8.3 8.4 8.5 8.6 Introduction The Sampling Theorem Template Matching and the Convolution Theorem Spatial Filtering Mean Square Estimation Bibliographical and Historical Remarks Problems DESCRIPTIONS OF LINE AND SHAPE

298
298 302 305 308 318 322 325 327 327 328 328 332 335 337 339 341 342 345 348 352 356 362 367 372 377 379 379 380 382 386 392 392 393 394 396

9.1 Introduction 9.2 Line 1 Description 9.2.1 Minimum-Squared-Error Line Fitting 9.2.2 Eigenvector Line Fitting 9.2.3 Line Fitting by Clustering 9.2.4 Line Segmentation 9.2.5 Chain Encoding 9.3 Shape! Description 9.3.1 Topological Properties 9.3.2 Linear Properties 9.3.3 Metric Properties 9.3.4 Descriptions Based on Irregularities 9.3.5 The Skeleton of a Figure 9.3.6 Analytic Descriptions of Shape 9.3.7 Integral Geometric Descriptions 9.4 Bibliographical and Historical Remarks Problems 10 PERSPECTIVE TRANSFORMATIONS 10.1 10.2 10.3 10.4 10.5 Introduction Modelling Picture Taking The Perspective Transformation in Homogeneous Coordinates Perspective Transformations With Two Reference Frames Illustrative Applications 10.5.1 Camera Calibration 10.5.2 Object Location 10.5.3 Vertical Lines: Perspective Distortion 10.5.4 Horizontal Lines and Vanishing Points

CONTENTS 10.6 Stereoscopic Perception 10.7 Bibliographical and Historical Remarks Problems

xvii 398 401 404

11 PROJECTTVE INVARIANTS 11.1 11.2 11.3 11.4 11.5 11.6 11.7 Introduction The Cross Ratio Two-Dimensional Projective Coordinates The Inter-Lens Line An Orthogonal Projection Approximation Object Reconstruction Bibliographical and Historical Remarks Problems

405 405 407 411 414 418 421 422 424

12 DESCRIPTIVE METHODS IN SCENE ANALYSIS

12.1 Introduction 12.2 Descriptive Formalisms 12.2.1 Syntactic Descriptions 12.2.2 Relational Graphs 12.3 Three-Dimensional Models 12.4 The Analysis of Polyhedra 12.4.1 Line Semantics 12.4.2 Grouping Regions into Objects 12.4.3 Monocular Determination of Three-Dimensional Structure 12.5 Bibliographical and Historical Remarks Problems

425
425 426 426 434 436 441 442 449 456 462 465

AUTHOR INDEX SUBJECT INDEX

467 472