Contents

Abbreviations and Symbols xvi i

1 Introduction 1

1.1 What Is a Neural Network? 1

1.2 Structural Levels of Organization in the Brain 6
1.3 Models of a Neuron 8
1.4 Neural Networks Viewed as Directed Graphs 13
1.5 Feedback 15
1.6 Network Architectures 18
1.7 Knowledge Representation 22
1.8 Visualizing Processes in Neural Networks 31
1.9 Artificial Intelligence and Neural Networks 32
1.10 Historical Notes 36
Problems 41

2 Learning Process 45

2.1 Introduction 45
2.2 Error-Correction Learning 47
2.3 Hebbian Learning 49
2.4 Competitive Learning 53
2.5 Boltzmann Learning 55
2.6 The Credit-Assignment Problem 56
2.7 Supervised Learning 57
ix
x Contents

2.8 Reinforcement Learning 59

2.9 Unsupervised Learning 65
2.10 Learning Tasks 66
2.11 Adaptation and Learning 69
2.12 Statistical Nature of the Learning Process 71
2.13 Learning Theory 75
2.14 Discussion 85
Problems 87

3 Correlation Matrix Memory 90

3.1 Introduction 90
3.2 Distributed Memory Mapping 92
3.3 Correlation Matrix Memory 95
3.4 Error Correction Applied to a Correlation Matrix Memory 100
3.5 Discussion 102
Problems 104

4 The Perceptron 106

4.1 Introduction 106

4.2 Basic Considerations 107
4.3 The Perceptron Convergence Theorem 108
4.4 Performance Measure 113
4.5 Maximum-Likelihood Gaussian Classifier 115
4.6 Discussion 118
Problems 120

5 Least-Mean-Square Algorithm 121

5.1 Introduction 121

5.2 Wiener-Hopf Equations 122
5.3 Method of Steepest Descent 124
 Contents xi

5.4 Least-Mean-Square Algorithm 126

5.5 Convergence Considerations of the LMS Algorithm 129
5.6 Learning Curve 131
5.7 Learning-Rate Annealing Schedules 132
5.8 Adaline 134
5.9 Discussion 135
Problems 136

6 Multilayer Perceptrons 138

6.1 Introduction 138

6.2 Some Preliminaries 140
6.3 Derivation of the Back-Propagation Algorithm 142
6.4 Summary of the Back-Propagation Algorithm 153
6.5 Initialization 156
6.6 The XOR Problem 157
6.7 Some Hints for Making the Back-Propagation Algorithm
Perform Better 160
6.8 Output Representation and Decision Rule 162
6.9 Computer Experiment 165
6.10 Generalization 176
6.11 Cross-Validation 179
6.12 Approximations of Functions 181
6.13 Back-Propagation and Differentiation 185
6.14 Virtues and Limitations of Back-Propagation Learning 186
6.15 Accelerated Convergence of Back-Propagation Through
Learning-Rate Adaptation 192
6.16 Fuzzy Control of Back-Propagation Learning 201
6.17 Network-Pruning Techniques 205
6.18 Supervised Learning Viewed as a Nonlinear Identification
Problem 212
6.19 Supervised Learning as a Function Optimization Problem 215
6.20 Supervised Learning of Probability Distributions by
Multilayer Perceptrons 217
6.21 Discussion 220
xii Contents

6.22 Applications 22 1
Problems 229

7 Radial-Basis Function Networks 236

7.1 Introduction 236

7.2 Cover’s Theorem on the Separability of Patterns 237
7.3 Interpolation Problem 242
7.4 Supervised Learning as an Ill-Posed Hypersurface
Reconstruction Problem 244
7.5 Regularization Theory 245
7.6 Regularization Networks 255
7.7 Generalized Radial-Basis Function Networks 256
7.8 The XOR Problem (Revisited) 260
7.9 Comparison of RBF Networks and Multilayer Perceptrons 262
7.10 Mixture Models 263
7.11 Learning Strategies 264
7.12 Computer Experiment 268
7.13 Factorizable Radial-Basis Functions 269
7.14 Discussion 272
7.15 Applications 274
Problems 28 1

8 Recurrent Networks Rooted in Statistical

Physics 285

8.1 Introduction 285

8.2 Dynamical Considerations 287
8.3 The Hopfield Network 289
8.4 Computer Experiment I 296
8.5 Energy Function 300
8.6 Error Performance of the Hopfield Network 303
8.7 Isomorphism Between a Hopfield Network and a Spin-
Glass Model 308
 Contents xiii

8.8 Stochastic Neurons 309

8.9 Phase Diagram of the Hopfield Network, and Related
Properties 311
8.10 Simulated Annealing 3 14
8.11 The Boltzmann Machine 318
8.12 A Markov Chain Model of the Boltzmann Machine 330
8.13 The Mean-Field-Theory Machine 335
8.14 Computer Experiments I1 341
8.15 Discussion 344
Problems 346

9 Self-organizing Systems I: Hebbian

Learning 352

9.1 Introduction 352

9.2 Some Intuitive Principles of Self-organization 353
9.3 Self-organized Feature Analysis 354
9.4 Discussion 362
9.5 Principal Components Analysis 363
9.6 A Linear Neuron Model as a Maximum Eigenfilter 370
9.7 Self-organized Principal Components Analysis 374
9.8 Adaptive Principal Components Analysis Using Lateral
Inhibition 382
9.9 Two Classes of PCA Algorithms 39 1
9.10 How Useful Is Principal Components Analysis? 392
Problems 394

10 Self-organizing Systems 11: Competitive

Learning 397

10.1 Introduction 397

10.2 Computational Maps in the Cerebral Cortex 398
10.3 Two Basic Feature-Mapping Models 400
10.4 Modification of Stimulus by Lateral Feedback 402

,
xiv Contents

10.5 Self-organizing Feature-Mapping Algorithm 408

10.6 Properties of the SOFM Algorithm 414
10.7 Reformulation of the Topological Neighborhood 422
10.8 Adaptive Pattern Classification 424
10.9 Learning Vector Quantization 427
10.10 Applications 430
10.11 Discussion 434
Problems 434

11 Self-organizing Systems II I: Information-

Theoretic Models 444

11.1 Introduction 444

11.2 Shannon’s Information Theory 445
11.3 The Principle of Maximum Information Preservation 452
11.4 Generation of Topologically Ordered Maps 458
11.5 Discussion 46 1
11.6 Spatially Coherent Features 464
11.7 Another Information-Theoretic Model of the Perceptual
System 469
11.8 Concluding Remarks 47 1
Problems 47 1

12 Modular Networks 473

12.1 Introduction 473

12.2 Basic Notions of Modularity 475
12.3 Associative Gaussian Mixture Model 478
12.4 Stochastic-Gradient Learning Algorithm 482
12.5 Hierarchical Structure of Adaptive Expert Networks 487
12.6 Piecewise Control Using Modular Networks 49 1
 Contents xv

12.7 Summary and Discussion 493

Problems 495

13 Temporal Processing 498

13.1 Introduction 498

13.2 Spatio-Temporal Models of a Neuron 501
13.3 FIR Multilayer Perceptron 506
13.4 Temporal Back-Propagation Learning 5 10
13.5 Temporal Back-Propagation with Adaptive Time Delays 518
13.6 Back-Propagation Through Time 520
13.7 Real-Time Recurrent Networks 521
13.8 Real-Time Nonlinear Adaptive Prediction of Nonstationary
Signals 526
13.9 Partially Recurrent Network 531
13.10 Discussion 532
Problems 533

14 Neurodynamics 537

14.1 Introduction 537

14.2 Dynamical Systems 539
14.3 Stability of Equilibrium States 542
14.4 Attractors 548
14.5 Strange Attractors and Chaos 550
14.6 Neurodynamical Models 551
14.7 Manipulation of Attractors as a Recurrent Network
Paradigm 555
14.8 Dynamics of Hopfield Models 556
14.9 The Cohen-Grossberg Theorem 56 1
14.10 The Hopfield Model as a Content-Addressable Memory 563
14.11 Brain-State-in-a-Box Model 57 1
14.12 Recurrent Back-Propagation 577
14.13 Discussion 585
Problems 589
xvi Contents

15 VLSl Implementations of Neural

Networks 593

15.1 Introduction 593

15.2 Major Design Considerations 594
15.3 Categories of VLSI Implementations 596
15.4 Neurocomputing Hardware 602
15.5 Concluding Remarks 612
Problems 615

Appendix A Pseudoinverse Matrix Memory 617

Appendix B A General Tool for Convergence
Analysis of Stochastic
Approximation Algor it hms 619
Appendix C Statistical Thermodynamics 625
Appendix D Fokker-Planck Equation 628
Bibliography 635

Index 691