Professional Documents
Culture Documents
I first read John von Neumann7sbook The Com- Von Neumann recognized that the reliance of
puter and the Brain in the summer of 1970, while the brain on analog-signal processing had far-
studying for the general examination for doctoral reaching significance for the style of computa-
candidacy in physics at Princeton. Ever since then, tion that the brain was capable of supporting. He
I have been thinking about the issues von Neu- pointed out that the logical depth of a calcula-
mann raised in his book. Rereading the book re- tion, for example, can be very great for a digital
cently has highlighted the progress that has been computer that retains high accuracy at each step
made on trying to understand information-pro- in the calculation; but for an analog system like
cessing in the brain, as well as the difficulties the brain, the compounding of errors causes se-
that remain. vere problems after only a few steps. Much of the
When von Neumann wrote the manuscript for work in artificial intelligence depends on the ef-
the Silliman Lectures a t Yale in 1956, the gen- ficient use of a sequential, symbol-processing ar-
eral character of electrical transmission and chitecture, and on tree searches that have great
communication between neurons had just re- logical depth. The model of computation based on
cently been elucidated through the seminal work logic that led to sequential architecture also served
of Alan Hodgkin and Andrew Huxley on the squid as a model for human reasoning in cognitive sci-
giant axon, and Bernard Katz on the frog neu- ence (Newel1 and Simon 1976). The recent avail-
romuscular junction. The all-or-none nature of the ability of digital computers with parallel archi-
action potential had suggested analogies with bi- tectures makes apparent the extent to which
nary gates in digital computers (McCulloch and cognitive science and artificial intelligence have
Pitts 1943), but the analog nature of neural in- been shaped by hardware that is based on se-
tegration was just beginning to be fully appre- quential symbol processing.
ciated. Typically, the accuracy of numerical cal- It is somewhat ironic that the sequential ar-
culation in a modern digital computer is 8 to 16 chitecture of digital computers is generally called
significant figures. But in a neuron, signaling by von Neumann architecture. Von Neumann was
means of the average firing rate has at best one well aware of the need for a broader science of
or two significant figures of accuracy. We still do computation and contributed not just to the de-
not understand how information is represented velopment of sequential architecture but also to
in the brain in such a way that low accuracy is the foundations of cellular automata (von Neu-
not a problem. mann 1963)-an early parallel architecture that
has only recently been exploited (Wolfram 1983). A synapse is activated when an action poten-
It is also apparent from The Computer and the tial-an impulse emanating from a neuron-in-
Brain that von Neumann was skeptical about se- vades a synaptic terminal, a chemical neuro-
quential architecture as a model for how the brain transmitter is released, and a signal is
works. communicated to the postsynaptic cell. The elec-
The use of memory in digital computers to store trical response produced in the postsynaptic cell
both sequences of instructions and data was a can be excitatory or inhibitory; that is, it can bring
breakthrough to which von Neumann made ma- the cell either closer to or farther away from the
jor contributions. Memory is one of the central threshold for initiating an action potential.
themes in his book. Very little was known at the Thousands of synapses converge on a single neu-
time concerning memory mechanisms at the cel- ron. The process of integrating information, of
lular level or the locations where memories are collecting signals from thousands of sources, in-
stored in the brain. "We are as ignorant," he volves filtering processes and nonlinear decisions
stated, "of its nature and position as were the that we are just beginning to understand at the
Greeks, who suspected the location of the mind molecular level. Von Neumann made many nu-
in the diaphragm." Today we have a much better merical estimates of memory capacity, neural
knowledge of the condition for neural plasticity power dissipation, and time scales in his book,
in many different areas of the brain. In the hip- but, curiously, he did not estimate the number of
pocampus, for example, a form of plasticity called synapses in the brain. Since it would be physi-
long-term potentiation has been found that re- cally impossible to count every synapse in the
sults in changes that can last for many days. brain, estimates are based on sampling tech-
Moreover, the plasticity in some parts of the hip- niques. When I first looked in the literature for
pocampus seems to depend on a learning rule first this number in 1970, I found estimates of around
proposed by Hebb (1949), and molecular mecha- 1013synapses, but this now appears to be an un-
nisms are being identified that control this plas- derestimate. The best current estimate is 1014
ticity (Brown et al. 1989). However, the organi- synapses, and the number may continue to change
zation of knowledge in networks of neurons is still as automated anatomical methods continue to
a mystery (Sejnowski and Tesauro 1989). improve. If we assume that synapses are sites of
Information in digital computers is stored at information storage, then we can make a rough
locations that can be individually addressed. In estimate for the total information stored in the
humans, information is organized in a complex brain. A single synapse can store only a few bits
web of structured associations, so that memory of information in the form of a coupling strength.
is accessed through content. Von Neumann cal- Thus, a rough estimate for the information stored
culated an upper bound for how much informa- in our brain is around 1014bits. Compared to the
tion could be stored in the brain. Assuming that estimate of 10' bits of information that is con-
a trace of all information impinging on our sen- sciously available to us, our nondeclarative mem-
sory receptors is stored, he arrived at a capacity ory could be larger by as much as lo5.
of about loz0bits, which is probably a vast over- One of the most difficult problems facing us is
estimate of what we actually retain. A lower that of trying to understand thought processes
bound on memory capacity, based on how much through understanding the brain (Sejnowski and
information can be consciously recalled, is 10' bits Churchland 1989). Here too, von Neumann stated
(Landauer 1986). This is, however, only one type a very clear position: "Thus, logics and mathe-
of memory, the declarative memory that we have matics in the central nervous system, when viewed
for facts and events (Squire 1987). Other memory as languages, must structurally be essentially
systems, such as motor learning and procedural different from those languages to which our com-
knowledge, appear to be organized separately from mon experience refers." This statement reflects a
the memory system for facts. It is more difficult lifetime of thinking about computation and
to quantify the capacity of nondeclarative mem- mathematics. It was perhaps his last thought on
ory sytems because they are not easy to dissect this issue and was in many ways deeply prophet-
into components. One approach to this problem ic of current research on brain models as the sub-
is to estimate the total amount of information that strate for cognition (Churchland 1986; Rumel-
can be stored a t synapses, the specialized con- hart and McClelland 1986). We are still very far
tacts between neurons that are used for signal- from knowing what the hidden language of the
ing. central nervous system might be. Given what we
in wafer-scale integration has the potential to book will also be ours: "However, the above re-
produce spectacular results (Mead 1988). marks about reliability and logical and arith-
It is apparent in Figure 1that we are also far metical depth prove that whatever the system is,
from performing simulations on a scale large it cannot fail to differ considerably from what we
enough to test our ideas of how the brain over- consciously and explicitly consider mathemat-
comes problems of low accuracy and shallow log- ics."
ical depth. Some progress has already been made
using simplified models of neural networks which
can be used to explore issues, such as how infor- Acknowledgments
mation about single items or relationships can be
represented in a distributed fashion over many I am grateful for many stimulating conversations
synapses and neurons (Hinton and Anderson 1981; with Drs. Patricia Churchland, Francis Crick, and
Rumelhart and McClelland 1986; Hopfield and Christof Koch.
Tank 1986). These models demonstrate that dif-
ficult problems can be solved with relatively sim-
ple network architectures, and that the perfor-
mance of a network model is surprisingly immune References
to damage and noise. However, they do not prove
that the brain solves these problems in the same Brown, T. H., A. H. Ganong, E. W. Kairiss, C. L.
way; the next step is to scale up these simula- Keenan, and S. R. Kelso. 1989. "Long-term Poten-
tions to more complex networks with more real- tiation in Two Synaptic Systems of the Hippocampal
istic assumptions. It is very likely that we are Brain Slice." I n ' J . H. Bryne and W. 0. Berry (eds.),
still missing a crucial insight into the informa- Neural Models of Plasticity, New York, Academic
Press, in press.
tion codes used by the brain to represent abstrac- Churchland, P. S. 1986. Neurophilosophy: Toward a
tions and symbolic relationships. There are al- Unified Science of the MindlBrain, Cambridge,
ready hints that spatially and temporally coherent Massachusetts, The MIT Press.
bursts of action potentials at high frequencies may Crick, F. 1984. "Function of the Thalamic Reticular
carry such a code (Brown et al. 1989; Crick 1984; Complex: The Searchlight Hypothesis." Proceedings
von der Malsburg 1987; Sejnowski 1976). Von of the National Academy of Sciences U S A 81, pp.
Neumann called this the problem of the "short 4586-4590.
code"; this is the problem of finding a represen- Hebb, D. 0. 1949. Organization of Behavior, New York,
tation in the brain sufficiently powerful to allow John Wiley & Sons.
the brain to imitate the behavior of another com- Hinton, G. E., and J. A. Anderson. 1981. Parallel Models
puting system. of Associative Memory, Hillsdale, New Jersey,
Lawrence Erlbaum Associates.
We owe much to von Neumann for the math- Hopfield, J. J., and D. W. Tank. 1986. "Computing with
ematical and computational tools that we now Neural Circuits: A Model." Science 233, pp. 625-633.
bring to bear on problems of the mind and brain. Landauer, T. K. 1986. "How Much Do People Remem-
The next few decades should prove to be very ex- ber? Some Estimates of the Quantity of Learned In-
citing ones for computational neuroscience. By formation in Long-Term Memory." Cognitive Science
2010 we should have enough computing power to 10, pp. 477-493.
simulate large brain systems, and many of the McCulloch, W. S., and W. Pitts. 1943. "A Logical Cal-
problems raised by von Neumann in The Com- culus of the Ideas Immanent in Nervous Activity."
puter and the Brain should become amenable to Bulletin of Mathematical Biophysics 5, pp. 115-133.
experimental investigation and modeling stud- Mead, C. 1988. Analog V L S I and Neural Systems,
ies. It is difficult to predict what methods for Reading, Massachusetts, Addison-Wesley.
studying complex systems may be needed before Newell, A,, and H. A. Simon. 1976. "Computer Science
, as Empirical Enquiry." CACM 19, pp. 113-126.
an understanding of cognition is achieved, and
Rumelhart, D. E., and J. L. McClelland. 1986. Parallel
what information will be found with these meth-
Distributed Processing: Explorations in the Micro-
ods. The traditional analytic techniques in math- structure of Cognition, Cambridge, Massachusetts,
ematics may not be sufficiently powerful for the The MIT Press.
exploration of nonlinear brain models. It may even Sejnowski, T. J. 1976. "On Global Properties of Neu-
be possible that all existing traditional tech- ronal Interaction." Biological Cybernetics 22, pp. 85-
niques based on the manipulation of symbols will 95.
be inadequate. The last words in von Neumann's Sejnowski, T. J., C. Koch, and P . S. Churchland. 1988.
"Computational Neuroscience." Science 241, pp. 1299- sis of Brain Organization." In J.-P. Changeux and
1306. M. Konishi (eds.), The Neural and Molecular Bases
Sejnowski, T. J., and P. S. Churchland. 1989. "Brain of Learning, New York, John Wiley & Sons, pp. 411-
and Cognition." In M. I. Posner (ed.), Foundations of 432.
Cognitive Science, Cambridge, Massachusetts, The Von Neumann, J. 1958. The Computer and the Brain,
MIT Press, in press. New Haven, Yale University Press.
Sejnowski, T. J., and G. Tesauro. 1989. "Building Net- Von Neumann, J. 1963. "The General and Logical
work Learning Algorithms from Hebbian Synapses." Theory of Automata." In A. H. Taub (ed.), John uon
In J. L. McGaugh, N. M. Weinberger, and G. Lynch Neumann: Collected Works, New York, Pergamon
(eds.), Brain Organization and Memory: Cells, Sys- Press, pp. 5, 288.
tems, and Circuits, New York, Oxford University Waltz, D. H. 1988. "The Prospects for Building Truly
Press, in press. Intelligent Machines." Daedalus 1, pp. 191-212.
Squire, L. R. 1987. Memory and the Brain, New York, Wolfram, S. 1983. "Statistical Mechanics of Cellular
Oxford University Press. Automata." Reviews of Modern Physics 55, pp. 601-
Von der Malsburg, C. 1987. "Synaptic Plasticity as Ba- 604.