This article appeared in a journal published by Elsevier.

The attached
copy is furnished to the author for internal non-commercial research
and education use, including for instruction at the authors institution
and sharing with colleagues.
Other uses, including reproduction and distribution, or selling or
licensing copies, or posting to personal, institutional or third party
websites are prohibited.
In most cases authors are permitted to post their version of the
article (e.g. in Word or Tex form) to their personal website or
institutional repository. Authors requiring further information
regarding Elsevier’s archiving and manuscript policies are
encouraged to visit:
http://www.elsevier.com/authorsrights
 Author's personal copy

New Ideas in Psychology 33 (2014) 8–20

Contents lists available at ScienceDirect

New Ideas in Psychology

journal homepage: www.elsevier.com/locate/newideapsych

Behavioral science integration: A practical framework of

multi-level converging evidence for behavioral science
theories
Gary L. Brase*
Department of Psychology, Kansas State University, 492 Bluemont Hall, Manhattan, KS 66506, USA

a b s t r a c t

Keywords: Several frameworks exist to help science in organizing known information, connecting
Theoretical framework previously disparate phenomena, promoting understanding, and identifying gaps in
Converging operations knowledge. This paper integrates previous frameworks that have been used in the
Levels of analysis
behavioral sciences to produce a more comprehensive, specific, and complete framework,
Marr
consonant with the process of scientific discovery as based on multiple, independent, and
Tinbergen
Conceptual integration converging lines of evidence. This multi-level convergence framework is designed specif-
ically for the analysis and understanding of human cognitive/behavioral traits, in contrast
to more general frameworks that are designed to be applied relatively widely (such as
across all information processing systems or across all biological organisms). This inte-
grated framework overcomes gaps in prior frameworks, provides a more complete picture
of the interrelationships between various aspects of the behavioral sciences, and can aid in
evaluating theories, both for comparison and identifying gaps in evidentiary support.
Ó 2013 Elsevier Ltd. All rights reserved.

1. A practical framework of multi-level converging is to develop a consistent, comprehensive, and enduring

evidence for theories in the behavioral sciences
What allows some scientific fields to advance seemingly
by leaps and bounds, whereas other sciences progress by
fits and starts or even at times regress? Certainly there can
be many factors, but the particular concern of this paper is
an impediment of progress due to the lack of a consistent,
enduring, and comprehensive framework for organizing
and understanding research findings. Some behavioral
sciences, in particular, have a tendency to cycle through a
parade of zeitgeists (e.g., functionalism, structuralism,
behaviorism, relativism, subjectivism, etc.), each enjoying
only transient popularity. As a result, there is considerable
wasted effort, amnesia for older research results, and a lack
of enduring scientific progress. The objective of this paper
* Tel.: þ1 (785) 532 0609; fax: þ1 (785) 532 5401.
E-mail address: gbrase@ksu.edu.
framework for research in the behavioral sciences.
The starting point for this paper is the principle of sci-
entific realism: the presumption that all the different areas
of science are in fact studying a single, real universe.
Although a few perspectives, such as constructive rela-
tivism, may not accept scientific realism as a basic fact, it is
otherwise almost universally acknowledged – particularly
among researchersdand a foundation for certain tenants of
the scientific method (e.g., testability). Scientific realism
has implications for how different areas of science are
related to each other. First, it is required that no sciences
directly conflict with one another. Given a single universe,
there cannot be two correct scientific explanations that
fundamentally contradict each other. Second, scientific re-
alism further requires that different bodies of knowledge
must be consistent with one another if they are over-
lapping. It is acceptable for two fields of knowledge to
merely be non-contradictory if they have no overlap (e.g.,
quantum physics and social psychology quite possibly have

0732-118X/$ – see front matter Ó 2013 Elsevier Ltd. All rights reserved.
http://dx.doi.org/10.1016/j.newideapsych.2013.11.001
 Author's personal copy
G.L. Brase / New Ideas in Psychology 33 (2014) 8–20
no overlap, hence there are no contradictions and no op-
portunity to ask about consistencies). When two areas do
overlap, however, the shared knowledge and understand-
ing of the world must be consistent.1
In many areas of science, this principle of fundamental
consistency has been effectively and powerfully realized.
The physical sciences such as chemistry, physics, geology,
and astronomy are organized and integrated relative to
each other, yielding tremendous advances in knowledge
and coherence. Information from any one field provides
guidance (and boundaries) for any other field in terms of
what is likely, possible, and impossible. The social sciences
have had more difficulty reaching a substantial and sus-
tained integration, and this has hampered advances in
knowledge and coherence.
This is not to say that no integration has occurred within
the social and behavioral sciences. The cognitive science
approach represents a fairly successful integration of spe-
cific fields (primarily computer science, neuroscience, and
cognitive psychology). Similarly, the evolutionary psy-
chology approach has tried to integrate a slightly different
set of fields (primarily evolutionary biology, psychology,
and anthropology; Pinker, 1997). Both these approaches
have so far experienced both unfulfilled potential and
stalled progress for various reasons. This paper suggests
that part of the difficulties these approaches have experi-
enced has to do with the particular frameworks and fields
included in each approach. Perhaps somewhat paradoxi-
cally, these ambitiously integrative approaches were in
certain respects too narrow. An integration of the behav-
ioral sciences, following the principle of scientific realism,
needs to include more than three or four fields.
How should we conceptually organize the many
research disciplines that make up the behavioral sciences,
though? It is, from one perspective, an ill-defined problem
(Simon, 1973): A problem with ambiguities in the initial
starting state, the permissible operations relevant to that
problem, and the desired goal state. Most real-world
problems are ill-defined to some degree (Simon, 1973).
(This is in contrast to artificial problems, such as most
invented games, in which the starting states, permissible
moves, and goal state are clearly defined in the rules of that
game.) So we need some sort of structure to organize ef-
forts at integrating the behavioral sciences and achieving
consistency. Such structures and frameworks have been
proposed, in various fields and at various times, to organize
findings and even organize fields relative to each other. The
following sections review a number of these and note their
individual strengths and weaknesses. Two overarching
themes can be discerned from these organizing frame-
works: a) principles of converging operations, and b)
multiple, complementary levels of analysis. Interestingly,
1
These implications of scientific realism lead to several topics and is-
sues within philosophy. Some of these are consistent with the present
paper (e.g., unity of science, consilience, and inference to the best
explanation; Whewell, 1847/1967, see also a more expansive treatment by
Wilson, 1998), whereas others are less clearly supported or not endorsed
at all (e.g., reducibility of scientific fields). The focus of this article,
however, is on developing a useful working framework for research; not
delving into these philosophical issues.
9
these two themes have developed largely along parallel
tracks and have not been particularly well integrated with
each other. Thus, after reviewing these past frameworks,
we will construct a framework that integrates these two
major approaches to organizing and evaluating scientific
knowledge. This multi-level model of converging opera-
tions seeks both to retain the strengths of prior frameworks
and to leverage aspects of different frameworks to over-
come some of their individual weaknesses.
2. Convergence frameworks
There is a general approach, often learned through
gradual inculcation, that much of scientific progress is
based on multiple, independent, and converging lines of evi-
dence. Theories and hypotheses that repeatedly find sup-
port across various studies, researchers, fields, and
methodologies (which includes surviving attempts at
refutation) are seen as better approximations of the true
state of the world, whereas hypotheses and theories that
fail to find support across one or more of these lines are
seen as inferior and eventually discarded. One way to think
about this approach is that it parallels the narrower concept
of convergent validity in research methods, in which a
proposed measure of a construct is validated by finding
that it correlates with other known measures of that same
construct. (One can also conceptualize this process as a
Bayesian model for scientific reasoning; e.g., Howson &
Urbach, 2006.)
The idea of “converging operations” in psychology-
dusing multiple studies to triangulate and validate a
particular research findingdtraces back to the perception
work of Garner (1954) (Garner, Hake, & Eriksen, 1956; see
also the philosophical work on consilience by Whewell,
1847/1967). Garner was clear from the onset:
Validation can be obtained by using converging opera-
tions to arrive at a single construct or concept. If two or
more independent sets of data, involving basically
different indicators of the nature of the sensory process,
lead to the same sensory scale, then we have a form of
validation. Such validation is probably the only mean-
ingful kind in this and in other areas of psychology. All
valid concepts are formed from independent observa-
tions and operations which allow convergence to the
single concept, although most techniques of validation
used in psychology (with the obvious exception of factor
analysis) do not make this process obvious. (Garner,
1954, p. 223)
This has been expanded, as foreseen by Garner, to a
more general approach. The rationale for this generalized
version of the principle of converging operations was
summarized by Sternberg and Grigorenko (2001) (see also
Sternberg, Grigorenko, & Kalmar, 2001) in a powerful ap-
peal for its use:
The basic idea is that any one operation is, in all likeli-
hood, inadequate for the comprehensive study of any
psychological phenomenon. The reason is that any
methodology introduces biases of one kind or another,
often of multiple kids. By using multiple converging
 Author's personal copy

10 G.L. Brase / New Ideas in Psychology 33 (2014) 8–20

methodologies (i.e. converging operations) for the study lines of evidence can be formalized as evidential “vectors”
of a single psychological phenomenon or problem, one that converge (or fail to converge) on a hypothesized
averages over sources of bias. (Sternberg & Grigorenko, adaptation (Fig. 1).
2001, p.1071) By adopting a broad, cross-disciplinary perspective, the
Schmitt and Pilcher framework provides a structure for
Unfortunately, whereas the criteria of multiple, inde-
organizing research findings that is both scientifically and
pendent, and converging lines of evidence is widely held as
intuitively attractive. In particular, this framework stresses
an ideal for research, actual research programs and reviews
the scientific realism principle that findings across different
of research areas seem to be much less well acquainted
areas of science should be mutually consistent with one
with this ideal. Sternberg and Grigorenko (2001) suggested
another, given that they are all aspects of the same unitary
that some reasons for this lack of use include narrow
real world.
training of researchers in a single methodology and norms
Another notable aspect of the Schmitt and Pilcher
within individual fields that discourage the use of tools and
framework is that it focuses specifically on the human
ideas from outside that particular area of psychology.
behavioral sciences. Although the behavioral sciences and
However, there are probably even more reasons for failures
physical sciences should certainly be non-conflicting, there
to use multiple, independent, and converging lines of evi-
are relatively few overlapping areas in which actual con-
dence in actual research. Indeed, Sternberg and Grigorenko
sistency can be an issue. The focus of this framework is to
allude to a further issue in noting that many psychological
be broadly inclusive of information from evolutionary
phenomena can be enriched by the ideas of other disci-
theory, biology, computer modeling, psychology, medicine,
plines, such as biology, anthropology, neuroscience, and so
anatomy, neurosciences, genetics, and anthropology. The
forth (p. 1073). This even broader construal of converging
behavioral science fields included of this framework are
evidence connects directly with the present paper’s start-
those that are more overlapping and therefore constitute
ing principle derived from scientific realism; that the sci-
bodies of knowledge that have a greater need for consis-
ences must be mutually consistent.
tency. A core goal of the framework is that these lines of
Put very simply, using multiple, independent, and
evidence all can be used to converge on the level of support
converging lines of evidence in actual research practice is
for a particular human behavior or trait. In sum, Schmitt
hard. A researcher (or research group) has to be cross-
and Pilcher provide a framework of specific, multiple lines
trained, cross-disciplinary, and resistant to comfortable
of evidence that is both easy to understand and concrete
simplicities of norms and panaceas. Without Dumbledorian
enough to immediately use as a research tool.
wisdom, it is often difficult to choose the hard but correct
The Schmitt and Pilcher framework contains clearly
option over the easier and more comfortable one. Thus,
independent, multiple, and converging lines of evidence,
anything that makes the hard work of multiple, indepen-
but these lines are actually representing entirely different
dent, and converging lines of evidence easier should be
fields. Gardner’s view (and Sternberg and Grigorenko’s
welcomed.
view) stressed more the idea of converging operations
With an emerging consensus on its value, we need to
within a field. Converging operations can certainly come
more clearly and explicitly delineate what constitutes
from within a single field, but converging operations work
“multiple, independent, and converging lines of evidence”.
even better when they include lines of evidence that come
Does the exact same experiment, conducted by different
from across different fields. In this way, the Schmitt and
people at different places qualify as two distinct lines of
Pilcher framework stresses the integrated causal structure
evidence? Do two very different experiments, but con-
of science and the principle of scientific realism; that
ducted by the same person in the same place, qualify as two
findings across different areas of science should be mutu-
distinct lines of evidence? Can lines of evidence come from
ally consistent, given that they are all aspects of the same
the same field or should (must?) different lines of evidence
unitary truth about the structure of the real world (also see
be demonstrated across different fields? Are these alter-
Cosmides, Tooby, & Barkow, 1992).
native procedures for generating lines of evidence equal in
Obliquely orthogonal pieces of evidence (coming from
weight? In short, how different should multiple lines of
different areas of science) are particularly powerful in that
evidence be in order to be considered truly independent
they are clearly additive, if not sometimes multiplicative.
lines that can therefore converge with either full additive or
Again, this does not exclude the existence of independent,
multiplicative strength? Such clarifications may well help
multiple, and converging lines of evidence coming from
advance the ideas of multiple, independent, and
within a single field of inquiry, but such pieces of evidence
converging lines of evidence firmly into day-to-day
will tend to be more strongly overlapping rather than
research programs.
completely additive (producing smaller increments of
converging support). Garner et al. (1956) recognized this
2.1. Schmitt and Pilcher’s multiple lines of evidence early on:
Ideally, the converging operations would be orthogonal
Schmitt and Pilcher (2004) have done a tremendous
(completely independent) since such operations are the
service by describing a framework with more specific
most efficient. In practice, however, it is difficult to
guidance about what can constitute the multiple, inde-
obtain truly orthogonal operations, because the world is
pendent, and converging lines of evidence so fundamental
so organized that all variable cannot be controlled
to scientific progress. Although not framed in the context of
completely independently. This fact does not seriously
Garner’s ideas, they propose that the strongest converging
 Author's personal copy

G.L. Brase / New Ideas in Psychology 33 (2014) 8–20 11

Fig. 1. Schmitt and Pilcher (2004) framework (p. 645; reprinted by permission).

change the nature of the problem, because a sufficient levels include its function (what it accomplishes in an
number of partially converging operations can still ultimate, evolutionary sense); its real-time operation
provide precise delimitation of alternative concepts. (p. (how it works proximately, from moment to moment);
151) how it is implemented in neural tissue; how it develops
in the individual; and how it evolved in the species.
.converging operations are rarely orthogonal, but are
(Pinker, 2002, p. 70)
usually only partially converging. If there are enough
partially converging operations, we can still arrive at a
single, well-defined concept, but more than two such 3. Levels of analysis
operations will be necessary. (p.152)
Aristotle is often credited with one of the first frame-
The lines of evidence in Schmitt and Pilcher’s frame-
works for understanding levels of analysis in scientific ex-
work (outside “boxes” in Fig. 1) are also not entirely rigid.
planations. He proposed both two-fold and four-fold
The development of these eight lines of evidence (as
frameworks for understanding causality in the world
opposed to splitting into a greater number, or lumping into
(Aristotle, (350 B.C.E.); Barnes, 1984). Aristotle’s four causes
a lesser number, of evidentiary lines) was motivated by
are the material cause (the physical materials that a thing is
histories and traditions in science, current patterns of
composed of; i.e., part-whole causation), the formal cause
contemporary research, and pedagogical usefulness
(what a thing is in terms of its definition, i.e., whole-part
(Schmitt, personal communication). Future directions of
causation), the efficient cause (the immediate agents or
research or changes in pedagogical goals could lead to
events that led to the existence of the item), and the final
further adjustments. In summary, the Schmitt and Pilcher
cause (the purpose or instrumental function of the item).
framework is a usefully concrete and concise model for
The last two of these, in particular, are often cast as prox-
converging operations, if not at the level of within-field
imate (efficient) causes and ultimate (final) causes. This
research, at the level of interdisciplinary research and
distinction between ultimate and proximate causes has
integration.
been a strong starting point for many subsequent
Finally, although this framework has tremendous po-
elaborations.
tential, it also notably lacks a key feature of many other
Stanovich (1999, pp. 9–12) summarized the history and
frameworks for theories; it lacks an organization into levels
terminology used in various descriptions of different levels
of analysis. A long history of conceptual frameworks have
of analysis in a cognitive system (these include Anderson,
utilized different levels of analysis:
1990, 1991; Dennett, 1978, 1987; Horgan & Tienson, 1993;
The big thinkers in the sciences of human nature have Levelt, 1995; Marr, 1982; Newell, 1982, 1990; Oaksford &
been adamant that mental life has to be understood at Chater, 1995; Pylyshyn, 1984; Sterelny, 1990). He concisely
several levels of analysis, not just the lowest one. The shows that these multiple frameworks all share a basic
linguist Noam Chomsky, the computational neurosci- dimension along which they parse their particular levels
entist David Marr, and the ethologist Niko Tinbergen (see Table 1).
have independently marked out a set of levels of anal- By comparing these frameworks side-by-side, several
ysis for understanding a faculty of the mind. These things become apparent. First, there clearly appears to be a
 Author's personal copy
12 G.L. Brase / New Ideas in Psychology 33 (2014) 8–20
Table 1
Different levels of analysis, as described by various cognitive theorists (adapted from Table 1.1 in Stanovich, 1999).
Anderson Marr
Rational Level Computational Level
Algorithmic Level Representation and Algorithm
Implementation Level
Biological Level Hardware Implementation
real issue and need for a levels of analysis framework to
help organize research. Second, the topmost and bottom-
most levels enjoy relatively consistent recognition as
distinct from the mid-level area, whereas this middle area
is a bit more unclear in terms of whether it should be one or
two levels. Finally, there are an abundance of names for
these levels, even as they are fundamentally referring to
about the same thing. Stanovich makes very cogent argu-
ments for the labels he adopts. (Note, he did not add to the
number of labels but simply selected the ones that seem
most descriptively apt.) Our integrated, multi-level
convergent framework will use Stanovich’s labels for this
reason.
To more fully discuss the nature, structure, and impli-
cations of multi-level frameworks, this paper will discuss
Marr’s (1982) framework, which has arguably been one of
the most influential and well-known of these approaches.
For the sake of consistency and clarity, though, this paper
will use the Stanovich labels.
3.1. Marr’s levels of analysis framework
Within the cognitive sciences, many researchers use
Marr’s (1982) framework to organize, understand, and
evaluate theories. Marr’s framework divides an information
processing task into three different levels of analysis: the
intentional (originally, computational) level, the algo-
rithmic level, and the biological (originally, hardware/
implementation) level (Fig. 2).
The Intentional Level describes what the purpose is of the
information processing system. In other words, what is the
problem to be solved and why is it a problem? For instance
(using Marr’s original example), the purpose of a cash
register – the problem it solvesdis to keep track of cash
flow. A computer, in the most general sense, has a
computational purpose of storing and processing infor-
mation. The processes entailed in this information man-
agement include the ability to create files, delete files, read
and write to files, and so on. Additionally, each software
program within a computer also has a purpose of its own.
For example, a word processing computer program is
designed to manipulate keyboard characters in various
ways and then send the “formatted” characters to a file or a
printer. It solves a problem of composition and efficient,
high quality printing of documents.
The Algorithmic Level describes: a) the relevant inputs to
the information processing system, b) the computational
transformations that input is subjected to, and c) the
resulting output of the system. For example, a cash register
must have algorithms that take input in the form of nu-
merical information, transform those numbers via some
operations (e.g., addition, subtraction, multiplication, and
Newell Dennett Stanovich
Knowledge Level Intentional Stance Intentional level
Program Symbol Level Design Stance Algorithmic Level
Register Transfer Level
Device Physical Stance Biological Level
division), and then produce an output (e.g., change due, or a
daily tally of receipts). There may be more than one algo-
rithm that is capable of accomplishing a particular
computational goal, but the number of possibilities can be
constrained by a detailed intentional level description. Each
candidate algorithm (or collection of algorithms) will have
particular properties (e.g., how long certain tasks take to
complete, particular circumstances under which the algo-
rithm will fail to solve the problem, and what input re-
quirements exist for the algorithm). These different
properties allow the possibility for experiments that can
determine which algorithms make up a given system.
The Biological Level describes the physical instantiation
of the algorithms. Again, there may be more than one way
to physically realize a given algorithm. For example, a cash
register or a computer could be constructed with gears and
levers, vacuum tubes, or circuit chips.
3.1.1. Benefits of Marr’s framework
Marr’s computational framework delivers a number of
benefits: (1) Each of the three levels is clearly defined as
dealing with a different aspect of the information pro-
cessing situation; (2) It can provide a skeleton onto which
theories may be developed in a clear and orderly way; and
(3) It can reduce the number of viable alternative theories.
The explicit attention given to the intentional- and algo-
rithmic-level descriptions of a theory (including the de-
mand that these levels be fully addressed), can sometimes
reveal that an existing theory is untenable (e.g., with
learnability analysis, Pinker, 1979, 1984).
This framework implicitly incorporates the Aristotelian
ideas of ultimate causes (intentional level) and proximate
causes (algorithmic and biological levels). The levels of
analysis framework also suggests that many arguments
over form, function, and design can be resolved by paying
attention to the level of analysis at which one is working.
(e.g., It can help to place different phenomena and
descriptive information in their proper relationships to one
another). For example, different “perspectives” are often
discussed as competing within psychology; Gray (2007)
describes seven different types of explanations that are
used in psychology (neural, genetic, evolutionary, learning,
social, cultural, and developmental). These different views
are, in large part, divisions based on levels of analysis:
Neuroscience and genetics are focused at the biological
level, cognitive sciences are focused at the algorithmic level
(with learning, social, and cultural explanations focusing on
the inputs into this level), and evolutionary psychology is
focused at the intentional level. This is neither to deny
there are some genuine conflicts between some of these
perspectives, nor is it claiming there are no in-
terconnections between these areas (note the bidirectional
 Author's personal copy
G.L. Brase / New Ideas in Psychology 33 (2014) 8–20
Fig. 2. Marr’s Computational framework, using the labels for different levels suggested by Stanovich.
arrows in Marr’s model), but aspects of these divisions –
and much of the conflictdare unnecessary and potentially
harmful.
3.1.2. Limits of Marr’s Framework
Marr’s computational framework was designed as an
account for information processing systems generally, not
only including the human mind, but also including com-
puters (such as in Artificial Intelligence), and cash registers
(one of Marr’s original examples). As such, this framework
has tremendous breadth of application, but this breadth is
attained at some cost of explanatory power. For one thing,
the recognition of developmental processes, a mode of
explanation in Gray (2007), is missing in Marr’s system.
Because Marr’s system is intended for any information pro-
cessing system, including not just brains/minds but com-
puters and cash registers, there is generally a relative lack of
emphasis on ontogenetic processes and constraints. More
broadly, there are a number of features that can be more
sharply framed (or “constrained,” in a sense), by considering
the specific situation of biological information processing
systems (including and especially, human brains/minds).
Cosmides and Tooby (1995) point out that, in the spe-
cific context of biological systems, evolutionary biology
places informative constraints on what constitute possible
and plausible computational theories. For example, evolu-
tionary biology can provide a more precise definition of
Marr’s “goal” of processing at the intentional level, it brings
into consideration issues of evolvability constraints,
phylogenetic comparisons, and if requisite algorithmic-
level inputs were available over evolutionary time. As
generalizations, when dealing with a biological system that
has evolved via natural selection, Marr’s three levels of
analysis can be viewed as the environmental selection
pressure, the evolved cognitive structures and processes,
and the physically evolved organ, respectively. In that light,
evolutionary biology provides additional constraints from
which computational theories of adaptive information
processing problems can be more effectively built. Simi-
larly, the physical instantiation of the human mind is the
human brain. The fields of neuroanatomy and neuroscience
are dedicated primarily to unraveling biological level
questions, and they thus provide further constraints about
13
what can occur at that level of biological information pro-
cessing systems.
As a final note, it now may be clear why both the present
paper and Stanovich (1999) chose to rename the top and
bottom levels of analysis from Marr’s original labels. When
focused on human information processing, the relevant
“hardware” is biological structures (neurons, etc.); the
realm of possible hardware implementations is incredibly
narrower. At the other end of the levels of analysis, several
people have criticized Marr’s use of “computational level”
(Anderson, 1990; Dennett, 1987) as confusing and
misleading; Dennett’s term (Intentional Level) works bet-
ter as an accurate descriptor.
3.2. Tinbergen’s four questions
The Aristotelian distinction between ultimate and
proximate causes, which can be seen as a progenitor of the
levels of analysis approach taken by Marr and others, is also
a starting point for one of the major conceptual frameworks
within the biological sciences. Mayr (1961, 1982) encour-
aged biologists to conceptualize research questions in terms
of proximate and ultimate (i.e., evolutionary) causality.
Tinbergen (1963) extended this insight further by proposing
that there were four different, but complementary, ways in
which one can address the issue of why a particular animal
has a specific trait (Tinbergen’s “four questions”):
1. One can focus on Adaptive Function by asking what that
trait does for the organism in evolutionary terms. That is,
how has it affected the organism’s functional capabilities
to survive and reproduce?
2. One can focus on Behavioral Ontogeny by asking how the
behavior develops within the individual’s lifetime. That
is, what genetic and environmental (and interactional)
factors guide its development?
3. One can focus on Behavioral Phylogeny by asking what
the evolutionary history of the behavior is. That is, how
does the phylogenetic history of the species help us
understand the trait’s present structure as a reflection of
ancestral features?
4. One can focus on Immediate Causes by asking what im-
mediate mechanisms underlie the expression of the
 Author's personal copy
14 G.L. Brase / New Ideas in Psychology 33 (2014) 8–20
behavior. That is, what are the neural, physiological, and
psychological factors within an individual that control
the behavior?
Each of these can be a valid and useful perspective to
take in answering the general question of where a trait or
behavior came from, but each is – by itselfdincomplete.
Answering this question in all four perspectives provides a
much more complete answer. (Marr’s framework is rarely
utilized within the biological sciences, just as Tinbergen’s
questions are rarely asked in the cognitive sciences.)
As with Marr’s framework, realizing that there are these
multiple ways of addressing the source of a trait also can
avoid debates and disagreements. To help clarify both the
Tinbergen framework and debates arising from research
speaking to different questions, Sherman (1988) restated
Tinbergen’s framework as four “levels of analysis” (see
Fig. 3). This organization more explicitly re-introduced the
proximate/ultimate levels distinction of earlier work, as
well as a distinction between questions primarily focused
on “how” a trait developed (i.e., ontogeny and phylogeny)
and questions primarily focused on “what” caused the trait
to exist (i.e., adaptive function and proximate causation).
3.2.1. Benefits of Tinbergen’s four questions
Tinbergen’s more specific focus on biological systems, as
compared with Marr’s framework that included any infor-
mation processing system, provides some more specific
guidelines for biological computational theories. Most
notably, Tinbergen’s recognition of developmental pro-
cesses, acknowledged in Tinbergen’s system as behavioral
ontogeny, provides a developmental aspect that does not
exist in Marr’s framework. (This also meshes with the
developmental mode of explanation noted by Gray (2007),
but missing in Marr’s system).
Tinbergen’s four questions, especially as organized by
Sherman (1988), also introduces the idea that each level of
analysis may be usefully conceptualized as being multi-
faceted. Tinbergen’s ultimate causality level is not just at a
generic level but rather broken down into adaptive func-
tion and behavioral phylogeny. Tinbergen’s proximate
causality level is divided into behavioral ontogeny and
immediate causes.
Fig. 3. Tinbergen’s four levels of analysis, as organized by Sherman (1988).
3.2.2. Limits of Tinbergen’s four questions
The Tinbergen framework is designed, as befitting an
ethological approach, specifically for biological organisms,
but it is focused heavily on observed behavior rather than on
the information processing that underlies that behavior. The
sociobiological approach of the 1970s ran into difficulties by
attempting to leap from ultimate considerations (i.e., of
adaptive functions and evolutionary history) directly to
proximate measures of human behavior (Symons, 1992).
What was crucially missing was a recognition that the
evolutionary process fundamentally works at the level of
gene frequencies in species populations, and genes do not
produce behaviors directly; they build bodies and brains.
Brains are lynchpins that take in information from the
environment, process it, and produce behavior based on that
processing. The more complex the brain/mind is in a species,
the more integral and involved that brain/mind will be in
mediating the relationships between the environmental
information, evolved mechanisms, and subsequent behav-
iors. Humans, who have very complex and developed brains,
can be expected to have non-arbitrary but quite complicated
relationships between evolved cognitive structures, envi-
ronmental information, and manifest behaviors.
3.3. Summary
Many research efforts in the human behavioral sciences
have used, and continue to use, Tinbergen’s, Marr’s, or
Schmitt and Pilcher’s frameworks with varying degrees of
utility and success. Each framework has been useful and
has enriched the development of scientific knowledge. But
each framework also has inherent strengths and limita-
tions, which tend to be reflected in the body of research
produced based on them. The following section combines
the features of these frameworks to take advantage of the
strengths of each and to minimize their weaknesses, spe-
cifically for the human behavioral sciences.
4. The multi-level convergence framework
It is possible to integrate these different frameworks to
realize the benefits of each and – importantlydfill con-
ceptual gaps in each of them. An integrated framework has
 Author's personal copy
G.L. Brase / New Ideas in Psychology 33 (2014) 8–20
Fig. 4. The multi-level convergence framework, incorporating considerations of Marr, Tinbergen, and Schmitt and Pilcher.
the more delineated multi-level framework of Marr’s
design, and also has the recognition of ontogeny (devel-
opment) and phylogeny (evolutionary history) from Tin-
bergen’s framework. It recaptures the idea of converging
operations (Garner et al., 1956), in the form of converging
lines of evidence. Finally, it also has the interdisciplinary
behavioral sciences approach from Schmitt and Pilcher’s
framework. The result is an interdisciplinary, multi-level,
converging-operations framework for the behavioral sci-
ences. For convenience, I will refer to this as the multi-level
convergence framework.
This multi-level convergence framework is designed
specifically for the analysis and understanding of human
cognitive/behavioral traits, in contrast to more general
frameworks that are designed to be applied relatively
widely (i.e., Marr across all information processing sys-
tems; Tinbergen across all biological organisms). Creating a
framework that is more narrowly focused on the human
behavioral sciences is strategically important. It has been
recurrently realized that specific problems tend to require
specific solutions for efficiency and accuracy (e.g., the frame
problem in artificial intelligence, the poverty of the stimuli
situation in linguistics, the problem of referential ambigu-
ity in semantics, the need for constraints on induction in
cognitive developmental psychology, stimuli under-
determining interpretations in perception, and the prob-
lem of indeterminacy in philosophy; see Tooby & Cosmides,
1992 for a review). Another way to view the importance of
a more narrowly focused framework is that, in terms of
problem solving techniques, it constitutes strong methods
– more powerful techniques that are tailored to the specific
structure of the domain to which they are applieddrather
than weak problem solving methods that must be used in
15
poorly understood domains (Simon, Langley, & Bradshaw,
1981). This same principle can be applied, even at the
level of theoretical frameworks. There are strong benefits
for a framework that is more specific than all things, all
animals, or all information processing systems. At the same
time, there is sufficient interest in the admittedly narrower
issue of human cognition and behavior to make such a
framework very useful.
Fig. 4 shows a high-level description of the multi-level
convergence framework. It includes the multi-level
framework of Marr’s computational design, the recogni-
tion of ontogeny (development) and phylogeny (evolu-
tionary history) from Tinbergen’s framework, and the
explicitly described multiple, independent, and converging
lines of cross-disciplinary evidence from Schmitt and
Pilcher’s framework. Thus, the center column indicates
general levels of analysis (intentional, algorithmic, and
biological levels, following Stanovich’s terminology), and
the outside columns indicate disciplinary lines of evidence
and the levels at which they typically contribute. Tinber-
gen’s “four questions” map onto two different lines of ev-
idence at the intentional level, and the remaining two are
distributed across the five lines of evidence at the algo-
rithmic level. (The comparatively larger number of
evidentiary lines at the algorithmic level may possibly be a
factor contributing to the greater proliferation of labels and
distinctions for this level in Table 1.)
4.1. The intentional level
The Intentional Level, as devised by Marr (1982), de-
scribes what the purpose is of the information processing
system; in other words, what is the problem to be solved
 Author's personal copy
16 G.L. Brase / New Ideas in Psychology 33 (2014) 8–20
and why is this a problem? In contrast to human-built
machines, biological information processing systems have
specific and well-known constraints regarding what the
relevant problems are and why the systems were designed
to solve those problems. The theoretical and phylogenetic
lines of evidence at this level take advantage of these
constraints to provide a more precise definition of Marr’s
“purpose” of processing at the intentional level. These more
specific guidelines for biological computational theories
also constitute, in many ways, an integration of Marr’s
framework with Tinbergen’s ideas of adaptive function and
behavioral phylogeny (i.e., ultimate causality level factors).
Theoretical Lines of Evidence for a trait include informa-
tion about selection pressures that could plausibly have
supported the development of such a trait over evolu-
tionary history, models from evolutionary biology, as well
as other theoretical modeling processes to establish
viability (e.g., cost-benefit analyses, game theory simula-
tions, and computer modeling). For example, some human
altruistic behaviors can be theorized, modeled, and pre-
dicted based on evolutionary biology considerations (e.g.,
Hamilton 1964; Trivers, 1971, 1972).
Phylogenetic Lines of Evidence for a trait include infor-
mation from (non-human) ethology, comparative psy-
chology, primatology, physical anthropology, and
paleontology. In other words, evidence for traits in non-
human species that are analogous human traits (due to
either common ancestry or convergent evolution in the
face of similar ecological circumstances). The logic of this
line of evidence is, for instance, the basis of research on
pharmaceuticals utilizing animal models and non-human
animal research on learning behavior as a means to pro-
vide insight into human learning (for reviews see
Maestripieri & Roney, 2006; Röska-Hardy & Neumann-
Held, 2009).
4.2. Algorithmic level
In Marr’s original framework the Algorithmic Level de-
scribes: a) the relevant inputs to the information processing
system, b) the computational transformations that input is
subjected to, and c) the resulting output of the system.
Adherents to Marr’s framework may note that the focus on
information flow from inputs, through a processor, and
then to outputs is less prominent in this framework than in
his original framework. This is not an issue of decreased
importance but rather a reflection of more details in the
multi-level convergence framework. Input/output se-
quences can and should be considered for each area of
evidence at the algorithmic level (note the text within the
boxes at this level in Fig. 4). The nature of these inputs,
processing, and outputs will tend to be specific to the area
of evidence under consideration. Within each area, slightly
different aspects of the physical, social, intrapsychic, and
cultural environment may be relevant. In this sense, then,
the input-transformation-output aspect of the algorithmic
level has actually been propagated across multiple areas in
this multi-level convergence framework and is just as (if
not more) important.
Cross-Cultural Lines of Evidence for a trait hinge on the
twin issues of universality and variation. The psychological
and physiological bases of a human trait should be reliably
developing in virtually all cultures but at the same time
there can be variations in a trait’s behavioral expression
(e.g., due to ecology-dependent variability and facultative/
conditional adaptations). In other words, cross-cultural
invariants indicate similar inputs to the cognitive and
physiological systems underlying a behavior (yielding
highly consistent outputs), whereas cross-cultural variants
help indicate how the underlying system responds to
different types of input (i.e., different environmental inputs,
fed into the same human brain and cognitive systems, will
result in variable cognitive/behavioral outputs).
Hunter–Gatherer Lines of Evidence can provide an
importantly different line of evidence from general cross-
cultural research. Research with hunter–gatherer societies
provide an approximation of ancestral human environ-
ments, including both information and environments that
are no longer common in industrialized societies and
excluding information and environments that have only
recently become common (i.e., more closely resembling the
nature of the inputs based on which the system was
designed). It is therefore important to specifically look at
hunter–gatherer societies (e.g., via cultural anthropology,
and human behavioral ecology) as a potential line of evi-
dence. Research with hunter–gatherer populations often
provides important information about the Environment of
Evolutionary Adaptation (EEA) of a trait (Crawford 1998;
Hagen 2005).
Psychological Lines of Evidence for a trait include infor-
mation from many sub-disciplines of psychology, utilizing
behavioral tests and surveys such as tests of cognitive
reasoning, judgment and decision-making, memory,
attention, language, emotional reactions, and other psy-
chological research. One way to view this situation is that
psychological research involves the more purposeful
manipulation of inputs into the cognitive system (e.g., by
manipulation of independent variables), albeit under often
more artificial circumstances. The dependent variables in
psychological research thus correspond to the output of the
system at the algorithmic level. Due to its breadth and
depth, psychological evidence can often be a major area of
relevant research.
Developmental Lines of Evidence for a trait includes in-
formation about clear, sequential progressions of abilities
and features across development (in typical environments,
and in the absences of developmental disorders). This can
include consistent patterns of development across cultural
variations, indications of developmental specificity, and
indications of developmental primacy. This separate line of
“Developmental Evidence” is an addition to the lines of
evidence in the Schmitt and Pilcher (2004) model, added to
address the issue of ontogeny so clearly specified in Tin-
bergen’s model. (Tinbergen’s fourth question, of causal
factors at the proximate level, is more broadly distributed
among the other lines of evidence at this level.)
Medical Lines of Evidence for a trait include indications
than some aspects of the trait are important for physical
and/or mental health. Such information includes trait-
relevant changes in physical health (including mortality),
mental health (including mental disorders), and health as it
relates to reproductive functioning (fertility and fecundity).
 Author's personal copy
G.L. Brase / New Ideas in Psychology 33 (2014) 8–20
Medical evidence is often characterized in terms of deficits
and disorders (i.e., faulty instantiations of the trait that
interfere with health and well being).
4.3. Biological level
The biological level in Marr’s framework describes the
physical instantiation of the algorithms. As with the
intentional level, the specific situation of biological infor-
mation processing systems constrains much more sharply
the possibilities at the biological level. Many potential
physical instantiations that are possible for other infor-
mation processing systems (using wood, metal, silicon,
plastic, etc.) are excluded from consideration in the case of
the human brain.
Physiological Lines of Evidence for a trait focus on brain
and behavior research, including physical morphology,
underlying neuroanatomical structures, neurotransmitters,
and hormonal bases. Examples of this type of evidence
include comparative and functional anatomy studies, brain
lesion studies, and research that measures levels of neu-
rotransmitters or hormones.
Genetic Lines of Evidence connect a trait to its genetic
underpinnings. This can include studies of behavioral/
population genetics, molecular genetics, experimental gene
mapping, gene manipulation, and gene replacement
studies. Although genetic research is a fairly recent science
compared to other lines of evidence, it adds a fundamen-
tally distinct line of inquiry (Bazzette, 2008). Genetic evi-
dence can help to show convergence with lines of evidence
such as evolutionary selection pressures (because natural
selection fundamentally works as the level of gene fre-
quencies), phylogenetic research (showing common ge-
netic underpinnings of a trait across species), and medical
research (showing genetic underpinnings of traits that
impact physical and mental health).
5. Advantages of using the multi-level convergence
framework
Is it actually the case that the combination of previous
frameworks has actually led to something conceptually
better, or is it a more complex model without any corre-
sponding gain? This section reviews some of the additional
value that the convergence model obtains, specifically
overcoming weaknesses in each of the earlier frameworks.
Tinbergen’s four questions framework highlights the
importance of evolutionary (ultimate) factors and devel-
opmental (proximate) factors, and this was an important
contribution to the multi-level convergence framework. A
weakness of the Tinbergian approach, particularly in the
details of how it has often been executed, is a tendency to
neglect the diversity of factors that come into play in be-
tween the high-level ultimate questions and the more
physiological proximate questions. Particularly in the case
of humans, one needs to devote significant considerations
to the cross-cultural, hunter–gatherer, psychological,
developmental, medical, physiological, and genetic in-
fluences. In some respects this was a primary obstacle for
the early sociobiological approach, which often relied on
Tinbergen’s model; a lack of consideration of the
17
intermediate steps between genetics (e.g., population gene
frequencies) and overt behaviors (Symons, 1992). The
multi-level convergence framework adopts the important
evolutionary and developmental features of Tinbergen’s
model and also includes the intermediate factors so
important for the human behavioral sciences.
The levels of analysis approach developed by Marr (and
several others) forms the backbone of the multi-level
convergence framework; it provides an essential organi-
zational structure for the various lines of evidence. These
levels of analysis by themselves are quite general, however,
and the more specific lines of evidence within each level
provide important details and informative fingerposts to-
ward different types of research. Of course, these more
specific lines of evidence also reflect the “constraints” of
the evolved, human biological information processing
system (Tooby & Cosmides, 1995), as compared with a more
abstract information processing system. For instance, the
issue of developmental factors is almost entirely invisible
within a traditional Marrian framework.
Schmitt and Pilcher (2004) showed that one can eval-
uate a theory by going systematically through each line of
possible evidence and assessing if evidence within that line
exists or not (or, indeed, if contrary evidence exists). They
suggest that theories can range from “minimal” evidentiary
breadth (one line of evidence), to “moderate” evidentiary
breath (two or three lines of evidence), to “extensive”
breadth (four–five lines of evidence), to “exemplary”
breadth (six or more lines of evidence). Such evaluations
must also take into consideration the depth of support
derived from each line of evidence, however, which in-
cludes the quality, quantity, and diversity of research
constituting that line of evidence (using a similar categor-
ical scale). This provides an organization by which one can
clearly and systematically assess strengths and weaknesses
of theories as descriptions of reality. Although the Schmitt
and Pilcher model intersects wonderfully with an inter-
disciplinary view of Garner’s converging-operations prin-
ciple, this connection had not previously been explored.
By adding the relationships across multiple disciplines,
Schmitt and Pilcher help move toward a system of
“strongly” orthogonal lines of evidence in which there are
few overlapping methodological biases and concerns. One
can even draw this principle out further; lines of evidence
that are not only interdisciplinary but also at different
levels of analysis (intentional, algorithmic, and biological)
are perhaps the strongest forms of converging evidence for
a phenomenon. Thus, it is possible to see a fuller spectrum
of evidentiary strength based on convergence. A strict
replication of a prior result is good for issues of reliability
but does not provide any substantial convergent support.
Replication and extension studies provide convergent
support, but only from only a very acute angle. Deliberately
different methodologies studying the same phenomenon
begin to provide much more meaningful convergent op-
erations, but these should inexorably lead to interdisci-
plinary lines of evidence (and then to evidence across level
of analysis).
A systematic comparison of Schmitt and Pilcher’s (2004)
model with the multi-level convergence framework reveals
that the former included a center of focus (the
 Author's personal copy
18 G.L. Brase / New Ideas in Psychology 33 (2014) 8–20
“hypothesized adaptation”) which does not exist in the
later. This central space for the phenomenon in question
was abandoned for several reasons. First, other models did
not include this feature, and close consideration suggests
that this center label is unnecessary and possibly even
undesirable. A model of a phenomenon does not need a
location for the phenomenon itself – the entire model is the
phenomenon (e.g., nowhere in the Krebs cycle is the name
“Krebs cycle”). Second, it is important to retain the strong
backbone of a levels of analysis approach rather than an
idea of a singular phenomenon. Even as Schmitt and Pilcher
took an explicitly evolutionary approach in their model, it
has been noted that evolutionary thinking is just as sus-
ceptible as other fields to confusions regarding levels of
analysis (Scott-Phillips, Dickins, & West, 2011). Finally, the
adaptationist terminology within that center label of the
Schmitt and Pilcher model could have been distracting to
some readers who could misperceive this as a strictly
“evolutionary” model. Other than a few items within the
“theoretical ‘evidence’” category (Evolutionary Biology
Theories and Theoretical Selection Pressures), there is little
in this model that is necessarily foundational to the central
target being an evolutionary adaptation, as opposed to
being more generally described as a cognitive or behavioral
trait.
Some people may question why it is not sufficient to
simply use the general principle of converging operations
as originally developed by Garner. The principle of
converging operations is taught as an ideal for scientific
research and reasoning (e.g., Stanovich, 2010; Sternberg &
Grigorenko, 2001), and thus one might think this suffi-
cient. The reality, however, is that actual implementation of
converging operations lags alarmingly distant from this
ideal. Twenty-five years after Garner first laid down the
explicit framework for converging operations, Pryor and
Ostrom (1981, p. 630) observed:
.despite the wide acceptance of the converging-
operations research strategy, there are almost no pub-
lished examples of its use in social psychological
experimental research.
This begs the question: Why are converging operations
so infrequency employed if it has such broad awareness,
respect, and advocacy? One possible reason is that the
general idea of converging operations has not been made
more explicit and supplemented with concrete guidance as
to its actual use in research. The present framework pro-
vides just this sort of detailed, specific outline for actual
employment of converging operations. (Also see Sternberg
& Grigorenko, 2001 for additional obstacles for an inte-
grated psychology.)
Lastly, some researchers in specific fields may initially
feel constrained – pigeonholed, evendto one area of evi-
dence. This would be a mistake, though. A researcher who
is focused on one area of evidence as a specialty is in no way
limited from pulling relevant information and knowledge
from other areas. In fact, this framework explicitly en-
courages this process, and as such it actually encourages
more freedom for researchers to take new approaches and
work with new perspectives. More broadly, this is a
framework for the development and advancement of all
behavioral sciences – our understanding of the human
condition.
6. Conclusion
One of the best known frameworks of how scientific
knowledge is developed and evaluated is Popper’s falsifi-
cationist strategy2; the idea that one should attempt to
falsify a proposed scientific hypothesis, and those hypoth-
eses that consistently defy falsification are likely to be true
(Popper 1959). Indeed, this is a basis – for better or wor-
sedfor the standard procedure of null hypothesis signifi-
cance testing (Gigerenzer & Murray 1987; Krueger, 2001;
Loftus, 1996; Newell, 1973; Nickerson, 2000).
It has repeatedly been pointed out, however, that
although the falsifiability criterion of Popper’s approach is
an important aspect of scientific methods, it is not sufficient
(Nickerson, 2000). The approximating approach of Lakatos
is often added to Popper’s ideas. Lakatos argued that the
purpose of science – and hence a characteristic that should
be exhibited in the scientific processdis the closer and
closer approximation of the true state of the world (see
Ketelaar & Ellis 2000 for a recent discussion of the Laka-
tosian approach in the behavioral sciences). The underlying
purposes of both Popper’s and Lakatos’s approach share a
basic tenet of scientific realism. They both hold that that
science is about discovering (or approximating the dis-
covery of) truth in the world, and this assumes that there is
a true state of the world to be discovered, that this true
state can be described and understood, and that current
scientific descriptions of the world are variously distant or
close to that true state.
The multi-level convergence framework represents a
synthesis of both philosophical ideas about how science is
conducted (culminating, in the most general sense, as
multiple, independent, and converging lines of evidence as
a touchstone for truth in reality) and a synthesis of prior
multi-level framework strengths. By recognizing both
multiple lines of evidence and distinct levels of analysis, it
becomes clear that one can assess the criteria of how well
lines of evidence truly are independent and convergent.
Lines of evidence are more independent when they not
only come from different particular sources (researchers,
labs, or methodologies), but also from different fields.
Similarly, the convergence of evidences is more powerful
when the “lines” of each piece of evidence are less acute
angles.
The multi-level convergence framework also recognizes
and takes advantage of some core defining features of the
2
Kuhn (1970) has also been highly influential as a model of how sci-
ence progresses. The idea of understanding science in terms of a suc-
cession of paradigms within any particular field can be useful, but tends
to emphasize the neglect or exclusion of information that conflicts with
the dominant paradigm. In contrast, a focus on scientific realism and the
interrelations between fields of knowledge tends to function, in part, as a
lens that make unfilled aspects more clearly evident (e.g., undiscovered
elements in the periodic table, phylogenetic missing links, and genetic
bases for natural selection). One conciliatory interpretation is that Kuh-
nian paradigms are often reflections of successive emphases on particular
dimensions or aspects of a (potential) larger program of scientific
discovery.
 Author's personal copy
G.L. Brase / New Ideas in Psychology 33 (2014) 8–20
human behavioral sciences. This framework can be used for
better organizing known information, making connections
between fields of the behavioral sciences, developing better
understanding and communication between researchers
(and between teachers and students), and more focused
identification of gaps in knowledge that call for further
research. This framework can also be used as scaffolding
onto which new theories can be developed, and as a
sounding board against which existing theories can be
evaluated (Schmitt & Pilcher, 2004; see Adair, Dillon, & Brase,
in press; Dillon, Adair, & Brase, in press for illustrations of this
implementation).
Acknowledgments
The author thanks Clark Barrett, Sandra Brase, Bruce
Glymour, Todd Schachtman, and Dave Schmitt for valuable
advice and comments on earlier drafts of this work.
References
Adair, L. E., Dillon, H. M., & Brase, G. L. (2013). I’ll have who she’s having:
mate copying, mate poaching, and mate retention. In M. L. Fisher-
MacDonnell (Ed.), Handbook on women and competition. Oxford Uni-
versity Press: New York, NY. in press.
Anderson, J. R. (1990). The adaptive character of thought. Hillsdale, NJ:
Lawrence Erlbaum Associates.
Anderson, J. R. (1991). Is human cognition adaptive? Behavioral and Brain
Sciences, 14, 471–517.
Aristotle (350 B.C.E.). Metaphysics. Trans. by W. D. Ross, available online
at: http://classics.mit.edu/ [downloaded 1 November 2007].
Barnes, J. (1984). The complete works of Aristotle (Vol. 2). Princeton, N.J:
Princeton University Press.
Bazzette, T. J. (2008). An introduction to behavior genetics. Sunderland, MA:
Sinauer Associates.
Cosmides, L., & Tooby, J. (1995). From function to structure: the role of
evolutionary biology and computational theories in cognitive
neuroscience. In M. Gazzaniga (Ed.), The cognitive neurosciences.
Cambridge, MA: MIT Press.
Cosmides, L., Tooby, J., & Barkow, J. (1992). Evolutionary psychology and
conceptual integration. In J. Barkow, L. Cosmides, & J. Tooby (Eds.), The
adapted mind: Evolutionary psychology and the generation of culture
(pp. 3–15). New York: Oxford University Press.
Crawford, C. (1998). Environments and adaptations: then and now. In
C. Crawford, & D. L. Krebs (Eds.), Handbook of evolutionary psychology:
Ideas, issues, and applications (pp. 275–302). Mahwah, NJ: Lawrence
Erlbaum Associates, Inc.
Dennett, D. C. (1978). Brainstorms: Philosophical essays on mind and psy-
chology. Cambridge, MA: MIT Press.
Dennett, D. C. (1987). The intentional stance. Cambridge, MA: MIT Press.
Dillon, H. M., Adair, L. E., & Brase, G. L. (2013). Operational sex ratio and
female competition: females, females everywhere, and not a male to
mate. In M. L. Fisher-MacDonnell (Ed.), Handbook on women and
competition. Oxford University Press: New York, NY. in press.
Garner, W. R. (1954). Context effects and the validity of loudness scales.
Journal of Experimental Psychology, 48, 218–224.
Garner, W. R., Hake, H. W., & Eriksen, C. W. (1956). Operationalism and the
concept of perception. Psychological Review, 63, 149–159.
Gigerenzer, G., & Murray, D. J. (1987). Cognition as intuitive statistics.
Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
Gray, P. (2007). Psychology (5th ed.). New York: Worth Publishers.
Hagen, E. H. (2005). Controversial issues in evolutionary psychology. In
D. M. Buss (Ed.), The handbook of evolutionary psychology (pp. 145–
173). Hoboken, NJ: John Wiley & Sons, Inc.
Hamilton, W. D. (1964). The genetical evolution of social behavior I and II.
Journal of Theoretical Biology, 7, 1–52.
Horgan, T., & Tienson, J. (1993). Levels of description in nonclassical
cognitive science. In C. Hookway, & D. Peterson (Eds.), Philosophy and
cognitive science (pp. 159–188). Cambridge, England: Cambridge
University Press.
Howson, C., & Urbach, P. (2006). Scientific reasoning: The Bayesian
approach. Chicago: Open Court.
19
Ketelaar, T., & Ellis, B. J. (2000). Are evolutionary explanations unfalsifi-
able? Evolutionary psychology and the Lakatosian philosophy of sci-
ence. Psychological Inquiry, 11, 1–21.
Krueger, J. (2001). Null hypothesis significance testing: on the survival of
a flawed method. American Psychologist, 56, 16–26.
Kuhn, T. S. (1970). The structure of scientific revolutions (2nd ed.). Chicago:
University of Chicago Press.
Levelt, W. (1995). Chapters of psychology. In R. L. Solso, & D. W. Massaro
(Eds.), The science of the mind: 2001 and beyond (pp. 184–202). New
York: Oxford University Press.
Loftus, G. R. (1996). Psychology will be a much better science when we
change the way we analyze data. Current Directions in Psychological
Science, 5, 161–171.
Maestripieri, D., & Roney, J. R. (2006). Evolutionary developmental psy-
chology: contributions from comparative research with nonhuman
primates. Developmental Review, 26, 120–137.
Marr, D. (1982). Vision: a computational investigation into the human
representation and processing of visual information. San Francisco:
Freeman.
Mayr, E. (1961). Cause and effect in biology. Science, 134, 1501–1506.
Mayr, E. (1982). The growth of biological thought: Diversity, evolution, and
inheritance. Cambridge, Massachusetts: The Belknap Press of Harvard
University Press.
Newell, A. (1973). You can’t play 20 questions with nature and win:
projective comments on the papers in this symposium. In W. G. Chase
(Ed.), Visual information processing (pp. 283–308). New York: Aca-
demic Press.
Newell, A. (1982). The knowledge level. Artificial Intelligence, 18, 87–127.
Newell, A. (1990). Unified theories of cognition. Cambridge, MA: Harvard
University Press.
Nickerson, R. S. (2000). Null hypothesis significance testing: a review of
an old and continuing controversy. Psychological Methods, 5, 241–301.
Oaksford, M., & Chater, N. (1995). Theories of reasoning and the compu-
tational explanation of everyday inference. Thinking and Reasoning, 1,
121–152.
Pinker, S. (1979). Formal models of language learning. Cognition, 7, 217–
283.
Pinker, S. (1984). Language learnability and language development. Cam-
bridge, MA: Harvard University Press.
Pinker, S. (1997). How the mind works. New York: W. W. Norton &
Company.
Pinker, S. (2002). The blank slate: The modern denial of human nature. New
York: Viking.
Popper, K. R. (1959). The logic of scientific discovery. New York: Hutchison
Education.
Pryor, J. B., & Ostrom, T. M. (1981). The cognitive organization of social
information: a converging-operations approach. Journal of Personality
and Social Psychology, 41, 628–641.
Pylyshyn, Z. (1984). Computation and cognition. Cambridge, MA: MIT Press.
Röska-Hardy, L. S., & Neumann-Held, E. M. (2009). Learning from animals?
Examining the nature of human uniqueness. New York: Psychology
Press.
Schmitt, D. P., & Pilcher, J. J. (2004). Evaluating evidence of psychological
adaptation: how do we know one when we see one? Psychological
Science, 15, 643–649.
Scott-Phillips, T. C., Dickins, T. E., & West, S. A. (2011). Evolutionary theory
and the ultimate-proximate distinction in the human behavioral
sciences. Perspectives on Psychological Science, 6, 38–47.
Sherman, P. W. (1988). The levels of analysis. Animal Behaviour, 36, 616–
619.
Simon, H. A. (1973). The structure of ill-structured problems. Artificial
Intelligence, 4, 181–201.
Simon, H. A., Langley, P. W., & Bradshaw, G. L. (1981). Scientific discovery
as problem solving. Synthese, 47, 1–27.
Stanovich, K. E. (1999). Who is rational? Studies in individual differences in
reasoning. Mahwah, NJ: Lawrence Erlbaum.
Stanovich, K. E. (2010). How to think straight about psychology (9th ed.).
Boston, MA: Allyn & Bacon.
Sterelny, K. (1990). The representational theory of mind: An introduction.
Oxford, England: Basil Blackwell.
Sternberg, R. J., & Grigorenko, E. L. (2001). Unified psychology. American
Psychologist, 56, 1069–1079.
Sternberg, R. J., Grigorenko, E. L., & Kalmar, D. A. (2001). The role of theory
in unified psychology. The Journal of Theoretical and Philosophical
Psychology, 21, 99–117.
Symons, D. (1992). On the use and misuse of Darwinism in the study of
human behavior. In J. H. Barkow, L. Cosmides, & J. Tooby (Eds.), The
adapted mind: Evolutionary psychology and the generation of culture
(pp. 137–159). Oxford: Oxford University Press.
 Author's personal copy
20 G.L. Brase / New Ideas in Psychology 33 (2014) 8–20
Tinbergen, N. (1963). On aims and methods in ethology. Zeitschrift für
Tierpsychologie, 20, 410–433.
Tooby, J., & Cosmides, L. (1992). The psychological foundations of culture.
In J. H. Barkow, L. Cosmides, & J. Tooby (Eds.), The adapted mind:
Evolutionary psychology and the generation of culture (pp. 19–136).
Oxford: Oxford University Press.
Tooby, J., & Cosmides, L. (1995). Mapping the evolved functional organi-
zation of mind and brain. In M. Gazzaniga (Ed.), The cognitive neu-
rosciences. Cambridge, MA: MIT Press.
Trivers, R. L. (1971). The evolution of reciprocal altruism. Quarterly Review
of Biology, 46, 35–57.
Trivers, R. L. (1972). Parental investment and sexual selection. In
B. Campbell (Ed.), Sexual selection and the descent of man 1871–1971
(pp. 136–179). Chicago: Aldine.
Whewell, W. (1847/1967). The philosophy of the inductive sciences, founded
upon their history (facsimily reprint) (2nd ed.). New York: Johnson
Reprint Corp.
Wilson, E. O. (1998). Consilience: The unity of knowledge. New York: Knopf.