Reason!

: Improving Informal Reasoning Skills

Tim van Gelder
Andy Bulka
University of Melbourne
tgelder@ariel.unimelb.edu.au
abulka@netspace.net.au
http://www.philosophy.unimelb.edu.au/reason

To appear in Proceedings of the Australian Computers in Education Conference, Melbourne July 2000.

Abstract
The goal of the Reason! project is to develop an effective and
affordable method for improving informal reasoning. In this paper
we sketch the background to the project, briefly describe the
Reason! software, and report positive results from a detailed study
of the first full-scale trial.

Introduction Seldom has there been such

One of the greatest challenges facing the Australian educational widespread agreement about
system in the coming years is to substantially improve higher-order a significant social issue as
cognitive skills. We need to continually improve what we do in the there is reflected in the view
classroom so that students at all levels will be better able to cope that education is failing in its
with the demands of employment in the Knowledge Economy. At the most central mission—to
same time, however, our resources seem to be constantly dwindling. teach students to think. -
Many people hope that creative use of information technology will Deanna Kuhn
help bridge this gap.
The Reason! project at the University of Melbourne is addressing the problem of general informal
reasoning—roughly, how to think through complex issues such as how to vote in the
republicanism referendum, or whether it is better to purchase an index fund or the latest hot stock.
Informal reasoning skills are very difficult to master, and appear highly resistant to instruction.
Traditional educational methods seem to do little good. However the skills themselves are too
important to neglect. Can we figure out a better way?
We decided to go right back to the drawing boards. We consulted the cognitive science literature to
find out how cognitive skills such as informal reasoning are acquired. The bottom line is that the
only real hope for improving general reasoning skills is intensive quality practice. We then
designed a new educational method around this approach. Unfortunately, the method seemed
completely impractical, since it apparently required lots of costly human experts. To get around
this problem we have been developing the Reason! software package, an interactive environment for
quality practice of general reasoning skills.
In a rigorous study just completed, students using the new Reason!-based method showed record
gains in informal reasoning. This indicates that thinking skills such as informal reasoning can be
improved, that intensive quality practice works, and that software such as Reason! can be crucial in
helping us meet the challenge in the face of tight resource limits.
What is informal reasoning?
Informal reasoning as we use the term, is most reasoning in a natural language. It contrasts with
formal reasoning, of the kind found in mathematics, computer science, and games such as chess.
Informal reasoning involves activities such as distinguishing principal claims from the reasons or
evidence provided in their support; supporting claims with reasons; evaluating the quality of
reasons; challenging reasons, and rebutting challenges; and evaluating the overall case for a claim.
Informal reasoning is closely related to informal logic (which tends to focus attention narrowly on
the claim/reason relation) and argumentation (the deployment of informal reasoning in attempts to
resolve disputes). Informal reasoning is a central component of critical thinking.
Why is informal reasoning important?
Informal reasoning is required for rational belief. Many philosophers have regarded rationality as
central to being human. Insofar as this is correct, some level of mastery of informal reasoning is
required for full self-realisation. Rationality is also integral to the ideal of citizenship. More
 2

prosaically, informal reasoning is a basic cognitive skill required for effective functioning in most
intellectual disciplines. Finally, competence in higher-order cognitive skills such as informal
reasoning is increasingly required for participation in Knowledge Economy (Hunt, 1995).
According to a recent government report, for example, the key skill separating successful job
applicants from their unsuccessful counterparts is critical thinking (Kemp, 1999).
How good are people at informal reasoning?
In the most thorough study of informal reasoning skills ever conducted, Deanna Kuhn found that
although some people have mastered informal reasoning, and many have some competence, the
average level of ability is dismal (Kuhn, 1991). The majority of her subjects could not reliably
exhibit general informal reasoning skills. This is not to say they have no reasoning skills at all.
Most people exhibit some kinds of reasoning from an early age (Andrews, 1995). Rather, it is to
say that they cannot reliably exhibit the kinds of skills sketched above. For example, although
most subjects would readily adopt an opinion on an issue such as the causes of crime, a majority
could not provide any evidence in support of their stance, seeming unable to distinguish genuine
evidence from other forms of discussion.
Kuhn's pessimistic assessment is supported by other large-scale surveys (see, e.g., Bruer, 1993
ch.1). We know of no comparable studies of informal reasoning in the Australian population, but
our own casual observation suggests that things are similar here. Anecdotally, in a class at the
University of Melbourne we gave 60 2nd/3rd year undergraduates a one-chapter reading and asked
them, as a homework exercise, to "analyse and evaluate the main argument." Very few were able to
do the task at all, let alone do it well. Our discussions with secondary school teachers suggest that
the problem is very familiar to them also.
Does education help?
It is widely assumed that responsibility for improving higher cognitive skills such as informal
reasoning rests with the educational system (e.g., West, 1998). Does it deliver? People who are
more highly educated are better at informal reasoning (Kuhn, 1991), but this may be only because
the education system selects for reasoning skill. Informal reasoning skills also improve during
education (Pascarella & Terenzini, 1991), but this may be largely due to maturation. There is
remarkably little evidence that education substantially improves informal reasoning once other
variables have been factored out, and some authors have argued for pessimism on this score (e.g.,
Perkins, 1985). In any case, the low average levels of ability indicate that general education is not
being effective enough.
What about educational strategies specifically targeted on informal reasoning? Although the
evidence here is mixed, on balance it indicates that such strategies are not very effective. For
example, McMillan reviewed all studies he could find of deliberate attempts to improve critical
thinking, and had a hard time finding significant improvements (McMillan, 1987). Nickerson et al.
reached a similar verdict in their review of general thinking skills programs (Nickerson, Perkins, &
Smith, 1985). We reviewed all prior studies of one-semester undergraduate courses in critical
thinking or informal logic, and found little reason to think they make any appreciable difference
(van Gelder, 2000).
On the need for a new approach...
The points made so far indicate that we need to try something new. Whatever we try ought to be
demonstrably effective at generating substantial gains. Additionally, it cannot impose excessive
burdens on an already overloaded educational system. In particular, it cannot present substantial
additional costs. Any such method is going to have to be solidly grounded in knowledge, from
cognitive science, of how informal reasoning skills are acquired (Bruer, 1993). So what do the
cognitive scientists say?
Research on Acquiring Informal Reasoning Skills.
Unfortunately, there has been very little research targeted on the problem of how general informal
reasoning skills are acquired. For example, a recent review chapter entitled "Acquiring Intellectual
Skills" (Voss, Wiley, & Carretero, 1995) contained just two pages on informal reasoning skills,
and says little about how those skills are improved. There has also been lots of research in the
psychology of reasoning, but this also has ignored the problem of acquisition. Still, there has been
considerable research on development of cognitive skills and of expertise in complex domains.
This research gives rise to two seemingly contradictory hypotheses:
The Quality Practice Hypothesis. One of the most important lessons from research on skill
acquisition has been that practice is the key factor (e.g., Chase & Simon, 1973). However, the
nature of that practice is crucial. First, psychologists have identified various features of practice
leading to higher rates of improvement and levels of achievement in cognitive skills. Thus practice
 3

should at least be (a) deliberate, i.e., self-monitoring with full concentration; (b) graduated, so that
mastery of easier levels precedes practice at more advanced levels; (c) under supervision providing
scaffolding, explicit guidance and feedback; and (d) based on learning by study of worked examples
(for review see Ericsson & Charness, 1994; Ericsson & Lehmann, 1996; VanLehn, 1996; Voss,
Wiley, & Carretero, 1995). Second, if we are to see improvements in general informal reasoning
skills, the practice must itself be suitably general. By this we do not mean that the practice should
be abstract and content-free. Rather the general skills are practiced in a wide variety of concrete
domains, so that the skills become abstracted from any particular content. Finally, it is clear that
mastering informal reasoning is a very challenging business, comparable to an adult gaining
fluency in a second language, or reaching a high level of expertise in a sport or profession. Practice
over a few hours or weeks is unlikely to produce dramatic gains. As with other complex,
challenging skills, we should expect appreciable gains to accrue only with intensive practice over
long periods.
The quality practice hypothesis, then, is that deliberate, graduated, supervised, worked-example-
based practice in a wide variety of concrete domains and sustained over long periods can generate
substantial improvements in general informal reasoning skills.
The Strong Situated Learning Hypothesis. Although the quality practice hypothesis can seem
obvious or trivial, it does stand in some tension with much of the thinking found in what is
known as the "situated cognition" literature. A range of views can be found in this literature, so in
order to maximize contrast with the quality practice hypothesis, we focus here on the most extreme
version, which we call the Strong Situated Learning Hypothesis. According to this hypothesis, all
thinking skills are inherently tied to particular contexts or contents, and so there is really no such
thing as general thinking skills (McPeck, 1990); for that reason, such skills cannot be acquired
through practice. One consequence is that practice on a skill in one context will not "transfer" to
other contexts. Another consequence is that training on general, abstract rules and principles will
yield little benefit (see, e.g., Brown, Collins, & Duguid, 1989; Lave, 1988; Lave & Wenger,
1991).
The strong situated learning hypothesis draws support from many studies finding failure to
transfer, and also from evidence of the kind described above concerning how bad people typically
are at general informal reasoning, and how difficult it is to improve such skills through education.
Still, in recent years it has become clear although the situated cognition literature has generated
many insights, the strong situated learning perspective is too simple (Anderson, Reder, & Simon,
1996; Nisbett, Fong, Lehman, & Cheng, 1987; Salomon & Perkins, 1989). Some general
cognitive skills can be acquired through training. The interesting issues are, roughly: how much,
what kind, and what does it take?
Implementing the Quality Practice Hypothesis: The Reason! Project.
Every year hundreds of University of Melbourne students enrol in Philosophy 003 Logic &
Rationality. This is a course about reasoning, but it is also intended to enhance students' ability to
reason. Yet nobody had ever stopped to ask how effective P003 was in this latter respect. Casual
observation was hardly encouraging; performance on the final exam generally seemed so poor it
was hard to imagine it was any worse
when they started.
In 1997 we began a project to
redesign P003 with the aim of
maximising improvement in thinking
skill, and to systematically evaluate
the new approach. The verdict from
cognitive science was that the new
course should be centred on intensive
quality practice. However,
transforming mere practice into
quality practice seemed to require
close supervision by expert
tutors—something like an Oxford
tutorial system. Such an approach
was obviously completely
impractical. We turned to information
technology and over the past three
Figure 1. Screenshot of the Reason! learning environment.
 4

years have been developing the Reason! software package.

Reason! is a PC program designed to (a) introduce students to the fundamentals of informal
reasoning, and (b) provide an environment for quality practice of emerging reasoning skills.
Reason! is designed to leverage the available human intelligence (that of the students themselves,
their peers, the instructor, and tutors) to maximum effect. Although Reason! may never be as good
as individual attention from an expert human coach, it does improve the quality of practice in
many ways. Currently it provides

• scaffolding for students' activities based on graphical tools for vividly representing reasoning
and users' evaluations of that reasoning
• capacity to handle arbitrarily complex reasoning on any topic, including multiple reasoning
"players," multiple evaluative perspectives, and multiple models of a given piece of reasoning
• support for analysing and evaluating texts, or for producing one's own arguments
• context-sensitive guidance at every point in the complex processes of informal reasoning;
• feedback mechanisms based on worked examples;
• graduated exercise sets;
• extensive help, including guides and a glossary.
A student using Reason! to evaluate the main argument of a text (say, a letter to the editor) would
first construct an argument tree, a graphical representation of the various parts of the reasoning and
how they fit together. The student then goes to the evaluation phase, and enters evaluative
judgements at appropriate places on the tree. At any time the student can click anywhere on the tree
and will be prompted to consider the most pertinent issue arising at that point (e.g., whether the
selected premise is true). The result is an easy-to-understand graphical representation of the
structure of the reasoning, overlaid with the students' own opinion as to its quality. In the final
phase of the process, the student prepares a report which includes the argument tree, a reformulation
of the reasoning in her own words, and a paragraph summarising her evaluation.
Using Reason! in Teaching
Although Reason! was designed for use in P003, it is a general learning tool. In 1999, for
example, it was incorporated into teaching as an auxiliary tool in Physiology and Agriculture. In
Philosophy 003 it was the primary learning vehicle for an entire semester. The course material was
divided into five two-week modules. Each module specified a set of concepts and skills to be
learned, and provided a set of exercises. Each module built on the skills developed in the previous
one. All course material was contained in the Reason! package, which could be downloaded from
the internet, or used in Arts faculty labs. Students were expected to devote 10 hours per week to
practice-based learning over a 12 week semester. Lectures were provided, but were for
administration and to supplement the students' own self-managed learning using the Reason!
environment.
Evaluating Reason!
Does Reason! work? In 1999 we conducted rigorous summative evaluation process. P003 was
taught using a traditional lecture/textbook/homework approach in semester one, and using the
Reason!-based quality practice approach in semester 2. Both groups of students were pre- and post-
tested using the Watson-Glaser Critical Thinking Appraisal (WGCTA), which for decades has been
the most widely used test of informal reasoning and critical thinking. The results are shown in
Table 1 and Figure 2. Students in the traditionally-taught course showed little gain, a result which
is broadly consistent with previous studies (van Gelder, 2000). Students in the Reason!-based
course showed a highly significant gain.
Concerns have been raised about the WGCTA, not least because it is a multiple-choice test. For
this reason we also pre- and post-tested students in the Reason! group with another test of our own
devising. Students were presented with a short (one paragraph) argumentative text, and were asked
to identify the main conclusion, reformulate the reasoning, and to evaluate the reasoning. Average
scores were low, partly because we used a very strict scoring scheme, but partly also because
students found the task very difficult. The test faithfully reflected the nature of the skills covered in
the course. The improvement on this test was similar in magnitude to that on the WGCTA, with a
gain of about 0.5SD over the semester (see Table 1).
 5

Discussion
The WGCTA improvement detected in the Reason!-based course was larger than any we have yet
found in any comparable study in the published literature. While 0.4-.05 SD is in general regarded
as a "medium-sized" effect, it is comparable to the typical effect size for some of the most effective
techniques in higher education, such as Keller's personalised system of instruction. Also, students
showed much stronger improvement in the Reason!-based course than in the traditional course.
These results increase our confidence (a) that the quality practice hypothesis is in fact correct, and
(b) that successful educational methods can be based on the quality practice hypothesis.

WGCTA Written test

Reason!-based Reason!-based
Traditional
course, n=53; 11 course, n=60; 15
course, n=32
weeks b/w tests weeks b/w tests
Score Pre Post Pre Post Pre Post
(WGCTA: # correct/80; Written test: raw
score) 63.8 64.0 61.3 64.9 1.81 3.00

Improvement (mean paired difference) 0.18 3.58 1.19

% improvement. 0.282% 5.84% 65%
Effect Size
0.02 0.41 0.51
(improvement/SD)
p 0.44 0.0004 .001
r 0.02 .45 .38

Table 1: Comparison of two versions of an informal reasoning course. The two courses
were very similar in content. The first version was taught by the traditional "chalk and talk"
method; the other using the Reason-based quality practice method. Both versions were pre-
and post-tested using the Watson-Glaser Critical Thinking Appraisal; the second course was
also tested using a written test of informal reasoning. Both tests had two forms; we used a
crossover design. In computing results we used only scores of students who had completed
both the pre- and the post-test.

Figure 2: Comparison of traditionally-taught and Reason!-based informal reasoning courses

on the Watson-Glaser Critical Thinking Appraisal.

We believe that the root cause of the students' improvement was their intensive, sustained quality
practice. In theory, the Reason! software package is unnecessary; it is just one environment in
which students can engage in quality practice. However, in reality, it is difficult to get students
 6

engaging in large amounts of quality practice. Reason! offers an environment in which practice can
take place, practice that is higher quality than could be achieved by any other feasible method. Put
another way, Reason! makes practice more efficient, and so makes substantial gains achievable.
Note that we are not attributing the strong gains in the Reason!-based course solely to the Reason!
software itself. The software was deployed in the context of a course that was different in many
respects from the traditionally-taught control group, and so there are a number of other variables
which might have been partly or even wholly responsible for the gains. Indeed, the gains might
simply be manifestations of the Hawthorn effect ("any change seems to work") or the founder's
effect ("founders always get the best results"). Systematically teasing out the effects of all the
possible confounding variables would require an extensive series of studies; we are undertaking
these as time and resources allow.
Notwithstanding the strong effect, our subjective impression when we look at the students'
performance, especially on the written post-test, is that they had improved less than we hoped. Part
of the explanation, we believe, is that teaching informal reasoning is not teaching something
totally new. Rather, we are attempting to change thinking from one set of habits to another. The
existing habits are very deeply ingrained, and changing them is difficult. On the bright side, we are
optimistic that, with improvements in the software and the Reason! method, we can produce even
greater skill levels. The current study reports only the first trial of a whole new method, and we
now have lots of ideas about how to improve the software and the method.
An interesting feature of our data, clearly apparent in Figure 2, is the fact that the students in the
control group scored higher on the initial test, and that the final score of students in the
experimental group was not greatly higher than the final score of the control group. This raises the
possibility gains in the control group were reduced due to a ceiling effect, and so we would have
little reason to think (from this data) that the Reason! approach is in fact more effective. However
we currently think it is implausible that gains are being capped by a ceiling effect. Mean scores of
60-65 out of 80 on the WGCTA still leave quite a bit of scope for improvement on that test. More
importantly, however, when we look at the students final performance on the written test (on which
students showed roughly the same gains as on the WGCTA) it is clear that most are still much
closer to the floor than any putative ceiling. There is still very considerable room for improvement.
Nevertheless, the possibility of a ceiling effect is one we will be actively exploring in subsequent
studies.

Future Directions
Major future directions for the Reason! project include:

• Refinement of the Reason! environment. 1999 was the first full-scale deployment of Reason!
as a course package. We are in the process of making numerous revisions suggested by
feedback from students, our observations of student activities, etc..

• Research. We will be conducting more studies of the efficacy of the Reason! approach. One
goal is to continue to test the hypothesis that the Reason! approach is effective (indeed, more
effective than any practical alternative). Another goal is to gain insights into what parts or
aspects of the total Reason!-based course are most responsible for the learning gains. Was it
the Reason! software, or the intensive quality practice (regardless of the software), or the
instructor, or something else entirely? And a third goal is to gain insights into what part or
aspects of the Reason! package are most responsible for learning gains.

• Automated assessment. Our ambition is to develop a method which is not only effective
but affordable for widespread use. The key cost hurdle at this time is assessment. Regular
assessment is a crucial feature of the Reason! approach, and currently it requires highly trained
experts. To deal with this problem we will be developing automated methods and integrating
them into the Reason! environment.

• Schools version. Reason! has been designed for use by university students. However the
skills are almost universally relevant and ought to be actively developed in secondary school
(if not earlier). In partnership with some schools we are currently producing a version suitable
for use at secondary level.

• Going Online. A major goal is to make Reason! an online learning environment, available at
low or zero cost to all students. To do this we will need to obtain suitable sponsorship.
 7

References
Anderson, J. R., Reder, L. M., & Simon, H. A. (1996) Situated learning and education.
Educational Researcher, 25, 5-11.
Andrews, R. (1995) Teaching and Learning Argument. London: Cassell.
Brown, J. S., Collins, A., & Duguid, P. (1989) Situated cognition and the culture of learning.
Educational Researcher, 18, 34-41.
Chase, W. G., & Simon, H. A. (1973) The mind's eye in chess. In W. G. Chase ed., Visual
information processing. New York: Academic, 215-281.
Ericsson, K. A., & Charness, N. (1994) Expert performance. American Psychologist, 49, 725-747.
Ericsson, K. A., & Lehmann, A. C. (1996) Expert and exceptional performance: evidence of
maximal adaptation to task constraints. Annual Review of Psychology, 47, 273-305.
Hunt, E. B. (1995) Will We Be Smart Enough? A Cognitive Analysis of the Coming Workforce.
New York: Russell Sage Foundation.
Kemp, D. A. (1999) Higher Education: Report for the 2000 to 2002 Triennium. Commonwealth
of Australia.
Kuhn, D. (1991) The Skills of Argument. Cambridge: Cambridge University Press.
Lave, J. (1988) Cognition in Practice: Mind, mathematics and culture in everyday life. New York:
Cambridge University Press.
Lave, J., & Wenger, E. (1991) Situated Learning: Legitimate Peripheral Participation. New
York: Cambridge University Press.
McMillan, J. (1987) Enhancing college student's critical thinking: A review of studies. Research in
Higher Education, 26, 3-29.
McPeck, J. E. (1990) Teaching Critical Thinking: Dialogue and Dialectic. New York: Routledge.
Nickerson, R. S., Perkins, D. N., & Smith, E. E. (1985) The Teaching of Thinking. Hillsdale NJ:
Erlbaum.
Nisbett, R. E., Fong, G. T., Lehman, D. R., & Cheng, P. W. (1987) Teaching reasoning. Science,
238, 625-631.
Pascarella, E. T., & Terenzini, P. T. (1991) How College Affects Students: Findings and Insights
from Twenty Years of Research. San Francisco: Jossey-Bass.
Perkins, D. N. (1985) Postprimary education has little impact on informal reasoning. Journal of
Educational Psychology, 77, 562-71.
Salomon, G., & Perkins, D. N. (1989) Rocky roads to transfer: Rethinking mechanisms of a
neglected phenomenon. Educational psychologist, 24, 113-142.
van Gelder, T. J. (2000) The Efficacy of Informal Reasoning Courses. Preprint No. 1/98,
University of Melbourne Department of Philosophy.
VanLehn, K. (1996) Cognitive skill acquisition. Annual Review of Psychology, 47, 513-39.
Voss, J. F., Wiley, J., & Carretero, M. (1995) Acquiring intellectual skills. Annual Review of
Psychology, 46, 155-81.
West, R. (1998) Learning for Life: Review of Higher Education Financing and Policy. Canberra:
Australian Government Publishing Service.