You are on page 1of 15

_______________________________________________________________________

EVALUATING THE EFFECTIVENESS OF


EDUCATIONAL INNOVATIONS: USING THE STUDY
PROCESS QUESTIONNAIRE TO SHOW THAT
MEANINGFUL LEARNING OCCURS
___________________________________________________________

David Kember, Margaret Charlesworth, Howard Davies, Jan McKay and


Vanessa Stott

Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong.

Educational Innovations

This paper advocates the use of repeated applications of inventories of approaches to learning as
an appropriate technique for the evaluation of educational innovations. The method is
particularly appropriate for innovations aiming to introduce more meaningful forms of learning
as this is a dimension measured by these instruments. The paper gives case studies showing how
the measure can be used for innovations introduced in naturalistic settings.

There is a voluminous literature on educational innovations. Different teaching methods, models


of teaching, the latest educational media, new curricula, alternative assessment methods and
many other forms of innovation have been proposed as leading to better teaching and
improvements in student learning outcomes.

For such claims to be believed they need some form of evaluation. The traditional approach to
seeking evidence is to use the science-based experiment and control method. This approach,
though, has been subject to criticism and suffers from limitations where the aim is to introduce
an innovation into a genuine educational setting, rather than to conduct a laboratory-type trial.

The first difficulty lies in ensuring that the experiment and control are genuinely comparable.
Educational media comparison studies, for example, have been criticized on this basis by a
number of writers (e.g., Levie & Dickie, 1973; Clark, 1983; 1985; Schramm, 1977). Clark
(1983) suggests that unintentional content differences between treatments often confound results.
Compelling evidence for this assertion comes from the observation that any positive effects for
new media more or less disappeared if the same teacher gave or produced the instruction for
each treatment group, but used different teaching methods or media.

Genuine educational settings are particularly difficult for control/experiment designs because of

1
the complex array of contextual variables which can interact with the variable under study. It is
sometimes possible to control statistically for extraneous variables but some variables are
difficult to measure and factors which were not anticipated can and often do play a part.

Making arrangements for control and experiment groups within a school or university can be
difficult. Cohorts of students are commonly taught within one class, so splitting them into two or
more groups requires special arrangements and extra resources. There are likely to be ethical
issues as one or more groups accorded different treatments may feel disadvantaged.

Taught courses normally last for extended periods such as a school year or a semester. Designing
different teaching programs, arranging for the separation of groups and holding extraneous
variables constant becomes more difficult the longer the trial. It is not surprising, therefore, that
many, if not most experiment/control tests are very short in duration. A common time-frame
appears to be as little as an hour or two. But, shortening the time-frame and resorting to
controlled conditions renders the experiment open to the charge of dubious relevance. The
validity of findings from artificial laboratory-type experiments for the complex classroom
environment have been severely questioned (e.g., Parlett & Hamilton, 1977). It is paradoxical
that the more the innovation is controlled to enhance experimental reliability the greater
becomes the discrepancy from the normal classroom setting.

A further criticism of experiment/control designs is the necessity for keeping conditions constant
between treatment groups. Firstly, it is often hard to know whether this condition has been met
because so many variables can play a part. It is also sometimes hard to know whether the control
and experiment groups are treated in the intended different way, as communication between the
two groups can nullify the experimental difference. The need for non-experimental variables to
remain constant across groups also makes it difficult to follow the adaptive processes and fine-
tuning which usually occur when innovations are implemented (Parlett & Hamilton, 1977, pp.
241-242). It is extremely difficult to foresee all eventualities so successful implementations
almost invariably require some modifications to initial plans in the light of experience. Gibbs
(1992) maintains that a process of fine-tuning is essential for a successful outcome to an
educational innovation.

If there is the intention to introduce a genuine educational innovation into a natural school or
college setting the classical experiment/control design has marked limitations. Rather than
allowing the research design to dictate the mode of implementation and prohibit fine-tuning, it is
preferable to adopt a more naturalistic approach which fits the investigation to the desired mode
of implementation and refinement.

A research approach which is consistent with normal implementation methods is the action
research cycle formulated by Lewin (1946). Lewins cycles contain steps for planning, action,
observation and reflection. Innovations would normally be implemented through a series of
cycles each incorporating refinements based upon lessons learnt from the previous cycle. Action
research differs from other paradigms in that change is an intended outcome. Lewin (1952) and
Rapoport (1970) maintain that it is both appropriate and legitimate for action researchers to
contribute to social change as well as making a contribution to theory.

The purpose of this article is to describe one method by which innovations introduced in
naturalistic ways can be evaluated. Our suggestion is that the method be used in conjunction with
other evaluation techniques as part of a triangulation procedure, as was the case with the case
studies described in this article. It is good practice in any evaluation design to include data of
different types. With naturalistic approaches it is particularly important as multiple questions are
usually addressed and the relationship between important factors may be complex.

It is our belief that educational innovations should be developed and tried out in naturalistic
settings and fine-tuned as the project proceeds. Such innovations can be introduced by groups of
involved participants reflecting critically upon their actions. But it should also be possible to
incorporate data gathering procedures within the observation phases of implementation cycles
which produce evidence of the effectiveness of an innovation which would satisfy a critical peer.

The value of such evidence is that it can convince both internal and external audiences. The

2
initiators themselves should seek evidence that their innovation is effective. They may well need
to convince colleagues within the same school or university department. Groups of people rarely
rise up as one with the same conviction that a particular change is necessary. Yet innovations can
often only be effective if they spread beyond the initial enthusiasts to a wider circle within a
school or department (Kember & Gow, 1992). Convincing evidence of the effectiveness of the
innovation is likely to be necessary for this widening involvement to occur. Finally, successful
innovations are worth communicating to a wide external audience so that accruing advantages
are widely shared and innumerable groups do not have to re-invent the wheel.

The examples given in the paper are all taken from educational action research projects in
universities. Action research initiatives by faculty evaluating some aspect of courses they teach
has been advanced as an approach to improving the quality of teaching and learning in higher
education (Gibbs, 1992; Kember & Gow, 1992; Kember & Kelly, 1993; Zuber-Skerrit, 1992).

Meaningful Learning

The method described in this paper is quite widely applicable in that it examines changes in the
learning approaches of the students. The learning approaches used by students are taken as an
indicator as to whether or not meaningful learning has occurred.

It is highly appropriate to describe a method for monitoring meaningful learning outcomes since
there has been much recent concern that graduates of higher education lack qualities like critical
thinking, an aptitude for self-managed learning, reflective thinking, and the ability to solve novel
problems. Doubts as to the quality of graduates, and hence teaching in universities, have been
expressed by several major reviews (e.g. Association of American Colleges, 1985; Boyer, 1987;
Daly, 1994; National Institute on Education, 1984). It is significant that a common theme within
these and most similar criticisms has been that graduates frequently lacked the very qualities
consistent with a deep approach to learning, which is a quality measured by the instruments
described in this article.

The procedure described is appropriate for innovations which have the aim of enhancing
meaningful or independent learning. This focus on meaningful learning is seen as particularly
apposite given that traditional experiment/control designs have often been criticized for using
verbatim recall of knowledge as a means of comparing treatment groups. While such tests may
produce statistically significant differences between groups, the type of learning measured may
be quite inappropriate to the type of innovation or the intended educational setting.

For example, Ausubel (1978) and Mayer (1979) were critical of many of the experiments with
advanced organizers. They claimed that a lot of the experiments had measured amount retained
or verbatim recall. However, advance organizers were proposed as a way of enhancing
meaningful learning by revealing the relationship between conceptual ideas.

Approaches to Learning

Approaches to learning are direct descriptions of learning processes used by students. The
categories used to describe approaches to learning were derived from interviews and observation
of students performing normal learning tasks such as reading academic articles. Marton and
Slj (1976) identified two discrete approaches to reading articles. Students who used a surface
approach concentrated on surface features of the learning task, such as key words or phrases.
Their strategy was to memorize and reproduce elements which seemed appropriate. When asked
about the content of an article they tended to give detail from examples but had often not grasped
the principle of the article.

3
Students adopting a deep approach concentrated on the underlying meaning of an article. The
intention was to understand the real message of a piece of writing or the underlying purpose of
an academic task.

Learning approaches have a motivation and a strategy element which are intimately related.
Students attempt to understand a topic if it is of real interest to them or if they can see its
relevance to their current or future professional roles. On the other hand a surface approach is
associated with limited interest in a task or an extrinsic motivation.

Learning approaches are not stable psychological traits. It is true that students normally have a
predisposition to either deep or surface approaches in general. However, this preferred approach
can be modified by the teaching context or learning environment for individual course or
particular learning tasks. The approach adopted therefore depends upon the students motivation
and the prevailing teaching context. It is reasonably common for students to use a deep approach
in one course and a surface approach in another (Laurillard, 1984; Ramsden, 1984).

Variables such as extrinsic motivation (Gow & Kember, 1990), lack of intrinsic interest
(Fransson, 1977), reproductive assessment questions (Entwistle & Ramsden, 1983, ch. 8;
Thomas & Bain, 1984), formal teaching (Ramsden & Entwistle, 1981), a focus on transmitting
information (Gow & Kember, 1993; Kember & Gow, 1994), excessively heavy workload
(Dahlgren, 1978; 1984) or lack of freedom in the learning environment (Entwistle & Ramsden,
1983) have all been shown to make the adoption of a surface approach more likely. Measures of
approaches to learning can, therefore, be related to aspects of the teaching and learning
environment.

Measures of learning approaches are, therefore, sensitive to the various contextual variables
which constitute the teaching and learning environment. Repeated application of the measures
will reveal changes to students approaches to learning which can then be related to innovations
or changes to a course.

Measuring Approaches to Learning

The original research (Marton & Slj, 1976) which characterized deep and surface approaches
used qualitative research methods. Questionnaires have since been developed which assess the
extent to which students use deep or surface approaches.

The Study Process Questionnaire (SPQ) (Biggs, 1987a) contains an achieving approach scale in
addition to ones for deep and surface approaches. Each scale is divided into sub-scales for a
strategy and a motive component. The SPQ has been designed for use in higher education. There
is an alternative version the Learning Process Questionnaire (LPQ) for use in schools. The LPQ
has the same scales and sub-scales as the SPQ but the questionnaire is shorter and the wording of
items is appropriate for a school setting.

Another instrument, the Approaches to Studying Inventory (ASI), was designed for a higher
education context (Ramsden & Entwistle, 1981; Entwistle & Ramsden, 1983). The ASI contains
scales for deep and surface learning in addition to scales measuring other learning approaches
and learning styles.

Each of these three instruments the SPQ, LPQ and ASI can be used to monitor changes in
learning approaches of students by repeated application of the instrument. The SPQ was used for
the case studies described in this paper because it had already been used extensively within our
institution for a longitudinal study. In the process of this use the cultural relevance for Hong
Kong students had been extensively researched (Biggs, 1992; Kember & Gow, 1990; 1991),
There has also been work in a variety of other settings on the factor structure of the instruments
and the dimensionality of approaches to learning. Richardson (1994) has recently published an

4
extensive review of this literature. Richardsons conclusion was that:
In short, both qualitative and quantitative research procedures have produced evidence from a
reasonable variety of national systems of higher education for the broad distinction between two
fundamental approaches to studying: first an orientation towards comprehending the meaning of
the materials to be learned; and, second, an orientation towards merely being able to reproduce
those materials for the purposes of academic assessment. (p. 463)

Richardson went on to note that the orientation towards comprehension was consistent between
countries, probably because of the concordance of the espoused aims of higher education
between systems (Kember and Gow, 1990). The less desirable approach appeared more variable
and less coherent, possibly because it is usually adopted as a consequence of students
perceptions that the learning environment is unsatisfactory, which will be manifest in different
ways in different institutions or countries. The orientation was seen as involving some
combination of surface process, learning pathologies and undesirable approaches (Harper and
Kember, 1989, p. 73).

The SPQ consists of 42 items, seven for each of the sub-scales surface strategy, surface motive,
deep strategy, deep motive, achieving strategy and achieving motive. Students respond to each
item on a five point scale. Main approach scales are calculated by summing scores on the
respective strategy and motive sub-scales. So the deep approach (DA) main scale, for example,
is the sum of the deep strategy and deep motive sub-scales. Norms for the SPQ for Hong Kong
students are available in Biggs (1992), which also gives further details of how to use the
instrument.

Establishing Comparison Data

To give a picture of normal trends over time, the SPQ data from a wide range of courses across
an institution is drawn upon. The SPQ was used for a survey of 2143 students in degree level
courses at the Hong Kong Polytechnic (Gow & Kember, 1990; Kember & Gow, 1991). Figure 1
is a graphical representation of the data given in Kember and Gow (1991) showing the mean
deep approach (DA) scores by year of study. The survey encompassed students in over a third of
the departments and included all three years of study. It can be seen that deep approach falls
consistently. Kember and Gow (1991, p. 126) reported that the decline in deep approach scores
was significant at the level p < 0.001.

For the case studies and the comparison base data, deep approach scores only are given. This is
partly to simplify presentation and interpretation. Deep approach and meaningful learning
outcomes can be closely related, so it is deep approach which is the most relevant indicator. For
the SPQ internal consistency of the deep approach scales have consistently been higher than
those for surface approach when the instrument has been used in a Western setting (Biggs,
1987b). In a non-Western setting, Kember and Gow (1990) concluded, from the factor structure
and reliability scores, that the deep approach construct and scale could confidently be considered
as having the same meaning and similar reliability as in the West. The reliability of the surface
approach scale was somewhat lower which may result from the emergence of a learning
approach by which students try to memorize material, but after seeking understanding. These
findings were confirmed by the review by Richardson (1994) of work in more diverse cultural
settings.

FIGURE 1: Mean SPQ deep approach scores by year of study for the Polytechnic.

5
mean DA 50

48

46 Polytechnic

44

42
0 1 2 3

year

The results of other surveys of learning approaches by year of study suggest that the findings
described above are certainly not unique to the one institution. Biggs (1987b) used the SPQ for a
survey of 2,365 students at ten Australian Colleges of Advanced Education (CAEs) and five
universities. University science students showed a general decline in the use of a deep approach
as they progressed through their course. For arts students, the mean deep approach scores rose in
the second year but dropped for the third year. In a longitudinal study, using the ASI, Watkins
and Hattie (1985) found that deep approach scores declined from first to third year. The decline
was attributed to increased workloads and perceived lack of employment opportunities.

It was even difficult to obtain increases in achieving and deep approach scores in all innovations
selected as case studies for a national project devoted to improving the quality of student
learning. Gibbs (1992) reported ten case studies funded by the British Council for National
Academic Awards project Improving Student Learning. A shortened version of the ASI was
used to monitor changes in the quality of student learning, in a similar way to that described for
the two courses in this study. Four of the case-studies did report increases in achieving and
meaning approach scores and a decrease in surface approach. In one project an experimental
group had significantly better ASI scores than a control. Three of the case studies reported no
significant differences in before and after achieving and meaning approach scores. One case-
study did not report ASI scores and one project had to be aborted.

Any innovation which can show an increase in the use of deep approach has, therefore, achieved
something quite significant. It even seems to be some sort of achievement in higher education if
deep approach scores do not decline.

As a word of caution, we would not view deep approach scores as unequivocal indicators of the
success of an innovation. Innovations are too diverse in their aims and nature for such faith to be
placed in individual indicators. Further we would not wish to be interpreted as advocating deep
approach scores as a universal measure of the quality of a coursea measure which has been
sought by several regulatory bodies.

6
Case Studies
Three case studies are given to illustrate the use of the SPQ as a monitor of the effectiveness of
innovations to promote meaningful learning. The examples refer to a whole innovatory course, a
course which replaced a previous one and a subject added to a course to promote reflection on
learning processes.

In each case the effect of the innovations upon the students approaches to study was examined
by repeated application of the SPQ to the same cohort of students. Repeated measures analysis of
variance (Norusis/SPSS Inc., 1988, ch. 7) was used to determine whether any changes in the
deep approach scores were statistically significant.

A Course in Design
The Diploma in Design is a foundation and pre-requisite for entry to one of the five separate
Honours Degreesin graphic, industrial, interior, photographic and fashion design. The course
aims to establish a foundation for all areas of design by encouraging a variety of responses,
drawing out individual abilities and aptitudes, rather than delivering a body of knowledge. A
capacity to generate, visually document, evaluate and locate responses in context is common to
all areas of design.

The curriculum is project based to emphasize the inter-relation of theoretical and practical
design elements, and to create a flexible framework for individual discoveries. Projects are
devised to pose problems, to identify topics for research, or to define materials and processes for
exploration. Each project is a new situation in which the individual student identifies resources,
methodologies for solving problems and originates ideas to be articulated verbally and visually.
In conjunction with a tutorial system, projects form a flexible framework to structure and
integrate the separate components of design studies, whilst at the same time encouraging variety
and performing a diagnostic process.

Students encounter design methodology by responding to a written brief. Each brief is a starting
point for investigation, from which concepts or experiences can be examined, the results
analyzed and re-structured. At the end of a project students are asked to communicate their
discoveries, in a visual and verbal presentation.

Project briefs are open-ended to provoke authentic responses. Pre-determined models for end
results are avoided, and evaluation criteria, set out in each brief, emphasize exploratory and
experimental approaches. Initially students tend to focus on more evident manipulative skills.
Project work emphasizes that the process of designing requires more than an accumulation of
techniques. Without the support of an expected outcome, students are challenged to re-appraise
preconceived values and attitudes. Without a correct outcome individuals often seek assurance
from approaches which have previously proved successful. Students who perceive project
outcomes to be demonstrations of technical skill, or direct efforts to predictable results sabotage
their own learning.

In the nature of materials and process-based projects, there are constraints on resources and
approaches which promote similar discoveries and invite students to influence one another. For
such projects students are usually directed to work cooperatively to extend individual experience
and enhance confidence.

The tutors role is to draw out students own experiences, cultivating individual responses and
transferring authority to the students. In tutorials and critiques students are asked to describe the
purpose of their work, how they initiate, organize and express the content, what other options
can be considered, and how such decisions might be weighed.

There is no value in deploying manipulative or technical skills inappropriately, however


attractive the products. The tutor helps students to identify stages in the decision-making process
and to appraise their methodologies and achievements. To satisfy course requirements students
are expected to demonstrate reflection on the appropriateness of the response together with

7
visual sensitivity, expressive qualities, and technical control.

The ability to develop an idea beyond its initial stages is crucial. Exploration provides
opportunities for self-discovery, testing an individuals capacity to respond to a problem and to
investigate the possibilities. Sophisticated techniques are unnecessary, in fact simple tools and
processes often stimulate creative responses in unexpected ways. It is a common experience to
find that accidents or unintentional developments can be used inventively. In this environment
students learn to communicate visually through drawing and model-making, to generate and
develop ideas creatively, and critically evaluate their own work and that of their peers.

Design studies are not assessed in formal examinations. Assessment of practical work is based
on a portfolio of all projects and exercises completed during a term. Study material does not lend
itself to conventional patterns of revising or rote learning, and it could be expected that students
are discouraged from developing surface approaches.

The course goals are clearly consistent with a deep approach. Repeated application of the SPQ
provided an indication as to whether, in broad terms, this innovatory course was meeting its
goals. Figure 2 shows the deep approach scores at the start, end of year one and conclusion of the
course. Repeated measures analysis of variance, employing a multivariate test of significance,
showed that the increase in deep approach scores was significant (F = 5.4, DF = 82, p < 0.005).
The results are for virtually the entire cohort of students enrolled in the course. The
questionnaire was given out in class on each of the three occasions it was administered, making
it possible to obtain such a high return.

In this instance the SPQ scores for the two year Design course are compared to the data for the
institution as a whole, shown in Figure 1. The rise in deep approach scores is seen as evidence
that the broad aims of the course were being achieved. Confirmation came from other evaluation
techniques, including student feedback questionnaires, interviews and peer review, against which
the SPQ data were triangulated. These other data are not presented here as the Design course is
not the main subject matter of the article.

FIGURE 2: Mean SPQ deep approach scores by year for the Design course.

mean DA 50

48

46 Design

44

42
0 1 2 3

year

8
A Course Replacing a Previous Course
The second case study concerns a degree level radiography course which replaced a previous
diploma course. The passage to mounting the course had not been smooth, with the first proposal
for the degree course being rejected by the institutional course approval procedure, and the
implementation being deferred for a year. Rejection of a course proposal in this way was an
unusual occurrence. In this instance, the rejected course proposal was seen as being very
procedurally based, with a heavy emphasis on lectures as the medium of instruction, and no
mechanism apparent for achieving the stated aim of integrating learning at all levels.

There was a change in leadership before the second proposal was started, after which it was
important to establish a cohesive course committee which could then progress in the development
of a program that would be academically challenging and vocationally relevant. Eventually the
second proposal passed through both internal and external course approval stages, and unlike the
previous proposal was praised by the panels.

The previous diploma course was heavily weighted towards lectures, with written examinations
being the main vehicle for assessment. The new program placed a much greater emphasis on
tutorial and practical classes, with lectures being used mostly to define topics and provide the
appropriate emphasis. Team teaching was introduced because of a need for integration between
subjects. The number of end-of-year examinations were considerably reduced, with a subsequent
increase in continuous assessment.

Students do not spend significant time in clinical placements until the end of the first year when
they have completed most of the professional subjects for that stage. On the diploma course
students were rostered to clinical settings after twelve weeks from the start of the course when they
had very little understanding of the professional subjects.

A comparison between the two programs given in Table 1, shows a progressive decrease in student
contact hours on the degree, which has been planned to allow for increasing autonomy for the
students in their learning.

TABLE 1: A comparison of contact hours per week


iploma Degree

24 23

29 14

26 6

To ensure that the course would be implemented as it was described and intended in the course
documentation, it was decided that an important step would be a program in staff development.
Several workshops were conducted including ones on tutorial management and assessment
methods. Action research was introduced as a means of monitoring teaching and learning, clinical
supervision and assessment, and project development.

The SPQ mean deep approach scores for the two courses are shown in Figure 3. The figure shows
data for the penultimate (RPD1) and final (RPD2) intakes to the Diploma course and for the first
(RBSC1) and second (RBSC2) intakes to the degree course. The apparent declines in deep
approach scores for the two Diploma intakes were not statistically significant (F = 1.74 for RPD1
and F = 1.64 for RPD2). These data are, though, quite sufficient to establish that a change has

9
occurred between the two courses, as the multivariate test of the repeated measures analysis of
variance showed that mean deep approach scores for the two intakes to the degree courses did have
a statistically significant increase (F = 5.3, DF = 41, p < 0.01 for RBSC1 and F = 7.7, DF = 38, p <
0.01 for RBSC2).

The two intakes to the Degree course were respectively 69 and 70. The DF figures in the test arise
because of the listwise treatment of missing values. Any student who fails to provide a usable
value for just one of the applications of the SPQ is deleted for the sample. The usable sample is,
nevertheless, representative since the mean SPQ scores for the sample remaining after listwise
deletion are virtually identical to mean values for the three or four applications of the SPQ
obtained when all students who completed the questionnaire on individual applications are
retained.

It is assumed that the increase in the mean deep score for the degree students can be attributed to
the change in teaching and assessment formats which have, as much as possible, tried to present
learning situations which are relevant to the required learning outcomes.

Students have been generally very accepting of the teaching and learning measures. A range of
students were interviewed at the end of the first year to gauge their opinion of the course. Overall
their comments reinforce the result of the SPQ as can be seen from the following examples:
I think we can practice what I have learned from the theory in clinical placement. .... In this course
they are very closely related with one another. (Student 1)

I think the discussion [in] the tutorial is quite helpful because in the tutorial, especially if the group
size is smaller, we can discuss more closely and the reaction is better. (Student 2)

The students have more chance to participate in the course like the practical classes .... we cannot
understand just through lectures. .... I want this kind of learning. So, it is learning and practice so
that I can understand the theory more. (Student 3)

FIGURE 3: Mean SPQ deep approach scores for the Radiography courses.

mean DA 50

48
RPD1

RPD2
46
RBSC1

RBSC2
44

42
0 1 2 3

year

10
A Subject Promoting Reflection
The third case study concerns a program which was developed in the first year of a generalist
undergraduate degree in Business Studies (see Davies, Sivan and Kember, 1994, for a more
detailed account). Its objectives were to:

a) make each student aware of the different approaches to learning which are
encountered in higher education;
b) provide each student with an appraisal of the approach to learning which they
employ most often;
c) develop students commitment to effective learning; and
d) involve the student in alternative forms of learning experience.
In order to meet these objectives the students were led through three steps. The first two of these
were explicitly concerned with objectives a) to c), while the third was concerned with d) and
included computer simulations, communications exercises, problem-solving techniques and
outside speakers.

The first step was to identify each individual students own view of learning, by asking them to
write down an unprompted answer to the question;

what is learning?

The answers were collected in and it was then explained to the students that research has
identified six distinct conceptions of learning (Slj, 1979; Marton, DallAlba and Beaty,
1993). Each of these was described and the lecturer emphasized that higher education seeks to
place less emphasis on learning as a quantitative increase in knowledge, and more on higher
level conceptions such as learning as applying, abstracting meaning or understanding reality.
After the lecture the students responses were read by two judges who allocated them to the six
different conceptions. These were then fed back to the students in the following week, both
individually and in the lecture.

The second step involved using the SPQ to measure students approaches to learning. The results
were again fed back on an individual basis. They showed that the business students exhibited a
very similar pattern to the Hong Kong norms, with a slightly less marked tendency to surface
approaches than Australian students and a slightly greater predilection for both deep and
achieving approaches. Students also examined their learning styles and related conceptions,
approaches and styles in reflective discussion.

These first two steps were completed within the first month of the course, and the program then
went on to the third step, which occupied the remained of the academic year.

The initiative was evaluated in a number of ways. Student views were elicited in mid-year using
the slip technique (Eitington, 1989) Each group of five students was given white cards on
which to write good things about the degree course as a whole and colored cards to write bad
things about the course. About 40% of the positive responses referred to the initiative
approvingly while about 18% of the negative comments expressed disapproval. Formal student
feedback questionnaires were also used, showing that students rated the initiative in much the
same way as they rated other modules.

The most formal, and appropriate, form of evaluation lay in further administrations of the SPQ,
which took place at the beginning of the students second year and near the end of the three year
course. The mean deep approach scores obtained during the second and third years of the course
were higher than that in the first year. The difference was just below the level normally seen as
necessary for statistical significance (F = 2.61, DF = 108, p = 0.078). The total enrollment for the
course was 120 so the usable questionnaire return rate was 92%.

At first sight this may not appear to be a particularly notable achievement. It does appear
creditable, though, compared to the statistically significant decline in the scores for deep
approach in the Polytechnic as a whole. An even more pertinent comparison is with the same

11
Facultys degree in Accounting. The Accounting course is very similar in its first year
curriculum content to the Business Studies degree, and is taught by a substantially overlapping
group of staff. Deep approach scores by year of study for the Accounting course have been
reported by Gow, Kember and Cooper (1994). Approaches to learning were monitored in a
longitudinal design by repeated application of the SPQ at three points in the three-year program.
Figure 4 includes a graphical representation of the data they give. It shows a steep decline in the
mean deep approach score from first to second year, followed by a leveling out in the third year.
It is reasonable to use the Accounting students as the most appropriate comparison, and in that
case the relative improvement of the Business course represents a substantial achievement. The
major difference between the two courses was the inclusion of the core reflective component in
the Business course. These deep approach data suggest that the addition of the reflective element
seems to have had an impact on student learning approaches of the desired type.

FIGURE 4: Mean SPQ deep approach scores for the Business and Accountancy courses.

mean DA 50

48

Accountancy
46
Business

44

42
0 1 2 3

year

Conclusion
The case studies have illustrated somewhat different ways of appraising the SPQ data. The data
for the innovatory design course are simply compared to that for the wider Polytechnic sample.
Showing that the SPQ scores rise by year of study is sufficient to show that the design course is
achieving its broad aims. This is especially true when the results are compared to the
Polytechnic-wide data where results fall by year of study.

The new radiography course replaced a previous one. In this instance mean SPQ scores from two
intakes of the previous course could be compared to those for the new course. Comparison
between the two courses is striking, indicating a very successful action research initiative.

The business case study also features a comparisonthis time between two courses with many
common subjects and lecturers. As the program promoting reflection upon learning was a major
difference between the two courses, it is reasonable to attribute differences between the SPQ
data to the introduction of this program. Since an aim of the program was promoting meaningful
learning approaches, use of the SPQ is particularly relevant.

12
The case studies show that the SPQ provides a useful evaluation of the outcomes of naturalistic
educational innovations. Each of the innovations had aims consistent with the deep approach
construct of the SPQ. The graphs show that each innovation can be seen as successful in meeting
its aims. Whether the nature of the innovations can be generalized to a wider context outside
Hong Kong is open to debate since the studies were conducted in a naturalistic style in a limited
setting. It certainly does, though, seem to be appropriate to extend the use of the SPQ and other
instruments measuring approaches to learning as an evaluation strategy for educational
innovations in other locations.

The action research approach to introducing and evaluating educational innovations is becoming
more widely accepted (e.g. Kember & McKay, in press) so it is timely to have greater attention
paid to appropriate evaluation strategies. Future work might look more closely at triangulation of
a number of evaluation measures or approaches.

Acknowledgments
This research was supported by grants from the Hong Kong Polytechnic University and the Educational
Development Fund of the Hong Kong Polytechnic University.

References

Association of American Colleges (1985). Integrity in the college curriculum: A report


to the academic community. Washington, DC: Report on redefining the meaning
and purpose of baccalaureate degrees.
Ausubel, D.P. (1978). In defence of advance organisers: A reply to the critics. Review of
Educational Research, 48, 251-257.
Biggs, J. (1987a). The study process questionnaire (SPQ): Manual. Hawthorn, Vic.:
Australian Council for Educational Research.
Biggs, J. (1987b). Student approaches to learning and studying. Melbourne: Australian
Council for Educational Research.
Biggs, J. (1992). Why and how do Hong Kong students learn? Using the Learning and
Study Process Questionnaires. Hong Kong: Hong Kong University.
Boyer, E.L. (1987). College: The undergraduate experience in America. New York:
Harper and Row.
Clark, R.E. (1983). Reconsidering research on learning from media. Review of
Educational Research, 54(4), 445-460.
Clark, R.E. (1985). Confounding in educational computer research. Journal of
Educational Computing Research, 1(2), 28-42.
Dahlgren, L.O. (1978). Qualitative differences in conceptions of basic principles in
economics. Paper read to the 4th International Conference on Higher Education
at Lancaster, August/September 1978.
Dahlgren, L.O. (1984). Outcomes of learning. In F. Marton, D. Hounsell & N. Entwistle
(Eds.), The experience of learning. Edinburgh: Scottish Academic Press.
Daly, W.T. (1994). Teaching and scholarship: Adapting American higher education to
hard times. Journal of Higher Education, 65(1), 45-57.
Davies, H., Sivan, A. & Kember, D. (1994). Helping Hong Kong Business students to
appreciate how they learn. Higher Education, 27, 367-378.

13
Eitington, J. E. (1989) The winning trainer. Houston: Gulf.
Entwistle, N.J. & Ramsden, P. (1983). Understanding student learning. London:
Croom Helm.
Fransson, A. (1977). On qualitative differences in learning. IV-Effects of motivation
and test anxiety on process and outcome. British Journal of Educational
Psychology, 47, 244-257.
Gibbs, G. (1992). Improving the quality of student learning. Bristol: Technical and
Educational Services.
Gow, L. & Kember, D. (1990). Does higher education promote independent learning?
Higher Education, 19, 307-322.
Gow, L. & Kember, D. (1993). Conceptions of teaching and their relationship to student
learning. British Journal of Educational Psychology, 63, 20-33.
Gow, L., Kember, D. & Cooper, B. (1994). The teaching context and approaches to
study of accountancy students. Issues in Accounting Education, 9(1), 118-130.
Harper, G. & Kember, D. (1989). Interpretation of factor analyses from the approaches
to studying inventory. British Journal of Educational Psychology, 59, 66-74.
Kember, D. & Gow, L. (1990). Cultural specificity of approaches to study. British
Journal of Educational Psychology, 60, 356-363.
Kember, D. & Gow, L. (1991). A challenge to the anecdotal stereotype of the Asian
student. Studies in Higher Education, 16(2), 117-128.
Kember, D. & Gow, L. (1992) Action research as a form of staff development in higher
education. Higher Education, 23(3), 297-310.
Kember, D. & Gow, L. (1994). Orientations to teaching and their effect on the quality
of student learning. Journal of Higher Education, 65(1), 58-74.
Kember, D. & Kelly, M. (1993). Improving teaching through action research. N.S.W.:
HERDSA Green Guide No. 14.
Kember, D. & McKay, J. (in press). Action research into the quality of student learning:
A paradigm for faculty development. Journal of Higher Education.
Laurillard, D. (1984). Learning from problem solving. In F. Marton, D. Hounsell & N.
Entwistle (Eds.), The experience of learning. Edinburgh: Scottish Academic
Press.
Levie, W.H. & Dickie, K. (1973). The analysis and application of media. In R.M.W.
Travers (Ed.) Second handbook of research on teaching. Chicago, IL: Rand
McNally.
Lewin, K. (1946). Action research and minority problems. Journal of Social Issues, 2,
34-46.
Lewin, K. (1952). Group decision and social change. In G.E. Swanson, T.M. Newcomb
& F.E. Hartley (Eds.), Readings in Social Psychology. New York: Holt.
Marton, F., DallAlba, G. & Beaty, E. (1993). Conceptions of learning. International
Journal of Educational Research, 19(3), 277-300.
Marton, F. & Slj, R. (1976). On qualitative differences in learning, outcome and
process I. British Journal of Educational Psychology, 46, 4-11.
Mayer, R.E. (1979). Can advance organisers influence meaningful learning? Review of
Educational Research, 49(2), 371-383.
National Institute on Education (1984). Involvement in learning: Realizing the potential
of American higher education. Washington, DC: Study group on the conditions of

14
excellence in American higher education.
Norusis, M.J./SPSS Inc. (1988). SPSS-X advanced statistics guide. Chicago: SPSS Inc.
Parlett, M. & Hamilton, D. (1977). Evaluation as illumination: A new approach to the
study of innovatory programs. In D. Hamilton et al. (Eds.) Beyond the numbers
game. London: Macmillan.
Ramsden, P. & Entwistle, N.J. (1981). Effects of academic departments on students'
approaches to studying. British Journal of Educational Psychology, 51, 368-383.
Ramsden, P. (1984). The context of learning. In F. Marton, D. Hounsell & N. Entwistle
(Eds.), The experience of learning. Edinburgh: Scottish Academic Press.
Rapoport, R.N. (1970). Three dilemmas in action research. Human Relations, 23(6),
499-513.
Richardson, J.T.E. (1994). Cultural specificity of approaches to studying in higher
education: A literature survey. Higher Education, 27, 449-468.
Slj, R. (1979). Learning in the learner's perspective, ISome commonsense
conceptions. Reports from the Institute of Education, University of Gothenburg,
No. 77.
Schram, W. (1977). Big media, little media. Beverly Hills, CA. Sage Publications.
Thomas, P.R. & Bain, J.D. (1984). Contextual differences of learning approaches: The
effects of assessments. Human Learning, 3, 227-240.
Watkins, D. & Hattie, J. (1985). A longitudinal study of the approaches to learning of
Australian tertiary students. Human Learning, 4, 127-141.
Zuber-Skerrit, O. (1992). Action research in higher education: Examples and
reflections. London: Kogan Page.

TEHE Ref.: R9/t2d2/t7c3

Reprinted from Studies in Educational Evaluation, 23(2), Kember, D., Charlesworth, M., Davies, H.,
McKay, J. & Stott, V., Evaluating the effectiveness of educational innovations: Using the study process
questionnaire to show that meaningful learning occurs, 141-157, Copyright (1994), with permission
from Elsevier Science http://www..elsevier.com/locate/studeduc
Single copies of this article can be downloaded and printed for the readers personal research and
study.

15

You might also like