Lewis and Scott

Evaluation during curriculum development

The importance of evaluation during curriculum development: the SNAB experience

Jenny Lewis and Anne Scott
An in-depth evaluation of implementation during the pilot of a new and innovative advanced biology course (SaltersNufeld Advanced Biology, or SNAB) allowed the developers to identify teachers needs and respond by designing suitable support materials

SaltersNuffield Advanced Biology (SNAB) is a new biology AS and A2 course which aims to modernise the post-16 biology curriculum in England through changes to content and a context-based teaching approach which encourages the development of students as active, motivated, autonomous learners. Development began in 2000 with wide-ranging consultation of expert biologists, teachers, lecturers, educators and students to determine the content and form of the course. Once the course structure, biological themes and outline contexts had been agreed, teams of writers mostly teachers or lecturers started preparation of pilot course materials. These included a textbook for each unit supported by extensive online resources (both downloadable paper-based activity sheets and interactive tutorials, animations, and tests). These draft course materials were edited by the central SNAB team and then
ABSTRACT The SaltersNuffield Advanced Biology course (SNAB) is a new biology A-level designed to encourage active, student-centred learning. It teaches biological content through topical real-life contexts, storylines and related socioscientific issues, and integrates ICT across all aspects of the course. As the approach requires teachers (and students) to modify their usual practices, an in-depth evaluation of implementation was undertaken. This evaluation had important consequences for the final revision of course materials and, more significantly, for the development of appropriate support, guidance and training for teachers. The implications for other curriculum development projects are considered.

reviewed by academics. The pilot specification was developed in parallel with this writing, in partnership with Edexcel, and was approved by the Qualifications and Curriculum Authority (QCA) in June 2002. The pilot AS began in September 2002, with the pilot A2 starting the following year. Approximately 1500 AS students and 850 A2 students, in 52 centres, piloted the course. Four pilot textbooks were used (two to cover AS, two to cover A2), each accompanied by a CD-ROM of electronic resources and supported by the pilot website. Feedback from pilot centres, combined with an independent evaluation, enabled a full review of the pilot specification and pilot materials to be undertaken to improve the specification and course materials. The revised AS/A2 specification was approved by QCA in November 2004 for teaching nationally from September 2005. A single, full-colour, textbook to support this new AS specification was published in April 2005 and a similar A2 textbook will be published in 2006. The new website, complete with additional interactive materials within a managed learning environment, went live in May 2005. More information about this course can be found at www. advancedbiology.org.

Aims and intentions of the course

SNAB was designed to engage and motivate students while developing and enriching their understanding of biology. The course combines the use of relevant, up-to-date and exciting content with teaching approaches that encourage the development of students as autonomous learners able to apply a
School Science Review, December 2006, 88(323)

119

Evaluation during curriculum development

Lewis and Scott

wide range of skills to a diversity of biological contexts (see Box 1). Assessment aims to reward understanding and application of knowledge, in addition to factual recall and routine use of practical work. The result is a course that is organised into topics based on real-life contexts such as coronary vascular disease (Lifestyle, health and risk), cystic fibrosis (Genes and health) and forensic biology (Infection, immunity and forensics). Biological content is split across topics, being presented as and when it is relevant to a particular context. For example:
q genetic variation and inheritance first appears at

the underlying principle of the electron-transport chain and the role of redox reactions in the creation of ATP for energy transfer is introduced; q this idea is revisited when looking at respiration in Topic 7, in the context of sporting activity, where the ideas presented in Topic 5 are developed to include detail relating to chemiosmosis and ATP synthesis. In this way, in addition to teaching the key reactions involved in the processes of photosynthesis and then respiration, the underlying principles of energy transfer are highlighted. To support the development of students as critical and independent thinkers, actively engaged in, and responsible for, their own learning:
q social and ethical issues, requiring students

q q

AS in Topic 2 where monohybrid inheritance is considered in the context of genetic disease; continuous and discontinuous variation is discussed in Topic 3 when looking at the effect of genes and the environment on phenotype; the generation of genetic diversity through mutations, recombination of genes and dihybrid inheritance is considered in the context of biodiversity and zoos in A2 Topic 5; DNA fingerprinting in Topic 6 links to the idea of genetic variation; the final topic in the course looks at polygenic inheritance.

Biological principles are developed across the course in a similar way:

q in Topic 5, where photosynthesis is considered

within the context of biodiversity in ecosystems,

BOX 1 A different approach to teaching and learning

A context-led approach: each topic uses a storyline or contemporary issue to engage and motivate students. Biological principles introduced when required, to aid understanding of the context: maintains relevance for the students; ensures content is presented in manageable chunks; builds on existing knowledge. Activities used as an integral part of the learning process: to introduce biological content; to develop wider skills. Use of discussion and debate of ethical issues: encourages students to justify decisions using ethical frameworks.

to critically evaluate different viewpoints (including any supporting evidence) and come to a reasoned view of their own, are integrated into the contexts; there is a strong emphasis on practical work, with the expectation that students will begin to develop scientific skills (including a critically analytical approach to their work) and a clearer understanding of the processes of science; there is a new approach to coursework, which rewards critical thinking and an understanding of the skills and processes of science, thus making practical work a valuable formative experience (see Box 2); extensive electronic resources and activities, such as on-line tutorials and interactive animations which illustrate and model demanding concepts, support learning and provide self-study opportunities for students (see Box 3); teachers are presented with a wide range of activities (both paper based and electronic) which they can select from according to purpose and need.

This extensive and innovative use of ICT is integral to the course as a whole providing teaching resources and materials; supporting student learning; providing additional information for teachers; supporting administration and marking. For a more detailed overview of the course, including a historical perspective, see Reiss (2005).

Feedback to the development team from the pilot centres

There were several sources of informal feedback, which allowed the development team to evaluate the pilot course. Individuals with a particular problem

120

School Science Review, December 2006, 88(323)

Lewis and Scott

Evaluation during curriculum development

BOX 2 AS

A new approach to coursework

A written report of: q a visit to a site of biological interest, focusing on the biology being used or q literature-based research into a biological topic. plus A Practical work review: q a seen exam paper which treats practical work as a formative process; q students draw on practical work undertaken during the year to demonstrate their knowledge and understanding of experimental and investigative skills. A2 An extended experimental investigation: q devised and carried out by students drawing on the skills developed during AS and submitted as a written report.

interactive resources and the activities. The result was a substantial set of data. This feedback network, drawing on a diverse sample and different sources of information, allowed the development team to identify and act upon the most commonly held views. Outcomes of this feedback include the following:
q There was high praise for the quality and range

BOX 3

ICT resources

The extensive electronic resources, which are designed to support teaching and learning, are available online through an annual subscription. To register for a 60-day free trial, go to www.snab. co.uk.
q

or concern often communicated directly with the project office by mail, email or telephone. Teachers, technicians and students all had their own e-groups in which they could exchange views, raise problems, give and receive support, etc. These were monitored regularly by the development team. School visits were organised by the development team to observe lessons and collect oral feedback from teachers and students. Further opportunities for direct (face-toface) feedback and discussion arose during training workshops and feedback meetings at the end of each year. In addition, a substantial sub-sample of the pilot schools were invited to take part in a more formal evaluation based on questionnaires. Teachers, technicians and students in these core centres completed questionnaires asking for comments on each topic, including feedback on the textbook, the

of resources and materials, particularly the ICTbased interactive animations. This was despite some initial teething problems with the use of ICT, many of which related to the diversity of ICT provision within schools. As the team began to appreciate the extent of this diversity and the problems that this sometimes caused, and to identify the different ways in which teachers wanted to make use of the resources, they began a major revision of the ICT materials. The resulting changes to organisation, presentation and format, based on feedback from teachers, have turned a good resource into an excellent resource, which is flexible, adaptable and easy to use. Teachers were mostly delighted with the upto-date contexts, which they found refreshing and stimulating, and the diversity of supporting activities but felt that it was possible to have too much of a good thing, particularly within the time constraints of the school curriculum. As the development team revised the pilot specification they reduced the number of AS topics from 5 to 4, reduced and reorganised content within topics (sharpening the focus in the process) and improved the signposting so helping teachers to prioritise more effectively. The team drew on detailed comments from teachers and students in revising the written materials. This involved a range of changes from editing text to including additional artwork and introducing revision style questions within the book. All activities were reviewed and edited, a few were deleted from the materials and some additional activities included. The teaching schemes were updated to reflect the changes made to the course. Additional support materials for students and teachers, including biochemistry, ICT, maths, practical skills and exam and coursework support, were included within the resources. To help new users find their way around the electronic resources, introductions to the course and website tours for students, teachers and technicians are now provided.

School Science Review, December 2006, 88(323)

121

Evaluation during curriculum development q The self-support offered by the e-group was

Lewis and Scott

considered extremely helpful in learning to use, modify and develop activities and approaches. This facility has now been integrated into the new website. In addition, both teacher and technician workshops were modified in the light of feedback. The feedback network, with its potential access to all participants and its combination of formal and informal sources of data, provided a good overview of strengths, weaknesses and problems. It had the advantage of being relatively quick, easy and cost effective to achieve but it also had disadvantages and limitations. Most questionnaires are rather blunt instruments. They may identify broad issues but they are usually less good at identifying underlying causes and problems information which could be crucial in informing the revision of the pilot. In addition, where information was based on what the teachers said, there was usually little facility for assessing the extent to which this reflected actual practice something that was particularly important in evaluating a course like SNAB, where both teachers and students were being asked to work in new and often unfamiliar ways. There was also an issue of independence. Pilot teachers were working very closely with the development team, meeting them at training workshops and talking to them on help lines. Given this relationship, how impartial (or critical) was their evaluation likely to be? The SNAB team decided to commission an independent evaluation to address these concerns.

purpose was to assess the impact of the course in the classroom and to inform both revision of the specification and the development of further support and training for teachers. Eight teachers from three different centres were invited to take part in this study. Four of these teachers were followed through the full AS/A2 cycle. To maximise the richness of the data the three centres a small rural selective grammar school, a medium sized semi-rural comprehensive and a large metropolitan sixth-form college were selected for their diversity. Data were collected at three points during the first two years of the pilot:
q Phase 1: the start of the pilot; q Phase 2: towards the end of Pilot Year 1 (first

year of AS);
q Phase 3: towards the end of Pilot Year 2 (second

year of AS, first year of A2). At each point the teachers were interviewed and observed in the classroom, and supporting data in the form of technician and student interviews were collected. Phase 1 provided baseline data relating to the teachers starting points their traditional approach, their initial understanding of SNAB and their expectations. Phase 2 reviewed the development of the teachers understanding and use of SNAB during their first experience of the pilot AS specification. Phase 3 reviewed progress as the teachers became more familiar with the specification as a whole, in particular probing the teachers own perceptions of the changes in practice which they needed to make, the extent to which they had achieved these and the factors which had influenced their ability to change. Key areas of difference between SNAB and traditional approaches to A-level relating to use and organisation of content, approaches to teaching and learning and use of ICT and other resources were used as the focus for analysis. After each phase a summary of findings was fed back to the development team and used to inform revision of the specification and development of training and support materials for teachers.

The independent evaluation

The agreed focus of the independent evaluation was the implementation of SNAB in the classroom. SNAB was designed to modernise the advanced biology curriculum by updating the content and developing less traditional approaches to teaching and learning. Inevitably the implementation of this course would require both teachers and students to make some readjustments to their usual practices. So how did teachers understand, interpret and use SNAB in the classroom? What challenges did they encounter and how might they be supported in overcoming these? While it was hoped that this evaluation would provide examples of good practice and justify the chosen approach, it was also expected to (i) corroborate and provide deeper insights into the findings from the general evaluation and (ii) identify the nature of any challenges faced by teachers as they implemented the course for the first time. The

Outcomes of the independent evaluation

While each of the teachers in this study showed their own quite distinctive way of responding to the demands of the course, there was a general concordance in their response to these particular

122

School Science Review, December 2006, 88(323)

Lewis and Scott

Evaluation during curriculum development

features of the SNAB approach (use and organisation of content, approaches to teaching and learning, use of ICT and other resources) and their responses resonated with findings from the broader evaluation. The teachers found some changes relatively straightforward to implement. At first they were a little anxious about the development of biological content across topics and contexts, and tended to over-teach. As they developed a better overview of the course, they became more relaxed and confident and for some this resolved the issue. However, many teachers shared the teaching, each taking different topics, and this made it more difficult for them to see exactly how concepts built up across topics. These teachers would have benefited from clearer guidance on how ideas built up across contexts. One teacher also noted that AS groups were the most diverse that she had to teach and that it was difficult to know how to pitch a topic without more guidance: if Id got those guidelines I could probably say right, I dont need to go to that extent [with that group] because in reality theyre never going to get it or I really need to do my groundwork because I really need to get up to that . In the review process these concerns were addressed with:
q additional guidance in the teaching notes; q the inclusion of concept maps indicating where

with A-level was dropped. While some teachers found it difficult to accept this, others were quick to see the advantages. One teacher noted that, at its best, SNAB works really well students learn more for themselves and are better at arguing, applying what they know and seeing the relationship between things: Although there is (over-all) less biological content, SNAB students are better equipped to find additional information for themselves. The teachers initially set out to use all the activities offered by SNAB but they quickly realised that this was not possible and that they were expected to be selective. At first they had a pragmatic approach to selection based on cost, resources, time, etc., but this quickly developed into a more sophisticated cost/ benefit approach in which educational purpose, and effectiveness in achieving this, were weighed against the pragmatic considerations. Where the teachers were clear about the educational purpose of an activity they were able to select, adapt and even develop alternatives of their own to achieve that purpose. With a clear sense of purpose they also came to recognise that the way they presented an activity would influence its effectiveness: the Carbon Cycle sheet I think I have used it much better this year first youve got to cut and stick and this [was] a bit beneath us [last year] ... We got into a discussion much more this time I got the feeling [that] at least some of the students were getting the idea I felt that them sitting there with that [the cut and stick] in front of them did help their understanding of this idea of moving from sink to sink my understanding of the sheets was much better [this time] The first time I dont think I made enough of the activity itself. I had been in a bit of a rush This is the carbon cycle, okay. This sheet enables you to do a carbon cycle and [then] we were off and back to my traditional, on the board [approach] erm these things produce carbon dioxide, we put carbon dioxide in the atmosphere, these things reduce it and etc. Where no sense of educational purpose developed, teachers found it difficult to recognise an activitys potential value: some activities seem to lack a clear educational purpose they may be fun but they feel like add-ons and leave you with a sense of so what?
School Science Review, December 2006, 88(323)

concepts are introduced and how they build up across topics within the teaching schemes; q the production of web-based staff development units designed to help teachers understand some of the underlying aims of the course, including contextualised biology teaching and the build-up of knowledge and principles. Generally the teachers coped well with the new biological content they were expected to teach, although they were initially less confident, but where their specialist biological knowledge was in a very different area they sometimes felt stressed: we cant be expected to be all-singing alldancing biologists, I mean, Im a biochemist you know everyone has their own areas of expertise. The revised materials include additional support materials, reading lists and web-links to help teachers develop confidence in less familiar areas of the course. To make room for new and up-to-date content and approaches some content traditionally associated

123

Evaluation during curriculum development

Lewis and Scott

The development team responded by making the purpose of each activity very explicit, by stating it on each activity sheet and providing further guidance within the teacher notes. Implementation in some areas was initially hampered by a lack of exemplar materials (particularly for practical exams and course work) and a lack of school-based resources (for example, ICT provision and access). As exemplar materials and then actual exam papers began to build up, most of the teachers became more confident in their approach to implementation. As they became clearer about the aims of the practical and course work and the criteria against which these would be assessed, they began to modify their practice, focusing on reasoning and critical thinking rather than the mechanics of the practical: in their write-ups students now refer to the details of the method in one line were referring to the method on Sheet X (attached) I dont mark their practicals, I give them feedback youve done this well next time, think about this question it [the course work] does seem to discriminate and the mark scheme has been quite effective the better investigations are scoring highly What it comes down to is [the students] ability to understand what the statistics are really telling them and [to] actually use it in the conclusion Some have used the statistics very mechanically and you dont get the feeling from reading it that they really understand why they needed that statistical test whereas others have explained it quite well, either in their rationale for the second part of the experiment or in their actual conclusion I think its better. Most teachers appreciated the intentions behind the SNAB approach to assessment: its good to see them out there doing individual bits of fieldwork [or projects] great to see that back rather than them all just jumping through a hoop. I like the idea of a synoptic paper I think its going to be a good part of the course . I asked them to read it several times before Friday and we went through asking words they misunderstood and that brought in discussions about different things. I think they found it quite interesting. (See Box 4.)

From the start teachers were appreciative and enthusiastic about the potential of the ICT resources, particularly in supporting student learning. Students had access to these resources in their own time through school-based networks or personal copies of stand-alone CDs, and were strongly encouraged to use these by their teachers, particularly for revision: the ICT is really, really good to do something in September, to be revising it in June and to have it on the screen, just as it was in September is amazing I wish Id had that! I keep saying to them you know, you should be using these for revision going back through the activities, using them like a resource, just as you would a book. The more enthusiastic teachers were a bit frustrated by the limitations of the pilot resources and pleaded for Web access (which is now available): if youre bringing all your marks and things together on the SNAB site you [need to] access it from home [because] the majority of us do our marking and assessment outside of class time. Where teachers did integrate ICT resources into their classroom teaching it was very successful: I know Ive taught that section [nerve impulse/ action potentials] much better than Ive ever taught it before. However, some teachers found integrating ICT into class teaching more challenging despite recognising the potential benefits. While they were clear about what they could use, where and for what purposes, they were less confident about how to manage the use of ICT within the classroom. One teacher described this as a process of evolution. At
BOX 4 The synoptic element

In preparation for the synoptic exam paper, students study a piece of published writing which is made available a month before the exam. This gives teachers the opportunity to discuss the article with their students and to consider the sorts of questions that might arise on the exam paper. The synoptic exam paper also includes an essay question and presents students with a range of stimulus material (data, photos, diagrams, quotes, etc.) with related questions.

124

School Science Review, December 2006, 88(323)

Lewis and Scott

Evaluation during curriculum development

his first attempt all the students rushed to individual computers and he had difficulty introducing the activity they were spread out, he had difficulty getting their attention (just checking my e-mail sir!), he had no eye contact and couldnt gauge their understanding. Next time, before letting them go to their computers, he brought them all together around the interactive whiteboard. This resulted in a more effective introduction to the activity but then he found that many students lacked focus when working on the task and often got diverted. The next time he broke the activity into short tasks with clear time limits, made sure that the students understood the purpose of each task and warned them in advance that they would be asked for feedback at the end of each task. All teachers are familiar with these strategies; the novelty (and the difficulty) was in having to use them more often, and in less familiar contexts. Similar challenges were experienced when teaching about social and ethical issues and when trying to promote active learning: its quite a challenging role for a teacher, I think, if youve not yet got the in-depth understanding of a lot of the ethics which go behind it it is quite un-nerving. The development team provided additional support in these challenging areas through the production of specific, Web-based, staff-development units. For example, a unit on ethical debate explores why teachers and students should engage in ethical debate in the classroom, looks at the identification of issues and the use of ethical frameworks in making ethical decisions and makes the aims (to encourage students to think about ethical issues, to come to a view which they can explain and justify, to appreciate that people will have different views on what is right and wrong when discussing ethical issues) explicit. Promoting active learning challenged teachers (and students) perceptions of the teachers role: some of the questions that students can ask you I find myself saying I might just have to look that up it sort of undermines your authority sometimes Im finding Im too interventionist still I ought to give them more time sometimes its much more comfortable for them to be given a list of things [and told] go away and learn these and youll come out alright at the end and SNAB doesnt do that and I think they sometimes find that difficult and then pressure comes back to us I have to try

and stop myself all the time. I could feel myself doing it this afternoon answering questions when I should have been giving them more time to think about it and come up with the answers themselves. Despite the challenges, teachers who succeeded recognised and appreciated the benefits of an active learning approach: [before] I was using the same approach in every lesson everything was me delivering facts to them I wasnt comfortable with that. you give them a very brief description of something and then they go away and find the answer and I do like that because they are having to go through the process and then we pull it all together at the end so its not all coming from me, its coming from them working together to generate ideas that is something that has massively changed in terms of my teaching. Despite the students resistance to being asked to think, the teachers felt they were generally more interested and motivated by SNAB, even when the work was difficult.

Implications for other curriculum development projects

Unlike some recent curriculum developments, SNAB was able to run a three-year pilot during which extensive evaluations could take place. This was a time-consuming and expensive exercise. Was it worthwhile? The answer must be a resounding Yes. Extensive piloting and evaluation have allowed the voices of teachers, students and technicians to be heard in many different contexts. The feedback confirmed that it is possible to teach biological content through contexts and that a more active, student-centred approach to learning can be effective. The detailed comments provided by pilot centres informed all aspects of the revision of the specification and course materials, allowing the production of high-quality post-pilot resources. The evaluation highlighted the difficulties experienced by teachers and prompted the production of additional materials to support the development needs of teachers. Some of the outcomes noted above could not have been achieved without the independent evaluation. This in-depth evaluation was challenging for those directly involved with it and commissioning it
School Science Review, December 2006, 88(323)

125

Evaluation during curriculum development

Lewis and Scott

was a brave move for the development team. The findings sometimes made for uncomfortable reading and this was accentuated by the focus of the study implementation as intended and the obstacles to achieving this. There was a natural wish to focus on the positives and downplay the negatives, and the team had to overcome this in order to maximise the usefulness of the results in informing revision of the specification and planning for teacher development and support. The experience was also potentially threatening and uncomfortable for the teachers. Their practices were being scrutinised at a time when their professional knowledge and expertise (in the context of SNAB, which was new to them) was low. The researcher needed to develop working relationships in which the teachers felt reasonably comfortable and secure and willing to talk about their experiences. The methodology itself is time consuming and expensive in terms of data collection and analysis qualitative data cannot be easily transferred to a spreadsheet and run through a statistical package. As a result, sample size is, inevitably, small and unrepresentative and this limits the claims that the data can support. This does not diminish the value of these data, when set in the context of the wider evaluation. The purpose of such in-depth studies is not to provide further broad generalisations but to investigate the detail which underpins those broad findings and so inform further developments. This level of detail was crucial in fine-tuning the revisions and ensuring that course materials and training met the needs of the teachers while also achieving the aims and intentions of the course. The extensive evaluation was appreciated by the teachers. They felt extremely well supported by the enthusiastic and speedy response from the

development team to their queries and concerns and were impressed with the extent to which these concerns were taken seriously and addressed during revision of the specification and course materials. The enthusiasm of the pilot teachers their contributions to development of the course, their use of the materials in often new and novel ways and their support for each other suggests that development of the course will not stop with the publication of the A2 materials.

Conclusion
Ogborn (2002) noted that whatever the aims of a course, however well designed it is for the purpose and regardless of the quality of the training which accompanies it, teachers must eventually take over the new curriculum and make it their own this is inevitable since it is the teacher who must implement the course in the day-to-day reality of their particular classroom. He considered the best preparation for this takeover was a clear ethos, a diversity of good activities and approaches and a teacher self-support network. Through their extensive evaluation of the pilot and the use they have made of this to inform revision of the final specification, course materials and training, the developers of SNAB have ensured that their aims are clear, achievable and credible, the course is well designed and structured and that training and guidance is carefully designed to meet the needs of teachers. The result is a course with a strong and clear ethos, a diversity of excellent activities designed for a range of purposes and a supportive network of teachers. They must now hand over to teachers to make it their own.

References Ogborn, J. (2002) Ownership and transformation: teachers using curriculum innovations Physics Education, 37(2), 142146. Acknowledgements

Reiss, M. J. (2005) SNAB: a new advanced level biology course Journal of Biological Education, 39(2), 5657. SNAB website: www.snab.co.uk

The authors would like to thank all teachers, students and technicians who took part in this evaluation. They would particularly like to thank those teachers who took part in the independent study for allowing access to their classrooms, for giving precious time willingly and for the patience and thoroughness with which they answered the many questions put to them.
Jenny Lewis undertook the independent evaluation and is a senior lecturer in science education in the Centre for Studies in Science and Mathematics Education, University of Leeds, Leeds LS2 9JT. Email: J.M.Lewis@education.leeds.ac.uk Anne Scott is project officer for SaltersNuffield Advanced Biology in the Centre for Innovation and Research in Science Education, Department of Educational Studies, University of York, York YO10 5DD.

126

School Science Review, December 2006, 88(323)