bs_bs_banner

International Journal of Applied Linguistics

Vol. 24

No. 1

2014

The effect of prociency on CLIL benets in

Engineering students in Spain
Marta Aguilar Universitat Politcnica de Catalunya
Carmen Muoz Universitat de Barcelona

This paper reports on a follow-up study of a Content and Language Integrated

Learning (CLIL) pilot experience with bilingual postgraduate engineering
students at a Spanish university. It aimed at examining learners gains in
listening and grammar skills after a CLIL course in English for a semester, in
particular whether students listening and grammar skills were affected
similarly and whether participants proficiency level played a role. Pairedsampled t-tests showed the difference between the mean scores in the pre-and
post-listening test was significant but it was not for the pre-and post grammar
tests. When students were distributed into three groups on the basis of their
pre-test scores, a repeated measures ANOVA showed that less proficient
students obtained higher gains in listening and grammar skills than more
proficient students.
Keywords: CLIL, proficiency level, tertiary education, listening skills, grammar
skills
Este artculo presenta los resultados de un estudio de seguimiento de una
experiencia piloto de AICLE con estudiantes bilinges de ingeniera
(postgrado) en una universidad espaola. El objetivo del estudio era examinar
las mejoras en las destrezas auditivas y gramaticales despus que los
estudiantes hubieran cursado un curso AICLE en ingls durante un semestre;
en particular, se trataba de examinar si las destrezas auditivas y gramaticales
cambiaban de forma similar y si el nivel de competencia lingstica era un
factor importante. Los t-tests de muestras pareadas mostraron diferencia entre
los valores medios de la comprensin auditiva en el pre-test y el post-test pero
no en los valores de la destreza gramatical. Cuando se distribuyeron los
estudiantes en tres grupos segn sus resultados en el pre-test, la ANOVA de
medidas repetidas mostr que los estudiantes con dominio ms bajo
mejoraban ms que los que tenan un dominio superior tanto en las destrezas
auditivas como en las gramaticales.
Palabras clave: AICLE, nivel de competencia, educacin terciaria, destrezas
auditivas, destrezas gramaticales

2013 John Wiley & Sons Ltd

doi: 10.1111/ijal.12006

Marta Aguilar and Carmen Muoz

Introduction
The approach known as Content and Language Integrated Learning (CLIL)
has been used in the past few years to enhance multilingualism and a
multilingual educational approach whereby one or two additional languages
are used as the medium of instruction. However, the term CLIL is used very
often as an umbrella term for very different teaching experiences, from total
and partial immersion to simply foreign language-medium instruction.
The use of an additional language, and in particular the use of English in
instruction at tertiary level, has a long tradition in many parts of the world from
India to some Arab countries. In recent times, the use of English as the language
of instruction in European university courses has also increased, motivated by
the internationalization process undertaken within the European Educational
Space (Wchter and Maiworm 2008). Many universities are increasingly
dependent on incomes generated from international programmes, yet not
many incorporate clear and explicit communication and language learning
goals (Fortanet-Gmez and Risnen 2008; Doiz, Lasagabaster and Sierra
2011). In fact, at tertiary level informal or tacit inclusion of language goals is
reported to be more usual than a CLIL approach proper (Wilkinson and Zegers
2008). Greere and Rsnen (2008) classified five major variants of CLIL, ranging
from LSP courses given by language specialists to (in-between) pre-CLIL
courses and adjunct-CLIL courses to fully dual CLIL approaches on the
grounds of main aims, target group, main actors, pedagogical approach, main
view of language, and learning outcomes expected. Grin (2005), for example,
using variables like intensity, compulsory status or starting age, identified up to
216 types of CLIL programmes.
To make things more complex, CLIL research seems to neglect that it
borrows from Content-Based Instruction (CBI) influenced by Swiss and
Canadian policies and common in the USA in the late 1990s. In this sense,
pioneer studies at tertiary level relate to CBI (Brinton, Snow and Wesche
1989; Kasper 1997; Rodgers 2006; Song 2006) and the ecological approach to
language (Garner and Borg 2005). Following Greere and Rsnen (2008)
classification, CBI could be regarded as a Pre-CLIL approach in the sense that
language specialists, not content specialists, teach and set the goals. Within
this amalgam of approaches, the following three are often found at tertiary
level: CLIL (with a fully content and language integration), English-Medium
Instruction (EMI) (a content-oriented approach with no linguistic goals) and
CBI (a language-oriented approach). In this paper CLIL is used as a cover
term, but strictly speaking EMI is the approach followed in the university we
are analysing.
A wealth of research exists at primary and secondary levels. Primary and
secondary school results seem to point to a better language proficiency of the
language of instruction if compared with results in formal (traditional)
instruction (Ruiz de Zarobe and Jimnez Cataln 2009; Prez-Vidal 2011).
Research has shown that CLIL students perform better in receptive tasks than
2013 John Wiley & Sons Ltd

The effect of prociency on CLIL benets

in productive tasks (Dalton Puffer 2007), in particular in listening skills, oral

production and organization skills (Httner and Rieder-Bnemann 2010;
Navs and Victori 2010; Ruiz de Zarobe and Lasagabaster 2010, Ruiz de
Zarobe 2011). CLIL students are seen to outperform non-CLIL students in
vocabulary acquisition and use (Jimnez Cataln and Ruiz de Zarobe 2007;
Mrquez 2007), lexical transfer (Agustn Llach 2009; Celaya and Ruiz de
Zarobe 2010) or higher lexical richness (Vrkuti 2010). This is in line with the
immersion outcomes described by Swain (1985). Swain noted that immersion
students progressed in content knowledge and in interpretive skills more
than in expressive (oral and written) skills and hypothesized that the reason
for this discrepancy was that students processed input semantically (rather
than syntactically), which implied that the key was making learners modify
their output (Swain 1995; Swain and Lapkin 1995). However, later research
points to secondary CLIL students outperforming Non-CLIL students in
written production as well (Jexenflicker and Dalton-Puffer 2010; Ruiz de
Zarobe 2010). Ruiz de Zarobe (2010) for example found out in a longitudinal
evaluation that CLIL students outperformed their counterparts in the written
competence despite differences in hours of exposure (80 hours in a 3-year
span) and age differences (CLIL students were 3 years younger), and that the
written outcomes of younger learners in CLIL were higher than those of
older Non-CLIL learners, hinting at the conclusion that under CLIL the rate
of acquisition is faster. Hence grade, linguistic outcomes and educational
programmes seem to be positively related even though the longitudinal
sample in the study was small and results have to be taken with caution.
As regards listening skills, Muoz (2001) reports on a survey with
students of English Philology which used a written questionnaire and oral
interviews. Students perceived they had improved particularly in listening
comprehension after having followed a number of CLIL courses (e.g.
literature and culture). The answers to the written questionnaire revealed that
students perceived higher gains in receptive skills than in productive skills,
and that gains in self-confidence were also high. In Aguilar and Rodrguez
(2012), engineering students most important self-reported gains pointed to
improvements in listening skills and acquisition of specialized vocabulary.
As Airey (2004) acknowledged out, however, many studies do not assess
language competence but rather base their conclusions on the participants
self-reported opinions. He also questions the reliability of many studies in
that researchers usually asked people involved in the experience to express
their opinions, and these people were naturally positive to CLIL.
Incidental learning in language outcomes has been acknowledged
(Pecorari, Shaw, Irvine and Malmstrm 2011a ; Pecorari, Shaw, Malmstrm and
Irvine 2011b), yet CLIL is fraught with problems. The areas that do not seem to
be favourably affected by CLIL are morphosyntax, pronunciation (foreign
accent), informal/non-technical language (Ruiz de Zarobe 2011) and sentencelevel dimensions in writing like cohesion, coherence, paragraphing, register
awareness, genre or style (Dalton-Puffer, Nikula and Smit 2010). Some
2013 John Wiley & Sons Ltd

Marta Aguilar and Carmen Muoz

researchers report no negative effect on content learning (Seikkula-Leino

2007; Bonnet 2012), while for others content knowledge is sacrificed (Jochems
1991;Vinke, Snippe and Jochems 1998; Airey 2004), in particular due to
lecturers slower pace of delivery (between 23 and 23.4% slower, Hincks 2010;
Thgersen and Airey 2011, respectively) and to increasing lecturer and student
workload (Vinke et al. 1998, Tatzl 2011). Lecturers inadequate qualification,
skills and language proficiency have repeatedly been pointed out (Airey and
Linder 2009; Hellekjaer 2010; Aguilar and Rodrguez 2012; Tatzl 2011).
University lecturers become less redundant, less expressive and clear in
English (Klaassen 2008) and seem to be quite unaware of students linguistic
deficiencies, deploying a limited stylistic lecturing variety, making a poorer/
different use of metadiscursive devices (Dafouz and Nez 2010) and seldom
relying on repetition or glossaries of key terms, which is why Tatzl (2011)
concludes that the three pillars on which teaching in English-medium higher
education rests are language proficiency, effective lecturing behaviour and
personal attitude. On the other hand, students are seen to have lecture
comprehension problems caused by poor pronunciation and unfamiliar
vocabulary (Hellekjaer 2010) and to understand content more superficially
(Airey and Linder 2006; Shaw et al. 2008). Finally, other hurdles of English as
lingua franca settings are that textbooks can inhibit rather than enhance content
learning due to the extra workload (Pecorari et al. 2011a) and that sometimes
university students problems in English do not differ much from their
problems in L1 lectures, basically due to their lack of academic literacy in L1
(Airey and Linder 2009; Airey 2010; Hellekjaer 2010).
Sometimes contradictory and inconclusive results have been obtained
across schools and across countries at all educational levels (Bonnet 2012). On
the other hand, it is important to note that in some studies reporting higher
gains, CLIL students have had more exposure to the target language than
non-CLIL students, so it is not clear whether these are effects of increased
exposure or of the CLIL intervention itself. An exception is the study by
Villarreal (2011) comparing groups of CLIL and non-CLIL secondary students
in the oral production of some features of verbal morphology. In a first
comparison with age-matched groups and different amounts of instruction
hours, the CLIL students showed a slight advantage over the non-CLIL
students. But in a second comparison in which CLIL and non-CLIL students
were matched for amount of instruction hours, the latter showed a clear
advantage over the former. Since the non-CLIL students were 3 years older
(1718 vs. 1415), Villarreal suggests this result may be attributed to their
superior cognitive maturity (see Muoz 2006). This contrasts with the
findings in Ruiz de Zarobes study (2010) mentioned above and at the same
time partly confirms the areas where CLIL has no or little effect, as an
important finding of this study is that CLIL effects seem weaker than age
effects, at least as far as verbal morphology is concerned. Another approach to
the time issue is to consider how much extra exposure needs to be provided
by CLIL courses, since it might be argued that slight differences in the number
2013 John Wiley & Sons Ltd

The effect of prociency on CLIL benets

of hours (in both CLIL and non-CLIL courses) might be insufficient for CLIL
benefits to clearly emerge among university students. In secondary education,
it has been demonstrated that types of instruction with more exposure to
English achieve higher levels of proficiency than instruction with less
exposure, provided students receive sufficient exposure to the target language
(Ruiz de Zarobe 2010), yet research on what type of instruction is most
beneficial and how intense exposure at tertiary level should be is to our
knowledge still limited. Studies point out that stay abroad allows for greater
benefits than CLIL and that CLIL yields better results than formal instruction
among children and teenagers (Prez-Vidal 2011). Evidence concerning the
period of time (and the intensity) needed for fully benefiting from long-term
university CLIL programmes is still under way.
All in all, little is known about the effect of CLIL on university students
overall proficiency. Given that learners in tertiary education in particular
come equipped with different language competencies and backgrounds,
learners proficiency level deserves consideration. Magnan (1986) probed the
relationship between speaking rates and level of study at tertiary level. After
investigating the level of speaking proficiency in French and the level of study
in undergraduates, he proposed sound foreign language requirements and
teacher certification. Nevertheless, few researchers have tackled the issue of
whether there is a threshold level that pupils should have attained before
optimally benefiting from CLIL classes (Muoz 2010), although the issue has
been addressed in reference to university immersion and study abroad
contexts. First, in the context of university immersion, research (Klee and
Tedick 1997; Lynch, Klee and Tedick 2001) has found that an initial proficiency
level of Intermediate High on the OPI1 is the minimum optimal level for
predicting social and academic success and target language proficiency
growth. Second, in the context of study abroad, DeKeyser (2010: 90) has
argued that students must have adequate basic knowledge of the structure of
the language if they are to have any meaningful practice experience abroad,
though he has not suggested a specific optimum proficiency level. Finally,
another important aspect in the CLIL approach that has not been sufficiently
researched is the relationship or interaction between students gains and
teachers proficiency, and the optimal (or minimal) teachers proficiency level
for linguistic gains to emerge. At tertiary levels, it is often the case that
stakeholders linguistic and cognitive levels differ. Adapting Cummins
matrix, Coyle (2007: 554) posits that in situations with a mismatch (the
language level of learners is lower than their cognitive level), the learning
environment must ensure that cognitive progression is maintained by
accessing content through a lower linguistic level (Quadrant 3), gradually
working towards higher linguistic demands (Quadrant 4). At university
level, therefore, the mismatch between lecturers and students linguistic
proficiency and the complexity of the content to be conveyed should be
brought to the fore in relation to student gains and to how lecturers speech
diverges from native and disciplinary speech. At both secondary and tertiary
2013 John Wiley & Sons Ltd

Marta Aguilar and Carmen Muoz

levels, as mentioned above, studies point to teachers pragmatically less

varied and less subtle language (Dafouz and Nez 2010; Nikula 2010; Dafouz
2011) and to inadequate materials, either because they are inadequately
adjusted or rediscursified materials (Lorenzo 2008) or because they are highly
demanding university textbooks (Pecorari et al. 2011a, b).
The present study aims at exploring the benefits of a CLIL experience on
the language outcomes of a group of engineering students. Specific attention
will be paid to the role of students proficiency level in English (the target
language), and to how proficiency may have an effect on the language gains
obtained through a CLIL course. Hence, with the aim of gaining insight into
the overall impact of the experience as well as of furthering knowledge about
the skills that seem to benefit the most, and about the role played by learners
general proficiency in the target language, the following research questions
were established:
1. What is the effect of following a CLIL course in English for a semester on
a group of Engineering students listening and grammar skills? Are there
any statistically significant gains after the CLIL experience?
2. Are the students listening and grammar skills similarly affected by the
CLIL experience?
3. What is the role played by the participants proficiency level?

Background of the study

This study was conducted as a follow-up of a CLIL experience in a school of
engineering at a Spanish university. In fact, this was the schools second
attempt to implement CLIL. A general call for volunteers made two years
earlier had been a total failure only seven lecturers out of 112 volunteered to
teach in English. This time it was deemed more cautious to implement CLIL
gradually, so the second step consisted in choosing a few courses at a pilot
stage. The criteria adopted to choose the courses were the following: (i)
courses where the lecturer agreed and felt prepared to teach in English; (ii)
courses with different groups so that students could choose the language of
instruction; (iii) postgraduate courses; and (iv) practical and interactive
courses where problem-solving, case studies or projects were usual teamwork
activities, rather than highly theoretical lessons. Accordingly, seven courses
from different disciplines engineering projects (two groups), business
organization, energy technology, planning and programming of logistic
activities, manufacturing technologies, world economy, and statistical
methods were formally offered in the autumn semester of 20082009.
Another decision made at that stage was to ask the university to collaborate in
two aspects: (i) provide internships (students) that would support lecturers in
the translation of their powerpoint material; and (ii) perform a follow-up
2013 John Wiley & Sons Ltd

The effect of prociency on CLIL benets

study of the experience. It was not until the following year that teachers were
offered teacher training courses.
Information about the kind of classroom activities was necessary to
document the type of instruction. Classroom observation was not feasible, so
discussion with the lecturers and some students shed some light. It is worth
mentioning that there was no CLIL methodological adaptation and
that materials were therefore not grounded in any language teaching
methodology: powerpoint slides usually shown in class had been merely
translated to English. Similarly, lecture instruction basically consisted in the
teacher explaining with the help of powerpoint slides and students taking
notes during the first part of the lessons; then students worked in pairs or in
teams, doing some related problem-solving or discussion tasks. Students
often had some recommended readings, and in three courses they worked in
teams on a given topic for some weeks and at the end of the semester
delivered an oral presentation and submitted a written summary (poster) of
their presentation. In the rest of the courses, every group had to gather
information on a specific topic and deliver an oral presentation.

Method
Participants
The original sample of students was composed of 205 students in an
engineering school, with an average age of 21.4, and with an upperintermediate level of English (B2) at the pre-test. However, not all of the
students took the post-test and thus the sample was reduced to 63 students.
Out of these 63 students, 82.6% were Spanish students and 17.4% were
foreign students in an Erasmus exchange. All of the Spanish students were
bilingual (Catalan-Spanish). The Erasmus students came from Germany,
France, Italy, Poland, the Netherlands, Denmark, Portugal and Slovenia.
Erasmus students were initially separated in a subgroup but since their
average pre-test results did not differ from those of Spanish students, they
were merged together with the latter in the same group.
The participants were distributed in six 15-week, 4 hour/week courses
(which is a total of 60 hours). Students followed the same degree programme
and had the same number of CLIL lectures.

Instruments
The test (Oxford Placement Test, OPT) consisted of a listening subtest and a
grammar subtest. Two equivalent forms (Allan 2005a, b) were used as pre-test
(first week of the semester) and post-test (last week of the semester),
respectively.
2013 John Wiley & Sons Ltd

Marta Aguilar and Carmen Muoz

Results
The results obtained by the whole group on the two subtests of the OPT at
both times (pre-test and post-test) are displayed in Table 1. The description of
the scores obtained at the beginning and at the end of the CLIL course shows
that the mean score in both the listening test and the grammar test is slightly
higher at the post-test. A paired-sampled t-test showed that the difference
between the mean scores in the listening test was significant, with a small
effect size (t = -3.468, p < 0.01, d = 0.30). In contrast, the comparison of the
mean scores in the grammar test by means also of a paired-sampled t-test
showed that the gains experienced by these learners during the course were
too small to be statistically significant.
In order to answer the second research question concerning the role played
by proficiency in the language gains, students were distributed into three groups
of similar size on the basis of their scores at the pre-test, both on the listening test
and on the grammar test. For the listening test, the lowest level groups scores
ranged from 53 to 68, the intermediate group scored between 70 and 77, and the
third group ranged in scores between 79 and 92. For the grammar test, the lowest
level group scored between 37 and 60, the intermediate group between 61 and 72,
and the more advanced between 74 and 96. The respective sub-groups were
separately entered in the analyses that explored the effects of proficiency on
listening and grammar skills, respectively.
First, a repeated measures ANOVA was conducted with the scores at Time
1 and at Time 2 in the listening test as the within-subjects factor, and the
proficiency group with three levels as the between-subjects factor. Table 2
displays the descriptive statistics (mean and SD) of the three groups into
which the sample was divided. While group 1 and group 2 can be observed to
experience some gains, higher in the former than in the latter, group 3 scores
slightly lower at Time 2.
The tests of within-subjects effects show a significant main effect for the
variable Time with a medium effect size (F1,60 = 13.78, p = 0.000, partial eta
squared = 0.19, power = 0.95). The interaction between the variable time and
the variable proficiency is also significant and with a large effect size (F2,60 =
7.9, p = 0.006, partial eta squared = 0.16, power = 0.85). This result indicates that
participants from the different proficiency groups did not obtain the same

Table 1. Descriptive statistics

Pre-listening
Pre-grammar
Post listening
Post-grammar

Minimum

Maximum

Mean

Std. deviation

63
63
63
63

53
37
59
36

92
96
97
96

72,75
66,30
75,57
67,16

9,670
14,166
9,129
12,746

2013 John Wiley & Sons Ltd

The effect of prociency on CLIL benets

Table 2. Descriptive statistics: listening scores

Pre-listening
Total
Post-listening

Group

Mean

Std. Deviation

1
2
3

62,77
72,24
84,25
72,75
68,41
75,33
83,70
75,57

4,231
2,221
5,046
9,670
5,124
6,256
8,529
9,129

22
21
20
63
22
21
20
63

1
2
3

Total

90

groups
1
2
3

Estimated Marginal Means

85

80

75

70

65

60
1

Time
Figure 1. Evolution of the three prociency groups in listening skills

2013 John Wiley & Sons Ltd

10

Marta Aguilar and Carmen Muoz

90

groups
1
2
3

Estimated Marginal Means

80

70

60

50
1

Time
Figure 2. Evolution of the three prociency groups in grammar skills

gains in listening skills after the course. Pairwise comparisons among the
estimated marginal means show that the effects of time are significant for
group 1, with the lowest level of proficiency (F = 23.70, p = 0.000, partial eta
squared = 0.26, power = 0.998), marginally significant for group 2 (F = 3.58, p
= 0.063, partial eta squared = 0.05, power = 0.46), and non-significant for group
3. Figure 1 shows the evolution of the three proficiency groups in listening
scores from Time 1 to Time 2.
A repeated measures Anova was conducted with the scores from the
grammar tests at time 1 and time 2, as above. Table 3 displays the descriptive
statistics (mean and SD) of the three groups into which the sample was
divided. Groups 1 and 2 experience some gains, larger in the former than in
the latter, but in group 3 the mean score at the pre-test is higher than at the
post-test.
The tests of within-subjects effects show no main effect for the variable
Time, indicating again that participants gains were not statistically significant.
However, a significant interaction is shown between proficiency group and
2013 John Wiley & Sons Ltd

The effect of prociency on CLIL benets

11

Table 3. Descriptive statistics: grammar scores

Pre-grammar
Total
Post-grammar

Group

Mean

Std. Deviation

1
2
3

51,05
66,65
81,95
66,30
55,82
66,95
79,24
67,16

7,410
3,329
6,368
14,166
7,391
8,587
9,027
12,746

22
20
21
63
22
20
21
63

1
2
3

Total

time with a large effect size (F2,60 = 7.9, p = 0.001, partial eta squared = 0.21,
power = 0.94). This result indicates that participants from the different
proficiency groups did not obtain the same gains in grammar after the course.
Pairwise comparisons among the estimated marginal means show that the
effects of time are significant for group 1, with the lowest level of proficiency
(F = 12.97, p = 0.001, partial eta squared = 0.18, power = 0.94), non-significant
for group 2, and significant for group 3, with a medium effect size (F = -2.7,
p = 0.05, partial eta squared = 0.06, power = 0.50). In other words, whereas the
gains in group 1 are significant, so are the losses in group 3, and no significant
changes are observed in group 2. Figure 2 shows the evolution of the three
proficiency groups in grammar scores from Time 1 to Time 2.

Discussion
The first research question of this study inquired into the gains in English
language proficiency obtained by students that followed a 60-hour course
taught in English at an engineering school. In order to answer this question,
participants scores were compared before and after the course by means of a
listening test and a grammar test. Results show that there was a small
improvement in test performance.
However, and in relation to the second research question specifically,
only gains in listening skills reached statistical significance, whereas the
improvement in grammar skills was too small. Therefore, the finding
concerning gains in listening skills provides confirmation that this is an area
in which students may benefit from a CLIL experience. The evidence
regarding improvement in listening skills is also in line with university
students self-perception of improvement in listening comprehension, as
shown in Muoz (2001) with students in an English degree, and in Aguilar
and Rodrguez (2012) with a group of Engineering students that included
participants in this study.
The third research question inquired about the effects of students
proficiency on their language gains after following the CLIL course. The
2013 John Wiley & Sons Ltd

12

Marta Aguilar and Carmen Muoz

findings in this study clearly revealed an effect of proficiency. Students with

an advanced initial level of proficiency in English (OPT Band 6) did not
experience any gains after the CLIL experience in either listening or grammar
skills, as measured by the standardized tests used in this study. Further, the
decrease in grammar scores is significant, indicating that those participants
who were advanced users of English at the beginning of the course performed
worse at Time 2. In contrast, participants with the lowest initial level of
proficiency in English (elementary level or OPT Band 3) obtained significant
gains in both listening and grammar skills, which indicates that they
benefited from following the CLIL course. As for the participants with an
upper intermediate level (OPT Band 5), their evolution in listening skills is
positive although it does not reach statistical significance, and their evolution
in grammar skills is extremely small.
This study has revealed the important role played by students proficiency
in university CLIL courses because the CLIL experience was more
advantageous for less proficient students, hinting at the time horizon required
to advance proficiency levels (less proficient students show greater gains in
shorter periods than more proficient students). Previous research has focused
on the minimum initial proficiency level needed for learners to benefit from
university immersion or partial immersion programmes (Klee and Tedick
1997; Lynch et al. 2001). Specifically in the context of adult advanced-level
university immersion, an initial proficiency level of Intermediate High
(ACTFL guidelines) has been suggested as the minimum optimal level for
predicting target language proficiency growth (Klee and Tedick 1997; Lynch
et al. 2001). However, in this study it has been shown that elementary level
students benefit more from the CLIL experience provided by just one 60-hour
course than those students with an Upper Intermediate or Advanced level on
the OPT. This discrepancy may be attributed to the participants different
characteristics in the two settings. One may hypothesize that the absence of
gains in the more proficient learners in the present study may be attributable
to the characteristics of the input received in class, probably more limited than
the input received by students in the university immersion programmes
reported above. As mentioned earlier, teaching materials (powerpoint
presentations) had been translated into English for the teachers, who in some
cases may have restricted their oral explanations to reading that material, thus
providing a poor linguistic environment. No adaptation along the lines of
Coyle (2007) was made.
In fact, the lecturers in this study had a relatively limited proficiency.
Scores in the OPT they also took at the beginning of term placed them in four
different bands: two in OPT Band 4 (lower intermediate), three in OPT Band
5 (upper intermediate), one in OPT Band 6 (advanced), and one in Band 7
(very advanced). This is lower than, for example, the recommended prerequisite for university teaching assistants in many US universities, which
is advanced plus or superior (Magnan 1986). If lecturers proficiency is
so limited that they cannot provide enriched input, students language
2013 John Wiley & Sons Ltd

The effect of prociency on CLIL benets

13

improvement may be endangered (Sercu 2004; Lasagabaster 2008). It may be

argued that for those students whose proficiency level was higher than their
teachers, the experience may not have led to much improvement, as in the
case of the advanced students grammar. In that context, CLIL benefits may be
restricted to students with a level of proficiency below the intermediate levels.
However, research into different measures of English proficiency other than
the OPT, with a greater sample of students, and with longer amounts of CLIL
exposure is necessary to corroborate this hypothesis.
When asked about the quality of delivered content, the lecturers in this
study stated that the same amount of topics had been covered in their English
group as in the Spanish groups and that the failure rate in final exams did not
vary significantly from that of the latter. It would be interesting to study the
quality of content and types of examinations in depth. Informal interviews
with some of the lecturers revealed that exams were essentially numerical or
following a multiple-choice format. Such formats may not display the deep
understanding of content as, for example, an exam requiring students to
explain the rich complexity of content in length and in depth. Research has
compared lecturers behaviour and experiences before and after receiving
CLIL teacher training (Klaassen 2008), but it would also be interesting to
compare the effects of lecturer training on students gains, and on differentability students gains.
However, and in spite of the shortcomings of the courses in this study, less
proficient students have been observed to benefit from the experience. It may
be argued that if benefits emerge even from a non-articulated CLIL
programme as the one in this study (and with not highly proficient lecturers)
maybe even better results could be expected if students followed a carefully
designed and planned CLIL course with a special emphasis on language
learning (Prez-Vidal 2007; Navs 2011). In relation to the implementation of
CLIL courses at university level, further comparative research between
university approaches like CLIL, EMI and CBI comparing the advantages of
each learning context could provide interesting insight.
This study has a number of limitations. One limitation is that the groups
were natural groups and there was no control of students allocation to
teachers. Although teachers were asked about their methodology in class
(mostly teacher-fronted lecturing with some variability in the opportunities
created for students oral contribution), classroom observations were not
possible and hence the actual use of the target language by students in class
remains unknown. A second limitation concerns the fact that students were
self-selected, which indicates a degree of motivation that cannot be assumed
to be generalizable to the whole population (Airey 2004). A third caveat is
the short length of the experience, since the period of 15 weeks per semester
may prove to be too short. It has been shown that, for example, two to three
months of university-level instruction provide negligible change in the
analysed syntactic complexity measures of L2 writing (Ortega 2003). Finally,
while this study has yielded interesting findings concerning the listening
2013 John Wiley & Sons Ltd

14

Marta Aguilar and Carmen Muoz

comprehension skills of university students as well as their performance on

a grammar test, further research should also examine how university
students with different entry proficiency levels benefit from CLIL, EMI, and
other instruction types, which should always be embedded in a robust
contextualized framework with clear aims and projected outcomes (Coyle
2007: 546).

Note
1. Oral Proficiency Interviews, following the American Council on the Teaching of
Foreign Languages (ACTFL) guidelines.

References
Aguilar, M. and R. Rodrguez (2012) Implementing CLIL at a Spanish university.
Lecturer and student perceptions. The International Journal of Bilingual Education
and Bilingualism 15.2: 18397.
Agustn Llach, M. P. (2009) The role of Spanish L1 in the vocabulary use of CLIL and
non-CLIL EFL learners. In Y. Ruiz de Zarobe and R. M. Jimnez Cataln (eds.),
Content and language integrated learning. Evidence from research in Europe. Bristol:
Multilingual Matters. 11229.
Airey, J. (2004) Can you teach it in English? Aspects of the language choice debate in
Swedish higher education. In R. Wilkinson (ed.), Integrating content and language.
Meeting the challenge of a multilingual education. Maastricht: Maastricht University.
97108.
(2010) The ability of students to explain science concepts in two languages. Journal
of Language and Communication Studies 45.1: 3549.
and C. Linder (2006) Language and the experience of learning university physics in
Sweden. European Journal of Physics 27.3: 55360.
(2009) A Disciplinary discourse perspective on university science learning:
achieving fluency in a critical constellation of modes. Journal of Research in Science
Teaching 40.1: 2749.
Allan, D. (2005a) Oxford placement test 1. Oxford: Oxford University Press.
(2005b) Oxford placement test 2. Oxford: Oxford University Press.
Bonnet A. (2012) Towards an evidence base for CLIL. International CLIL Research Journal
1.4: 6678.
Brinton, D., M. Snow and M. Wesche (1989) Content-based second language instruction.
Rowley, MA: Newbury House.
Celaya M. L. and Y. Ruiz de Zarobe (2010) First languages and age in CLIL and
non-CLIL contexts. International CLIL Research Journal 1.3: 606.
Coyle, D. (2007) Content and language integrated learning: towards a connected
research agenda for CLIL pedagogies. The International Journal of Bilingual
Education and Bilingualism 10.5: 54362.
Dafouz, E. (2011) English as the medium of instruction in Spanish contexts: a look
at teacher discourses. In Y. Ruiz de Zarobe and R. M. Jimnez Cataln (eds.),
Content and Foreign Language Integrated Learning. Contributions to Multilingualism in
European Context. Bern: Peter Lang. 189210.
2013 John Wiley & Sons Ltd

The effect of prociency on CLIL benets

15

and B. Nez (2010) Metadiscursive devices in university lectures: a contrastive

analysis of L1 and L2 teacher performance. In C. Dalton-Puffer, T. Nikula and
U. Smit (eds.), Language use and language learning in CLIL classrooms. Amsterdam:
John Benjamins. 21332.
Dalton-Puffer, C. (2007) Discourse in content and language integrated (CLIL) Classrooms.
Amsterdam: John Benjamins.
T. Nikula and U. Smit (eds.) (2010) Language use and language learning in CLIL
Classrooms. Amsterdam: John Benjamins.
DeKeyser, R. (2010) Monitoring processes in Spanish as a second language during a
stay abroad program. Foreign Language Annals 43.1: 8092.
Doiz, A., D. Lasagabaster and J.M. Sierra (2011) Internationalisation, multilingualism
and English-medium instruction. World Englishes 30.3: 34559.
Fortanet-Gmez, I. and C. Risnen (eds.) (2008) ESP in European higher education.
Integrating language and content. Amsterdam: John Benjamins.
Garner, M. and E. Borg (2005) An ecological perspective on content-based instruction.
Journal of English for Academic Purposes 4.2: 119134.
Greere, A. and A. Rsnen (2008) Lanqua subproject on content and language integrated
learning. Redefining CLIL. Towards multilingual competence. Year One Report.
Retrieved July 2012 from (http://www.lanqua.eu/files/Year1Report_CLIL_
ForUpload_WithoutAppendices_0.pdf ).
Grin, F. (2005) Added value of CLIL. Paper presented at The Changing European
Classroom. The Potential of Plurilingual Education Conference. Luxembourg, 911
March.
Hellekjaer, G. O. (2010) Language matters: assessing lecture comprehension in
Norwegian English-medium education. In C. Dalton-Puffer, T. Nikula and U. Smit
(eds.), Language use and language learning in CLIL classrooms. Amsterdam: John
Benjamins. 23358.
Hincks, R. (2010) Speaking rate and information content in English lingua franca oral
presentations. English for Specific Purposes 20.1: 418.
Httner, J. and A. Rieder-Bnemann (2010) A cross-sectional analysis of oral narratives
by children with CLIL and Non-CLIL instruction. In C. Dalton-Puffer, T. Nikula
and U. Smit (eds.), Language use and language learning in CLIL classrooms.
Amsterdam: John Benjamins. 6179.
Jexenflicker, S. and C. Dalton-Puffer (2010) The CLIL differential: comparing the
writing of CLIL and non-CLIL students in higher colleges of technology. In C.
Dalton-Puffer, T. Nikula and U. Smit (eds.), Language use and language learning in
CLIL classrooms. Amsterdam: John Benjamins. 16989.
Jimnez Cataln, R.M. and Y. Ruiz de Zarobe (2007) Does the type of instruction have
any bearing on EFL learners receptive vocabulary? Paper presented at the ELIA 10
Conference. Issues in teaching, learning, and using vocabulary in an L2. University
of Seville, Spain.
Jochems, W. (1991) Effects of learning and teaching in a foreign language. European
Journal of Engineering Education 16.4: 30916.
Kasper, L. F. (1997) The impact of content-based instructional programs on
the academic progress of ESL students. English for Specific Purposes 16.4: 309
20.
Klaassen, R. (2008). Preparing lecturers for English-medium instruction. In R.
Wilkinson and V. Zegers (eds.), Realizing content and language integration in higher
education. Maastricht: Maastricht University: 3242.
2013 John Wiley & Sons Ltd

16

Marta Aguilar and Carmen Muoz

Klee, C. and D. Tedick (1997) The undergraduate foreign language immersion program
in Spanish at the University of Minnesotta. In S. B. Stryker and B. L. Leaver (eds.),
Content-based instruction in foreign language education. Washington, DC: Georgetown
University Press. 14173.
Lasagabaster, D. (2008). Foreign language competence in content and language
integrated courses. The Open Applied Linguistics Journal 1.1: 3142.
Lorenzo, F. (2008) Instructional discourse in bilingual settings. An empirical study of
linguistic adjustments in content and language integrated learning. Learning
Language Journal 36.1: 2133.
Lynch, A., C. A. Klee and D .J. Tedick (2001) Social factors and language proficiency
in postsecondary Spanish immersion: issues and implications. Hispania 84.3:
51024.
Magnan, S. S. (1986) Assessing speaking proficiency in the undergraduate curriculum:
data from French. Foreign Language Annals 19.5: 42938.
Mrquez, M. (2007) Models of CLIL: an evaluation of its status drawing on the German
experience. A critical report on the limits of reality and perspectives. RESLA.
Volumen monogrfico: 95111.
Muoz, C. (2001) The use of the target language as the medium of instruction.
University students perceptions. Anuari de Filologia 23.1: 7182.
(2006) The effects of age on foreign language learning: the BAF project. In C. Muoz
(ed.), Age and the rate of foreign language learning. Clevedon: Multilingual Matters.
140.
(2010) Challenges of bilingual education in Spain: CLIL from the perspective of
language acquisition. Plenary talk given at SEDLL (Sociedad Espaola de Didctica
de la Lengua y la Literatura). Universidad de Jan, December.
Navs, T. (2011) How promising are the results of integrating content and language
for EFL writing and overall EFL proficiency? In Y. Ruiz de Zarobe, J. M. Sierra
and F. Gallardo del Puerto (eds.) Content and foreign language integrated
learning. Contributions to multilingualism in European context. Bern: Peter Lang.
15586.
and M. Victori (2010) CLIL in Catalonia: an overview of research studies. In D.
Lasagabaster and Y. Ruiz de Zarobe (eds.), CLIL in Spain. Implementation, results and
teacher training. Newcastle upon Tyne: Cambridge Scholars Publishing. 3054.
Nikula, T. (2010) Effects of CLIL on a teachers classroom language use. In C. DaltonPuffer, T. Nikula and U. Smit (eds.), Language use and language learning in CLIL
classrooms. Amsterdam: John Benjamins. 10521.
Ortega, L. (2003) Syntactic complexity measures and their relationship to L2
proficiency: a research synthesis of college-level L2 writing. Applied Linguistics
24.4: 492518.
Pecorari, D., P. Shaw, A. Irvine and H. Malmstrm. (2011a) English for academic
Purposes at Swedish universities: teachers objectives and practices. Ibrica 22.1:
5577.
H. Malmstrm and A. Irvine. (2011b) English textbooks in parallel-language
tertiary education. TESOL Quarterly 45.2: 31333.
Prez-Vidal, C. (2007) The need for focus on form (FoF) in content and language
integrated approaches. An exploratory study. RESLA Volumen monogrfico:
3954.
(2011) Language acquisition in three different contexts of learning: formal
instruction, stay abroad, and semi-immersion (CLIL). In Y. Ruiz de Zarobe,
2013 John Wiley & Sons Ltd

The effect of prociency on CLIL benets

17

J. M. Sierra and F. Gallardo del Puerto (eds.) Content and foreign language integrated
learning. Contributions to multilingualism in European context. Bern: Peter Lang.
10327
Rodgers, D. (2006) Developing content and form: encouraging evidence from Italian
content-based instruction. The Modern Language Journal 90.3: 37386.
Ruiz de Zarobe, Y. (2010) Written production and CLIL. In C. Dalton-Puffer, T. Nikula
and U. Smit (eds.), Language use and language learning in CLIL classrooms.
Amsterdam: John Benjamins. 191209.
(2011) Which language competencies benefit from CLIL. In Y. Ruiz de Zarobe, J. M.
Sierra and F. Gallardo del Puerto (eds.) Content and Foreign Language Integrated
Learning. Contributions to Multilingualism in European Context. Bern: Peter Lang.
12953.
and R. M. Jimnez Cataln (eds.) (2009) Content and language integrated learning.
Evidence from research in Europe. Bristol: Multilingual Matters.
and D. Lasagabaster (2010) CLIL in a bilingual community: the Basque autonomous
community. In D. Lasagabaster and Y. Ruiz de Zarobe (eds.), CLIL in Spain.
Implementation, results and teacher training. Newcastle upon Tyne: Cambridge
Scholars Publishing. 1229.
J. M. Sierra and F. Gallardo del Puerto (eds.) (2011) Content and foreign language
integrated learning. Contributions to multilingualism in European context. Bern: Peter
Lang.
Seikkula-Leino, J. (2007) CLIL learning: achievement levels and affective factors.
Language and Education 21.4: 32841.
Sercu L. (2004) The introduction of English-medium instruction in universities.
A comparison of Flemish lecturers and students language skills, perceptions
and attitudes. In R. Wilkinson (ed.), Integrating content and language. Meeting
the challenge of a multilingual education. Maastricht: Maastricht University. 54755.
Shaw, P., C. Benson, S. Brunsberg, R. Druhs and D. Minugh (2008). International
masters in Stockholm. In I. Fortanet-Gmez and C. A. Risnen (eds.), ESP in
European higher education. Amsterdam: John Benjamins. 26782.
Song, B. (2006) Content-based instruction: long-term effects and outcomes. English for
Specific Purposes 25.4: 42037.
Swain, M. (1985) Communicative competence: some roles of comprehensible input
and comprehensible output in its development. In S. Gass and C. Madden (eds.),
Input in second language acquisition. Rowley, MA: Newbury House. 23553.
(1995) Three functions of output in second language learning. In G. Cook and B.
Seidholfer (eds.), Principles and practice in applied linguistics: studies in honour of H.G.
Widdowson. Oxford: Oxford University Press. 12544.
and S. Lapkin (1995) Problems in output and the cognitive processes they
generate: a step toward second language learning. Applied Linguistics 16.3: 371
91.
Tatzl, D. (2011) English-medium masters programmes at an Austrian university of
applied sciences: attitudes, experiences and challenges. Journal of English for
Academic purposes 10.4: 25270.
Thgersen, J. and J. Airey (2011) Lecturing undergraduate science in Danish and in
English: a comparison of speaking rate and rhetorical style. English for Specific
Purposes 30.3: 20921.
Vrkuti, A. (2010) Linguistic benefits of the CLIL approach. International CLIL Research
Journal 1.3: 6779.
2013 John Wiley & Sons Ltd

18

Marta Aguilar and Carmen Muoz

Villarreal, I. (2011) Tense and agreement in the non-native English of Basque-Spanish

bilinguals: content and language integrated learners vs. English as a school subject
learners. Doctoral Dissertation, Universidad del Pas Vasco.
Vinke, A., J. Snippe and W. Jochems (1998) English-medium content courses in nonEnglish higher education: a study of lecturer experiences and teaching behaviours.
Teaching in Higher Education 3.3: 383394.
Wchter, B. and F. Maiworm (2008) English-taught programmes in European higher
education. The picture in 2007. Bonn: Lemmens.
Wilkinson, R. and V. Zegers (eds.) (2008) Realizing content and language integration in
higher education. Maastricht: Maastricht University.
email: marta.aguilar@upc.edu
munoz@ub.edu

[Received 7 October 2012]

2013 John Wiley & Sons Ltd