Royce, T. (2007). Multimodal communicative. Competence in second language contexts. In T. Royce & W. Bowcher (Eds.

), New directions in the analysis of multimodal discourse (pp. 361-390). New Jersey: Lawrence Erlbaum Associates, Inc.

Chapter

12

Multimodal Communicative Competence in Second Language Contexts

Terry D. Royce
Teachers College, Columbia University

Recent discussions on multiliteracies have focused on the future of literacy teaching in the wider global village, with little concentration on multiliteracy in second and foreign language contexts. Notable exceptions to this are discussions by Lo Bianco (2000), Stenglin and Iedema (2001), and Royce (2002). Lo Bianco (2000), as part of the Multiliteracies Project by the New London Group, discusses the role of multiliteracies and personalsocietal multilingualism, and considers the effects of globalization on multiliteracy practices in multicultural contexts, suggesting a need to create a metalanguage to unite disparate areas of communication and representation, multimodally as well as multiculturally, into a new pedagogy (p. 99). Stenglin and Iedema (2001) address the necessity for TESOL (Teaching English to Speakers of Other Languages) teachers to understand and systematically use visual analysis tools in their classrooms to help students to read visuals and to develop in-class teaching materials/techniques to facilitate that process (p. 195). They propose three sets of tools which can be used to analyze images, and make the point that the knowledge of how to analyze visuals is crucial to students understanding of how meanings are made in multi-modal texts (p. 207). Royce (2002), via an analysis of a multimodal text extracted from an introductory environmental science textbook, examines some of the ways that TESOL professionals can explore with their students the copresence of visual and linguistic modes in their textbooks, and suggests that teachers should be increasingly concerned with developing their students multi361

LEATHE TYPE HOUSENEW DIRECTIONS IN THE ANALYSIS OF MULTIMEDIA DISCOURSE (ROYCE/BOWCHER)

362

ROYCE

modal communicative competence as a result of the technologizing of modes of communication (p. 192). As the New London Group (2000) suggests, one of the central missions of education is to equip students with the tools to participate fully in public, communal, and commercial life, and for literacy pedagogy specifically this has traditionally meant teaching and learning to read and write in page-bound, official, standard forms of the national languagein other words a restriction to formalized, monolingual, monocultural, and rulegoverned forms of language (p. 9). The New London Group argues for a broadening of this understanding of literacy to one which focuses on a multiplicity of discourses, with two central aspects of concern: The first concern is to extend the idea and scope of literacy pedagogy to account for the context of our culturally and linguistically diverse and increasingly globalized societies, [and] to account for the multifarious cultures that interrelate and the plurality of the texts that circulate (p. 9). The second is the need for literacy pedagogy to account for the burgeoning variety of text forms associated with information and multimedia technologies, as well as the proliferation of communication channels and media [which] supports and extends cultural and subcultural diversity (p. 9). This applies to all educational spheres where appropriate multiliteracy pedagogies need to be developed, whether they are largely monolingual, or multilingual. The second-language classroom is no less a source of multimodal meanings than the first language classroom, particularly, with the increase in attention to and provision of computer-assisted language learning and media-based teachinglearning materials and methodologies. Although visuals have often been used as a basis for various teaching techniques to stimulate discussion or build vocabulary, or to encourage students to use their social knowledge to generate predictions about the [often accompanying] written text (Stenglin & Iedema, 2001, p. 195), little attention has been paid to visuals as socially and culturally constructed products which have a culturally specific grammar of their own (p. 194). Furthermore, competence in the second language, being the target, has naturally taken primacy in the second-language classroom, but this has been at the expense of any real attention being paid to the interrelationship between language and other semiotic systems. Rather, competence in a second language, or communicative competence as it has come to be called, has been the primary focus, often to the exclusion of other modes of meaning (Royce, 2002). In this chapter I argue for an extension of communicative competence beyond its traditional (and narrow) linguistic view, to one which incorporates a recognition of the need to focus on multimodal literacy.

12.

MULTIMODAL COMMUNICATIVE COMPETENCE

363

THE BACKGROUND: COMMUNICATIVE COMPETENCE AND L2 CONTEXTS Communicative competence (hereafter CC), as proposed by Dell Hymes (1972) in his seminal article, On Communicative Competence, asserts that speakers of a language need to have more than just grammatical knowledge to be able to communicate effectively in a language; they also need to have knowledge of how language is used by the members of a speech community to enact social purposes. They need the ability to use speech appropriately in varying social contextsthey should know what to say, to whom they should say it, and the way to say it. Hymes referred to rules of use which enable actual speakers to use language effectively for communication, and proposed four criteria for this knowledge of use. The first of these is whether and the degree to which something is formally (grammatically) possible. Most view this in linguistic terms, but Hymes widened this to include nonverbal and cultural grammaticality, which incorporates meaningful rules of behavior (pp. 284285). The second criterion refers to whether and the degree to which a language instance can be feasibly implemented. This refers to psycholinguistic factors to do with memory limitation, perceptual device(s), effects of properties such as nesting, embedding, branching and the like, and relates to the feasibility of processing, which declines the greater and longer the input produced becomes (p. 285). For example, a long sentence with multiple embedded clauses may well be grammatical, but it may be too long to be successfully processed by a listener or reader. The third criterion relates to whether and the degree to which something is appropriate in terms of its context, both in terms of the immediate context of the communication event, and in terms of the wider culture (with the implication that the appropriateness is not a binary choice of appropriate/not appropriate, but is a question of position along a continuum of appropriacy). The fourth criterion is whether and the degree to which something is in fact done, and what its performance involves. As Hymes (1972) suggests, something may be possible, feasible, appropriate and not occur (p. 286). Hymes also assumed that language users have knowledge of which language forms occur most commonly, as well as some sense of the probability of occurrence. Since Hymes article there has been much discussion and refinement of this concept of CC, and it has been applied in many educational spheres. An important and influential reformulation and critique of CC by Canale and Swain (1980, pp. 2931), presents a three-part model consisting of grammatical competence [knowledge of the language code], strategic competence [knowledge of linguistic and nonlinguistic ways to deal with communication breakdowns etc.], and sociolinguistic competence, which is fur-

364

ROYCE

ther classified into two sets of rules: sociocultural rules of use [knowledge of the relation of language use to its nonlinguistic context] and rules of discourse [knowledge of the ways that combinations of utterances and communicative functions are organized to create coherent communication]. Subsequent reformulations of CC have addressed possible confusion with the competence [knowledge]/performance aspect of this notionthe question of whether one is referring to static intrapersonal knowledge or dynamic interpersonal skill when discussing for example the knowledge involved in strategic competence, which implies some ability for use ( Johnson & Johnson, 1998, pp. 6667). The application of CC to the teaching of English as a second or foreign language can be traced to Savignon (1972), who initially defined CC as the ability to function in a truly communicative settingthat is, in a dynamic exchange in which linguistic competence must adapt itself to the total informational input, both linguistic and paralinguistic, of one or more interlocutors (p. 8). The literature on developing CC in L2 contexts focuses on areas that are commonly categorized under the generic headings of linguistic and pragmatic competencies. These are summarized in Table 12.1 (Richards & Schmidt, 2002, pp. 9091; Savignon, 1983, pp. 3548). These CCs are typically interpreted as being both productive and receptive in nature, and are focused around listening, reading, speaking and writing skills and how these are used to send and receive messages in interaction. In L2 contexts the competencies are often discussed in terms of communicative language teaching and communicative methodology. Communicative language teaching is basically viewed as an APPROACH to teaching a second or foreign language (Richards & Rodgers, 2001, pp. 153177), while communicative methodology is concerned with the classroom techniques developed by teachers who adopt a communicative language teaching approach. Since the mid-1970s there have been many discussions of what constitutes communicative methodology, but overall there seems to be one standard view which is characterized by a view of language rather than a view of language learning (although some argue for a more cognitive idea of the methodology, a view which incorporates an information-processing approach for more discussion of this see Richards & Rodgers, 2001, pp. 153177). The view of language adopted in communicative methodology can of course be traced to Hymes seminal discussions of CC, but perhaps the most influence has been the functional view of language taken by British applied linguists since the 1970s. From this perspective, language is viewed as an instrument for enacting or realizing social activityor language as social semiotic (Halliday, 1978). In communicative methodology much emphasis is placed upon achieving MESSAGE-FOCUS in teaching methods as opposed to the more traditional FORM-FOCUS. In message-focus emphasis is placed on using lan-

TABLE 12.1 Linguistic and Pragmatic Competencies in CC

Linguistic Competency Competency Phonological Orthographic Definition The ability to recognize and produce the sounds of a language. The ability to decipher and use the writing system. The ability to effectively recognize, produce and use the grammatical structures of a language. The ability to recognize and use vocabulary in a language appropriately, as well as have an understanding of word families and collocational relationships. Textual: at the level of generic structure, the ability to comprehend and compose texts which realize different genres. Coherence : the ability to use the various textual features which operate to make the text coherent, and others which can be used to emphasize certain points in the genre. Pragmatic Competency Competency Functional Sociolinguistic Definition The ability to use a language to perform some task or action. The ability to interpret what is happening in social terms through the linguistic varieties being selected by the interlocutors and to respond to and produce appropriate language for that situation. The knowledge of and ability to use the interactional rules assumed for various communication situations by a specified speech community and culture. The ability to comprehend how the members of a particular culture behave with each other, and to interact with them in acceptable (and recognizable) ways. Examples Ask directions, make greetings etc. The appropriate use of formal/informal registers. Examples Consonants/vowels, intonation, rhythm, and stress patterns etc. Graphological script and formatting uses such as bolding, italics, CAPS etc. Tense and aspect, Mood, word classes etc. Morphology, spelling, topic-based vocabulary etc.

Grammatical

Discourse

Genres such as descriptions, narratives, expositions, reports, etc. Reference, substitution, ellipsis, discourse markers, etc.

Interactional

Conversational management skills (e.g. turn-taking rules, repair, fillers etc.), non-verbal cues (gestures, eye contact, interpersonal space). A general appreciation of a cultures social structure, the way of life espoused, and the typical rules which govern how society is organized.

Cultural

365

366

ROYCE

guage skills to produce and receive what Widdowson (1978) refers to as use (language units as carriers of messages or meanings to someone for some purpose in some context), and not the form-focus of usage (reproducible and correct sounds and grammatical structures). Probably the most well-known classroom exercise derived from this functional view is the information-gap exercise, where students are given two separate and different pieces of information which together provide a complete whole; for example, two incomplete maps where the students must interact with each other to complete the missing information on their respective maps. The language learners in that situation must both use and receive language to find the needed informationthey are also placed in a situation where a need to use language for problem solving is created, rather than using language in the traditional presentation-practice-production sequence of structural teaching methods. In addition to this messagefocus, communicative methodology also emphasizes the appropriateness of usage, the simulation of the psychological processes involved in problem solving and risk taking, and opportunities for free-practice of the language ( Johnson & Johnson, 1998, pp. 6972).

THE LINGUISTIC AND THE VISUAL: TWO VIEWS Most language teaching professionals would of course maintain that they are concerned with developing their students communicative abilities, and that those abilities would be primarily linguistic in nature. Language teacher education programs world-wide reflect this emphasis. However, given the changes in communication modes and conventions in recent years, language teaching professionals need to be increasingly concerned with developing students multimodal communicative competence. It is not enough, to meet students needs, to focus only on language; teachers should begin to focus on and develop students abilities in visual literacy, and to develop a pedagogical metalanguage to facilitate these abilities when images cooccur with spoken and written modes. Furthermore, as the discussion by Ferreira (chap. 10, this volume) demonstrates, and as Stenglin and Iedema (2001) have suggested, visuals should be interpreted as socially and culturally constructed products which have a culturally specific grammar of their own (p. 194). This view has implications for the second-language classroom in that it recognizes that images (and in fact any kind of semiotic coding used within a cultural sphere) are culturally bound in the sense that what makes sense in one culture may not in another, or it may be differently framed. Additionally, the way in which language interrelates with other semiotic systems differs across cultures, and can be a rich source of detail for language teaching. This means that an approach to communication

12.

MULTIMODAL COMMUNICATIVE COMPETENCE

367

which takes these points into account ultimately provides the doorway to the target culture; thus the ways in which multiple modes pattern in combination would seem to be fertile ground for even understanding the social practices and ideologies of the target culture. This would indeed be a new direction for the TESOL profession to explore, and could open up more effective ways to meet students emerging needs. I would like to suggest therefore that language teaching professionals move away from a primary focus on CC as it relates to linguistic communication, to a more developed view which focuses on multimodal CC. As already mentioned, CC has been characterized by a view of language that has been derived from Hymes insights and by a social semiotic view of language. I propose that in the same ways that a social semiotic view of language has been extended to develop a grammar of the visual (Kress & van Leeuwen, 1996; OToole, 1994), a systemic functional (hereafter SFL) view of language can also be extended to explain the multimodal competencies needed for second-language contexts, and can be used to develop a metalanguage for multimodality which language teachers can use in designing pedagogical programs for their students. In an SFL view of language, the term social suggests two things simultaneously: first it refers to the social system, which is synonymous with culture as a system of social meanings, and second, it refers to the dialectical relationship between communication (language) and social structure. The SFL perspective therefore involves an attempt to relate language primarily to one particular aspect of human experience, namely that of social structure (Halliday & Hasan, 1985, p. 4). This relationship between language and social situation implies that language is viewed as a system of choices or options made against a background of other potential options, and against other ways of communication which human beings have developed over time and in various cultural contexts. Halliday (1978, pp. 16, 21, 2729, 109) makes four central claims about language: 1. Language is functional in terms of what it can do or what can be done with it. 2. Language is semantic in that it is used to make meanings. 3. Language is semiotic in that it is a process of making meanings by selecting from the total set of options that constitute what can be meant (p. 53). 4. The meanings generated and exchanged are motivated by their social and cultural contexts. These claims about language are represented in the tri-stratal model in Fig. 12.1. Here, language is interpreted as a complex semiotic system com-

368

ROYCE

FIG. 12.1. Levels of language and realization in a tri-stratal model. Adapted from Butt, Fahey, Spinks, and Yallon (1995).

posed of multiple LEVELS or STRATA in which the central stratum, the inner core of language, is that of grammar (Halliday, 1994, p. 15). This central stratum is referred to as the LEXICOGRAMMAR, because it incorporates both grammar and vocabulary (p. 15). The key concept used to describe the ways that these strata are related in the overall model is the concept of realization. As Fig. 12.1 shows, the linguistic levels are related to each other in that the level of phonology and graphology realizes the level of the lexicogrammar, and this lexicogrammar itself realizes the level of semantics or meanings, which also realizes the extralinguistic features of the context. Looking at this from the opposite perspective, the extralinguistic features of the context are realized in the choices made in the semantic level, these meanings are realized in choices made in the lexicogrammar, and the lexicogrammar is realized by choices that are made in the soundings and graphology (p. 15). A major strength of the SFL model is that the concept of a text in terms of metafunctional meaning also permits an analysis of semiosis from three

12.

MULTIMODAL COMMUNICATIVE COMPETENCE

369

different metafunctional perspectives, the IDEATIONAL, INTERPERSONAL, and TEXTUAL, with the assumption that an analytical focus on any one necessarily implies that the other two are and should be considered as operating simultaneously. Furthermore, communication involves systems of meanings and the act of communication involves making simultaneous selections from those systems in terms of what is going on (the field of discourse), who is taking part (the tenor of discourse), and the role assigned to language (the mode of discourse), and this suggests that it is a paradigm which can perhaps be usefully applied to other systems of meaning besides language. This applicability has been demonstrated clearly by the work of OToole (1994, 1995) and Kress and van Leeuwen (1996), in which they show that the SFL model can be utilized not only for the sociolinguistic analysis of natural language, but can also offer a powerful and flexible model for the study of other semiotic codes (OToole, 1995, p. 159). An adaptation and application of the SFL model to the visual semiotic is presented in Fig. 12.2. In line with the focus of this discussion on CC in L2

FIG. 12.2. Levels of visual expression and realization in the SFL model. From Royce (1989b, p. 118).

370

ROYCE

contexts, this adaptation and extension is extended to the kinds of visual meanings that students in an L2 educational context will most likely encounter: page and screen-based visual meanings. These meanings typically are realized via commercially produced textbooks, CALL (Computerassisted language learning) materials, and in-house page-based teaching resources. Of course an important aspect of learning to communicate is the use of other nonlinguistic modalities such as gesture, but these can be represented in another way as their visual affordances are different to page or screen-based modalities. At the lowest level of this model of visual expression and realization (which in the SFL or linguistic model is the level of phonology), the term Representational Symbology is used. The assumption here is that each visual instance consists of choices that have been made from the systems of fundamental display elements which are available to the producers of a visual. The term Representational is derived from a common classification used in communication and media studies research. Generally, this research identifies a communication medium as the physical or technical means of converting a MESSAGE into a signal capable of being transmitted along a given CHANNEL (Watson & Hill, 1997, p. 139). It also typically identifies a communication medium as being Presentational, Representational, or Mechanical. Presentational media refers to linguistic features (the voice or the spoken word) and gestural features (the face and body) which are involved in acts of communication, with the person doing the communicating viewed as the medium. Representational media however is concerned with works of communication, where the medium is a book, painting, photograph, or drawing which creates some kind of text that is independent of its author or designer, and it is in this sense that it is directly relevant to an examination of page- or screen-based communication. Mechanical media refers to the physical channel that is used, such as radio, television, film, telephone, and computer media which act as transmitters for the presentational and representational media (Fiske, 1982, p. 18). The visual expression level of Representational Symbology in Fig. 12.2 specifically refers to the various display elements of which a work of visual communication, at its most basic level, is constructed. These are the visual elements which in a sense are the compositional source for all kinds of visual materials and messages and objects and experiences (Dondis, 1973, p. 15). Representational Symbology is therefore concerned with the ways that visual signs and symbols (or works) are produced through the use of various primary display elements, which Dondis (1973, pp. 1516) summarizes as:

the dot; which is the minimal visual unit, pointer or marker of space on the page.

12.

MULTIMODAL COMMUNICATIVE COMPETENCE

371

the line; this can be a fluid restless articulator of form in sketches, or a rigid line which is used to tightly control visual space (as in a technical drawing). shape; this includes the basic geometrical shapes of the circle, square, triangle and their various combinations and dimensional versions. direction; this is the thrust of movement (vectors) which arise from the nature of the various circular, diagonal and perpendicular shapes. tone; the presence or absence of light. color; allied with tone, this element is important for its chromaticity (purity and intensity of hue). texture; the surface characteristics, which can be optical or tactile. scale or proportion; is concerned with salience, or relative size and measurement. dimension and motion; the use of perspective to give a sense of depth, and the use of depth of field in still and moving film.

The artist, craftsperson or graphic designer is thus the visualizer who, through the choices he or she makes, manipulates these basic visual elements to create an intended effect or to project any number of specific messages. In both the visual and linguistic systems, there is plenty of opportunity for creating meaning, for reiterating existing meanings, for generating original meanings; the meaning potential is thus limitless. Like the linguistic system, the visual system relies on a set of intersubjective conventions constrained by a specific relevant context. Like language, any number of existing or new visual messages can be created, and in the same ways that each spoken or written text is an instance of the language system, so too is each visual an instance of the visual system. To paraphrase Halliday (1991, p. 7), the context for this meaning potentialfor visual language as a systemis the context of culture, and the context for the particular instancesfor visual language as processes of textis a context of use. Just as a sketch or a diagram is an instance of visual language, so is a situation of visual representation an instance of culture. Thus, the context for an instance of visual language is an instantiation of choices made constrained by a specific situation, and the context for the system that lies behind each visual is the system which lies behind each situationnamely, the culture. The level of visual grammar, which is characterized as the system of visual design in Fig. 12.2, relates to the ways that the various systems of display elements in Representational Symbology are combined to realize visual message syntagms, or the ways in which visual elements are organized into recognizable structures (Dondis, 1973, pp. 2038). In the same ways that a linguistic grammar combines sounds into words which then combines these into clauses, sentences, and whole texts, a visual-grammar looks at the

372

ROYCE

ways that people, creatures, mythological beings, inanimate objects, and spatial representations (places, scenes, landscapes) have been created by the basic visual elements combining in meaningful ways to produce coherent visual phenomena of varying degrees of complexity. Both language and visual phenomena rely on specific intersubjective sign systems (their respective semiotic systems), and the choices available in these systems are organized in specific ways that make sense to members of a culture. My earlier use of the term meaningful is important in that it is the primary focus of this chapters approach to visual interpretation and its application to the notion of multimodal CC. The aim here is to view instances of visual communication, in line with the SFL view of language, as instances of meaning which are structured according to function, not, as analysts in various structuralist schools of semiotics have generally done, to examine visuals in terms of their isolated elements (and not in the same ways as structural approaches to teaching language have done, to view language as a collection of rules to be learned, usually in isolation from social contexts). As much work on multimodal meaning demonstrates, verbal and visual modes utilize the meaning-making features peculiar to their respective semiotic systems, in the sense that there are some individual meanings which can be expressed only visually, and some which can be expressed only through language. It also recognizes that there are areas where they both share meanings. Like the lexicogrammar however, visual-grammar operates as a means of representing patterns of experience which enables human beings to build a mental picture of reality, to make sense of their experience of what goes on around them and inside them (Halliday, 1994, p. 106). Visual-grammar also works as a means of projecting and exchanging messages, to generate forms of address to potential viewers, and to color those forms of address in modal and attitudinal terms. Additionally, visualgrammar works as a means to project a unified, coherent visual message, to organize the elements of its composition in such a way that the viewers will be able to see how one part of the visual fits with every other part, leading to a sense of visual coherence. In the lexicogrammar of the SFL model, the clause plays a central role in embodying experience, organizing the nature of the exchange, and in sequencing the message. In visual-grammar, meanings are organized into what could be viewed as visual syntagms, which are realized by various arrangements of the core visual display elements outlined before. Thus, in visual-grammar, as there is in lexicogrammar, there are various ways of relating the participants portrayed (through visual Transitivity systems), of relating the viewer and the viewed (through visual Mood and Modality systems), and of relating the elements on a page to each other (through visual composition systems). At the level of the semantics in Fig. 12.2, the metafunctions will be interpreted in similar ways to those used by Kress and van Leeuwen (1996), and

12.

MULTIMODAL COMMUNICATIVE COMPETENCE

373

OToole (1994), and subsequently by Royce (1999a, 1999b, 2002). The IDEATIONAL metafunction is seen as the function of language to represent the goings on in the world. In analyzing visuals the starting point is to identify the represented participants, or all the elements or entities that are actually present in the visual (animate or inanimate), as well as the processes in which they are engaged and the circumstances in which they are found. The INTERPERSONAL metafunction is the function of language to represent the roles and statuses that participants hold in any form of interaction, and here the interactive participants are the focithis includes the participants who are interacting with each other in the act of reading a visual, one being the graphic designer or drawer, and the other the viewer, and the social relations between the viewer and the visual. The TEXTUAL metafunction is that function of language through which a text can be recognized as having coherence and as making sense. The focal point in a consideration of page-based multimodal text is the coherent structural elements or composition. This relates to aspects of layout and design which combine and integrate the elements on the page in a way in which the graphic designers or drawers wish to present at a particular point in time (Kress & van Leeuwen, 1996, p. 183). Reading (or viewing) a visual therefore involves the simultaneous interplay of three elements that correlate with Hallidays (1985) three metafunctions: the ideational, the interpersonal, and the textual. For the visual mode these relate to the represented participants, the interactive participants, and the visuals coherent structural elements (Kress & van Leeuwen, 1990, pp. 1621). This social semiotic view of communication implies that whether a text contains only verbal, or both verbal and visual modes, it can and should now be viewed as embodying the pattern of purposeful choices made by its constructors in order to make meanings for others to receive and respond to in some way. It is multimodal communication, and as such it is addressed to a viewer/reader. In the context of an L2 classroom or in noneducational contexts, the student who has to view/read it must have the necessary competencies to be able to use its meanings for effective communication, either receptively or productively.

A MULTIMODAL VIEW OF COMMUNICATIVE COMPETENCE A multimodal view of CC makes a number of important assumptions that are derived directly from the SFL model. First, it assumes that multimodal communication is constructed with a view to exchanging, projecting, or sending meanings within a social context. As we are dealing with multi-

374

ROYCE

modal meaning here, and not only linguistic meaning, projecting or sending meanings within a social context to some other person, whether that person be a listener, a reader, or a viewer, the channel used for the transmission of these meanings can be a combination of two or more modalities, and each channel will communicate the socially based meanings in a form that is appropriate to the medium. Second, it assumes that these social meaning selections are activated by the cultural context in which they are situated. The resultant multimodal text is an instantiation of these choices, and can as a result be viewed as a realization of the contexts of situation and culture-bound choices made by its constructors. At the same time, and in line with the dialectical relationship between text and culture in the SFL model, a multimodal text also construes the context of situation and culture in which it occurs. Thus, one can say that visual and verbal language is in culture, and culture is in visual and verbal language (cf. Hasan, 1981, 1995, 1996). Third, it assumes that the ways people communicate in various visual and verbal modes are the result of the choices they have made or the options they have taken up from each particular semiotic system. If the orthographic mode of writing is chosen, then the person making the choice will have made a range of choices from a variety of meaningful options available in that orthographic system. In the same way, someone who designs, draws, or develops a visual has made a range of choices from a visual social semiotic system, choices which, like those from the written mode, are situated in the social, cultural, and ideological contexts in which they have been made and which they share with others. They are intersubjective sign systems, by virtue of the fact that in a community [they] serve to define the nature of the world for its members and have a role in the mediation of meanings between the members (Hasan, 1981, p. 107). These meanings and choices, realized in differing modes, will necessarily have a message and interactional-focus, and will draw on the textual or compositional conventions appropriate to the mode. Clearly, students in second-language contexts need to be able to develop visual communicative competence. As Stenglin and Iedema (2001) suggest, however, multimodal communicative competence is not simply concerned with the need for students to deal with the ways that the respective modes individually realize their contextualized meanings. Rather, it has to do with how students can become competent in interpreting and constructing appropriate meanings multimodally. A multimodal text (e.g., page or screen-based) is a text where the modes utilized work together in various ways to produce comprehensible meaningsthere is a synergy in their combined meanings, which, it has been suggested, is realized by the intersemiotic complementarity between the modes.

12.

MULTIMODAL COMMUNICATIVE COMPETENCE

375

As the previous section demonstrates, the SFL paradigm has been successfully and usefully applied to systems of meaning other than language, and one of these is the visual semiotic. If we can assume that the verbal and visual modes in combination are social, purposeful, and contextualized, we can begin to talk about the ways we can clarify just what is meant when we refer to multimodal communicative competence, and what this means in terms of students multimodal CC needs. Clearly, both modes represent meaning in Ideational terms; they both realize meanings related to experience, meanings which can be construed as being concerned with the identification of participants (who or what is involved in any activity), the activity (the processes in terms of what action is taking place, events, states, types of behavior), the circumstances (where, who with, by what means the activities are taking place), and the attributes (the qualities and characteristics of the participants). Students faced with having to read (or produce) a text which includes both verbally and visually realized meanings will not of course produce two totally unrelated visual and verbal instances of meaning, but will attempt to ensure that the resultant multimodal text is coherent for any potential viewer/reader in terms of the subject matter represented. It is here that the teacher needs to be especially concerned with multimodal CC. Students need to have the knowledge and skills to be able to interpret or produce a text which coherently construes Ideational meanings that are culturally and contextually meaningful. In order to do this students should be aware that they need to draw on an understanding of what makes a multimodal text informationally coherent. Ideationally, this involves the various lexico-semantic ways of relating the experiential and logical content or subject matter represented or projected in both visual and verbal modes through experiential meaning that is repeated and synonymized, the ways that experiential oppositions are developed, as well as the ways that partwhole and classsubclass relationships are set up between the information in the modes (see Royce, chap. 2, this volume). Allied to this are the ways that the respective meanings form collocational relations across the modes. So in Ideational terms, an important aspect of multimodal CC is the ability to both process and produce these kinds of intersemiotic relationships. Both verbal and visual modes can also represent meaning in concert in Interpersonal terms; they both address their viewers/readers, express degrees of involvement, and realize various power relations. They also articulate degrees of social distance between the participants in the interaction, and express meanings related to modality: degrees of possibility, probability, and certainty. Students faced with having to read (or produce) a text which includes these interpersonal aspects need to be able to identify and use the various ways of intersemiotically relating the reader/viewers and the

376

ROYCE

text through various forms of Address (offers, commands, statements, questions), and Attitude (modality, where ideas are real or unreal, true or false, possible or impossible, necessary or unnecessary). Students also should be able to draw on an understanding of how both modes, within the boundaries of a single text, may maintain the same forms of address to viewers/ readers (reinforcement of address), or realize both similar and opposite [or even ironic] attitudes (attitudinal congruence and dissonance relations). In certain contexts they may even need to produce their own multimodal texts (e.g., in some EAP/ESP (English for Academic Purposes/ English for Specific Purposes) courses, such as geography, or business courses students are asked produce texts with accompanying diagrams (see also Ferreira, chap. 10, this volume). Finally, students will need skills in compositional meanings in multimodal texts, and need to know the ways that both the visual and verbal modes can combine to produce coherent meanings on the page or screen. They need to be aware of the various ways that multimodal texts map the modes to realize a coherent layout or composition and indicate degrees of information valuation. Allied to this is the use of salience principles, degrees of framing of elements, and intervisual synonymy to help the reader move around the page. There is also the important culturally based issue of potential reading paths, and the need to develop skills for moving from the most important information to the least important (see Ferreira, chap. 10, this volume). Additionally, there is a discourse dimension to this where intersemiotic complementarity may be realized via multimodal discourse patterning, as in a text where chronological (past-present-future) and GivenNew complementarity may be realized between a multimodal texts verbal discourse stages and any narrativization stages displayed in the visual. A quick example would be the ways that a grade reader in schools would verbally present a narrative which is complemented intersemiotically by the series of images used at various stages of the plotthese images can be placed at the point in the verbal text where the focus of the image can be read, and their sequence approximates the plot as it unfolds. Multimodal CC is therefore concerned directly with the ways that the two modes interact semantically on the page or screen, the skills and awareness that students and teachers need to be able to address the fact that the two modes co-occur, that they project their meaning in concert, and that these combined meanings often realize a visualverbal synergy which provides in many ways a richer and fuller expression of meaning than would be extant if a single mode were used. Allied to this is the fact that students will come to their classes with their own culturally situated understandings of multimodality; this dimension can and should be drawn upon as a rich source of detail that can be used for comparison and contrast of English

12.

MULTIMODAL COMMUNICATIVE COMPETENCE

377

multimodal texts, and it suggests an important future direction for discussions of multimodal CC in the TESOL classroom.

MULTIMODAL CCSOME CLASSROOM APPLICATIONS What then does this multimodal view of CC mean for teaching practice in a language learning context? As suggested in this chapter, almost every image can be analyzed in terms of what it presents, who it is presenting to, and how it is presenting, and that the concept of metafunctions can be suggestive for the language teacher in developing pedagogical resources targeted to help students extract just what the visuals are trying to say, to relate these messages to the verbal aspect, and then use them to contribute to developing students multimodal communicative skills. Some methodological suggestions arising from this approach follow, centered around a pagebased extract adapted from a textbook approved by the Monbukagakusho (Education Ministry) for High Schools in Japan. The text of the storys first pages is presented in Fig. 12.3, and Fig. 12.4 shows some adapted story images in the same sequence as they appear in the text. The text is basically an abridged short story by Kurt Vonnegut Jr. about a new super-computer EPICAC, which falls in love with a beautiful computer technician and starts to write love poems to her. A male computer technician is also attracted to the woman and uses the computergenerated love poems to form a relationship with herthe woman of course thinks that the love poems are from the young man and they both fall in love and leave together, all of which the computer observes. The computer experiences unrequited love, blows its fuses, and in the end dies of a broken heart. The students in Japanese high schools are asked to read this kind of narrative text in government-approved textbooks (which only recently started to include color pictures) in order to learn the required vocabulary and structural/grammatical points (via grammar-translation methods) for English language entrance exams, and as the Monbukagakusho has recently mandated, to develop their communicative language skills. Now, at first glance this kind of page-based text, produced basically for learning about the English language for testing purposes, would seem to be a rather deficient source for teachers to use to focus on developing students multimodal literacy skills. However, I would argue that this text is in fact a rich source of multimodal meanings which can be approached in terms of multimodal CC. To demonstrate this, some multimodal classroom activities, focusing on specific receptive and productive skills follow.

378
FIG. 12.3. Epicac, copyright 1950 By Kurt Vonnegut Jr., from Welcome to the Monkey House by Kurt Vonnegut Jr. Used with permission of Dell Publishing, a division of Random House Inc.

12.

MULTIMODAL COMMUNICATIVE COMPETENCE

379

FIG. 12.4. Kumi Yasuda. Story images [adapted] from Mainstream IIB: The new English reading course (2nd ed., pp. 114126). Osaka: Zoshindo, 1992.

Pre-Reading Skills Activities could be organized which involve the students asking questions of the visuals, and then using their answers to assist in their reading development. The richest source of information can obviously be derived from those questions that focus on the message-focus (or Ideational) aspects of a visual (who or what do you see in the visual frame; what are they doing; who/what are they doing it with etc.), and since many school subjects are concerned with information, its organization, and its relationship to other information, many classroom activities could be centered around extract-

380

ROYCE

ing just what the visuals are trying to say to the viewers in terms of their informational content. The information derived from extracting these visual meanings could then become the focus of further reading, writing, and speaking activities (and indirectly listening). In the EPICAC story, we can see that in the sequence of images in Fig. 12.4 there are various characters/ roles portrayed (genders; bosses; computer technicians; computer hardware), actions, reactions and interactions (discussing; reading; processing; showing affection; showing displeasure; etc.), and projecting other associated representational meanings (attributeslab coats; business suits; long hair; etc.). Aspects of the visuals are picked up in the verbal aspect of the text shown in Fig. 12.3 through the intersemiotic complementarity relations of Repetition (EPICAC, super-computer), Synonymy (Government people), Meronymy (Tubes, wires, switches) etc. These and other relations are consistently realized in the following pages of the story with Repetition of portrayed processes such as ask, crying, decoding, get married, and so forth. In the classroom, questions can be addressed concerning whether all the students in the class agree with what the visuals represent and what the verbal actually says; they are therefore being asked to look for the kinds of relations mentioned earlier and to compare their interpretations. For developing reading readiness, asking these kinds of message-focus questions of visuals can activate the students existing background knowledge in the L2, and working with what is familiar can thus reduce textshock with new and unfamiliar texts. The students can ease themselves into a reading and get some idea of what to expect in terms of the who, what, where, why, how, and with whom in the image. The effect is that expectancies are being set up in the students minds, and the process of reading the text will then either give them a confirmation of their interpretation of the information (or story), or in rare cases introduce ambiguities, which the class can then explore in more depth through discussion and follow-up written activities. In doing these kinds of activities students are not just focusing on the language, but are looking at the text multimodally. They are drawing meanings for the visual individually, they are drawing meanings for the verbal, and they are drawing meanings in combination, or meanings that have been formed by the interaction of the modes on the page (such as the ways that Intersemiotic Repetition is realized). They are therefore interacting with the text multimodally, as a text which coherently uses visual and verbal modes to project its meanings. Vocabulary Development Associated with this latter point on activating background knowledge is vocabulary development. The interpretation of a visual in relation to any associated verbal text will necessarily involve encounters with new words, which

12.

MULTIMODAL COMMUNICATIVE COMPETENCE

381

the students can immediately associate with a visual representation. This allows for cognitive associations to be set up, facilitating vocabulary learning. Reading skills such as skimming and vocabulary sight-recognition development can also be engendered. Skimming through a reading and identifying words that relate to the visual in terms of whether they be participants, processes, and circumstances can also stimulate students prereading vocabulary development. Supporting this of course is the possibility of pronunciation practice, both in terms of single words and fluency development. In Fig. 12.3, which presents the first page of the EPICAC text, we see that in the first few paragraphs the following lexical items occur, which relate in some way to the first two visuals: EPICAC; computer; electronic tubes; wires; switches; plugged into; 110-volt line; super-computer, etc. Again, the focus has been shifted from a singular concentration on either the visual or the verbal, to a view of the text as an interaction between them. Students can now be trained to use both modes as meaning-making resources in combination; this means that they are developing the skills needed to improve their multimodal CC.

Comprehension and Genre Knowledge Skills The EPICAC reading is clearly of the narrative genre, and follows the generally recognizable schematic stages of the narrative genre: orientation complicationresolutioncoda (Martin & Rothery, 1981, p. 11). For supporting reading comprehension of narrative genres, students understanding of a specific plot or of plot structure in general could be developed by looking for visual sequencing. For example, if there is a sequence of visuals in a short story, as is common in many graded readers and abridged versions of novels used in schools, the students could be asked to start their reading of the novel by looking at the visuals only in their story sequence, and then to interpret them by figuring out who the actors are, what they are doing, and why, before they start to actually read the story. This would work well with those readers who use images in this waythe sequence of images could be related at a discourse level in terms of Intersemiotic Repetition in the verbal aspect of the text, however, the teacher would need to be careful as some publishers may (though rarely) include image sequences which do not approximate the story as it unfolds. This is clearly not the case with the sequence of visuals in the EPICAC story, which obviously approximates the unfolding the stages of the narrative of the computer love affair. Students could even be presented with the visuals out of order so the students can create their own sequences, and they could then write what they think the actual story is and explain or discuss why. As an English teacher in the Japanese educational context suggests,

382

ROYCE

For the presentation of new material, I think multimodal analysis is very effective to let students know the content of the text. Using multimodality, or showing the relationship between the text and the pictures or drawings in the textbook, students can cultivate the ability of reading or even listening by guessing the context. Sometimes I draw pictures which are related to the content and ask the students put them in order along with the textbooks listening tape. Their reaction to these activities is very good. They say they can understand the content of the text if they have these activities before the grammar explanation. I also give them other information sheets such as quizzes, historical stories, maps, biographies, pictures and so on. (Yokoyama, personal communication, May 2004)

Although this kind of activity has been used by teachers for various purposes for many years, it can be useful and relevant for developing in students a multimodal understanding of story/narrative structures: The image sequences and the writing activities which arise from them could be used to introduce the students to other genres (description for example), based on the ways that visuals are organized, and the ways they relate to the verbal aspect of the text. In terms of expository writing development, an expository visual like The Water Cycle (see chapter 6 by Mohan et al. in this volume) can be used to explicate the way that the cycle actually operates, since it does in a sense tell a story. The students could start at some point and tell the process sequence of the water cycle, with the sun (solar energy) as a starting point, for example. This kind of story could then perhaps be a basis for changing the writing to a more acceptable scientific form of writing, thus showing the students the differences between narrative and expository writing. Another multimodal activity would be for the students to see what stages of the story have been represented in the visuals provided, to see which stages may have been left out or included and why, and what details were included in the visuals that were/were not included in the written text (and the reasons why). Writing Development Closely related to the development and understanding of genres is the students writing development, especially in the area of creative narrative writing. The same sequence of pictures could be extracted from the reading that the students are required to cover, and using these decontextualized images they could construct their own story individually or in groups, then write the story in class or as a journal or as the basis for a class story magazine. They could even draw their own visuals to go with their stories. This story-writing or class magazine production could be used as a writing process activity where drafting and re-drafting is carried out in consultation with teachers, or in peer-editing groups. This kind of activity is relevant to students developing an understanding of story/narrative structures: The im-

12.

MULTIMODAL COMMUNICATIVE COMPETENCE

383

age sequences and the writing activities that arise from them could be used to introduce the students to various genres (narrative, description), based on the ways that visuals are organized. What is very important here is the choice of which visuals are relevant to the story being createdthis activity would be the focus on the relations between the two modes, with the students asking themselves questions about how the two relate at both the lexical and discourse levels. The sequence of this kind of classroom activity could be to start with visuals which can readily tell a story, either one story only or a number of possible plots. Then the teacher could work with the students to answer the visual questions showing who are the main characters, what are they doing and with whom, why are they doing it, and how, etc. Following on from this, the students could try to place the pictures in some order, which could be organized into their own spoken and then written storythis is an area where students creativity can be allowed full reign. Some interesting questions may arise here also; for example, what governs their choice of visuals that fit with a story, and is there a rank scale to the choices (note the role of insets, which usually highlight small details or bring to the foreground small details). Questions can be raised here about what small details in the insets would be interesting to highlight and why, as well as a consideration of whether the details brought to the foreground through insets are culturally determined or bound in some way. Speaking and Listening Skills The students speaking and listening skills could also be developed through all the preceding activities, which provide ample opportunities for students to converse with the teacher and with their peers. The reading-readiness activities for example could also be used for reporting back to the class, for giving short speeches, explaining, describing, etc. where the students refer directly to the verbal and visual aspects of the presentation, and the students of course have to listen to them, so the development of listening skills is another positive result (especially if the listening is task focused and perhaps evaluated through follow-up worksheets which focus on the ways that the presenters used images and language together). Images can also be used for evaluating speaking skills in a testing formatthis could involve showing a student a picture from a story already read and asking him or her to talk about it in an allotted time. This will test both production and understanding of the readings content, and could also be used in a class-wide evaluation to see which group understands a storys content and sequence best, and if required, the allocation of various grades. Space prohibits extensive examples of the ways that multimodal CC can be facilitated in L2 environments, especially since many of the activities presented are isolated and generalized activity examples divorced from spe-

384

ROYCE

cific L2 educational contexts. One of the best ways to demonstrate how multimodal classroom activities can be carried out is by providing examples of actual teaching practice, so two specific examples of these follow. Specific Classroom Examples To illustrate the ways that a focus on multimodality can be incorporated into lesson planning and classroom activities, two classroom applications that have been either used or developed for use in schools in the Japanese EFL context follow. Both applications have been developed by professional English teachers who designed them as part of their coursework in the Teachers College, Columbia University MA in TESOL Program in Japan. The activities are either based on or are extensions from the required textbooks to be used in the Japanese school system, and both have the underlying aim of developing the students multimodal CC. The first example contains a multimodal activity which is embedded in a series of reading [language] skills classes, whereas the second is designed to focus specifically on developing multimodal CC over two class sessions.

CLASSROOM APPLICATION ONEMULTIMODAL READING My school is a public school located in a rural area, and is one of 12 academicallyoriented schools in the prefecture. Many students want to pursue higher education after they graduate. Parents expectations are that the school will improve their academic ability and prepare them with the skills necessary for the entrance exams, especially the national college entrance exams. The teachers in the school are expected to make every effort possible to motivate the students and improve his/her academic performance throughout the 3 years of high school education. English is a required subject for any future exams for college or employment, so English is considered to be key for success. The class where this activity was attempted is a General English Class consisting of 25 first year high schools students (11 boys and 14 girls). The class is one of three advanced courses and each class is taught by a different teacher. A Monbukagakusho textbook is used and 5 common exams are given to all students. The specific class where this activity was carried out is the fifth of a sequence of six classes: the first and second classes dealt with activating background knowledge via prereading activities and group research on the topics in the WWW and library, along with presentations in L1 on what they found; the third and fourth classes dealt with metacognitive awareness raising and fast reading practice through two multimodal readings from the assigned textbooks; the fifth class was aimed at extensive reading and involved a multimodal text from an alternative non-textbook source on the Nazca Lines in Peru; the sixth class involved intensive reading of the same text.

12.

MULTIMODAL COMMUNICATIVE COMPETENCE

385

The Aim of the Class and the Activity The specific aims were to:

have the students appreciate their classmates interpretation of the text have the students learn about discourse patterns

The general specific aims were to help the students become interested in reading further in the topics given in the textbook; to develop the students metacognitive and metalinguistic awareness so they can approach new readings with confidence and appropriate approach skills; to increase the students motivation towards reading in a foreign language; to increase students reading skills and their knowledge of discourse and grammar structures; and to develop the skills to be able to draw upon the visual and verbal information given in the reading text. The Preactivity Setup. to connect to the previous classes work the students are asked to review the ideas presented in the multimodal text they have already read. The discourse patterns by paragraph in that text are reviewed diagrammatically on the blackboard, with the aim to show that the readings they have in their classes have larger patterns and fit together in coherent ways which they can use to help them read and decipher. The Classroom Activity. the class is divided into small groups and told that the title of the days class is Lets become textbook editors, and that they are going to make their own textbook passage using the ideas they have been looking at in the previous four classes. The topic of the text is related to the text read in the previous class, which is: Mysteries of the World. Each group is given two packets of materials which deal with the topic: The Nazca Lines of Peru. Packet one contains a number of separated color and monochrome pictures pasted onto cardboard. Packet two contains the written part of the text, which has been cut up into five paragraphs. Four of the paragraphs are intact and have been pasted onto cardboard, while one of the paragraphs has been cut up by sentence into seven separate strips of paper (seven sentences). The students are then told that they are going to design their own textbook reading using their understanding of the language and the pictures in the packets. The Activity Instructions 1. Jigsaw readingone person from each group takes one of the four reading cards and reads it for understanding (leave the seven sentence paragraph for later). 2. Reading and reporting to the groupeach person reads aloud the paragraph he or she has and gives an interpretation of its topic and content to the group in L1. Once each group has completed this the groups then try to put the four paragraphs in an order that makes sense to them. Once this is done, the groups try to organize into a

386

ROYCE

coherent paragraph the jumbled seven-sentence strips, and then try to fit that into the whole text. 3. Once the groups have used their understanding of discourse structure to organize all the paragraphs, they are then asked to open the packet with the pictures and as a group select the pictures which seem to go with the ideas in the language. The paragraphs and the pictures chosen should then be spread out on the desk to make a page-design space, and the paragraphs and pictures are arranged in the form of a textbook page. The students then go through a lengthy negotiation about which pictures should go with the paragraphs and why, and decisions are made about ordering and which pictures should be omitted. The end result is a range of multimodal (text and image) presentations which students can then present to the rest of the class as a completed textbook passage. The Review and Conclusion Activity. once all the groups are ready, the students are asked to walk around the room and check others textbook pages and ask questions about the reasons why particular configurations were used (Muto, personal communication, May 2003)

CLASSROOM APPLICATION TWOMULTIMODAL INFORMATION GAP ACTIVITY In my school all teachers are required to keep pace with each other using the same textbook for the same examinations at the end of each trimester. So it is difficult to introduce activities such as this one on a long-term basis. The activity is aimed at a class of 42 third grade high school boys in two 50 minute periods. These students were selected for the activity as they have more background lexical and grammatical knowledge as a result of their earlier studies, so it was felt that they would be better prepared for the introduction of this kind of multimodal activity. A further reason is motivationalthey were quite tired of the use of traditional textbooks and drills, and this would be a fresh approach for them. The Aim of the Activity. to develop in the students a basic understanding of the fact that when a combination of visual and verbal elements are included in a textbook, they can use both modes to develop their understandings about what is going on in the multimodal text they have to study. They could hopefully transfer these skills to their textbooks. It was also felt that in this activity it is important to not introduce specialist linguistic or visual terms, as the students are not specialists in that sense. The need is for them to get the overall and specific message of the text, as presented by both the visual and verbal modes. In this EFL context also, it was felt that the students should be allowed to use, at various stages, their L1 in group discussions. There were also motivational reasons for this, as well as the fact that their language abilities were not so advanced that they could do all the work in the L2.

12.

MULTIMODAL COMMUNICATIVE COMPETENCE

387

The Preclass Activity. before the class, the 42 students are told to break up into 7 groups of 6 members each. Each group is then divided into further groups labeled Group A and Group B. Then all Group A students will be asked to move to the next room so they cannot see Group B students activities. There will be 7 three-member A Groups in one room, and 7 three-member B Groups in another room. The Classroom Activity. Groups A and B will have 20 minutes to discuss and complete their exercises. Both groups will have an activity sheet which has been derived from a magazine advertisement. Group A will have a sheet with the language removed, and are asked, based on the visual elements, to predict or guess the product advertised, and to try to brainstorm as many words as possible which could be expected to appear in the language which was removed Each three-member Group B has a sheet with the visual removed and the language intact, and is asked to predict what kind of visual would be most likely placed in the space where the visual should be. They will be asked to consider the following listed features (which will be explained in their L1), as well as who would be the most likely people who would read/view the advertisement (the customers). The features to consider are: 1. 2. 3. 4. 5. the the the the the represented participants (human or not; gender; age etc.); actions portrayed; circumstances (background and other things used by people etc.); attributes of the participants (clothes, physical features etc.); colors used and why.

The Presentation Activity. All students in Groups A and B are then asked to go back to the main home room and to re-join their counterparts from the original groups. They are to discuss the findings of each of their members and to present them to each other and compare. They are then asked to prepare as a group a short presentation in English about what they discussed and found, to present that as a group in English, and then show the whole class their completed multimodal advertisement texts. These are then put up on the classroom walls. The Conclusion Activity. after the groups presentations, the teacher can then interact with the whole class with questions about the groups findings and presentations. This could be done in terms of the following focus questions: 1. 2. 3. 4. What is the purpose of this text? Why did the text creator make the text with both images and language? Why did the text creator make the images as they are (use color etc.)? What are the important words in the language part of the text? Why?

388

ROYCE

5. How are the image and the language similar or different in the meanings they send? How do you know this? An Alternative Conclusion Activity. after the groups presentations, the teacher could assign the above focus questions for homework as a written assignment, or could ask them to bring in their own versions of the same kinds of text and discuss and compare them (Sakai, personal communication, May 2004). The two classroom applications presented above are effective examples of the ways that lesson activities can focus on aspects of multimodality for developing multimodal CC in EFL contexts. There is a clear focus on multimodal ideational, interpersonal, and compositional features, and students work with these features in specific and general ways. For example, in the first application, the students use their general understandings of salience and reading paths when they work to place the images of the Nazca Lines on their desks (representing the page space) in certain positions. These placements indicate that they understand discourse organization and reading paths, and the ways that the various stages of the writing should relate to the images they chose. In both applications they use specific knowledge of multimodal relations by placing images in close proximity to the words that relate to them intersemiotically (in Repetition, Synonymy, Meronymy, Hyponymy, etc.), or in a reversal of the process, by predicting which words go with a certain advertising image. They also focus on forms of address, which in application one both visually and verbally takes the form of statements about the Nazca Lines to the readers and viewers. The sequencing of the images organized by the students in this application also approximated the discourse stages of the expository verbal text, and mirrored the generalizationdetail relations organization (see comments on clause relations in Hoey, 1983).

CONCLUSION Gunther Kress (2000) suggested that given the changes in the modes and affordances of communication systems in todays world, it is now impossible to make sense of texts, even of their linguistic parts alone, without having a clear idea of what these other features might be contributing to the meaning of a text (p. 337). To focus on one mode, without interpreting the other, could mean that the full message is not being received. Many language teachers would suggest that they are concerned with communication, with language central of course, but combined with awareness that other modes of communication are important too. However, this can tend to be incidental, and is often related to teacher training or previous studies

12.

MULTIMODAL COMMUNICATIVE COMPETENCE

389

in another discipline. As one graduate TESOL student who has a background in art studies put it,
I never realized it but I have used the idea of multimodal literacy in my classes before. In my adult classes, I would frequently have discussions about the picture of a text, before reading the text. This is a great way to get the students to focus on the topic and imagine what the text is about. For lower level students, a vocabulary list might be a good idea. Have the students make a list of all related words they can think of, to a related picture or a series of pictures. This is also a good way for students to review related vocabulary, before reading a story ( Jones, personal communication, 2004)

Teachers are becoming increasingly aware that they should be more concerned with developing students multimodal communicative competence, and that there is a need for specific and systematic approaches which can help them to raise students consciousness of the fact that not only are there alternative ways of communicating meanings, but that those alternatives can be interpreted in concert with language. Teacher education should play its part here, and TESOL graduate schools are increasingly offering courses which focus on the ways that various visual media enrich the language learning experience and work in concert with other modes, in both ESL and EFL contexts.

REFERENCES
Butt, D., Fahey, R., Spinks, S., & Yallop, C. (1995). Using functional grammar: An explorers guide. Sydney: NCELTR, Macquarie University. Canale, M., & Swain, M. (1980). Theoretical bases of communicative approaches to second language teaching and testing. Applied Linguistics, 1(1), 147. Dondis, D. A. (1973). A primer of visual literacy. Cambridge, MA: MIT Press. Fiske, J. (1982). Introduction to communication studies (2nd ed.). London: Routledge. Halliday, M. A. K. (1978). Language as social semiotic. London: Edward Arnold. Halliday, M. A. K. (1985). An introduction to functional grammar. London: Edward Arnold. Halliday, M. A. K. (1991, March/April). The notion of context in language education. In Thao L & M. McCausland (Eds.), Language education: Interaction and development. Proceedings of the international conference, Vietnam (pp. 126). Launceston, Tasmania: University of Tasmania. Halliday, M. A. K. (1994). An introduction to functional grammar (2nd ed.). London: Edward Arnold. Halliday, M. A. K., & Hasan, R. (1985). Language, context and text: Aspects of language in a socialsemiotic perspective. Oxford, England: Oxford University Press. Hasan, R. (1981). Whats going on: A dynamic view of context in language. In J. E. Copeland & P. W. Davis (Eds.), The seventh LACUS forum (pp. 106121). Columbia SC: Hornbeam Press. Hasan, R. (1995). The conception of context in text. In P. H. Fries & M. Gregory (Eds.), Discourse in society: Systemic functional perspectives. Meaning and choice in language: Studies for Michael Halliday (pp. 183296). Vol. L, Norwood, NJ: Ablex. Hasan, R. (1996). Ways of saying, ways of meaning: Selected papers of Ruqaiya Hasan. London: Cassell.

390

ROYCE

Hoey, M. (1983). On the surface of discourse. Boston: George Allen and Unwin. Hymes, D. (1972). On communicative competence. In J. B. Pride & J. Holmes (Eds.), Sociolinguistics (pp. 269293). Harmondsworth: Penguin Books. Johnson, K., & Johnson, H. (Eds.). (1998). Encyclopedic dictionary of applied linguistics: A handbook for language teachers. Oxford, England: Blackwell. Kress, G. (2000). Multimodality: Challenges to thinking about language. TESOL Quarterly, 34, 337340. Kress, G., & van Leeuwen, T. (1990). Reading images. Geelong, Victoria: Deakin University Press. Kress, G., & van Leeuwen, T. (1996). Reading images: The grammar of visual design. London: Routledge. Lo Bianco, T. (2000). Multiliteracies and multilingualism. In B. Cope & M. Kalantzis (Eds.), Multiliteracies: Literacy learning and the design of social futures (pp. 92105). Melbourne: Macmillan. Martin, J., & Rothery, J. (1981). Working Papers in Linguistics No. 2. Sydney: University of Sydney. New London Group. (2000). A pedagogy of multiliteracies: Designing social futures. In B. Cope & M. Kalantzis (Eds.), Multiliteracies: Literacy learning and the design of social futures (pp. 937). Melbourne: Macmillan. OToole, M. (1994). The language of displayed art. London: Leicester University Press. OToole, M. (1995). A systemic-functional semiotics of art. In P. Fries & M. Gregory (Eds.), Discourse in society: Systemic functional perspectives (pp. 159179). Vol. L in the series Advances in Discourse. Norwood, NJ: Ablex. Richards, J. C., & Rodgers, T. S. (2001). Approaches and methods in language teaching. Cambridge, England: Cambridge University Press. Richards, J. C., & Schmidt, R. (Eds.). (2002). Longman dictionary of language teaching and applied linguistics. London: Pearson Education, Longman. Royce, T. (1999a). Visual-verbal intersemiotic complementarity in The Economist Magazine. Unpublished doctoral dissertation. University of Reading: United Kingdom. Royce, T. (1999b). Synergy on the page: Exploring intersemiotic complementarity in pagebased multimodal text. In JASFL Occasional Papers No. 1 (pp. 2549). Tokyo: Japan Association of Systemic Functional Linguistics ( JASFL). Royce, T. (2002). Multimodality in the TESOL classroom: Exploring visual-verbal synergy. TESOL Quarterly, 36(2), Summer 2002, 191205. Savignon, S. J. (1972). Communicative competence: An experiment in foreign language teaching. Philadelphia: The Center for Curriculum Development. Savignon, S. J. (1983). Communicative competence: Theory and classroom practice. Menlo Park, CA: Addison-Wesley. Stenglin, M., & Iedema, R. (2001). How to analyse visual images: A guide for TESOL Teachers. In A. Burns & C. Coffin (Eds.), Analysing English in a global context (pp. 194208). London: Routledge. Vonnegut Jr., K. (1950). Epicac. In Welcome to the monkey house (pp. 297299). New York: Dell, a division of Random House. Vonnegut Jr., K. (1992). EPICAC. In Mainstream IIB: The new English reading course (2nd ed., pp. 114126). Osaka: Zoshindo. Watson, J., & Hill, A. (1997). A dictionary of communication and media studies (4th ed.). London: Arnold. Widdowson, H. (1978). Teaching language as communication. Oxford, England: Oxford University Press.

Heberle, V (2010). Multimodal literacy for teenage EFL students. Caderno de Letras (UFRJ), 27, 101-116.

Multimodal literacy for teenage EFL students

Viviane Heberle
[g]iven the importance of visually displayed information in so many significant social contexts, there is an urgent need for developing adequate ways of talking and thinking about the visual (KRESS and van LEEUWEN, 1996, p. 33).

Nowadays literacy has been understood as a social practice, as a complex set of reading, writing and technological skills which joins verbal, visual, and other meaningmaking resources. Williams and Hasan (1996), Rios (2009) and Kleiman and Baltar 2008) point to the social nature of literacy processes. The New London Group (1996) also argue that literacy pedagogy now must account for the burgeoning variety of text forms associated with information and multimedia technologies. Studies in many English-speaking countries have emphasised the relevance of visual grammar in English language contexts (UNSWORTH, 2001; KRESS, 2003). The New London Group (1996), for example, refer to the increasing multiplicity and integration of significant modes of meaning-making, where the textual is also related to the visual, the audio, the spatial, the behavioral, and so on. Likewise, as suggested by Andrews (2004:63), "... it is the visual/verbal interface that is at the heart of literacy learning and development for both computer-users and those without access to computers. Visual literacy can be linked to multimodality, which refers to the use of different semiotic resources to produce or interpret meanings. In this article, following Royce (2007), first I expand the notion of communicative competence to include multimodal communicative competence as an important skill for ESL/EFL1 students to develop, in order for them to interact more effectively with members of English-speaking discourse communities. Then I provide readers with a brief discussion on the relevance of multimodality in ESL/EFL teaching and offer suggestions on how to relate visual literacy specifically to teenage learners of ESL/EFL, as found in video games and advertisements, drawing attention to pedagogical, task-based activities which can be incorporated in the EFL syllabus. Multimodal communicative competence Communicative competence, a well-known term in second or foreign language teaching and research, coined by Hymes (1972) and extended by Canale and Swain (1980), has undergone refinements regarding the specific skills and strategies involved (BROWN, 1994; KRAMSCH, 1995; WESCHE and SKEHAN, 2002). The different aspects related to the term converge to propose that learners need specific strategies and
Cadernos de Letras (UFRJ) n.27 dez. 2010
http://www.letras.ufrj.br/anglo_germanicas/cadernos/numeros/122010/textos/cl301220100viviane.pdf

102

skills in order to understand and produce oral and written texts. These strategies include use of appropriate language forms (lexis, grammar, syntax), pragmatics and negotiation of meaning, which must be adequate to the context of situation. Thus, if we refer to Halliday (1985; 1994)s ideational, interpersonal and textual metafunctions, we could say that they allow us to perceive the link between grammar and functions of language, necessary requirements for any communication. As Christie (2002:11) explains, any language use serves simultaneously to construct some aspect of experience [related to the ideational metafunction], to negotiate relationships [related to the interpersonal metafunction] and to organize the language successfully so that it realizes a satisfactory message [related to the textual metafunction, my comments]. Multimodal communicative competence involves the knowledge and use of language concerning the visual, gestural, audio and spatial dimensions of communication, including computer-mediated-communication. Literacy practices nowadays incorporate these semiotic meanings, which ESL/EFL learners should be familiar with. In a study of a Chinese immigrants electronic textual experiences and the construction of his identity in ESL, for instance, Lam (2000: 458) emphasizes the contextual nature of reading and writing and the way literacy is intimately bound up with particular sociocultural contexts, institutions, and social relationships. It seems that multimodal communicative competence will allow ESL/EFL learners to be better prepared for different literacy practices in their professional and sociocultural experiences with native and non-native speakers of English. In classes where students have easy access to the Internet, teenage students may interact with other students around the world through ORKUT, Facebook, fotologs, blogs, and other e-environments. The experience with different kinds of multimodal texts in the English-speaking world can become a very productive means for developing students multimodal communicative competence in English. The relevance of multimodality in ESL/EFL learning and teaching Following Kress and van Leeuwen (1996) and The New London Group (1996), in relation to new demands in literacy, Kress (2003) explains:
It is no longer possible to think about literacy in isolation from a vast array of social, technological and economic factors. Two distinct yet related factors deserve to be particularly highlighted. These are, on the one hand, the broad move from the now centuries-long dominance of writing to the new dominance of the image and, on the other hand, the move from the dominance of the medium of the book to the dominance of the medium of the screen. These two together are producing a revolution in the uses and effects of literacy and of associated means for representing and communicating at every level and in every domain. (KRESS, 2003:1)

With this new paradigm for learning, my concern in this article is to open up new possibilities for ESL/EFL teaching. ESL/E FL teachers need to understand that [t]o communicate is to work in making meaning (KRESS, 2003:11) and that [v]isual structures realize meanings as linguistic structures do also, and thereby point to different interpretations of experience and different forms of social interaction (KRESS and van LEEUWEN, 1996:2).
Cadernos de Letras (UFRJ) n.27 dez. 2010
http://www.letras.ufrj.br/anglo_germanicas/cadernos/numeros/122010/textos/cl301220100viviane.pdf

103

Multimodality as a significant educational feature is also emphasized by MackenHorarik (2004:24):

Whatever the subject, students now have to interpret and produce texts which integrate visual and verbal modalities, not to mention even more complex interweavings of sound, image and verbiage in filmic media and other performative modalities

She emphasizes the need for learners to go beyond practical expertise in computer-based technologies of the school curriculum to encompass an understanding of semiotic frameworks of analysis, such as the grammar of visual design by Kress and van Leeuwen. In order to implement visual meaning-making resources and reconceptualize literacy in EFL as a joint role of verbal and visual meaning-making, teachers should become familiar with basic principles of systemic-functional linguistics and understand the three basic metafunctions proposed by Halliday, as previously mentioned:
We use language to talk about our experience of the world, including the worlds in our own minds, to describe events and states and the entities involved in them (the ideational metafunction). We also use language to interact with other people, to establish and maintain social relations with them, to influence their behaviour, to express our own viewpoint on things in the world, and to elicit or change theirs (the interpersonal metafunction). Finally in using language, we organize our messages in ways which indicate how they fit in with other messages around them and with the wider context in which we are talking or writing (the textual

metafunction) (THOMPSON, 1996:28). In terms of visual grammar (KRESS and van LEEUWEN, 1996; 2006), the ideational metafunction is known as representational, the interpersonal as interactive, and the textual, as compositional. Unsworth (2001:18) explains:
representational/ideational structures verbally and visually construct the nature of events, the objects and participants involved, and the circumstances in which they occur; interactive/interpersonal verbal and visual resources construct the nature of relationships among speakers/listeners, writers/readers, and viewers and what is viewed; compositional/textual meanings are concerned with the distribution of the information value or relative emphasis among elements of the text and image.

Understanding the ideational, interpersonal and textual metafunctions in relation to verbal and visual semiotics and how they are jointly used in communication represents a valuable asset for EFL teachers and students.

Cadernos de Letras (UFRJ) n.27 dez. 2010

http://www.letras.ufrj.br/anglo_germanicas/cadernos/numeros/122010/textos/cl301220100viviane.pdf

104

Meaningful opportunities in multimodality for ESL/EFL learners Whether an ESL/EFL course is designed for teenage or adult students, it must provide a link between linguistic form and function (understood here as language use, speech function). In ESL/EFL classes, the link between these two important aspects of language learning should be included in clas sroom activities, as learners must acquire both to be fully competent in their second language (DOUGHTY, 1998: 128). As SLA (Second Language Acquisition2) studies have shown, a focus on form which promotes meaningful communication seems to be the best pr actice. Striking a balance between emphasizing accurate production of second language forms and promoting meaningful communication in real contexts has become a vital concern (DOUGHTY, 1998:149). In this respect, using visuals and developing a metalanguage of visual literacy in the classroom may lead to meaningful language practice. During the last thirty years or so, communicative language courses around the world have proliferated, and pedagogy has changed from language in isolation to language as communication (DOUGHTY, 1998:134), offering the opportunity for ESL/EFL learners to use the target language in class activities. However, even though the tenets and methodologies of the communicative approach arguably constitute the dominant paradigm in current English language teaching (POOLE, 2002:75), in many parts of the world EFL teaching is still focussed on forms only, on grammatical terminology per se, in detriment of the development of meaningful, fluent use of English. We see the need to be concerned with a focus on form as scaffolding, as a tool for interaction and practice, not as monotonous unstimulating lists of grammatical terms. The importance of social interaction in learning processes, a shift from laboratory-based notions of learning to learning within the learners social world is also emphasised by Van Lier (1998:157). Van Lier refers to two types of interaction in the language classroom, teacher-learner interaction and learner-learner interaction, and concludes that teachers should att empt to make classroom interaction varied and multidimensional (p. 178). From a systemic-functional linguistic (SFL) perspective to second language pedagogy, interaction in the classroom is also important, and a fundamental premise is that language development arises from general circumstances of use and communicative interaction (PERRETT, 2000:93). Christie (2002), who analyses classroom discourse from SFL theory and Bernstein work on pedagogic discourse, also sees the need for teachers to provide scaffolding and important pedagogic opportunities, make students understand technical language and integrate instructional and regulative registers 3. From her analysis of several classes in the English-speaking world, she proposes an appropriate integration of the regulative and instructional registers, with teacher intervention to effectively help students to develop, classify and frame knowledge. She claims for an education that values knowledge and that values the learner, by seeking to make available to learners as explicitly and unambiguously as possible significant and useful information and ideas (CHRISTIE, 2002:179). Meaningful interactional practice with students active participation in class contributes to improve their competence in the foreign language. In recent communicative language teaching (CLT) two different trends have prevailed: task-based
Cadernos de Letras (UFRJ) n.27 dez. 2010
http://www.letras.ufrj.br/anglo_germanicas/cadernos/numeros/122010/textos/cl301220100viviane.pdf

105

instruction (TBI) and content-based instruction (CBI) (WESCHE and SKEHAN, 2002). According to Wesche and Skehan (2002), communicative classrooms usually give prominence to:
Activities that require frequent interaction among learners or with other interlocutors to exchange information and solve problems; Use of authentic (nonpedagogic) texts and communication activities linked to real-world contexts, often emphasizing links across written and spoken modes and channels; Approaches that are learner-centered in that they take into account learners backgrounds, language needs, and goals and generally allow learners some creativity and role in instructional decisions (WESCHE and SKEHAN, 2002:208)

These authors go on to say that in order to support the above mentioned aspects, CLT may include:
Instruction that emphasizes cooperative learning such as group and pair work Opportunities for learners to focus on the learning process with the goal of improving their ability to learn language in context Communicative tasks linked to curricular goals as the basic organizing unit for language instruction Substantive content, often school subject matter from nonlanguage disciplines, that is learned as a vehicle for language development, as well as for its inherent value. (WESCHE and SKEHAN, 2002:208)

Both task-based and content-based activities can contribute to students development of multimodal communicative competence, as they engage learners in negotiation of meaning through language use. While TBI activities foster interaction among learners to exchange information and solve problems in the foreign language, CBI is concerned with the integration of school or academic content with language -teaching objectives and can be effective for both language and content learning (WESCHE and SKEHAN, 2002:220) Incorporating multimodal skills in the teaching of English One way for teachers to start the journey towards the use of visual literacy in the EFL classroom specifically for teenagers is to collect all different kinds of pictures from newspapers, magazines, leaflets so as to organize a data bank, in sets which may emphasize the three functions as proposed in Kress and van Leeuwens (1996; 2006) grammar of visual design: representational (which corresponds to actions or concepts), interactive (which refer to the relations between the interlocutors) and compositional (arrangement of elements in a visual space, with subsets of different categories). Students are invited to contribute to the data bank, and this can become a motivating factor for interaction. Just as is done with professional databanks (MACHIN, 2004) students may create their own databank of pictures so as to integrate them in activities. Whether EFL teachers who subscribe to forms of communicative language teaching tend to support task-based or content-based instructions (or even a combination
Cadernos de Letras (UFRJ) n.27 dez. 2010
http://www.letras.ufrj.br/anglo_germanicas/cadernos/numeros/122010/textos/cl301220100viviane.pdf

106

of the two), it seems that incorporating a multimodal approach can be a rewarding experience, as I point out (HEBERLE, 2006). My experience in EFL teacher training courses in Brazil has shown that teachers have responded positively to an integration of TBI or CBI with multimodality and multiliteracy. The main points emphasized are that: 1) The approach may give opportunities for groups or pairs of students to negotiate meaning in English, when they have to solve a problem. For instance, the teacher may assign them specific task-based activities which could include pictures of people, places or objects in advertisements. Students would have to choose one specific picture, giving reasons for their choices, based on their intuitions and interests but their analysis should be based on the multimodal metalanguage. Two interesting books on the use of multimodal metalanguage for educational purposes are Callows (1999) and Unsworths (2001); 2) Regarding a link with real world contexts and topics of students interests, a s proposed by CLT, students may be asked to bring their favourite magazines to class so as to discuss the visual-verbal synergy and the different meanings available in them. Heberle and Meurer (2007) present a brief multimodal analysis for EFL classes. 3) In terms of joining academic texts and EFL, following CBI, teachers may ask students to discuss aspects of their cultural heritage, such as of their town, their family background or any culture that interests them. They would have to carry out research in books and magazines, as well as in on-line museums and/or art galleries, and then present their findings in oral or written assignments. Christie (2002) provides a rich discussion of secondary school geography lessons which can be adapted to EFL classes. 4) Students may access different hyperlinks to http://www.allposters.com/ and choose a category to discuss (for instance, Entertainment or Art, as seen below). First they would give a general view of the category itself regarding the main topics listed and then they would select one picture for a more detailed analysis. Two of the categories are:
Entertainment College Comics Humor Movies Music People Sports Television

and

Cadernos de Letras (UFRJ) n.27 dez. 2010

http://www.letras.ufrj.br/anglo_germanicas/cadernos/numeros/122010/textos/cl301220100viviane.pdf

107
Art Contemporary Art Traditional Art Museum Art Photography Vintage Art & Style Fine Art Reproductions Artists

These different activities have been suggested by EFL teachers in Brazil, although more detailed findings still need to be reported. Engaging students in either task-based or in content based instructions, working with visuals and using the appropriate metalanguage, students may certainly be more involved and motivated to learn not only EFL and school content, but also be more aware of sociocultural aspects related to their own communities or other foreign communities, especially if their classes are multicultural. Video games as a stimulus for learning English Video games represent one of the most popular worldwide phenomena for teenagers, and if EFL teachers can increment aspects of these games in their classes, chances are that students may be indeed motivated to participate. These games may be a suitable and motivating resource for EFL students to use visual grammar and learn English. As suggested by Unsworth (personal communication) [t]he first phase of the task with video games would involve understanding the game instructions and strategies in English a good opportunity for collaborative peer work and actually playing the game. Unsworth also adds that [i]t is useful to start off using games that are based on movies since the background story and characters are likely to be well known to teenagers and they can use their background knowledge to assist them to negotiate the English and the new work on reading images grammatically. The video game Ultraviolet is based on the movie released by Sony, at http://www.sonypictures.com/movies/ultraviolet/site/ (access June, 2010). In case the video game is no longer available, the teacher can use the film, available in video rental stores.

Cadernos de Letras (UFRJ) n.27 dez. 2010

http://www.letras.ufrj.br/anglo_germanicas/cadernos/numeros/122010/textos/cl301220100viviane.pdf

108

Picture 1 Violet, from the film and video game Ultraviolet

While looking at this picture, the students can give suggestions as to what Violet is doing or what is going on. The picture is a clear example of a visual narrative representation, but also emphasizing the concept of beauty and power: Violet, an elegant long-haired, sexy brunnette is positioned in an oblique angle in relation to the viewers, but looking directly at the viewers, demanding a reaction from them, and pointing a sword towards us/them. In this case, viewers can foresee her courage and power. The teacher can also discuss different aspects of visuals with students, such as the colours involved, the descriptions of the scenes and the actions performed by Violet. For this specific game, as in the film, each frame could be used by groups or pairs of students for visual analysis and discussion. They can discuss the visual and the verbal actions. In http://xbox.gamespy.com/ (access June 2010) the teacher and students can also find several games to analyse. For instance, students can watch demo videos of the game The Godfather Xbox. If students know the movie and/or the book, they can compare the similarities and differences between the different media. Another popular game which may serve as an effective stimulus for the development of students practice in visual grammar and in EFL is SimCity: (http://simcity.ea.com/about/simcity4/overview.php, (access March 2006 and June 2010). Unlike the other games mentioned above, this game seems particularly adequate if teachers are concerned with non-violence and a more peaceful setting. Here the players can act as mayor and carry out tasks to govern their own virtual metropolis. Notice that on the site, shown below, there are a series of commands (imperative form) for students to examine, and visually speaking the player has power over the scenes, as s/he is looking down, from a high angle.

Cadernos de Letras (UFRJ) n.27 dez. 2010

http://www.letras.ufrj.br/anglo_germanicas/cadernos/numeros/122010/textos/cl301220100viviane.pdf

109

Overview of SimCity 4
Play God Sculpt mountains, gouge valleys, and seed forests to lay the groundwork for your creation. Drop a disaster anywhere in your city--set buildings ablaze, pelt your metropolis with meteors, unleash a twister, or wreak havoc with a volcano. Populate your terrain with animals that graze and roam and sometimes stampede into your city! Play Mayor Create, build and run the most realistic city you can imagine. Connect your metropolis with other cities you've created to form a massive region of SimCities, each sharing and competing for resources. Deploy police cruisers and fire trucks to the scene--youre up close and part of the action as they deal with fires, striking mobs, crime, and more. Play With Your Sims Place your Sims in your city--they'll give you valuable feedback. Follow your Sims to work and home. Experience life in the big city, from mellow traffic to commuter hell, noontime crowds to nighttime calm.

These examples are just an illustration of the possibilities available for teachers. If they do not have easy access to the Internet in their classes, they can provide transparencies for the overhead projector and discuss them in class with their students. EFL students interviewed by Heberle (2006) explained that they learned a lot of English through video games, as they wanted to play but had to understand the instructions, the verbs related to actions and the descriptions of the scenes. Different frames of the selected site for analysis can be used for classroom discussion. The class can observe the actions in the narrative sections and the teacher can then help students to construct a narrative with different frames, using specific metalanguage from visual grammar to make their stories vivid. Alternatively, students may be asked to examine the site outside class, as homework assignments, and bring pictures of their own favourite video games to comment on in class.
Cadernos de Letras (UFRJ) n.27 dez. 2010
http://www.letras.ufrj.br/anglo_germanicas/cadernos/numeros/122010/textos/cl301220100viviane.pdf

110

From different kinds of video games, besides analysing the scenes visually in terms of actions, the class can discuss the verbal actions, the wording used to describe the main characters and their missions. The class could, for instance, refer to the different kinds of interpersonal meanings, the verbal resources used to establish a relationship between the writers and the readers, following the APPRAISAL system network proposed by Martin (2000; MARTIN and ROSE, 2003; ROTHERY and STENGLIN, 2000), constituted by three broad options: attitude, amplification and source. Attitude includes AFFECT (related to feelings), JUDGEMENT (related to character) and APPRECIATION (related to value) 4. Another discussion might be proposed regarding the visual and verbal meanings and violence, whether the violent scenes shown in a specific game can also be detected in the visuals. Based on APPRAISAL choices, students could examine different kinds of adjectives and explain how the visual representations correlate with the verbal choices. From a critical discourse analysis perspective, reading involves a resistant, subversive, non-cooperative view of the text and its social-historical context (KRESS, 1989; WALLACE, 1995; HEBERLE, 2000). Video games such as StarWars, Ultraviolet, The Godfather Xbox and Perfect Dark Zero can be critically discussed in ESL/EFL classes in terms of representation of violence. Teachers might include the three general questions by Meurer (2001), regarding representations, relations and identities in these games. The questions are: (1) how does this text represent the specific reality it relates to?
(2) what kind of social relations does this text reflect or bring about? (3) what are the identities, or the social roles, involved in this text?

One might add a further question to query the textual metafunction as well. For instance: (4) How is the text organized to create certain representation(s), relation(s) and identity(ies).
Studying English through advertisements

In the twenty-first century, there is no need to state the pervasiveness of advertising in peoples lives, whether we refer to billboards, TV, magazines, leaflets and other kinds of semiotic resources which reinforce the promotional character of contemporary culture (WERNICK, 1991; FAIRCLOUGH, 1995; 2003). Advertisements are ubiquitous, an inevitable part of everyones lives (WILLIAMSON, 1978:11). Berger tells us:
In the cities in which we live, all of us see hundreds of publicity images everyday of our lives. No other kind of image confronts us so frequently. In no other form of society in history has there been such a concentration of images, such a density of visual messages. (BERGER, 1977: 129)

To Wernick (1991:92-5), advertising has impacts on contemporary culture, and at the same time that [w]e are bombarded with images which convert consumer goods and services into seductive tokens of psychological and social values, there is also an intertextual link with other ads, forming a promotional chain, a communicative complex not only with ads
Cadernos de Letras (UFRJ) n.27 dez. 2010
http://www.letras.ufrj.br/anglo_germanicas/cadernos/numeros/122010/textos/cl301220100viviane.pdf

111

but also with non-advertising contents (WERNICK, 1991:92-5). Wernick (1991:181) extends the definition of advertising beyond its competitive, commercial, business and economic roles to a more comprehensive sense, to a broader cultural complex, referring no t only to a type of message but to a type of speech, and, beyond that, to a whole communicative function which is associated with a broader range of signifying materials than just advertisements stricto sensu. Likewise, Cook (1992) emphasizes the prominence of advertising in contemporary society and how it provides information about our society, and Leiss, Kline and Jhally (1997: 5) see advertising as an integral part of modern culture, linking images of persons, products, and well-being. Taking into account the promotional condition of contemporary culture (Wernick, 1991:181), the fact that advertising combines and appropriates different symbols, discourses and cultural forms (LEISS, KLINE and JHALLY, 1997) and that new literacy programs include analysis of different semiotic/multimodal practices, teachers should encourage EFL students to explore the meanings in ads, applying the visual metalanguage proposed for the description of ads, along with the discussion of social values, lifestyles, and ideological meanings. As multiliteracy learning can be bound up with ESL/EFL learning since both processes involve learning how to understand and produce texts in different situations, it seems advertising can serve as an interesting literacy resource in the context of EFL learning. Describing objects, places, people, feelings and actions is a very important function in ESL/EFL materials, and advertisements may provide a rich repertoire of topics for ESL/EFL classes. From New Interchange Book 3, for instance, some of the functions include describing personalities, past events, problems, challenges, frustrations, rewards and qualities for success. These different kinds of description can be integrated with the analysis of language/image relations in the advertisements. Here are some suggested activities with ads. Task 1: Matching words and pictures Preliminary work by the teacher: 1. Select ten different ads which contain words, phrases or full clauses; 2. Write these words, phrases or clauses in separate individual cards; 3. Cross out these words or clauses from the ads so that students cant see them In class: 1. The teacher spreads the ads and the written cards around the desks; 2. Students have to match the verbal and the visual meanings; 3. Class discuss the image/text relations, in terms of ideational meanings: are the pictures adding new information or complementing the verbal text?; 4. If desirable, class could discuss interpersonal/interactive meanings or still textual/compositional meanings too This activity has been applied in different EFL classes in Brazil, and with a discussion of visual grammar, it has become a motivating factor for teenage students. Task 2: Matching descriptions
Cadernos de Letras (UFRJ) n.27 dez. 2010
http://www.letras.ufrj.br/anglo_germanicas/cadernos/numeros/122010/textos/cl301220100viviane.pdf

112

Another task refers to a matching activity. On one pile (which can be spread out around the desks or given to groups of students) students will find short descriptions of pictures, containing metalanguage of visual grammar. We believe six or seven might be a reasonable number of ads for class discussion. The teacher can simplify the descriptions or pre-teach some of the potentially difficult vocabulary. After students discuss the ads and their corresponding descriptions, the class can give more details about the pictures, in terms of visual metalanguage. Depending on the ads selected and students interest and motivation in the discussion, teacher and class can decide which specific kind of meanings will be preferred for analysis: the representational, the interactive or the compositional meanings. The next step concerns analysis of the broader sociocultural context and possible questions here include: 1. What is the ad about? What are its representational meanings? 2. Who is the ad specifically addressed to and what kind of life style does the ad represent or suggest? 3. Are the image/language relations concurrent, in other words, is there an equivalence of participant-process-phenomenon-circumstance representation in the image and the language? 4. Who are the represented participants? Who is looking at whom? What kind of relationship is established between the represented participants and the viewer? Which personal characteristics (also known as attributes, in multimodal theory) or symbols are present in the ad? 5. What kind of clothes are the participants wearing? Are the clothes related to concepts such as elegance, romance, casual-ness (based on Hall, 1997), or adventure, danger, calmness, power, wealth, freedom, etc.? 6. What kind of background is there? Is there any part in the picture which is framed, separated from the central figure? 7. What sociocultural meanings are represented? Are these meanings conveyed in the ad similar to social values in your community or country? An alternative visual-verbal activity includes students analysis of the three metafunctions, as proposed by Kress and van Leeuwen (1996), and also used in multiliteracy studies by Unsworth (2001) and Callow (1999), following Halliday. Possible questions related to these metafunctions are: - What is the picture about? Who are the participants involved, and what circumstances are represented in the photograph? - What is the relationship between the viewer and what is viewed? - How are the meanings conveyed? How are the representational structures and the interactive/interpersonal resources integrated into a whole? After students answer these questions and present their analyses of the ads, the whole class can discuss broader social issues and the relevance of images to make people better understand them.
Cadernos de Letras (UFRJ) n.27 dez. 2010
http://www.letras.ufrj.br/anglo_germanicas/cadernos/numeros/122010/textos/cl301220100viviane.pdf

113

Conclusion: Visual-verbal tasks for ESL/EFL teenage learners Teenagers may find that examining visual-verbal meanings can be stimulating in terms of their productive and receptive skills in English. I understand that applying visual literacy in the case of EFL teaching for teenagers represents a valid path for meaningful learning. Task-based or content-based instruction with analysis of images and their interpretive possibilities and classroom discussions around them, such as the ones suggested in this paper, hold the potential to expand students skills in le arning English. What I am also proposing is the integration of visual literacy skills in students learning, to make them understand and explore the notion that images are not evident and obvious, but socioculturally constructed (KRESS and van LEEUWEN, 1996; STENGLIN and IEDEMA, 2001). Thus visuals are not to be seen as a separate or add-on strategy, but as a valid tool in EFL teaching and learning. I hope to have emphasized the relevance of multimodal meaning-making in different literacy pract ices in teachersand students academic, social/cultural and civic life. No doubt the classroom activities presented are just suggestions, not strict rules to be followed, and teachers need to take into account their own institutional demands but most importantly their own students sociocultural and personal values. For this, it is important to redraw the disciplinary and cultural boundaries of foreign language study towards what Kramsch (1993:12) calls the pursuit of communicative happiness. We believe that incorporating a multimodal, multiliteracy approach to the TESOL context may contribute to a better understanding of these boundaries. Viviane M. Heberle UNIVERSIDADE FEDERAL DE SANTA CATARINA (UFSC)

This paper is partially funded by the Brazilian Research Council CNPq (Project n. 305756/2008-7)

Notes
11 2

. ESL stands for English as a Second Language, while EFL refers to English as a Foreign Language. According to Perrett (2000: 88) In most schools of linguistics, apart from SFL, research into SLD [second language development] is commonly known as second language acquisition (SLA) researchDevelopment is preferred by SFL linguists since it connotes the social nature of language learning. 3 Based on Bernstein, Christie uses the term register and not discourse as Bernstein proposes. She also explains that regulative register refers to the mechanisms and management of classroom activities, and instructional discourse refers to academic content and its necessary skills. 4 According to Martin and Rose (2003:22), Appraisal is concerned with evaluation: the kinds of attitudes that are negotiated in a text, the strength of the feelings involved and the ways in which values are sourced and readers aligned We use resources of APPRAISAL for negotiating our social relationships, by telling our listeners and readers how we feel about things and people.

References BERGER, J. Ways of seeing. London: Penguin Books, 1977.

Cadernos de Letras (UFRJ) n.27 dez. 2010
http://www.letras.ufrj.br/anglo_germanicas/cadernos/numeros/122010/textos/cl301220100viviane.pdf

114

BROWN, H. D. Principles of language learning and teaching (3rd ed.). Englewood Cliffs, NJ: Prentice Hall Regents, 1994. CALLOW, J. (Ed.). Image matters: Visual texts in the classroom. Marrickville, NSW: Primary English Teaching Association, 1999. CANALE, M.; SWAIN, M. Theoretical bases of communicative approaches to second language teaching and testing. Applied Linguistics, 1(1), 1980, 1-17. CHRISTIE, F. Classroom discourse analysis. A functional perspective. London and New York: Continuum, 2002. COOK, G. The discourse of advertising. London: Routledge, 1992. DOUGHTY, C. Acquiring Competence in a Second Language. In BYRNES, H. (Ed.). Learning Foreign and Second Languages New York: The Modern Language Association of America, 1998. 128-156. FAIRCLOUGH, N. Media Discourse. London: Edward Arnold, 1995. FAIRCLOUGH, N. Analysing discourse: Textual analysis for social research. London and New York: Routledge, 2003. HALL, S. Representation, cultural representations and signifying practices. London: Sage, in association with The Open University, 1997. HALLIDAY, M. A. K. Language as social semiotic : the social interpretation of language and meaning. London: E. Arnold, 1978. HALLIDAY, M. A. K. An introduction to functional grammar (2nd ed.). London: E. Arnold, 1994. HEBERLE, V. M. Critical reading: Integrating principles of critical discourse analysis and gender studies. In Revista Ilha do Desterro - A Journal of English Language, Literatures in English and Cultural Studies, ISSN - 0101-4846, N. 38, 2000. HEBERLE, V. M. Multimodality and EFL teaching in Brazil: Classroom and conference notes. Florianpolis, mimeo., 2006. HEBERLE, V. M.; MEURER, J. Aspects of visual analysis for the EFL class. In Anais do I Congresso Internacional da ABRAPUI. Belo Horizonte: UFMG/FAPEMIG, 2007, 1-11. HYMES, D. On communicative competence. In PRIDE, J.; HOLMES, J. (Eds.). Sociolinguistics. Hasrmondsworth, UK: Penguin, 1972, 269-293. KLEIMAN, A.; BALTAR, M. Letramento e formao de professores. Linguagem em (Dis)curso, 8(3), 2008. KRAMSCH, C. J. Context and culture in language teaching. Oxford: Oxford University Press, 1993. KRAM,CH, C. The applied linguist and the foreign language teacher: Can they talk to each other? In COOK, G. ; SEIDLHOFER, B. (Eds.). Principle and practice in Applied Linguistics. Oxford: Oxford University Press, 1995. KRESS, G. Linguistic Processes in Sociocultural Practices. Oxford: Oxford University Press, 1989. KRESS, G. Literacy in the New Media Age. London/New York: Routledge, 2003. KRESS, G.; van LEEUWEN, T. Reading images : The grammar of visual design. London: Routledge, 1996. KRESS, G. R.; van LEEUWEN, T. Reading images : the grammar of visual design (2nd ed.). London ; New York: Routledge, 2006.

Cadernos de Letras (UFRJ) n.27 dez. 2010

http://www.letras.ufrj.br/anglo_germanicas/cadernos/numeros/122010/textos/cl301220100viviane.pdf

115

LAM, W. S. E. L2 Literacy and the Design of the Self: A Case Study of a Teenager Writing on the Internet. Tesol Quarterly, 34(3), 2000, 457-482. LEISS, W., KLINE, S.; JHALLY, S. Social communication in advertising. ondon/New York: Routledge, 1997. MACHIN, D. Building the world's visual language: the increasing global importance of image banks in corporate media. Visual Communication, 3(3), 2004, 316-336. MACKEN-HORARIK, M. Interacting with the multimodal text: reflectioins on image and verbiage in ArtExpress. Visual communication, 3(1), 2004, 5-26. MARTIN, J. R. Beyond exchange: Appraisal systems in English. In HUNSTON, S.; THOMPSON, G. (Eds.). Evaluation in text: Authorial stance and the construction of discourse (Vol. 1). Oxford: Oxford University Press, 2000, 142-175. MARTIN, J. R.; ROSE, D. Working with discourse: Meaning beyond the clause. London/New York: Continuum, 2003. MEURER, J. L. The three non-mystifying questions you can ask and explore in the texts you bring to your EFL classrooms. Florianpolis, Brazil. APLISC Newsletter, 8(2), 2001, 3-4. PERRETT, G. Researching second and foreign language development. In UNSWORTH, L. (Ed.). Researching language in schools and communities. London/Washington: Cassell, 2000. POOLE, D. Discourse analysis and applied linguistics. In KAPLAN, R. B. (Ed.). The Oxford Handbook of Applied Linguistics. Oxfrord/ New York: Oxford University Press, 2002, 73-84. RIOS, G. Literacy Discourses: A Sociocultural Critique in Brazilian Communities. Saarbrcken, Germany: VDM Verlag Dr. Mller, 2009. ROTHERY, J.; STENGLIN, M. Interpreting literature: the role of APPRAISAL. In UNSWORTH, L. (Ed.). Researching language in schools and communities. London and Washington: Cassell, 2000, 222-244. STENGLIN, M.; IEDEMA, R. How to analyse visual images: A guide for TESOL teachers. In BURNS, A. ; COFFIN C. (Eds.). Analysing English in a global context: A reader. London: Routledge, 2001, 194-208. THE NEW LONDON GROUP. A pedagogy of multiliteracies: Designing social futures. Harvard Educational Review,, 66(1), 1996, 60-91. THOMPSON, G. Introducing functional grammar. London/ New York: Arnold, 1996. UNSWORTH, L. Teaching multiliteracies across the curriculum: Changing contexts of text and image in classroom practice. Buckingham, UK: Open University, 2001. van LIER, L. Constraints and resources in classroom talk: Issues of equality and symmetry. In BYRNES, H. (Ed.). Learning foreign and second languages: perspectives in research and scholarship. New York: The Modern Language Association of America, 1998. WALLACE, C. Reading with a suspicious eye: Critical Reading in the foreign language classroom. In COOK , G.; STEIDLHOFER, B. (Eds.). Principle and Practice in Applied Linguistics. Oxford: Oxford University Press, 1995, 335-347. WERNICK, A. Promotional Culture: Advertising, ideology and symbolic expression. London: Sage, 1991. WESCHE, M. B.; SKEHAN, P. Communicative, task-based, and content-based language instructions. In KAPLAN, R. B. (Ed.), The Oxford Handbook of Applied Linguistics. Oxford/New York, 2002, 207-228. WILLIAMS, G.; HASAN, R. Literacy in society. London ; New York: Longman, 1996.

Cadernos de Letras (UFRJ) n.27 dez. 2010

http://www.letras.ufrj.br/anglo_germanicas/cadernos/numeros/122010/textos/cl301220100viviane.pdf

116

WILLIAMSON, J. Decoding advertisements: Ideology and meaning in advertising. London/ New York: Marion Boyars, 1978.

Abstract

Literacy nowadays is understood as a complex set of reading, writing and technological skills which joins verbal, visual, and other meaning-making resources. Visual literacy can be linked to multimodality, which refers to the use of different semiotic resources to produce or interpret meanings. In this paper, following Royce (2007), first I discuss the notion of multimodal communicative competence as an important skill for ESL/EFL students to develop, in order for them to interact more effectively with members of English-speaking discourse communities. Then I provide readers with a brief discussion on the relevance of multimodality in ESL/EFL teaching and offer suggestions on how to relate visual literacy specifically to teenage learners of ESL/EFL, as found in video games and advertisements, drawing attention to pedagogical, taskbased activities which can be incorporated in the EFL syllabus. Keywords: Multimodal literacy, semiotic resources, EFL teaching, visual grammar.

Resumo Letramento atualmente entendido como um conjunto complexo de habilidades de leitura, escrita e de tecnologia, que inclui recursos verbais e visuais. Letramento visual pode estar ligado a multimodalidade, que se refere ao uso de diferentes recursos semiticos para produzir ou interpretar significados. Neste trabalho, com base em Royce (2007), primeiramente discuto a noo de competncia comunicativa multimodal como fator importante para o estudo de ingls como lngua estrangeira ou segunda lngua, a fim de que os estudantes possam interagir mais eficazmente com membros de comunidades discursivas falantes de ingls. A seguir, brevemente discuto sobre a relevncia da multimodalidade no ensino de ingls e apresento sugestes sobre como relacionar letramento visual especificamente para adolescentes aprendizes de ingls, atravs de videojogos e propagandas, com ateno voltada para atividades pedaggicas baseadas em tarefas, que podem ser incorporadas no programa da disciplina de ingls. Palavras-chave: Letramento multimodal, recursos semiticos, ensino de ingls como lngua estrangeira, gramtica visual.

Cadernos de Letras (UFRJ) n.27 dez. 2010

http://www.letras.ufrj.br/anglo_germanicas/cadernos/numeros/122010/textos/cl301220100viviane.pdf

Bezerra, F. (2011). Multimodality in the EFL classroom. BELT Journal, 2(2), 167-177.

Multimodality in the EFL classroom Multimodalidade na sala de aula de ingls Fbio BEZERRA1

Abstract: Due to the increased multimodal nature of communication in todays globalized and culturally diverse world (Christie, 2005), it is paramount to devote attention in the classroom to how semiotic resources (van Leeuwen, 2005) other than verbal language have been used to create identities and to position people socially, especially for the fact that the school plays or should play a vital role in peoples individual, social, cultural and political development. In this paper, I contextualize the work with images and present metalanguage from the Grammar of Visual Design (Kress & van Leeuwen, 2006) in order to enable teachers to develop activities to foster their students multimodal communicative competence (Royce, 2007; Heberle, 2010). Besides, I carry out analysis of a movie poster so as to put the theory presented into practice. Keywords: multimodality; classroom; grammar of visual design; multimodal communicative competence Resumo: Devido crescente natureza multimodal da comunicao no mundo globalizado e culturalmente diverso de hoje (Christie, 2005), fundamental que dediquemos ateno na sala de aula maneira como recursos semiticos (van Leeuwen, 2005) alm da linguagem verbal tm sido utilizados para criar identidades e posicionar as pessoas socialmente, especialmente pelo fato de que a escola desempenha - ou deveria desempenhar - um papel vital no desenvolvimento das pessoas nos mbitos individual, social, cultural e poltico. Neste artigo, contextualizo o trabalho com imagens e apresento metalinguagem da Gramtica Visual (Kress & van Leeuwen, 2006) a fim de capacitar professores para desenvolverem atividades que promovam a competncia comunicativa multimodal de seus alunos (Royce, 2007; Heberle, 2010). Alm disso, analiso um cartaz de filme com o objetivo de colocar a teoria apresentada em prtica. Palavras-chave: multimodalidade; sala de aula; gramtica visual; competncia comunicativa multimodal

1 Introduction In our globalized and culturally diverse world, communication is increasingly multimodal (Christie, 2005, p.123), hence the importance of devoting attention in the classroom to how semiotic resources (van Leeuwen, 2005) other than verbal language have been used to create identities and to position people socially, especially for the fact that the school plays or at least should play a vital role in peoples individual, social, cultural and political development. In this context, more actions should be taken to foster students multimodal communicative competence (Royce, 2007), which involves the knowledge and use of language

concerning the visual, gestural, audio and spatial dimensions of communication, including
1

I'm currently in the last year of my doctorate in English and Applied Linguistics at Universidade Federal de Santa Catarina (UFSC) and my PhD in Linguistics at the University of Sydney (cotutelle).

BELT Journal Porto Alegre v.2 n.2 p. 167-177 julho/dezembro 2011

167

computer-mediated-communication (Heberle, 2010, p. 102) besides the other communicative

competences described by Hymes (1972) and Canale and Swain (1980). Thus, this new reality demands immediate actions from all of us, teachers and students, in order to develop a better understanding of how verbal language and images construe representations of our experience and relationships between social actors as well as how these are brought together in a textual manifestation as a cultural construct (Bezerra, Nascimento & Heberle, 2010; MottaRoth & Nascimento, 2009, p. 320). It should also be emphasized that the work with multimodal texts in the EFL classroom can be used to develop not only students reading skills, but also their writing, speaking and listening competence, which could help consolidate a holistic approach to the teaching of the four skills (DAndrea, 2010). Facing this challenge, the New London Group2 published an article in the Harvard Educational Review, where they present a theoretical overview of the connections between the changing social environment facing students and teachers and a new approach to literacy pedagogy that they call multiliteracies (1996). In this new pedagogy, the following four components are suggested for the work with the students: situated practice, overt instruction, critical framing and transformed practice. Situated practice refers to the need to approach whatever meaning-making resource from the starting point of the personal experiences of students so that they can locate themselves in relation to the study to be done, especially for the fact that meanings belong to culture, rather than to specific semiotic modes (Kress and van Leeuwen, 2006, p.2). This term is actually closely related to Gees discussion of situated meaning, where he states that meaning is always situated in specific sociocultural practices and experiences (2000, p.195). Overt instruction would be the moment to provide students with the metalanguage to carry specific investigations. This is exactly the main objective of the present paper, since the analysis carried out is done by using the metalanguage proposed by Kress and van Leeuwen (2006) in their visual grammar, which we must teach our students in order to provide them with the tools to understand that certain images and ideas about the world are favoured or presented as normal, while other possibilities as excluded, either implicitly or explicitly (Callow, 1999, p.2). Critical framing is fostered by having students interpret the contextual background and values which inform whatever social practice and its related text. In other words, students are engaged in the activity of critically deconstructing what is given to them, whether in the form of

For information about the New London Group and access to their seminal article on multiliteracies, see http://wwwstatic.kern.org/filer/blogWrite44ManilaWebsite/paul/articles/A_Pedagogy_of_Multiliteracies_Designing_Soci al_Futures.htm#11

BELT Journal Porto Alegre v.2 n.2 p. 167-177 julho/dezembro 2011

168

written texts, images, sound etc. Therefore, being literate means being able to understand, enjoy and critique the kinds of messages that these images convey (Callow, 1999, p.2). Transformed practice is the idea that students, having learned how texts of various kinds are constructed, start designing their own practices based on the new knowledge in the same or in new contexts. This is the moment when the teacher will really have the chance to see just how much students could demonstrate an appropriation of not only the metalanguage which was taught, but most importantly of a new mindset towards the reading of varied texts, which is especially important in such a fast-changing world, where many of the texts presented to students in textbooks are non-linear (Christie, 2005).

2 The grammar of visual design Taking all the aforementioned points into consideration, especially the one about the situated practice, as visual language is not () transparent and universally understood; it is culturally specific (Kress and van Leeuwen, 2006, p.4), I aim to demonstrate the usefulness of the categories for the analysis of images described in the Grammar of Visual Design (Kress & van Leeuwen, 2006). For that matter, I analyze a movie poster used to promote the motion picture Sex and the City, since it is based on the homonymous world famous3 TV series (Bezerra, 2008) and has been strongly advertised by the media. Besides that, this can also be seen as an example of how to use different genres in the EFL classroom, which may contribute to making the work with another language more relevant for the students in the new communicative configuration that permeates the learning environment nowadays (Lange, 2010). Before starting the analysis, it is important to have an overview of the metalanguage used, for it facilitate[s] systematic attention by teachers to the multimodal nature of texts in developing critical literacy practices (Unsworth, 2001, p.72). It is important to state, though, that it is the teachers responsibility to consolidate their knowledge of the categories used for analysis by doing additional readings (see References). Even though all meaning-making processes have always been multimodal, not much attention has been paid to the meanings of regularities in the way image elements are used (Kress and van Leeuwen, 2006, p.1). This is due to the longstanding misconception that images are transparent representations which do not require instruction to be understood. It is precisely one of
3

Sex and the City is viewed in the following countries, as informed in Wikipedia The Free Encyclopedia (http://en.wikipedia.org/wiki/Sex_and_the_city): U.S.A., Canada, Mexico, Brazil, Germany, United Kingdom, Ireland, France, Netherlands, Sweden, Italy, Belgium, Spain, Finland, Russia, Romania, Bulgaria, Turkey, Australia, New Zealand, Japan, Malaysia, Thailand, Singapore, Hong Kong, India, Pakistan, Philippines, Lithuania, Latvia, Denmark and Hungary.

BELT Journal Porto Alegre v.2 n.2 p. 167-177 julho/dezembro 2011

169

the aims of a project for multiliteracies that students should have a chance to be introduced to descriptions of images, and other semiotic resources, in much the same way that their verbal literacy is fostered in schools. Therefore, in the Grammar of Visual Design (GVD), Kress and van Leeuwen (2006) propose a more systematic approximation of the image in a tri-functional perspective, as in Halliday and Matthiessens (2004) Introduction to Functional Grammar, of which their work is an extrapolation. In other words, they develop visual analysis having the following three metafunctions as realizing meanings: the representational, the interactional and the compositional.

2.1 The representational metafunction

The representational metafunction concerns how participants and objects as well as events (processes) and their associated circumstances are realized, which may be done through narrative or conceptual representations. The narrative representations concern actions, reactions, thought and speech. Actions and reactions are represented by the presence of a vector connecting the participants. In actions, the vector departs from the actor and is directed towards the goal, in the case of the bears left arm touching the other bear in Figure 1. In some images, the vector can be bi-directional, with both participants being at the same time actor and goal, which would be the case if Figure 1 both bears were touching each other. In reactions, the vector is formed by the eye line, that is, one participant (reactor) is looking at another (phenomenon), or both are looking at each other (bi-directionality). In Figure 2, for instance, some people are foregrounded and we see that they are gazing at something. It is also important to add that Figure 2

actions and reactions can be either transactional or nontransactional, depending on whether or not both represented participants can be seen by the viewer. Thus, in Figure 2, it is a non-transactional process of reaction, since we do not see what those people are looking at. Figure 3 Figure 4

BELT Journal Porto Alegre v.2 n.2 p. 167-177 julho/dezembro 2011

170

Now, turning to thought and speech, one must know that the former is depicted by the use of a thought cloud (Figure 3) and the latter by means of a speech balloon or bubble (Figure 4), both being connected to the participants (senser and sayer, respectively) also by a vector. The conceptual representations, in turn, refer to images that either classify Figure 5

(covert or overt taxonomy), show part-whole relationships (analytical structured or unstructured) or attribute/suggest values (symbolic). In Figure 5, we see a covert representation of types of car. It is covert because it is not overtly stated Figure 6 in the image what it is classifying. In Figure 6, the focus is only on part of an aircraft (its engines), while in Figure 7 the golden hue arguably adds a symbolic value of wealth to the city. I do not extend on these categories in the present paper, since they are not found in the posters analyzed. However, for further information, see Kress and van Leeuwen (2006). Figure 7

2.2 The interactional metafunction

The interactional metafunction refers to the interrelation between the image and the viewer. This relationship can be analyzed in terms of interaction (contact), social distance, attitude, power and realism. Contact can be defined as either demand or offer, depending on whether or not the represented (human, human-like or animal) participant is looking directly at Figure 8 the viewer. Figure 8 is an instance of offer. In the case of a demand, there is the construal of a stronger relationship. Also, such gaze directed at, or away from, the viewers can be further emphasized by the participants facial expression. Social distance may happen in one of three levels: intimate (close shot), social (medium shot) or impersonal (long shot). In Figure 9, we see a long shot. Attitude may either show Figure 10 involvement (frontal angle) or detachment (oblique angle as in Figure 10), that is, it depends on whether or not the frontal angle of the represented participant and the viewer coincide. Power may be attributed to the represented participant (low angle), to the Figure 9

BELT Journal Porto Alegre v.2 n.2 p. 167-177 julho/dezembro 2011

171

viewer (high angle) or there may be a sense of equality (eye-level view). Finally, realism is measured by how color, context, depth, detail and light interplay, with low modality as a representation located away from a naturalistic view in a continuum, which happens, for instance, in black and white images.

2.3 The compositional metafunction

The compositional metafunction has to do with the distribution of the elements and information in the image, that is, the way in which the representational and interactive elements are made to relate to each other, the way they are integrated into a meaningful whole (Kress and van Leeuwen, 2006, p.176). Three elements are analyzed in the compositional organization of images: information value, framing and salience. When dealing with the information value in the image, we observe the layout in regard to the left-right, top-bottom and center-margin positioning. In this perspective, while what is located on the left-hand side of the page is taken as given information (like books in Figure 11), the rightFigure 11 hand side brings the new information being introduced to the viewer (computers in Figure 11). Also, while the bottom of the page represents the real information, that is, what the viewer identifies as more concrete, the top part brings the ideal, what is supposed to be aspired to by the viewer. The layout may also foreground a central positioning, giving prominence to certain elements, while having others as a marginal surrounding structure. Besides information value, an image also has framing, which can be strong or weak, depending on whether the elements are shown as being connected or disconnected. Finally, we can also analyze salience in the image, by identifying which factors may give prominence to specific elements in the image, which can be done, for instance, through the use of relative size, color and foregrounding (like the glass of wine in Figure 12). Figure 12

3 Analysis

BELT Journal Porto Alegre v.2 n.2 p. 167-177 julho/dezembro 2011

172

Having contextually situated the analysis and presented the metalanguage needed, we can now focus on the poster chosen for the present paper (Figure 13). In this analysis, I aim to demonstrate how the knowledge of the metalanguage can help the reader be aware of the classificatory and constitutive nature of both language and images, which, hopefully, can be a step towards having more teachers designing activities including the analysis of images in the classroom based on sound terminology and theory. In this figure, I bring not only the movie poster, but also a summary of the points discussed, so as to provide an overview of the image construal.
Interactional
(Contact) Offer

VISUAL GRAMMAR

Representational

(Social Distance) Impersonal/Long Shot (Attitude) Point of View/Oblique Angle Detachment (Power) Eye-level Vi ew (Realism) Medium Modality

(Process) Narrative: Acti onal > Non-transac tional (Participant) Actor: Carrie (Circumstance) SATC (skyscraper)

Compositional SFL
Ideational
What: Get Carried Away Where: Only in Theaters When : May 30
(Information Value) Given/N ew & Ideal/Real (Framing) Connec ted (Salien ce) Size, Color and Foreground

Figure 13

In the poster of Figure 13, there is a narrative representation of an actional process, as can be seen through the vector formed by Carries dress and legs, showing her (actor) walking; however, this action is non-transactional, since we dont see any other represented participant (goal). The name of the movie (Sex and the City) can be seen as a circumstance of location, especially for its relative size in comparison to Carrie, which suggests an idea that it might be a skyscraper behind her, being integral to a glamorous and sparkling city. This focus on the action is also corroborated by a transitivity analysis (Halliday and Matthiessen, 2004) of the verbal language used in the poster: Get

BELT Journal Porto Alegre v.2 n.2 p. 167-177 julho/dezembro 2011

173

Carried Away Only in Theatres May 30. There is the use of a material process aiming to entice the viewer into watching the movie by a play of words with the main characters name (Carrie). This way, a choice of action is being given to the viewer, but while watching the movie, this action is consequently passed on to Carrie, who will then carry us away throughout the narrative. However, it is highlighted that this encounter can only happen in theatres and on a specific day, which are the circumstantial elements of the event. Looking at the interactive meanings construed in the image, we note that there is not eye contact with the viewer, which suggests only an offer. This offer being the possibility of seeing the movie and having the opportunity to be carried away. This invitation is done from an impersonal distance, as it is depicted in a long shot. This point is also reinforced by the detachment created through the oblique angle between Carries body and the viewer. However, this detachment is softened by the fact that there is an equal relation of power, created by the eye-level view, which makes sense as the creator of the image probably wanted to foster proximity with the movie audience. What calls attention, though, in this poster is the fact that there is use of medium to low modality, as the representation of both Carrie and the city is not done in a very naturalistic manner (both are placed against a black artificial background), which, as a matter of fact, does not seem to work in favor of the portrayal of Carrie as a real woman who goes through the same ups and downs as many others. Although we can highlight this possible contradiction, there is also the fact that it might be important for the success of the movie to show its main character as someone to be admired precisely for the fact that she represents something the audience may aspired to be like, which is actually done compositionally. Therefore, in a compositional perspective, we see Carrie on the right-hand side of the poster and the play with her name in the clause Get Carried Away on the left-hand side, suggesting that even though Carrie is someone we are already familiar with, she always has something new to present, either in her appearance or in her life. The place where Carrie is inserted the city is located at the top (the ideal) of the image, since New York City (where the story in the movie takes place) is a much sought-after destination for its glamorous and sizzling life style. However, the information about the movie itself (where and when it can be seen) is given at the bottom of the poster, indicating that this is a real element, on which viewers can rely. The weak framing of the image shows Carrie and the city completely connected, which is further indicated by the use of the color pink in the name of the movie (the city skyscraper) and her dress, also suggesting a mixture of romance and modernity, as the shade of pink used is really strong and bright. Also discussing the salience in the image, we can notice that, even though Carrie is smaller in comparison to the name of the movie, she is foregrounded. However, this foregrounding is not so salient, which may be

BELT Journal Porto Alegre v.2 n.2 p. 167-177 julho/dezembro 2011

174

understood as the portrayal of Carrie as an integral part to the city and vice-versa, a fact which was strongly emphasized in the TV series of which the movie is a continuation. In sum, we see the interconnection of the three metafunctions towards a representation of Carrie as a woman who both acts and represents new things to come, which are to be seen by those who will eventually watch the movie. The city of New York is portrayed as a glamorous and shining place, basically though the use of relative size, color and light, that is, through compositional elements. Again, it is worth stressing that the representation of Carrie and the city in a medium to low modality seems to have been an element of contradiction in the whole message conveyed by the poster, for a higher modality could have added to the idea of bringing the viewer closer to her and the circumstance where she is situated. However, this fact is somehow softened by the use of other representational, interactional and compositional resources.

4 Final Remarks

In conclusion, it is important to bring forth once more the need for systematic actions towards the preparation of teachers to work with multimodal texts in their classes (Heberle & Meurer, 2007), especially for the fact that, as previously put, our students will undoubtedly have increased contact with texts which not only use varied semiotic resources, but whose access has also changed, including to a greater extent the computer-mediated technologies. Additionally, it is vital for teachers to be constantly reviewing their practice so as to keep up with the new challenges, not only in regard to the myriad of semiotic resources and media available in todays society, but also in respect to the role of the English language in the new world order (Salles & Gimenez, 2010) and to their role to prepare [students] to function effectively in

such contexts (Ur, 2010). Therefore, the ultimate aim of this paper has been to provide teachers
with some theoretical basis and related metalanguage as well as to demonstrate that the work with multimodal texts can be not only stimulating, but most importantly emancipating.

References BEZERRA, F. SEX AND THE CITY: an investigation of womens image in Carrie Bradshaws discourse as narrator. Dissertao de Mestrado. Ps-graduao em Ingls. Universidade Federal de Santa Catarina, Florianpolis, SC, 2008. BEZERRA, F.; NASCIMENTO, R. G.; HEBERLE, V. M. Anlise multimodal de anncios do programa Na Mo Certa. Revista Letras (UFSM), Santa Maria, v. 20, n. 40, 2010, p. 9-26.

BELT Journal Porto Alegre v.2 n.2 p. 167-177 julho/dezembro 2011

175

CALLOW, J. Reading the visual: an introduction. In Callow, J. (Ed.). Image matters: visual texts in the classroom. Newtown, NSW: Primary English Teaching Association (PETA), 1999, p. 1-13. CANALE, M.; SWAIN, M. Theoretical bases of communicative approaches to second language teaching and testing. Applied Linguistics, v. 1, n. 1, 1980, p. 1-17. CHRISTIE, F. Language education in the primary years. Sydney: UNSW Press, 2005. DANDREA, L. Using writing to develop communicative competence in the foreign language classroom. BELT JOURNAL, Porto Alegre, v. 1, n. 2, 2010, p. 139-146. GEE, J. P. Discourse and sociocultural studies in reading. In Kamil, M. L., Mosenthal, P. B., Pearson, P. D., & Barr, R. (Eds.), Handbook of reading research, vol. III. New Jersey, USA: LEA, 2000, p. 195-207. HALLIDAY, M. A. K.; MATTHIESSEN, C. An introduction to functional grammar. 3. ed. London: Edward Arnold, 2004. HEBERLE, V. Multimodal literacy for teenage EFL students. Caderno de Letras (UFRJ), n. 27, 2010, p. 101-116. HEBERLE, V., MEURER, J. L. Aspects of visual analysis for the EFL class. Anais do I Congresso Internacional da ABRAPUI. Belo Horizonte: UFMG/FAPEMIG, 2007, p. 1-11. LANGE, C. A sala de aula contempornea. BELT JOURNAL, Porto Alegre, v. 1, n. 2, 2010, p. 99105. KRESS, G., van LEEUWEN, T. Reading images: the grammar of visual design. (2ed.). London/NY: Routledge, 2006. HYMES, D. On communicative competence. In: PRIDE, J.; HOLMES, J. (Eds.). Sociolinguistics. Hasrmondsworth, UK: Penguin, 1972, p. 269-293. MOTTA-ROTH, D.; NASCIMENTO, F. S. Transitivity in visual grammar: concepts and applications. Linguagem & Ensino UCPel, Pelotas, v. 12, n. 2, 2009, p. 319-349. NEW LONDON GROUP. A pedagogy of multiliteracies: designing social futures. Harvard Educational Review, 66(1), 1996, p. 60-91. ROYCE, T. Multimodal communicative competence in second language contexts. In: ROYCE, T.; BOWCHER, W. (Ed.). New directions in the analysis of multimodal discourse. Mahwah, NJ: Lawrence Erlbaum, 2007, p. 361-403. SALLES, M.; GIMENEZ, T. Ensino de ingls como lngua franca: uma reflexo. BELT JOURNAL, Porto Alegre, v. 1, n. 1, 2010, p. 26-33. UNSWORTH, L. Teaching multiliteracies across the curriculum: changing contexts of text and image in classroom practice. Buckingham, UK: Open University Press, 2001. UR, P. English as a lingua franca: a teachers perspective. Caderno de Letras (UFRJ), n. 27, 2010, p. 85-92.

BELT Journal Porto Alegre v.2 n.2 p. 167-177 julho/dezembro 2011

176

van LEEUWEN, T. Introducing social semiotics. London/New York: Routledge, 2005.

Received: September 20, 2012 Accepted: March 21, 2012

E-mail: fabes10@yahoo.com.br

BELT Journal Porto Alegre v.2 n.2 p. 167-177 julho/dezembro 2011

177