Multimodal Interaction Language and Modal Configurations

Sigrid Norris
5. Multimodal Interaction – Language and

Modal Configurations
Abstract: This chapter illustrates how language builds one system with all other
modes in interaction. Elucidating this point, the chapter examines language and
modal configurations in multimodal actions in a small art school in Germany. First,
the historical development of multimodal (inter)action analysis is outlined. Then, rel-
evant theoretical concepts are discussed. Following, the theory is exemplified by an
analysis of (inter)action in the art classroom. Here, the chapter investigates practices
of speaking about colour in the art school based on audio and video data collected
over four months. Then, it hones in on one representative sample in which the art
teacher interacts with a new art student. The chapter then zooms in further, examin-
ing how language in (inter)action shifts within modal configurations. The conclusion
critically assesses the explanatory value and practical use of multimodal (inter)action
analysis.
1 Introduction
2 Overview – Historical Developments
3 Theory – Multimodal (Inter)action Analysis
4 Applications – Interactions and Modal Configurations in an Art School
5 Conclusion – Language in Multimodal (Inter)action
6 References
1 Introduction
This chapter examines language and modal configurations in interaction using mul-
timodal (inter)action analysis as the theoretical/methodological framework (Norris
2004, 2009, 2011a, 2011b, 2013a, 2014b).
First, this chapter gives a brief overview of how multimodal (inter)action analy-
sis developed, elucidating some pertinent background literature. Second, the chapter
defines and explicates those concepts of the theory that are used to analyse the exam-
ples: site of engagement (Scollon 1998, 2001; Norris/Jones 2005), practice, nexus of
practice (Scollon 1998, 2001) and community of practice (Lave/Wenger 1991), low-
er-level and higher -level actions (Norris 2004, 2011a), and modal configurations
(Norris 2009, 2014b) from multimodal (inter)action analysis (Norris 2013a, 2014b).
Beginning with the concept of site of engagement, I investigate how participants
in art classes in a particular art school speak about colour. This investigation leads me
to zooming in on one representative, higher-level action in which the art teacher inter-

Brought to you by | Staatsbibliothek zu Berlin Preussischer Kulturbesitz
Authenticated
Download Date | 3/22/19 2:54 PM
122 Sigrid Norris
acts with a new student. Here, I provide audio and multimodal transcripts (Norris
2002, 2004, 2011a), illuminating the heuristic value of multimodal (inter)action anal-
ysis. The data comes from a 4-month long ethnographic study that was conducted in
a small private art school in Germany. My main interest at the time was the art teacher
and her (inter)actions. The art school consists of about 45 art students per week; with
the art teacher teaching adult classes, classes for children, classes for teens, and
classes for families. The data consists of observations with detailed observational
notes, 40 hours of audio/video of naturally occurring interaction, and 30 hours of
video-recorded interviews.
2 Overview – Historical Developments

Multimodal (inter)action analysis (Norris 2004, 2009, 2011a, 2013a, 2014) is a theo-
retical and methodological framework to analyse interaction in its complexity. The
framework originally grew out of mediated discourse analysis (Scollon, 1998, 2001),
interactional sociolinguistics (Goffman 1963, 1974; Gumperz 1982; Tannen 1984), and
social semiotics (Kress/van Leeuwen 2001); and also builds on work by Merleau-Ponty
(1962, 1963), Vygotsky (1978, 1986), Birdwhistell (1970), Bateson (1972), McNeill (1992),
and Wertsch (1998).
This theory, takes the mediated action as its unit of analysis, strictly following
mediated discourse analysis (Scollon 1998, 2001; Wertsch 1998). Other theoretical
concepts in mediated discourse analysis are also largely taken on board by multi-
modal (inter)action analysis; albeit, quite a number of them are developed and/or
refined in Norris (2004, 2011a, 2013b, 2014b), Norris & Jones (2005), and Geenen
(2013). Multimodal (inter)action analysis takes from interactional sociolinguistics its
interest in the social origin of interaction, its ability to work on a micro level to dis-
cover instantiations of minute details in interaction, and linking these to the social.
With a theoretical backbone that clearly grew out of mediated discourse analysis, the
theory fosters the joining of micro, intermediate and macro in research. Originally,
multimodal (inter)action analysis (Norris 2004, 2009, 2011a) took on the definition
of mode from social semiotics (Kress/van Leeuwen 2001); i.e.: a mode is a semiotic
system with rules and regularities attached. However, through further development
of the theory, a mode is now defined as a system of mediated action in multimodal
(inter)action analysis (Norris 2013b). The defining of a mode as a system of mediated
action embeds the social actor, allowing for the following:
Looking at modes in this way, we can begin to analyse how there are individual differences as
well as modal overlap. We can also become very clear that the objects in the world, the settings,
or the web pages are infused with socio-cultural histories and with possibilities to act with and
through. However, how social actors use these objects, settings, or web pages, depends upon the
social actors’ modal development. (Norris 2013b, 167)

Authenticated
Multimodal Interaction – Language and Modal Configurations 123
Multimodal (inter)action analysis, thus is a theory of human action that, with its the-
oretical underpinnings and theoretical/methodological tools allows the investigation
of how social actors act and interact.
3 Theory – Multimodal (Inter)action Analysis

The theory of multimodal (inter)action analysis consists of two postulates (Norris
2014b, 183):
1. Social actors are a part of their physical environment, acting with and through it.
2. All modes in use are interconnected, forming one system of communication.
The first postulate disperses the duality between social actors and the environment/
objects within. It is hypothesized that social actors and environment are closely inter-
connected, building a unity. The second postulate hypothesizes that social actors,
as they act with and through the environment, their bodies, objects and psychologi-
cal mediational means, weave all parts together while acting and communicating and
in order to act and communicate. Similarly to McNeill (1992, 2), ‘who asserted that
gestures and language form one system, I theorize that all modes in use build one
system…’ (Norris 2014b, 184).
Multimodal (inter)action analysis itself is made up of many theoretical/method-
ological tools, allowing us to analyse interaction in its complexity. For space reasons,
however, I shall only discuss those tools that are used in the analyses below. These
tools are the site of engagement, practice (including nexus of practice and community
of practice), lower-level and higher-level actions, and modal configurations.
3.1 Site of Engagement
The site of engagement is an analytical tool that allows us to illuminate a moment in

interaction. Building on Scollon (1998), a site of engagement is defined as the window
opened up to make particular concrete actions possible. A site of engagement simul-
taneously comes about through and produces the intersection of multiple converg-
ing practices that make concrete actions possible. A site of engagement, the opened
window, includes place, time, and social actors – their physical, psychological and
historical make-up (Norris 2011a; Makboon 2015).
However, we can envision the closed window as the research question that the
analyst is trying to address. Placing the closed window thus is the first step for a mul-
timdoal (inter)action analysis. It can then be opened wide, partially, narrowly or any-
thing in between to illuminate sites of engagement in which practices converge and
make particular actions possible. In the example below, the closed window is placed

Authenticated
124 Sigrid Norris
over an art school. With this notion, we are able to examine various levels of concrete
(inter)actions.
As analysts, we thus begin our examination by analytically delineating a par-
ticular site of engagement (placing the closed window) before actually illuminating
it (opening the window). Here, I first take a look at the art school, by taking a wide,
yet constricted view on colour-utterance use over time. I then take a closer look at a
particular interaction by narrowing and lengthening the window opening to gain a
focused view of one particular teaching moment in one adult art lesson. As the teach-
ing moment progresses, I zoom in even more, closing the window to only a narrow
slit, to take a very detailed look at modal configurations.
3.2 Practice, Nexus of Practice and Community of Practice
Following Scollon (1998, 2001), practice is defined as an action with a history. Scollon
(2001) illustrated this notion by looking at handing. A social actor hands a ball to
another social actor. A social actor hands a key to another social actor. A social actor
hands a glass of water to another social actor. Each time one social actor hands
something to another social actor. It really does not matter what is handed by who to
whom, we all know and understand what is happening. The reason is that handing
is a practice. Handing is an action with a history that we learn from very young. But
when investigating the people, places, discourses, ideas and objects as they come
together, we speak of a nexus of practice. R. Scollon and S. Scollon (2004) explain:
A nexus of practice is the point at which the historical trajectories of people, places, discourses,
ideas, and objects come together to enable some action which in itself alters those historical tra-
jectories in some way as those trajectories emanate from this moment of social action. (Scollon/
Scollon 2004, 159)
With this explanation, R. Scollon and S. Scollon point to the simultaneity of practices
that construct at the same time as they are constructed. A nexus of practice does not
assume membership or groupings of people, but only a similar way of doing things.
Whereas, the concept of a community of practice (Lave/Wenger 1991) was developed
to illustrate that learning is based on, and in, social interaction within social groups.
For a community of practice to form, a mutual undertaking is needed.
With a multimodal (inter)action approach, we begin our analysis by examining
concrete actions, leading us to uncover practices and nexus of practice, which then
may lead us to the discovery of a community of practice. A community of practice is
thus not presumed, but instead, may be discovered.

Authenticated
3.3 Lower-Level and Higher-Level Actions
Practice, the action with a history, may be a lower-level action as the action of handing
or it may be of a higher-level such as an art lesson. Neither the lower-level, nor the
higher-level action is prior to the other; rather, these two levels constitute each other
simultaneously.
A lower-level action (Norris 2004) is defined as the smallest pragmatic meaning
unit of a mode. Examples of a lower-level action would be the utterance for the mode
of spoken language, the gesture unit for the mode of gesture, or the postural shift for
the mode of posture. Each lower-level action has a beginning and an ending point.
A higher-level action (Norris 2004) is defined as the coming together of a multi-
tude of chains of lower-level actions. Examples of a higher-level action may be an art
lesson or a teaching moment in an art class. Particular levels of higher-level action
are delineated by the analyst, marking the beginning and the ending point as they
are relevant to their research question. However, this does not mean that an analyst
randomly assigns beginning and ending points. Rather, beginning and ending points
are demarcated through the analysis of the data. In the example below, where I am
interested to see what is actually taught at a particular moment, I delineate the high-
er-level action as beginning at the point when the art teacher addresses the student
and ending at the point when the art teacher finishes her explanation. Or, when
interested in understanding the modal configurations during a particular utterance,
I delineated the higher-level action as beginning with the beginning of the utterance
linked to arm/hand/finger movement and ending it with the ending of the utterance
linked to arm/hand/finger movement.
3.4 Modal Configurations
Modal configuration is the hierarchical configuration of lower-level actions (or their

chains) in relation to other lower-level actions (or their chains) within a higher-level
action (Norris 2009, 2011b, 2014b). In order to determine modal configuration, we first
analyse the meaning that has been conveyed in a particular (inter)action. Thus, while
we are interested in investigating the unfolding of interactions, we actually begin our
examination of the interaction as it has occurred and then work backwards.
Modal configurations, I show below, may fluctuate quickly or slowly from one
higher-level action to the next. When investigating modal configurations, we some-
times need to narrow our site of engagement, closing our window to only a narrow
slit, in order to be able to examine these changes. Important here, is that we always
investigate higher-level actions, i.e.: the coming together of multiple lower-level
actions (or chains thereof).

Authenticated
126 Sigrid Norris
4 Applications – Interactions and Modal Configura-

tions in an Art School
For a study in which I investigated the identity production of an artist, I spent much
time in her art school, and the art school became a field site in two respects: 1. It was a
site in which the artist produced her multiple identity elements; and 2. It was a site in
which the artist interacted with many different kinds of art students. For this chapter,
I am interested in the second part of the study.
For this investigation, I first place my closed window over the art school and open
it wide yet constricted so that, when opened, it illuminates the site of engagement,
the art school over the period of four months, showing how various students and
the art teacher speak about colour. This speaking about colour many concrete times
and across many students gives us insight into the practices of how students and the
teacher speak about colour.
I then close the window quite narrowly, illuminating the site of engagement of
the art teacher interacting with a new adult art student, who intends to add a little
black to her painting but cannot decide where to do so (cf. also: Norris, 2014a, 2014b).
This time, the site of engagement focuses on one particular higher-level action, the
higher-level action of the art teacher teaching the student where and where not to
add black.
During this examination, I narrow the window even more, only allowing a small
slit to illuminate the site of engagement of a brief higher-level action in which the
art teacher is re-drawing a line and producing one utterance. I then move the closed
window, opening it again to only a small slit to illuminate the site of engagement of
another brief higher-level action in which the art teacher re-draws a second line while
also producing one utterance. Here, the sites of engagement allow us to illuminate
the chains of lower-level actions and their hierarchical relationships to one another.
Right after this, I again open the window a little, now illuminating the site of
engagement of the art teacher teaching the student which line not to paint black and
why not. Continuing to illuminate the chains of lower-level actions and their hierar-
chical relationships, but this time showing that these relationships can also last over
longer stretches in interaction.
4.1 Talking About Colour
During my observations and video-ethnography over the period of four months in a

small art school in Germany, it was striking, though maybe not surprising, just how
much individuals spoke about colour. I began by noting down all colour utterances
in a notebook and, later, analysed audio and video data for the same. Some sample
utterances are presented here.

Authenticated
Excerpt 1: Sample colour-utterances in an art school in Germany

Social actor A: Vielleicht würde ich da (+ Geste) eventuell ein bisschen mehr
gelb reintun.
Perhaps I would possibly put a little more yellow in there (+
deictic gesture).
Social actor B: Das (+ Geste) malt man nicht blau. Das must du grün
machen.
One doesn’t paint that (+ deictic gesture) in blue. You have to do
it in green.
Social actor C: Du könntest da (+ Geste) vielleicht noch mit einem Hauch rot
reingehen.
You maybe might also go into it there (+ deictic gesture) with a
touch of red.
Social actor D: Mach das (+ Geste) blau.
Make that (+ deictic gesture) blue.
Social actor E: Ich würde da (+ Geste) eventuell noch blau nehmen.
I would there (+ deictic gesture) possibly also use blue.
Social actor F: Ich würd das (+ Geste) in rosa malen.
I would paint that (+ deictic gesture) in pink.
Social actor G: Du könntest da (+ Geste) vielleicht noch grün nehmen.
You could there (+ deictic gesture) possibly use some green.
When analysing the colour utterances in connection with the non-verbal and manual
actions that individuals performed at this nexus of practice, and relating these to the
historical data that I collected over the four months, a community of practice emerged.
The art teacher, or the expert, had the strongest modal use, the most adverbial
use, and the most use of hedges. She would say things as ‘perhaps I would possi-
bly put a little more yellow in there (+ deictic gesture)’. While new students, or the
novices, had the strongest use of directives, saying things such as ‘make that (+
deictic gesture) blue’. All other students were nicely stratified somewhere between
expert and novices in their modal, adverbial, hedging, and directive use, showing
how close they either were to the expert or to the novices.
The art teacher thus used language carefully as not to direct, to allude to possibil-
ities, and to show that there is no right or wrong. While new students used language
to demonstrate their own knowledge about the world, in which they perceived a clear
right and wrong.
However, when seeing that the art teacher clearly verbalises that there is no right
or wrong, alluding to the fact that anything is possible when painting, and when
always hedging, we may want to ask what it is that she actually teaches.

Authenticated
128 Sigrid Norris
4.2 Teaching How to Paint
As discussed above, the art teacher does not give direct instructions about the use of
colour. She uses modals, adverbials, and hedges to open up possibilities for students
rather than answering colour questions straightforwardly.
This hedging has brought me to take a closer look at the teaching practice of this
art teacher. Looking at many hours of video, I found a very similar pattern: a pattern
that can be found when she teaches children of all ages and a pattern that can also be
found when she teaches new adult students. The language that the art teacher uses is
quite striking in that she does not actually seem to say much of substance. She often
falls into colloquialisms; and she often seems to emphasise her own reliance on her
intuition and often appears to be quite vague.
Here, I shall now examine one representative higher-level action illuminated by a
site of engagement in an adult art class, where a new student indirectly requests help.
I first illustrate how the art teacher teaches, by narrowing my window, illuminating
the site of engagement of one higher-level action that I demarcate as beginning when
the art teacher starts to engage with the new student’s request, and ending when she
has finished the teaching-moment.
I first constrict my site of engagement to illuminate the verbal exchange between
art teacher and student, and thus begin with the examination of an audio transcript.
Just before the art teacher engages with the student, the student had stepped into
the pathway of the art teacher. This action is discussed in Norris (2014a) with a focus
on the student learning tacit practices. In this chapter, I focus on the art teacher and
discuss the moment in detail in section 4.2.1 below. However, this moment precedes
the exchange detailed in Audio Transcript 1; and therefore, this verbal exchange is
necessary for the understanding of the following exchange. Here, the art teacher,
finding the student in her way, begins to speak with the student saying du guckst so
kritisch (you are looking so critically) and the student responds by saying mm. The art
teacher moves to a different position and gazes at the painting for 12 seconds before
the student explains da müsste noch son bisschen Schwarz rein nur son ganz kleiner
Hauch aber ich weiss noch nicht wo (there should be a little more black in there just a
very little touch but I am not yet sure where) (Figure 1). The art teacher takes this as
an indirect request for help, takes the five steps towards the painting and the dialogue
in Audio Transcript 1 develops.
Audio Transcript 1: das is so bblub (that is so bblub)

(1) Art teacher: das Thema is ja,
well the point is
(2) wenn du da hier rein gehen würdest
if you’d go in here
(3) das wär auch doof.
that would be idiotic too.

Authenticated
(4) das war ja jetzt so.

that was now so.
(5) Student: ja das passt da nirgendwo ne?
yes it doesn’t fit there anywhere right?
(6) Art teacher: ja aber ich hab grad
yes but I just have
(7) ich hatte grad in ge-
I had just in th-
(8) hier aussen,
here on the outside,
(9) oder da
or there
(10) watte
wait
(11) muss i nomal gucken
have to look again
(12) also ich glaube
well I believe
(13) hier aussen
here on the outside
(14) könnt ich mir vorstelln
I could imagine
(15) wesse wenn de dir jetz vorstellst
you know when you imagine
(16) weil-
because
(17) das is soo
that is soo
(18) bblub
bblub
(19) Student: weil da
because there
(20) nee
noo
(21) Art teacher: hier geht au nich
here isn’t possible either
(22) weil dann ham wir son komisches
because then we’ll have such a strange
(23) weiss ich nich
not sure
(24) Gezacke
zigzag

Authenticated
130 Sigrid Norris
(25) hhh
hhh
(26) ne also
no so
(27) das wär auch doof
that would be idiotic too
(28) also ich hab jetzt grade so ausm Gefühle heraus
well I’m thinking so from my own feeling
(29) irgendwie hier gedacht
about here
Audio Transcript 1 illustrates that the art teacher uses many colloquial terms such as
doof (idiotic), bblub (blubb), or komisch (strange) when speaking with the student.
She further speaks by hedging, as discussed previously and with much modal use.
But what is more striking in this transcript, is the fact that the art teacher in lines (26)
and (27) ends her teaching by stating also ich hab jetzt grade so ausm Gefühle heraus
irgendwie hier gedacht (well I’m thinking so from my own feeling about here).
At first sight, we notice the relevance of audience design (Bell 1984): The teacher
speaks to the student in a way that is understandable to her. Since the teacher is
speaking with a new student, she likely assumes that the student has little knowl-
edge about art and/or the language connected to the teaching/learning of art prac-
tice. However, that does not explain how or what the art teacher actually teaches. It
appears, especially from lines (26) and (27) that the art teacher grounds her advice
regarding to where the student might want to add some black in her own intuition.
Thereby, we could say that she teaches the new student to trust in her own intuition as
well. But, while such trust in intuition may be very important when learning how to
paint, we may wonder what else – if anything – the art teacher teaches the student.
Thus, when we investigate Transcript 1, we find that the art teacher utilises the medi-
ational means of spoken language to chiefly teach the new student the need to trust
her intuition.
However, when examining other modes, we realise that the art teacher teaches
much more than trust in intuition. Specifically, we find that she teaches the student a)
how to look at a painting; b) where to add more black; and c) where not to add black
and why not. Figure 1 illustrates the moment when the art teacher first engages with
the student, and it is at this moment, when she teaches the student where to position
herself in order to correctly look at a painting.

Authenticated
4.2.1 Teaching How to Look at a Painting
Narrowing the site of engagement (closing our window a bit), we can delineate the
concrete moment of interaction as a higher-level action with a beginning at the point
where the art teacher addresses the student (Figure 1 image 1); and ending at the point
where the student finishes her utterance (Figure 1 image 6).
The new student is positioned at an angle to her painting and about two steps
away from it (Figure 1 images 1 and 2). As the art teacher is getting ready to observe
the painting, she moves to a position directly across and five paces away from the
painting.
Fig. 1: Teaching how to look at a painting

Authenticated
132 Sigrid Norris
As the art teacher positions herself across from the painting and gazes at it for an
extended time (12 seconds), she employs a painting-proxemics-gaze modal aggregate
(Figure 2) (discussed in detail in Norris 2014b). At this instance, the art teacher engages
in her practice of looking at a painting. Thereby, she tacitly teaches the correct phys-
ical distance and positioning that the student needs to take up in order to be able to
see the painting as a whole. This perceiving a painting as a whole is accomplished
through the painting-proxemics-gaze modal aggregate (Figure 2).
Fig. 2: Employing a painting-proxemics-gaze modal aggregate (from Norris 2014b, 189)
When recognizing the painting-proxemics-gaze modal aggregate employed in this

concrete higher-level action, we are compelled by our analytical tool, the site of
engagement, to realise that it is made possible by converging practices. These con-
verging practices are, at this moment re-created and/or shaped. In other words, by
analysing this concrete instantiation of a modal aggregate, we trace and link it to
practices (actions with a history), and may further trace and link it to discourses such
as the discourse of art (when for example thinking of a museum visit, where the visi-
tors stand away from paintings to view them) (cf. also: Norris 2014b).

Authenticated
4.2.2 Teaching Exactly Where to Add More Black
Changing our site of engagement a bit by moving and closing our window to a narrow
slit, we can zoom in on lines (8) and (9) of Audio Transcript 1. In line (8), the art
teacher utters the words hier aussen, (here on the outside,) with slightly rising into-
nation, indicating that there is more to come. At the same time, she moves the little
finger of her right hand along a line, starting at one particular point on the line and
stopping at another specific point as illustrated in Figure 3.
Fig. 3: Teaching the exact positioning of a line through precise re-drawing (from Norris 2014b, 193)
The art teacher redraws a specific part of the outside line of a light green half circle
with her little finger, indicating precisely where she believes the student could add
some black. At this moment, the arm/hand/finger movement takes on much weight
in producing the meaning of illustrating the exact position of a possible black line
(shown in Figure 5).
At this moment, the painting-proxemics-gaze modal aggregate is highly relevant,
but the meaning of the particular part of the green line that the art teacher suggests
the student to paint black, is created through the movement of her little finger. At the
same time, as the art teacher begins to retrace the line, she begins her utterance hier
aussen (here on the outside). This positioning of a pointing finger coinciding with
the deictic term hier (here) could be a classic pointing gesture (McNeill 1992) and is
misunderstood as such by the student (analysed in detail in Norris 2014b). However,
in this instance language is not superordinate to gesture and the two modes also do
not take on an equal position. While the spoken language is clearly linked to the arm/
hand/finger movement, the modes painting, proxemics, and gaze are needed to actu-
ally produce the arm/hand/finger movement along the intended line, while the mode

Authenticated
134 Sigrid Norris
of spoken language in fact is not needed to convey the meaning that the art teacher
is trying to convey. For example, when we think about Transcript 1 just before this
instance happens as represented in Transcript 2, the art teacher could have conveyed
the same meaning of hier aussen (here on the outside) by only re-drawing the part of
the line without uttering the words.
Transcript 2: Excerpt of Transcript 1

(6) Art teacher: ja aber ich hab grad
yes but I just have
(7) ich hatte grad in ge-
I had just in th-
Since the same meaning could have been conveyed without the utterance, spoken
language takes on less weight here than the other modes. However, the mode of
spoken language takes on as much weight as the arm/hand/finger movement when
the art teacher next says oder da (or there).
Fig. 4: Giving an alternative
The art teacher moves her little finger to the other side of the green half circle and
suggests that the student could alternatively add black to a part of that line (Figure 4).
Here, the art teacher shows exactly which part of the inside line of the green half
circle the student could alternatively redraw in black. Once more, the art teacher is
very specific in her suggestion indicated by the movement of her little finger, while
she says ‘oder da’ (or there) ending in lowered intonation, denoting the end of the
suggestion.

Authenticated
However, here the modal configuration has changed from the moment preceding
this one. In this current instance, when the art teacher gives the student an alterna-
tive, the mode of spoken language grows in importance and intensely merges with
the mode of arm/hand/finger movement. In this instance, the meaning of an alter-
native can only be conveyed when these two modes build an aggregate and wholly
function together. In other words, the mode of spoken language alone cannot convey
the meaning that the art teacher constructs at this moment, and neither can the mode
of arm/hand/finger movement. Figure 5 and 6 illustrate the lines that the art teacher
suggests. In Norris (2014b), I have shown how the new student understood this teach-
ing moment.
Fig. 5: hier aussen (here on the ouside) Fig. 6: oder da (or there)
Opening up the site of engagement more now to see the continuation of the art teach-
er’s teaching moment, we find that next, the art teacher says (lines 9 and 10) watte
muss i nomal gucken (wait I have to look again) as she takes five steps away from the
painting and looks at it. She then steps forward again and begins during her third
step, (line 11) saying also ich glaube (well I believe). As soon as the art teacher reaches
the painting, she re-draws the exact same part of the outside line of the green half
circle as shown in Figures 3 and 5, again saying (line 12) hier aussen (here on the
outside) as she moves her little finger from bottom to top along the line. She then
re-draws it back (top to bottom) and forward (bottom to top) again, while saying (line
13) ‘könnt ich mir vorstellen’ (I could imagine). Then, later, when the art teacher says
(in lines 26 and 27) also ich hab jetzt grade so ausm Gefühle heraus irgendwie hier
gedacht (well I’m thinking so from my own feeling about here), she again redraws the
outside line of the green half circle in a very similar way as she did before (illustrated
in Figures 3 and 5). Now, she redraws it four times, moving her little finger up, down,
up, and down again.

Authenticated
136 Sigrid Norris
When examining the multimodal higher-level action, we thus detect that the
art teacher repeatedly demonstrates the exact positioning of a possible place for a
black line. While the art teacher uses the mode of spoken language colloquially and
vaguely irgendwie hier (about here), she utilises the mode of arm/hand/finger move-
ment very exactly, conveying precisely where the student should add a black line. In
this instance, the art teacher teaches the student confidence, as the modes of spoken
language, arm/hand/finger movement, and all other modes in the interaction inter-
link, although, the modal configurations vary and sometimes change quite quickly as
illustrated in the two examples above.
But the art teacher does not only teach where the student should add a black line,
she also teaches where the student should not add a black line and gives a detailed
explanation of why not.
4.2.3 Teaching Where Not to Add Black And Why Not
When moving our window a bit and opening a narrow gap, we can open up another
site of engagement. This time, we take line (20) of Audio Transcript 1, where the art
teacher says hier geht au nich (here isn’t possible either) as the beginning of the high-
er-level action (Figure 7); and take the instance when the art teacher has retracted
her hands as the ending point of the higher-level action (last image in Figure 8). In
between these points, (line 22–24) the art teacher says weil dann ham wir son komis-
ches weiss ich nich (because then we’ll have such a strange not sure) Gezacke (zigzag).
The reason that we delineate the higher-level action in this way is because it is this
instance, in which the art teacher conveys where the student should not add black
and why not.
First, the art teacher shows the student where a black line should not be added
as illustrated in Figure 7. Here, the art teacher draws her finger along a line from top
to bottom as she says hier geht au nich (here isn’t possible either), opening her right
hand and turning it palm up as she finishes the utterance with lowered intonation.

Authenticated
Fig. 7: Teaching where a black line should not be drawn
As the art teacher says hier geht (here possible), she moves her arm/hand/finger along
a line from top to bottom. At this point (Figure 7 images 1 and 2), the modes of spoken
language and arm/hand/finger movement again build an aggregate as the message
cannot be understood unless both modes are used together to indicate the position
of the line. But then, the mode of spoken language takes on more weight as the mode
of arm/hand/finger movement, still supporting the verbal message, is not absolutely
necessary for the meaning construction. In other words, if one had not seen the wave
of the hand, the message of auch nich (isn’t either) (Figure 7 bottom row), namely that
a line is not possible here, is still understood.
She then continues with an explanation beginning with weil (because) as her left
hand is moving upwards and her right hand is moving downwards as seen in Figure
8 from image 1. She continues this utterance with a hypothetical beginning with dann
(then) as both hands are now moving up as seen in Figure 8 image 2, saying ham wir
(we’ll have). Her hands change very little until she says more quietly son (such a), and
right after this begin to move faster, first coming closer together, before she moves her
right hand along a black line to the left as illustrated in images 1 and 2 of the second
row in Figure 8. Now she adds komisches (strange) and moves her hand along an
adjacent black line to the right.
As the art teacher follows one line after the other with her right hand, and utters
the words weiss ich nich (not sure), her left hand first rests in a post-stroke hold before
it enters a retraction phase during this utterance that is completed in image 1 of the 5th
row in Figure 8. Only after much gesturing with her right hand, does the art teacher
verbalise what she has been showing with her arm/hand as she says Gezacke (zigzag)

Authenticated
138 Sigrid Norris
and laughs. But even as she utters the word Gezacke (zigzag), verbalising what she
has been and still is showing with her arm/hand movement, she continues the zigzag
movement with her right hand until she has completed her re-drawing of the black
lines.
With this elaborate gesturing and air-re-drawing of lines, the art teacher gives
an elaborate visual explanation, illustrating what would happen if the student was
painting the line black that the art teacher has pointed out not to draw in Figure 7
(images 1 and 2).
5 Conclusion – Language in Multimodal (Inter)action

Multimodal (inter)action analysis allows the investigation of interaction on a great
variety of levels. With this theory, we do not need to build bridges between the micro,
the intermediate and the macro. (For a detailed analysis of these three levels in the
instances shown in Figures 2 and 3, cf. Norris 2014b. There the explicit connection
between modal configurations, practices, and discourses is made.) While the micro,
intermediate and macro are quite incommensurate levels of analysis for some the-
ories, multimodal (inter)action analysis ecologically incorporates these and moves
back and forth between ostensible levels with ease.

Authenticated
Fig. 8: Explaining a hypothetical through hand-motion plus naming it Gezacke (zigzag)

Authenticated
140 Sigrid Norris
With the analytical tool of site of engagement, the analyst begins the actual investi-
gation by looking at concrete actions, and at the same time, the analyst never loses
awareness of the fact that the actions that are being investigated are made possible
through converging practices, and that these practices in turn link to larger discourses
(Norris 2014 a and b).
Thinking of the research question that is addressed as the closed window that
is positioned in a certain way before it is opened to illuminate a site of engagement,
demonstrates that any kind of positioning is possible. This notion is exemplified in
the chapter, for example, by positioning the closed window in such a way that, when
opened, the site of engagement only comprises utterances about colour; or, by posi-
tioning the closed window in such a way that, when opened, it illuminates the site of
engagement of the art teacher showing the student exactly where to add a black line.
However, it is not only the ease with which the analyst moves across various levels
of analysis, but also, and most importantly, the notion that all modes in interaction
build one system of communication, which makes this theory unique. Thinking of
interaction in these multimodal terms, we notice that language is just one mode in the
much larger system. Language, just as any other mode, can take on a super-ordinate
or a sub-ordinate position to other modes in interaction (Norris 2011b). Language,
always integrated into the larger system, can easily function in close connection with
other modes such as arm/hand/finger movements as illustrated in the example of
the art teacher giving an alternative of where the student could add a black line. Or
language can function as an add-on to another mode, such as the arm/hand/finger
movement as exemplified in the example of the art teacher demonstrating to the
student exactly where to add black. But language can also function differently from
other modes such as arm/hand/finger movements as illustrated in the example where
the art teacher quite vaguely states irgendwie hier (about here) when, in fact, she
demonstrates exactly where the student should add a black line by re-tracing only a
part of a line in the painting. Here, the art teacher is simultaneously exact and vague,
achieving the fostering of confidence in the student. However, it is not the case, as
one may assume, that she utilises the mode of spoken language in order to teach con-
fidence, and the mode of arm/hand/finger movement to teach exactly where to posi-
tion the black line. Rather, the teaching of confidence is produced by giving a very
clear answer to the student through the mode of arm/hand/finger movement at the
very same time as giving a vague explanation through the mode of spoken language.
In other words, was the art teacher vague in both modes, the vagueness would not
result in teaching confidence; and was the art teacher exact in both modes, it would
also not result in teaching the student confidence. This teaching of confidence is thus
only possible through the interplay of these modes in complexly interlinked ways,
giving both a vague and an exact explanation simultaneously.
Multimodal (inter)action analysis thus is a theoretical framework with linked
methodological tools that allows us to investigate interaction in its complexity. With

Authenticated
this, the theory opens up new directions and scope to the inquiry into language and
interaction.
Acknowledgement: I would like to thank the participants for their engagement and
the Faculty of Design & Creative Technologies at Auckland University of Technology,
New Zealand for their financial support of the project.
6 References
Bateson, Gregory, (1972): Steps to an Ecology of Mind. Chicago.
Bell, Allen (1984): Language style as audience design. In: Language in Society 13 (2), 145–204.
Birdwhistell, Ray (1970): Kinesics and Context. Essays on Body Motion Communication.
Pennsylvania.
Geenen, Jarret (2013): Kitesurfing: Actions, (Inter)actions, and Mediation. Unpublished PhD Thesis.
Auckland University of Technology. Auckland, New Zealand.
Goffman, Erving (1963): Behaviour in Public Places. New York.
Goffman, Erving (1974): Frame Analysis. New York.
Gumperz, John (1982): Discourse Strategies. Cambridge.
Kress, Gunther/Theo van Leeuwen (2001): Multimodal Discourse. The Modes and Media of
Contemporary Communication. London.
Lave, Jean/Etienne Wenger (1991): Situated Learning. Legitimate Peripheral Participation.
Cambridge.
Makboon, Boonyalakha (2015). Spiritual Vegetarianism. Identity in Everyday Life of Thai
Non-Traditional Religious Cult Members. Unpublished PhD thesis. Auckland University of
Technology. Auckland, New Zealand.
McNeill, David (1992): Hand and Mind. What Gestures Reveal About Thought. Chicago.
Merleau-Ponty, Maurice (1962): Phenomenology of Perception. Transl. by Colin Smith. New York.
Merleau-Ponty, Maurice (1963): The Structure of Behavior. Transl. by A.L. Fisher. Boston.
Norris, Sigrid (2002): The implication of visual research for discourse analysis: Transcription beyond
language. Visual Communication. 1 (1): 97–121.
Norris, Sigrid (2004): Analyzing Multimodal Interaction. A Methodological Framework. London.
Norris, Sigrid (2009): Modal density and modal configurations. Multimodal actions. In: Jewit, Carey
(ed): Routledge Handbook for Multimodal Discourse Analysis. London.
Norris, Sigrid (2011a): Identity in (Inter)action. Introducing Multimodal (Inter)action Analysis. Berlin/
Boston.
Norris, Sigrid (2011b): Three hierarchical positions of deictic gesture in relation to spoken language.
A multimodal interaction analysis. In: Visual Communication 10 (2), 1–19.
Norris, Sigrid (2013a): Multimodal (inter)action analysis. An integrative methodology. In: Cornelia
Müller/Ellen Fricke/Alan Cienki/David McNeill (eds): Body – Language – Communication.
Berlin/New York.
Norris, Sigrid (2013b): What is a mode? Smell, olfactory perception, and the notion of mode in
multimodal mediated theory. Multimodal Communication 2(2): 155–169.
Norris, Sigrid (2014a): Learning tacit classroom participation. WCLTA. Procedia Social and Behavioral
Sciences. Elsevier. Volume 141, 166–170.
Norris, Sigrid (2014b): The impact of literacy-based schooling on learning a creative practice. Modal
configurations, practices and discourses. In: Multimodal Communication 3 (2), 181–195.

Authenticated
142 Sigrid Norris
Norris, Sigrid/Rodney H. Jones (2005): Discourse in Action. Introducing Mediated Discourse

Analysis. London.
Scollon, Ron (1998): Mediated Discourse as Social Interaction. London.
Scollon, Ron (2001): Mediated Discourse. The Nexus of Practice. London.
Scollon, Ron/Suzie Scollon (2004): Nexus Analysis. Discourse and the Emerging Internet. London.
Tannen, Deborah (1984): Conversational Style. Analysing Talk among Friends. Norwood, NJ.
Wertsch, James. V. (1998): Voices of the Mind. A Sociocultural Approach to Mediated Action.
Cambridge, Mass.
Vygotsky, Lev (1978): Mind in Society. The Development of Higher Psychological Processes. Boston.
Vygotsky, Lev (1986): Thought and Language. Revised and Edited by Alex Kozulin. Boston.

Authenticated

Multimodal Interaction Language and Modal Configurations

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Multimodal Interaction Language and Modal Configurations

Uploaded by

Copyright:

Available Formats

Sigrid Norris

5. Multimodal Interaction – Language and

2 Overview – Historical Developments

3 Theory – Multimodal (Inter)action Analysis

The site of engagement is an analytical tool that allows us to illuminate a moment in

3.2 Practice, Nexus of Practice and Community of Practice

3.3 Lower-Level and Higher-Level Actions

Modal configuration is the hierarchical configuration of lower-level actions (or their

4 Applications – Interactions and Modal Configura-

4.1 Talking About Colour

During my observations and video-ethnography over the period of four months in a

Excerpt 1: Sample colour-utterances in an art school in Germany

4.2 Teaching How to Paint

Audio Transcript 1: das is so bblub (that is so bblub)

(4) das war ja jetzt so.

4.2.1 Teaching How to Look at a Painting

Fig. 1: Teaching how to look at a painting

Fig. 2: Employing a painting-proxemics-gaze modal aggregate (from Norris 2014b, 189)

When recognizing the painting-proxemics-gaze modal aggregate employed in this

4.2.2 Teaching Exactly Where to Add More Black

Transcript 2: Excerpt of Transcript 1

Fig. 4: Giving an alternative

4.2.3 Teaching Where Not to Add Black And Why Not

Fig. 7: Teaching where a black line should not be drawn

5 Conclusion – Language in Multimodal (Inter)action

Fig. 8: Explaining a hypothetical through hand-motion plus naming it Gezacke (zigzag)

Norris, Sigrid/Rodney H. Jones (2005): Discourse in Action. Introducing Mediated Discourse

You might also like

5. Multimodal Interaction – Language and

2 Overview – Historical Developments

3 Theory – Multimodal (Inter)action Analysis

3.2 Practice, Nexus of Practice and Community of Practice

3.3 Lower-Level and Higher-Level Actions

4 Applications – Interactions and Modal Configura-

4.1 Talking About Colour

4.2 Teaching How to Paint

4.2.1 Teaching How to Look at a Painting

4.2.2 Teaching Exactly Where to Add More Black

4.2.3 Teaching Where Not to Add Black And Why Not

5 Conclusion – Language in Multimodal (Inter)action