Wolfgang Hallet

34 Non-verbal Semiotic Modes and Media in

the Multimodal Novel
Abstract: Since the 1990s a new kind of novel has emerged in considerable number,
which integrates a wide range of non-linguistic symbolic forms and non-narrative
modes into the narrative discourse. These forms encompass visual images of all sorts,
but also diagrams or maps and reproductions or imitations of non-narrative texts
and genres. Whereas intermediality theories regard and describe such other forms
of symbolization as ‘media,’ multimodality theories (originating mainly from social
semiotics) conceptualize them as semiotic modes that serve (more or less convention-
alized) signifying and communicative purposes. The present contribution introduces
the concept of multimodality and demonstrates how it can be applied to understand
and describe the sub-genre of the multimodal novel.

Key Terms: Multimodality, multimodal novel, semiotic mode, multimodal narrative

discourse, multiliterate reading, transmodal meaning

1 Features of the Multimodal Novel

From its early beginning, the genre of the novel has been associated with the written
word in printed form in the medium of the paperbound book. Thus, although in the
nineteenth century quite a few novels were published with additional (non-diegetic)
illustrations to popularize the genre (↗20 The Nineteenth-century Illustrated Novel),
reading has always been a primarily linear, page-turning act of decoding alphabetic
signs, of word-based imagination and of making meaning of letters, words and sen-
tences. However, since the end of the twentieth century there is a growing number of
novels that integrate other, non-linguistic symbolic forms and modes of representation
into the narrative discourse (↗7 Postcolonial Ekphrasis in the Contemporary Anglo-
phone Indian Novel; ↗10 Nesting – Braiding – Weaving; ↗21 Intermedial Encounters
in the Contemporary North American Novel). These forms include visual images of all
sorts, often photographs, but also diagrams, cartographic maps, screenshots, draw-
ings or cartoons and comics. Apart from visual images, reproductions or imitations of
other kinds of texts and non-narrative genres can be integrated, like, e.g., handwrit-
ten letters, e-mails or other ephemera, but also formulaic languages like algorithms or
mathematical calculations. There is, of course, no limited set of modes or media that
can occur in the multimodal novel; the collage of written text, images, reproductions
of documents and a large range of visual and distinct other textual elements (e.g.

Brought to you by | Lund University Libraries

Authenticated
Download Date | 11/17/15 3:32 PM
638   Wolfgang Hallet

footnotes) make it difficult to identify a text as a novel in the traditional sense at all
and to regard the novel as ‘a text.’ Still, the systematic and recurrent integration of
non-verbal and non-narrative elements in novelistic narration makes it necessary to
expand the notion of ‘the novel’ and ‘the literary text’ beyond word-based forms of
representation and meaning-making. On the one hand, the kind of novel described
above obviously constitutes a sub-genre, categorized as ‘the multimodal novel.’ On
the other hand, the integration of non-linguistic signs and even whole ‘texts’ calls
for a re-conceptualization of ‘diegesis’ in literary studies in general and of novelistic
narration and the theory of the novel in particular (cf. Hallet 2009, 149–152; Hallet
2014, 168–169).
It is important to note that in the kind of novel that is labelled as ‘multimodal,’
these non-linguistic elements are not extra-textual, additional illustrations (like, e.g.,
illustrated editions of novels) or complementary editorial elements, and that they are
substantially different from paratexts. Rather, there are two defining features that
make a novel ‘multimodal.’ Narratologically speaking, these other, non-linguistic or
non-narrative elements
– are distinct visual or textual entities that form an integral part of the narrative dis-
course; they are at the narrator’s disposal and displayed as the narrative unfolds.
Often, the verbal narrative refers to them or addresses them more or less explicitly;
– are an intrinsic part of the fictional world at different diegetic levels. They are
artifacts that are produced, used and located in the fictional world of the novel
and are thus related to the characters’ actions and perceptions or to the narrator’s
ways of thinking, communicating and making sense of the world.

In other words: Unlike in the traditional (‘monomodal’) novel, the fictional world is
not represented and constituted in verbal form only. The integration of different types
of symbolization and semiotic forms leads to a multisemiotic, more comprehensive, at
times also more authentic representation of the fictional world that tries to imitate or
resembles the multifarious ways in which the non-fictional reality is perceived by the
reader and in which knowledge and experiences are represented and communicated
in the reader’s lifeworld. Therefore, it is appropriate to categorize this type of multi-
semiotic narrative as a literary subgenre and designate it as the multimodal novel (cf.
Hallet 2009a, 2011a, 2011b; Gibbons 2010a, 99; Gibbons 2010b, 287; Nørgaard 2010b).
Michael Ondaatje’s Running in the Family (1982; Hallet 2015), Mark Z. Dan-
ielewski’s House of Leaves (2000; Gehring 2009; Gibbons 2012, 46–85), W. G. Sebald’s
Austerlitz (2002; cf. sections 2 and 3 in Denham and McCulloh 2006; Hallet 2011a,
2014), Mark Haddon’s The Curious Incident of the Dog in the Night-Time (2003; cf.
Hallet 2014) or Jonathan Safran Foer’s Extremely Loud & Incredibly Close (2005; cf.
Hoth 2006; Glorig 2007; Hallet 2009, 2014; Nørgaard 2010a; Gibbons 2012, 127–166)
are among the most renowned and popular multimodal novels. However, Young Adult
Novels and fictional autobiographies in diary form in particular appear to be the glob-
ally most successful type of multimodal novel. Sherman Alexie’s The Absolutely True

Brought to you by | Lund University Libraries

Authenticated
Download Date | 11/17/15 3:32 PM
 34 Non-verbal Semiotic Modes and Media in the Multimodal Novel    639

Diary of a Part-Time Indian and Jeff Kinney’s Diary of a Wimpy Kid engage in obviously
highly recognizable and popular communicative and notational practices and styles
of self-narration, imitating the handwritten diary, scrapbook practices, occasional
lifeworld-related graphics (such as, for instance, teacher or peer caricatures) and inte-
grating more ambitious forms of hand-drawn portraits or comics.

Fig. 1: Multimodal sample page from The Selected Works of T. S. Spivet.
Larsen 2009, 3.

Brought to you by | Lund University Libraries

Authenticated
Download Date | 11/17/15 3:32 PM
640   Wolfgang Hallet

A brief description and analysis of a prototypical page from Reif Larsen’s novel The
Selected Works of T. S. Spivet (2009) may serve to demonstrate how a multimodal
novel is composed and narrates (fig. 1). The main body of the page consists of verbal
narrative discourse in the traditional style, presenting the autodiegetic narrator and
his family; the first sentence also introduces the situation (the phone call) that trig-
gers the young narrator’s adventurous journey to the East. The second paragraph
further specifies the narrator’s character and simultaneously serves as a backdrop
to the visual elements on the page since the narrator describes his habit of mapping
everything he experiences: “maps of people doing things” in blue, “zoological, geo-
logical and topographical maps” in green and anatomic sketches of insects in red
notebooks. The verbal text shares the main body of the page with a topographical
map of the narrator’s home region in Montana in which his hometown Divide and
his parents’ Coppertop Ranch is located. Later in the novel, U.S. America’s east-west
divide is thematized as an existential experience and a determining factor in the col-
onisation of the American West (cf. Hallet 2014, 158–159).
A detailed plan of the narrator’s bedroom with the color-coded bookshelves
authenticates his claim that his life is organized around his mapping practices and
the art of drawing maps and scientific sketches. This room plan is placed in the
margin, which the narrator continuously uses for remarks, additional notes, reflec-
tions and excerpts from his notebooks, as in the example in fig. 1, where more details
about the young topographer’s equipment that he stores in his room are disclosed.
Sometimes the margin is also an important space to present graphic or topograph-
ical details, genealogical or personal information about other characters or about
the narrator’s (Spivet’s) mapping practices. The sketch of a sparrow skeleton at the
bottom of the margin alludes to and explains the background to the narrator’s second
name, ‘Sparrow,’ and is a demonstration of the narrator’s skill in the art of scientific
drawing. Often, as in the line between the topographical map and the map of the
bedroom at the top of the page, a particular item in the main text is assigned to a mar-
ginal detail through a dotted line; in other cases, particularly in the middle part of the
novel containing the journey, the margin is used as a navigation device that provides
the reader with geographical information about the narrator’s east-west itinerary and
the locations and places he visits or passes.
The sample page from Larsen’s novel demonstrates to what extent the way the
storyworld is represented and constituted differs from the monomodal novel; but it
also becomes evident at first sight that what is commonly regarded as ‘narrative dis-
course’ takes on a completely different shape. It is not only delinearized, but impor-
tant details concerning the setting, the characters or their actions and even the nar-
rator’s thoughts, reflections and knowledge are now, through the layout, relegated
to specific places on the page and presented in various symbolic forms, urging the
reader to decode other ‘languages’ and transforming reading into a hypertextual
activity. This is why the shift from monomodal to multimodal storytelling in the novel
affects both story and discourse, the representation of the storyworld and the (delin-

Brought to you by | Lund University Libraries

Authenticated
Download Date | 11/17/15 3:32 PM
 34 Non-verbal Semiotic Modes and Media in the Multimodal Novel    641

earized) presentation of information on the page, the creation of a narrative and the
act of reading (cf. sections 3 and 4 for details).

2 Multimodality as a Theoretical Framework

The concept of multimodality originates from various theory strands and different dis-
ciplines, the three most important of which are discourse analysis in linguistics, mul-
timedia technology for man-machine interaction (Ventola, Charles, and Kaltenbacher
2004; Scollon and LeVine 2004) and the functional grammar approach in social semi-
otics (Kress and van Leeuwen 2001; van Leeuwen 2005; Hallet 2008; Gibbons 2010,
8–25; Kress 2010; Bucher 2011, 132–135). Two major developments have led to the
establishment of multimodality as a theoretical framework in the humanities. On the
one hand, multimodality theories try to account for the shortcomings of monomodal
disciplinary approaches in which, as in the philologies, “language was (seen as) the
central and only full means for representation and communication” (Kress and van
Leeuwen 2001, 45; cf. also Bucher 2011, 123-125) or, in case other modes are studied
(music, photography, painting), “representation was treated as monomodal: discrete,
bound, autonomous, with its own practices, traditions, professions, habits.” (Kress
and van Leeuwen 2001, 45). However, even theories and disciplines specialized in the
study of one particular type of artifact like, e.g., paintings, could hardly ignore the
involvement of other ways and forms of meaning-making. This is why in his picture
theory, W. J. T. Mitchell contended that the

image / text problem is not just something constructed ‘between’ the arts, the media, or differ-
ent forms of representation, but an unavoidable issue within the individual arts and media. In
short, all arts are ‘composite’ arts (both text and image); all media are mixed media, combining
different codes, discursive conventions, channels, sensory and cognitive modes. (Mitchell 1995,
94–95)

The social-semiotic theory of multimodality has taken up this notion of the intrinsic
combination of ‘different codes’ and ‘modes’ involved in a single act of signification
and communication.
On the other hand, the emergence of new multimedia technologies, and the elec-
tronic hypertext in particular, has led to the insight that theories of symbolic rep-
resentation and communication need to account for the combination of different
media and symbolic forms in displays and environments in which ‘meaning’ can no
longer be explained as resulting solely from the natural human language alone. In
electronic multimedial environments, the contribution of other codes and symbolic
languages such as, e.g., sound and music, maps and diagrams, photographs and
moving images (as in videos, for example), needs to be considered, too: “Multimodal
production is now a ubiquitous fact of representation and communication. That

Brought to you by | Lund University Libraries

Authenticated
Download Date | 11/17/15 3:32 PM
642   Wolfgang Hallet

forces us urgently to develop precise tools requisite for the description and analysis of
texts and semiotic entities of contemporary communication.” (Kress 2010, 102) There-
fore, any theory of cultural semiosis and communication must explain and describe
how meaning is made across (and simultaneously through) a variety of different
semiotic systems, medial and generic modes, and how a combination of all of these
modes and media is able to produce one whole, integrated meaning. This applies to
both, single successful and efficient (multimodal) acts of communication, as, e.g. in
a newspaper article that combines verbal text, a diagram and a photograph, and to
the broader production of cultural meaning in discourses which “may be realised in
different ways. The ‘ethnic conflict’ discourse of war, for instance, may be realised as
(part of) a dinner-table conversation, a television documentary, a newspaper feature,
an airport thriller, and so on.” (Kress and van Leeuwen 2001, 5)
In social semiotics, the concept of multimodality is closely tied to a functional
definition of the semiotic mode. It is regarded as “a socially shaped and culturally
given resource for making meaning in representation and communication” (Kress
2010, 53), which is “used in recognisably stable ways as a means of articulating
discourse” (Kress and van Leeuwen 2001, 25). Although the status of some of these
resources (like, e.g., color or typography) remains rather unclear (cf. Bucher 2011,
131; Nørgaard 2010b), the vast majority of modes, particularly in the context of her-
meneutic approaches, can be seen as textual or medial (generic) entities. By contrast,
media are defined as merely physical and material resources “used in the production
of semiotic products and events, including both the tools and the materials used (e.g.
the musical instrument and the air; the chisel and the block of wood)” (Kress and
van Leeuwen 2001, 22), as is the case for an oil painting on canvas, a black and white
paper-print photograph, a printed newspaper article, a video blog in the electronic
environment of the Internet and so forth. Thus, a semiotic mode is always tied to a
specific material or medial carrier, but media in themselves do not produce meaning.
This is a substantial conceptual difference between intermediality theories and mul-
timodality theories (↗0 Introduction). Whereas in the former the verbal text and a
visual image are regarded and described as different, interrelated media, text-image
relations in the multimodal novel (as in multimodal texts in general) are not concep-
tualized as intermedial relations, but as an interplay of two distinct semiotic modes
(textual entities) in the same ‘medium,’ i.e. the printed book, which jointly contribute
to the production of one whole meaning in a single act of communication (cf. Bucher
2011, 125). In multimodality theories, therefore, the emphasis is on the meaning that
the single mode (textual entity) produces, and on the combination of various modes
that result in one (transmodal) meaning of a multimodal text. An analysis of the mul-
timodal novel has to account for the contribution of these single semiotic modes to
the constitution and characterization of signifying, communicative and socio-cultural
practices in the fictional world as well as to the kind of meaning it produces there, and
the role of a specific mode in the text-reader interaction (cf. sections 3 and 4).

Brought to you by | Lund University Libraries

Authenticated
Download Date | 11/17/15 3:32 PM
 34 Non-verbal Semiotic Modes and Media in the Multimodal Novel    643

The distinctness of the single mode is a pre-condition of the constitution of a

multimodal text, but it also implies that its communicative efficiency, its epistemo-
logical potential and its capacity to produce meaning in a way that is specific to a
particular mode plays an important role (‘affordance’). According to M. A. K. Halli-
day (2004, 29–31) a mode needs to comply with three metafunctional principles of
communication: the ideational (or referential or representational) function, i.e. the
ability to communicate content, knowledge and experiences; the interpersonal func-
tion, i.e. the capacity to establish social interaction between interlocutors; and the
textual function, which concerns the coherence of a communicative entity as a dis-
tinct textual (or medial) unity.
The distinctness of single semiotic modes and their contribution to a larger, more
comprehensive meaning in a given multimodal text or communicative unity (among
them the multimodal novel) leads to three levels of analysis, interpretation and
description:
– firstly, the specific meaning of the single semiotic mode like a photograph, a map,
a handwritten letter or other elements. Since they can be treated and occur as
independent texts (in the wide, semiotic sense) they produce a meaning of their
own. For instance, a topographical map or a photograph represent specific fea-
tures or aspects of a particular slice of the world and thus produce meaning rela-
tively independent of the context in which they are deployed. A map of Montana
represents an identifiable part of the United States, whether it occurs in the mul-
timodal novel or in some other discursive and cultural context;
– secondly, the interrelatedness of the single mode with other modes and the spe-
cific kind of relation that can be identified (cf. Bucher 2011, 128–132; Chan 2011).
Such intermodal relations always depend on the given communicative or discur-
sive context (i.e. on the way they are presented in the multimodal novel) and can
be described as, for instance, redundant (image and verbal text providing the
same information), hierarchical (e.g., the verbal text framing an image), elaborat-
ing (additional information provided by the image) or juxtapositional (the image
contrasting with the text in some way). A literary analysis of the multimodal novel
will have to provide a detailed description of the respective role and relatedness
of a single mode to other modes in the novel;
– thirdly, the transmodal (synthesized, holistic) meaning, in which the interplay of
the multiple modes in the novel results. This overall (hypertextual) meaning of
the novel is more than an addition of the individual meanings of the single modes
and also of the specific kinds of intermodal relations that can be observed in a
given novel. The manifold and intricate kinds of interplay of semiotic modes and
their individual meaning (comparable to, e.g., the film) lead to a higher level of
meaning that neither resides in these modes themselves nor in their relations, but
in a level of meaning that is constituted through the interplay itself.

Brought to you by | Lund University Libraries

Authenticated
Download Date | 11/17/15 3:32 PM
644   Wolfgang Hallet

Because of all of these observations, the multimodal novel can, on the one hand,
be regarded as just one instance of multimodality, participating in a general cul-
tural practice of signifying and making meaning and as cultural and communicative
normality. On the other hand, because of its specificity as a literary text, it can be
expected to not only mirror such social and cultural practices in fictional form, but
also to reflect and comment upon them critically and in a self-reflexive manner. The
narratological implications are multifarious, since the multimodality of novelistic
narration affects the way readers imagine the storyworld and its various constituents,
while also altering the reader’s pathways of making sense of signs. These are now not
only arranged in a hypertextual manner, as opposed to the linearity of signs in the
conventional, word-based novel, but they also belong to different sign systems which
the reader must be able to decode.

3 The Multimodal Constitution of the Storyworld

The sample page from Larsen’s novel in fig. 1 demonstrates that the information and
data the reader is offered in multimodal novels to construct and imagine the story-
world are substantially different from those in the traditional novel. A brief examina-
tion of Mark Haddon’s novel The Curious Incident of the Dog in the Night-Time serves
to demonstrate how all of the main constituents of a fictional narrative are affected.
Apart from traditional verbal discourse, this novel presents the reader with a broad
range of other symbolic forms and semiotic modes, among them street-plans of the
narrator’s neighborhood, diagrams and curves, drawings of all kinds of objects, even
of cow patterns or seat patterns, but also lists of all sorts, time-tables, mathemati-
cal formulae, calculations and algorithms, handwritten letters and many more. All of
these modes affect basic constituents of the narrative.
(1) Actions and the Story: As the title indicates, the narrator regards the death of
a dog in his immediate neighborhood as a murder mystery case which he sets out to
solve and narrate. Therefore he employs and presents various modes that are suited
for his detective work, systematizing the data and his observations. Since they thus
serve as problem-solving tools they enhance Christopher’s investigations and thus
propel the action. For instance, Christopher produces a map of his neighborhood
to plan his investigation or to clarify the positions of possible witnesses. One of his
favorite problem-solving strategies is algorithmic reasoning, formally developed
(Haddon 2004, 78–82) and represented as a “Chain of Reasoning” (Haddon 2004,
53–54), to come to conclusions about the main suspect. Furthermore, key experiences
in Christopher’s life are (re-)presented in the mode of the handwritten letter (typo-
graphically set off as such): Letters from his mother are central objects in the story
(Haddon 2004, 118–144) since they not only reveal that, in contrast to what his father
had always claimed, his mother is still alive, they also testify to his mother’s will to

Brought to you by | Lund University Libraries

Authenticated
Download Date | 11/17/15 3:32 PM
 34 Non-verbal Semiotic Modes and Media in the Multimodal Novel    645

keep in touch with him after leaving his father; furthermore, these letters also trigger
his father’s confession that he killed the dog (Haddon 2004, 150). Finally, these letters
also lead to Christopher’s decision to travel to London in order to live with his mother
(Haddon 2004, 161–163). Once again, this journey is not only related in words, but also
represented through a large number of visual elements, maps in particular, but also
other modes like traffic or Underground railway signs, electronic timetables and so
forth. Therefore it can be contended that in this multimodal novel (as in many others)
non-verbal semiotic forms represent actions, key-stages of and central objects in the
story, whereas in the traditional novel actions and the story are exclusively rendered
in verbal form.
(2) Character: Apart from representing key elements and stages of the murder
mystery case and of the subsequent family story, the non-verbal semiotic modes are
of paramount importance for the narrator’s and protagonist’s ways of making sense
of the world. Since Christopher shows deviant forms of world apprehension and signs
of autism, resorting to non-linguistic ways of understanding, thinking, communi-
cating and expressing himself is this character’s cognitive and communicative key
strategy. For instance, a set of smileys helps him to typify people’s emotions, which
are otherwise difficult or almost impossible for him to decode (Haddon 2004, 2–3).
Since problem solving is a constant challenge in his everyday life, he has developed
cognitive excellence in reasoning, logical, mathematical and algorithmic thinking.
All of these cognitive activities are conducted in formal languages, which he also pre-
sents in the course of and as an intrinsic part of the narrative discourse. This is why
a key-decision in his life – whether to stay with his father or with his mother in the
future – is presented in the form of an algorithm (Haddon 2004, 162–163). This way,
readers have direct access to the narrator’s mind and the specific epistemological
tools Christopher applies to solve all the puzzles and mysteries in his life; readers
are able to observe ‘how the mind works’ (another of Christopher’s major areas of
interest; cf. Haddon 2004, 146). Thus the semiotic modes in this novel represent the
narrator’s (often solipsistic) cognition and one of this character’s important (or even
pre-dominant) features. Since he frequently also reflects upon his own cognitive abil-
ities and thinking strategies, they also form a dominant dimension in his personality
and identity development – one of the reasons why this novel can also be regarded as
a Bildungsroman.
(3) Social and Communicative Practices: Whereas in the traditional novel readers
access the fictional world via linguistic signs only, the multimodal novel makes it
possible to study all sorts of semiotic and communicative practices and artifacts
employed or produced in the textual world in a direct, unmediated way. Thus readers
are able to observe and recognize a multitude of semiotic and social practices in the
storyworld and relate them to those in their own lifeworld. For instance, the mul-
timodal youth novel in particular imitates, mirrors or popularizes youth cultural
practices like keeping diaries, creating scrapbooks or designing comics. However,
the integration of non-verbal semiotic forms in the novel not only serves to represent

Brought to you by | Lund University Libraries

Authenticated
Download Date | 11/17/15 3:32 PM
646   Wolfgang Hallet

social, cultural and communicative practices like mapping, visualizing statistical

data or designing buildings and objects in the storyworld (cf., e.g., the seat patterns
or the plan of Christopher’s mother’s flat in Haddon 2004, 227, 235). The multimodal
novel also displays and highlights the semantic, cognitive or epistemological surplus
and the specific affordances of semiotic modes in a given (fictional) sociocultural and
societal environment. This way it also subjects the semiotic modes presented in the
course of the novel to metacognitive and metasemiotic observation and critique (cf.
section 5), particularly if the communicative or epistemological affordance of a spe-
cific mode is explicitly addressed by a character or the narrator, like, e.g., the smileys
or stellar constellations in The Curious Incident (Haddon 2004, 2–3, 156–157).
(4) Space and Setting: As in Larsen’s novel, self-made maps feature prominently
in Haddon’s novel because they play a key-role in the young narrator’s life. Due to
his special cognitive disposition (namely, Asperger Syndrome), he needs to anticipate
and systematize spatial perceptions and to plan his itineraries carefully. Therefore,
he has developed the habit of mapping his environment and the places he visits. As a
result, the spatial dimension of this novel is, in many cases, presented in the semiotic
form of maps and plans. Thus, readers are equipped with often very detailed archi-
tectural, geographical and even astronomic information about railway stations, the
narrator’s hometown Swindon and London, a zoo, and even the Milky Way and stellar
constellations of the universe. Thus, in this multimodal novel as in others, in order to
“orient themselves on the map of the fictional world” and to “picture in imagination
the changing landscape along the routes followed by the characters” (Ryan 2001, 123),
readers can make use of a whole range of semiotic modes, including verbal descrip-
tion, to construct mental models of the places and spaces (cf. Hallet 2008b, 2009b,
2011a, 2014). Regarding the spatial dimension, multimodality is definitely a feature
that brings the novel closer to the reader’s experiences since, in the lifeworld, space
is always ‘real-and-imagined,’ constructed in symbolic form (cf. Hallet 2009b, 2011a),
including conventionalized semiotic modes like maps of all kind, graphic routes or
electronic navigation devices.
As is the case for the reader’s imagination of the fictional spaces and places, it can
be generalized that the multimodal novel offers the reader a broader range of modes
than the traditional novel as a basis for their construction of the storyworld. Since all
textual worlds are of a more or less holistic nature (cf. Hallet 2008b), comprising a
“connected set of objects and individuals; habitable environment; reasonably intel-
ligible totality for external observers, field of activity for its members” (Ryan 2001,
91), readers intuitively draw upon their real world experiences in which the use of
different types of signs and sensory channels and of different symbolic languages is a
semiotic and communicative rule. The multimodal novel, at least to a certain degree,
imitates the multiplicity of modes that are involved in everyday cognitive processes
and enhances the experiential dimension of the reader’s construction of the fictional
world (cf. fig. 2; cf. Hallet 2008b, 2014).

Brought to you by | Lund University Libraries

Authenticated
Download Date | 11/17/15 3:32 PM
 34 Non-verbal Semiotic Modes and Media in the Multimodal Novel    647

fictional world

reader,
reading as a multiliterate act

verbal photograph diagram handwritten

…
discourse (reproduction) (reproduction) letter (repr.)

Fig. 2: The multimodal constitution of the fictional world.

Hallet 2014, 167.

4 Multimodal Narrative Discourse

A look at any multimodal page of a novel instantly demonstrates that its narrative
discourse no longer unfolds in a linear manner and that this kind of novel can no
longer be regarded as a coherent text in the traditional sense. Rather, the novel’s nar-
rative discourse is now organized as a hypertextual ensemble of different types of
symbolic representations and textual elements that the reader must interrelate (cf.
Hallet 2009a, 150–151; Hallet 2011b), and text-reader interaction no longer relies on
linguistic signs only or on a linear act of reading. Although longer passages of the
novel are still word-based narration, a typical double page in Larsen’s T. S. Spivet
novel (2009, 174–175) consists of a variety of modes, i.e. bounded elements that can be
read as coherent texts of their own. Apart from verbal text in the main body of these
two pages, the most conspicuous element is a family tree designed by the narrator as
a “Genealogy Placemat that I had made for Father on the occasion of his forty-eighth
birthday” (Larsen 2009, 175). This object is part of an ongoing subplot that traces the
narrator’s family history and scientific tradition, and is inserted in the main text like
a quote. The two margins are filled by a note on the physical experiences of being
slapped in the face and a school bus hitting a squirrel; a hand-drawn sketch of a
school bus “from Notebook G29” (Larsen 2009, 174); and a short history of the place-
mat gift rendered in another marginal note (175) which is linked to the depiction of the
Genealogy Placemat in the main body by a dotted line. In other instances, the multi-
modal page of a novel may only be moderately hypertextual, e.g. by inserting a typo-
graphically represented e-mail. By contrast, in Danielewski’s House of Leaves, pages
of the novel do not present coherent verbal text at all nor do they resemble traditional
book pages any longer; rather, in the most advanced form of the multimodal novel,
pages look like poetic collages, labyrinths or jigsaw puzzles, the pieces of which the

Brought to you by | Lund University Libraries

Authenticated
Download Date | 11/17/15 3:32 PM
648   Wolfgang Hallet

reader has to study and interconnect to make meaning (cf. Gehring 2009; Gibbons
2010b; Gibbons 2012, 46–85).
A more recent and most advanced type of multimodal novel even expands nar-
ration and the act of reading beyond the medium of the printed book (i.e. ‘medium’
in terms of the physical and material quality of the carrier of a mode) and refers the
reader to the Internet in order to involve them in electronic interactive formats like
websites or blogs. Nota bene, these Internet formats are not non-diegetic additional
components (like, e.g., an interview with the author or some kind of background
information); instead, they are existents in the fictional world and part of the diegesis
so that, for instance, the reader is enabled to communicate with characters from the
novel or with the narrator, or to look at artifacts and documents from the fictional
world. In such novels, like Jeffery Deaver’s Road Side Crosses or Jennifer Cowan’s
Earthgirl, narration and reading transgress the medial boundary of the printed book
and become truly transmedial.
Therefore, ‘reading’ multimodal novels not only requires multiple literacies, i.e.
the reader’s ability to decipher a large variety of codes and symbolic languages, but
it also transforms the act of ‘reading’ into a hypertextual activity and the reader into
a ‘user’: The transmodal construction of coherent narrative discourse and meaning
now depends on the reader’s reading paths and decision-making. The reader is con-
stantly challenged to proceed from one semiotic element on the page to the other,
to identify or define the kind of interrelation between these different elements (cf.
section 2) and to assign meaning to the single semiotic element as well as to the com-
bination of all of these modes on a page and in the book as a whole.

5 Meta-semiotic Critique and Epistemological

Scepticism
The experiential dimension of multimodality in a novel implies that the reader, while
decoding the signs, always, at least to a certain extent, engages in meta-semiotic,
meta-communicative or epistemological reflection, even if the novel does not the-
matize the modes and languages it deploys. It comes as no surprise, though, that
most multimodal novels display a deep interest in the discussion and critical reflec-
tion of the affordances of semiotic modes and their cultural role and impact. In the
young adult novel The Absolutely True Diary of a Part-Time Indian, the young narrator
declares that he draws pictures “all the time”: “I draw because words are too unpre-
dictable. I draw because words are too limited.” (Alexie 2004, 5) Likewise, young
Spivet, at the beginning of Larsen’s novel, is deeply convinced of the universal epis-
temological potential of maps. And in Sebald’s novel Austerlitz, the protagonist of the
same name is obsessed with the idea of a photographic cultural history of European
architecture.

Brought to you by | Lund University Libraries

Authenticated
Download Date | 11/17/15 3:32 PM
 34 Non-verbal Semiotic Modes and Media in the Multimodal Novel    649

However, a lot of these novels also develop a considerable semiotic and episte-
mological skepticism. While, in Larsen’s novel, Spivet is completely obsessed with
mapping practices, he is, at the same time, aware of their short-comings. He accuses
George Washington of producing and “imagining all sorts of false geographies”
(Larsen 2009, 33) and generalizes his observations into an epistemological and polit-
ical critique of cartography as a cultural practice in a historical perspective: “[T]hese
early cartographers of the Corps of Topographical Engineers […] were conquerors in
the most basic sense of the word, for over the course of the nineteenth century, they
slowly transferred the vast unknown continent piece by piece into the great machine
of the known, of the mapped, of the witnessed  – out of the mythological realm of
empirical science.” (Larsen 2009, 16)
In the same vein, the protagonist of Sebald’s novel not only gives up his photo-
graphic project (handing over his huge collection of photographs to the anonymous
narrator), but his confidence in languages and signs in general is completely lost
when his writing project fails too: “[T]he exposition of an idea by means of a certain
stylistic facility […] now seemed to me nothing but an entirely arbitrary or deluded
enterprise. I could see no connections any more, the sentences resolved themselves
into a series of separate words, the words into random sets of letters, the letters into
disjointed signs.” (Sebald 2002, 175–176) This way, multimodal novels are often also
meta-semiotic and meta-cultural narratives that reflect upon the semiotic and com-
municative practices in which they engage. Multimodality in the novel is therefore
not only a way of representation and storytelling, but also of epistemological and
meta-semiotic reflection and critique.

6 Bibliography
6.1 Works Cited
Alexie, Sherman. The Absolutely True Diary of a Part-Time Indian. New York: Little, Brown Books for
Young Readers, 2007.
Bucher, Hans-Jürgen. “Multimodales Verstehen oder Rezeption als Interaktion: Theoretische und
empirische Grundlagen einer systematischen Analyse der Multimodalität.” Bildlinguistik:
Theorie – Methoden – Fallbeispiele. Ed. Hajo Diekmannshenke, Michael Klemm, and Hartmut
Stöckl. Berlin: Erich Schmidt, 2011. 123–156.
Chan, Eveline. “Integrating Visual and Verbal Meaning in Multimodal Text Comprehension: Towards
a Model of Intermodal Relations.” Semiotic Margins. Meaning in Multimodalities. Ed. Shoshana
Dreyfus, Susan Hood, and Maree Stenglin. London: Continuum, 2011. 144–167.
Cowan, Jennifer. Earthgirl. Toronto: Groundwood, 2009.
Danielewski, Mark Z. House of Leaves. A Novel. New York: Pantheon, 2000.
Deaver, Jeffery. Roadside Crosses. London: Hodder & Stoughton, 2009.
Denham, Scott, and Mark McCullough, eds. W. G. Sebald: History, Memory, Trauma. Berlin and New
York: De Gruyter, 2006.
Foer, Jonathan Safran. Extremely Loud & Incredibly Close. London: Penguin, 2005.

Brought to you by | Lund University Libraries

Authenticated
Download Date | 11/17/15 3:32 PM
650   Wolfgang Hallet

Gehring, Melina. “Das Labyrinth als Chronotopos. Raumtheoretische Überlegungen zu

Mark Z. Danielewskis House of Leaves.” Raum und Bewegung in der Literatur: Die
Literaturwissenschaften und der Spatial Turn. Ed. Wolfgang Hallet and Birgit Neumann.
Bielefeld: transcript, 2009.
Gibbons, Alison. “‘I Contain Multitudes’: Narrative Multimodality and the Book that Bleeds.” New
Perspectives on Narrative and Multimodality. Ed. Ruth Page. London and New York: Routledge,
2010a. 99–114.
Gibbons, Alison. “The Narrative Worlds and Multimodal Figures of House of Leaves: ‘– find your
own words; I have no more.’” Intermediality and Storytelling. Ed. Marie-Laure Ryan and Marina
Grishakova. Berlin and New York: De Gruyter, 2010b. 285–311.
Gibbons, Alison. Multimodality, Cognition and Experimental Literature. London and New York:
Routledge, 2012.
Glorig, Lorenz. “Narrative Konstruktion, Bild und Photographie in Jonathan Safran Foer’s Extremely
Loud & Incredibly Close.” Komparatistik Online 2 (2007): n. pag. freenet-homepage.de/
Komparatistik-online/page11937d5bdf2.html/. (15 Mar. 2012).
Haddon, Mark. The Curious Incident of the Dog in the Night-Time. London: Vintage, 2004.
Hallet, Wolfgang. “Multimodalität.” Metzler Lexikon Literatur- und Kulturtheorie. Ed. Ansgar
Nünning. Stuttgart: Metzler, 52008a. 546–547.
Hallet, Wolfgang. “The Multimodality of Cultural Experience and Mental Model Constructions of
Textual Worlds.” Ed. Jürgen Schlaeger. The Literary Mind. Tübingen: Narr, 2008b. 233–250.
Hallet, Wolfgang. “The Multimodal Novel. The Integration of Modes and Media in Novelistic
Narration.” Narratology in the Age of Interdisciplinary Narrative Research. Ed. Sandra Heinen
and Roy Sommer. Berlin and New York: De Gruyter, 2009a. 129–153.
Hallet, Wolfgang. “Fictions of Space: Zeitgenössische Romane als fiktionale Modelle semiotischer
Raumkonstitution.” Raum und Bewegung in der Literatur: Die Literaturwissenschaften und der
Spatial Turn. Ed. Wolfgang Hallet and Birgit Neumann. Bielefeld: transcript, 2009b. 81–113.
Hallet, Wolfgang. “Visual Images of Space, Movement and Mobility in the Multimodal Novel.”
Moving Images – Mobile Viewers. 20th Century Visuality. Ed. Renate Brosch. Berlin: LIT Verlag,
2011a. 227–248.
Hallet, Wolfgang. “Medialisierung von Genres am Beispiel des Blogs und des multimodalen
Romans: Von der Schrift-Kunst zum multimodalen Design.” Medialisierung des Erzählens
im englischsprachigen Roman der Gegenwart: Theoretischer Bezugsrahmen, Genres und
Modellinterpretationen. Ed. Ansgar Nünning and Jan Rupp. Trier: wvt, 2011b. 85–116.
Hallet, Wolfgang. “The Rise of the Multimodal Novel: Generic Change and Its Narratological
Implications.” Storyworlds across Media. Towards a Media-Conscious Narratology. Ed.
Marie-Laure Ryan and Jan-Noël Thon. Nebraska: University of Nebraska Press, 2014. 151–172.
Halliday, Michael K. An Introduction to Functional Grammar. London: Holder Education, 32004.
Hoth, Stephanie. “From Individual Experience to Historical Event and Back Again: ‘9/11’ in Jonathan
Safran Foer’s Extremely Loud & Incredibly Close.” Kulturelles Wissen und Intertextualität.
Theoriekonzeptionen und Fallstudien zur Kontextualisierung von Literatur. Ed. Marion Gymnich,
Birgit Neumann, and Ansgar Nünning. Trier: wvt, 2006. 283–300.
Kinney, Jeff. Diary of a Wimpy Kid: A Novel in Cartoons. New York: Amulett Books, 2007.
Kress, Gunther. Multimodality: A Social Semiotic Approach to Contemporary Communication.
London and New York: Routledge, 2010.
Kress, Gunther, and Theo van Leeuwen, eds. Multimodal Discourse: The Modes and Media of
Contemporary Communication. London: Arnold, 2001.
Larsen, Reif. The Selected Works of T. S. Spivet. London: Harvill Sacker, 2009.
LeVine, Philip, and Ron Scollon, eds. Discourse and Technology: Multimodal Discourse Analysis.
Washington: Georgetown University Press, 2004.

Brought to you by | Lund University Libraries

Authenticated
Download Date | 11/17/15 3:32 PM
 34 Non-verbal Semiotic Modes and Media in the Multimodal Novel    651

Mitchell, W. J. T. Picture Theory: Essays on Verbal and Visual Representation. Chicago: University of
Chicago Press, 1995.
Nørgaard, Nina. “Multimodality and the Literary Text: Making Sense of Safran Foer’s Extremely Loud
and Incredibly Close.” New Perspectives on Narrative and Multimodality. Ed. Ruth Page. London
and New York: Routledge, 2010a. 115–126.
Nørgaard, Nina. “Modality: Commitment, Truth Value and Reality Claims Across Modes in Multimodal
Novels.” Journal for Literary Theory 4.1 (2010b): 63–80.
Norris, Sigrid. “Multimodal Discourse Analysis: A Conceptual Framework.” Discourse and
Technology: Multimodal Discourse Analysis. Ed. Philip LeVine and Ron Scollon. Washington:
Georgetown University Press, 2004. 101–115.
Ondaatje, Michael. Running in the Family. 1982. London: Vintage, 1993.
Page, Ruth, ed. New Perspectives on Narrative and Multimodality. London and New York: Routledge,
2010.
Ryan, Marie-Laure. Narrative as Virtual Reality. Immersion and Interactivity in Literature and
Electronic Media. Baltimore and London: Johns Hopkins University Press, 2001.
Ryan, Marie-Laure. “Cognitive Maps and the Construction of Narrative Space.” Narrative Theory and
the Cognitive Sciences. Ed. David Herman. Stanford: CSLI, 2003. 214–242.
Ryan, Marie-Laure. “Fiction, Cognition and Non-Verbal Media.” Intermediality and Storytelling. Ed.
Marie-Laure and Marina Grishakova. Berlin and New York: De Gruyter, 2010. 8–26.
Ryan, Marie-Laure, and Marina Grishakova, eds. Intermediality and Storytelling. Berlin and New
York: De Gruyter, 2010.
Sebald, W. G. Austerlitz. London: Penguin, 2002. [German original 2001].
Scollon, Ron, and Philip LeVine. “Multimodal Discourse Analysis as the Confluence of Discourse and
Technology.” Discourse and Technology: Multimodal Discourse Analysis. Ed. Philip LeVine and
Ron Scollon. Washington: Georgetown University Press, 2004. 1–6.
Van Leeuwen, Theo. Introducing Social Semiotics. London and New York: Routledge, 2005.
Ventola, Eija, Cassily Charles, and Martin Kaltenbacher, eds. Perspectives on Multimodality.
Amsterdam and Philadelphia: John Benjamins, 2004.

6.2 Further Reading
Baldry, Anthony, and Paul J. Thibault. Multimodal Transcription and Text Analysis: A Multimedia
Toolkit and Coursebook with Associated On-line Course. London: Equinox, 2006.
Bateman, John. Multimodality and Genre: A Foundation for the Systematic Analysis of Multimodal
Documents. Houndmills and New York: Palgrave Macmillan, 2008.
Herman, David, and Ruth Page. “Coda/Prelude: Eighteen Questions for the Study of Narrative and
Multimodality.” New Perspectives on Narrative and Multimodality. Ed. Ruth Page. London and
New York: Routledge, 2010. 217–220.
O’Halloran, Kay L., and Bradley A. Smith. “Multimodal Studies.” Multimodal Studies: Exploring
Issues and Domains. Ed. Kay L. O’Halloran and Bradley A. Smith. London and New York:
Routledge, 2004. 1–13.

Brought to you by | Lund University Libraries

Authenticated
Download Date | 11/17/15 3:32 PM
Brought to you by | Lund University Libraries
Authenticated
Download Date | 11/17/15 3:32 PM