You are on page 1of 22

497665

2013

VCJ12410.1177/1470357213497665Visual CommunicationJessen and Graakjr

visual communication
ARTICLE

Cross-media communication in
advertising: exploring multimodal
connections between television
commercials
and websites

I b en Breda h l J essen and N icolai J rgensgaard


G raa k j r
Aalborg University, Denmark

A b stract

The aim of this article is to advance current understandings of cross-media


communication in advertising. The study is based on a sample of 80 television commercials and their announced websites, and the article is inspired
by recent studies of cross-media advertising effectiveness as well as semiotic perspectives on multimodal analysis. The authors present three dimensions to be considered when examining multimodal connections between
television commercials and websites: announcements, participants and
address strategy. These dimensions are brought together in an analytical
framework that can serve as inspiration for further research on cross-media
communication in advertising and possibly in other types of communication.
Ke y words

Advertising communication cross-media mode multimodal connections


television commercials websites

I ntroduction

The aim of this article is to advance current understandings of cross-media


communication in advertising. Cross-media communication is a field of study
with a growing significance. Enabled by the advent of new media technologies and an intensified interest in combining media platforms, content in different media increasingly seem to connect both by means of announcements
(e.g. web addresses and QR codes) and by presenting content with qualities in
common (e.g. a common visual identity). The article demonstrates how three
dimensions announcements, participants and address strategy function to
create multimodal connections between television commercials and websites.
SAGE Publications (Los Angeles, London, New Delhi, Singapore and Washington DC:
http://vcj.sagepub.com) Copyright The Author(s), 2013.
Reprints and permissions: http://www.sagepub.co.uk/journalspermissions.nav/
Vol 12(4): 437458 DOI 10.1177/1470357213497665
Downloaded from vcj.sagepub.com at E-nformation SRL on January 24, 2016

Let us begin with an example to illustrate the kinds of connections we will


bring into focus.
The example advertises a Danish provider of cable television and
broadband solutions called YouSee, and it includes a television commercial
and its announced website. The television commercial is composed of a drama
sequence and a final graphic still accompanied by a voice-over. The drama
part leads to a humorous point in which the main character is revealed as saying one thing, but doing another. We follow a father who enters his sons and
his daughters rooms kindly asking them what they are doing (Figure 1a). As
the answers from both children are very dismissive (I watch football Dad,
do you mind! / watch Paradise Dad, do you mind!), the father enters
the sitting room, shouting about how children are becoming slaves of the

Figure 1. From the YouSee television commercial (12 May 2011). Reproduced courtesy
of IK Jensen, YouSee.

438

Visual Communication 12(4)

Downloaded from vcj.sagepub.com at E-nformation SRL on January 24, 2016

media instead of doing something clever. The father then turns on the television, and we witness from his face how the television programme instantly
holds his attention (Figure 1b). Consequently, when his wife comes home after
a run and asks her husband what he is doing, he gets annoyed and tells her not
to disturb him ( do you mind!). The television commercial concludes with
a graphic still in which the characteristic green, white and yellow stripes of
the YouSee brand are displayed as background. First, the stripes appear with a
transparent television screen in front, co-branding a myriad of channel logos,
and with a male voice-over encouraging the viewer to order now (Figure 1c).
Next, the YouSee logo is displayed together with the top-level domain name
.dk (Figure 1d).
If we now visit the announced website (see Figure 2), the green, white
and yellow stripes appear as background, and the YouSee logo appears in
the top left corner of the page. In a semi-transparent box a special offer is
announced in written text, followed by an orange button to click and purchase.
The user is addressed in a direct manner by means of imperatives (get, order
now) and personal address ( you save / what are you waiting for?), and,
on the right, the message free of charge is visualized by a three-dimensional
zero in which a collage of still pictures from different television channels is
displayed.
From the brief description of the YouSee example, it appears that the
television commercial and the website connect in various ways. The website is
announced as a place to order now, but also other kinds of connections seem
to exist, most remarkably, the characteristic stripes that make up the final
graphic still in the television commercial and the background of the website.
Yet, the television commercial and the website also differ in many respects.
The characters from the drama sequence are absent on the website as well as
the implicit mode of address put forward by the drama.
The YouSee example illustrates the kind of cross-media communication to be examined in this article. In particular, we aim to explore how television commercials and websites relate by means of multimodal connections
across media and how these connections can be described and analyzed. To
our knowledge, there exists no fully fledged model for describing and analyzing connections between advertising content appearing in different media.
Therefore, the following study is explorative in the sense that it (1) synthesizes
perspectives from different research areas, most significantly those of advertising research and multimodal analysis; and (2) draws on a sample of actual,
observed multimodal practices (i.e. recent examples of connected advertising content appearing in different media). Our study will focus on the television commercial and the website as two separate texts (following Geertz,
1973: 9ff.) which relate in various ways via what we shall coin multimodal
connections (see more below). Hence, in this article, we will neither include
perspectives of production (e.g. the intention of producers) nor include actual
reception patterns (e.g. viewer experience, use and effect). Instead we will

Jessen and Graakjr

Downloaded from vcj.sagepub.com at E-nformation SRL on January 24, 2016

439

Figure 2. Front page of the YouSee website (12 May 2011). Reproduced courtesy of IK Jensen, YouSee.

440

Visual Communication 12(4)

Downloaded from vcj.sagepub.com at E-nformation SRL on January 24, 2016

concentrate on how advertising messages are connected on a textual level.


From advertising research, we are inspired by the growing interest in media
synergy, but we will expand this interest to include more than merely address
strategy (Kim et al., 2005) and, as implied above, we will take no further notice
of issues of effectiveness. From the field of multimodality, we are inspired most
notably by the multimodal principle of framing (Kress, 2009, 2010; Kress and
Van Leeuwen, 2006; Van Leeuwen, 2005), but we will adjust the principle to
apply to cross-media communication, a hitherto rather neglected issue within
studies of multimodality (see, for example, Engebretsen, 2012).
Whereas this article may contribute to the understanding of crossmedia communication in general, the findings will of course be marked by
the particular genre (advertising) and the particular media referred to (television and the web). Generally, television and the web can be understood as
media dominated by push and pull, respectively. The web most significantly
offers content to be searched for by attentive users with a high level of control
(users usually pull the content), while television most significantly offers a
predefined flow to be observed by easily distracted viewers with a low level
of control (viewers are pushed by the content). While this might apply as a
general and somewhat crude distinction, it shows even more evidently when
considering the genre of advertising in television. Specific television commercials are not announced and they are not necessarily the subject of the viewers
attention and interest. Indeed, we inadvertently come across television commercials, whereas websites are purposely visited. Symptomatically, television
commercials often explicitly refer to websites, whereas websites rarely explicitly refer to television commercials. From a strategic point of view, the television commercial craves the viewers attention, and the possible announcement
of the website represents an effort to redirect the viewer from potentially
competing commercials while instantaneously allowing for purchase.
In the following, we will explore multimodal connections between television commercials and websites. Firstly, we will present a selection of concepts that we find applicable from advertising research and the field of multimodal analysis. With this presentation, we intend to establish an otherwise
absent well-grounded terminology to describe cross-media communication in
advertising. Secondly, informed by observations of communicative practices
in a sample of 80 television commercials and websites, the article presents a
range of analytical dimensions to be considered when examining multimodal
connections and eventually systemize them in an analytical framework. In
conclusion, we will briefly evaluate the scope of the framework by considering
its relevance to other genres and media platforms.

A dvertising R esearc h and C ross - M edia


C ommunication

Within advertising research, the study of effectiveness a dominant issue


within this field of research has changed from an intra-media approach to an
Jessen and Graakjr

Downloaded from vcj.sagepub.com at E-nformation SRL on January 24, 2016

441

inter- or cross-media approach with an increased focus on the synergy effect


that the combinations of different media bring about (Assael, 2011). Crossmedia effects between television and web advertising have been studied predominantly on the basis of measurements of consumer responses (e.g. Chang
and Thorson, 2004; Dijkstra et al., 2005; Wang, 2011). A characteristic feature
of this line of research is the focus on the effects of independent variables on
dependent variables. For example, Wang (2011) examines the effects of the
independent variable cross-channel integration i.e. whether participants in
a laboratory setting are exposed to a television commercial with a subsequent
exposure to the advertisers website or solely the television commercial on
the dependent variables brand attitude, attitude toward the television spot
and media engagement (pp. 280ff.). Wang concludes that the study indicates
that the use of cross-channel integration of a television spot and additional
advertising messages on an advertisers website indeed enhances attitude
toward the television spot, perceived media engagement of overall advertising messages, and brand attitude (p. 286). We suggest that what matters when
examining cross-media communication is not only the fact that a television
commercial and a website are explicitly integrated by being part of the same
campaign, but also in which way they are integrated.
Only a few studies have examined the more detailed multimodal communication of television commercials and websites in a cross-media perspective. Kim et al.s (2005) study is an inspiring example. It examines the overall
strategies of advertising during Super Bowl 2003. Television commercials and
their associated websites are scored respectively in regard to whether they are
primarily informational or transformational and the extent to which message
strategies and creative strategies are made use of. In applying the concepts of
informational and transformational strategies, Kim et al. here subscribe to the
widely acknowledged dichotomy within advertising research offered by Puto
and Wells (1984) (see Laskey et al., 1989, for a similar approach on television
commercials and see Kim et al., 2005: 48ff., for an overview of related typologies). Puto and Wells (1984) propose that specific advertising messages will
be characterized by one of these strategies more than the other, and hence
the strategies are not mutually exclusive. Advertising following an informational strategy provides consumers with factual (i.e., presumably verifiable),
relevant brand data in a clear and logical manner, whereas advertising following a transformational strategy associates the experience of using (consuming) the advertised brand with a unique set of psychological characteristics (p.
638). Kim et al. (2005) focus on the distribution of strategies in their sample
(40 television commercials and websites) and aim at describing differences
and similarities (p. 56). It is concluded that the television commercials were
more transformational, while the web sites were more informational (p. 52).
Although the study shows merits in dealing to a considerable degree with the
multimodal communication of television commercials and websites, we suggest that the perspective should be both supplemented and nuanced.

442

Visual Communication 12(4)

Downloaded from vcj.sagepub.com at E-nformation SRL on January 24, 2016

While the conceptual distinction between informational and transformational strategies identifies the type of content and argument involved
in advertising, it does not detail the ways in which the addressee is in fact
spoken to. The issue of modes of address can be further specified by identifying differences in the ways in which the addressee is spoken to in terms
of verbal address. In television commercials, verbal address is predominantly
auditory and this is indicative of the importance of sound in television commercials more generally. It is not unusual for television commercials to appear
intelligible even to viewers looking elsewhere. Speech and music are the types
of sounds that ensure that glancing or distracted viewers to some extent are
kept informed on what is going on in the television commercial sound can
function as a sort of umbilical cord between the television commercials and
viewers.
In television commercials, the auditory verbal direct or explicit address
can take the form of a presenter, a testimonial, or a voice-over (Stigel, 2001:
332ff.). The speech of the presenter and the testimonial are both visualized
through appearances of speech performing characters. They differ in the way
in which the speech is directed: the presenter has eye contact with the camera and is thus demanding something from the viewer (cf. the conception of
demand-pictures in Kress and Van Leeuwen, 2006: 117ff.), whereas the testimonial e.g. a celebrity endorser usually has eye contact with an imagined interviewer sitting behind the camera. Contrary to the visually immersed
speech of the presenter and testimonial, the speech of the voice-over originates from outside the visualized context. The three types of verbal direct
address are all primarily factual in the sense that the speech is identifying
and commenting on i.e. providing information on products and brands.
Accordingly, they all three correspond to the broad category of informational
commercials. In television commercials, speech might also appear in a more
indirect and implicit manner as is the case in dramas and montages (Stigel,
2001). In small-scale dramas, for example, speech takes the form of a dialogue
between characters in a dramatic situation (as in the YouSee example), and
usually the role of the product and brand is not declared explicitly. Small-scale
dramas and montages tend to correspond to the broad category of transformational commercials.
Generally, a restricted focus on strategy may lead us to ignore otherwise
important aspects. Thus, we suggest that an analysis of strategy or what we
in the following will coin address strategy should be (1) nuanced by a focus
on the way in which the addressee is spoken to, and (2) supplemented with an
analysis of other types of multimodal connections between the television commercial and the website considered as degrees of similarity and/or dissimilarity
between one or more modes across media (to be expanded on below).
The referred examples of relevant advertising research on cross-media
communication indicate that knowledge about how connections are established is restricted. To obtain a more distinct focus on the organization of texts

Jessen and Graakjr

Downloaded from vcj.sagepub.com at E-nformation SRL on January 24, 2016

443

and their connections we will now present a selection of concepts that we find
applicable from the field of multimodal analysis.

M ultimodal A nal y sis and C ross - M edia


C ommunication

When pointing at similarity and dissimilarity (see Kim et al., 2005) to describe
connections between television commercials and websites, we find resemblance to the concepts of connection and disconnection related to the multimodal principle of framing (Kress and Van Leeuwen, 2006) and Kresss (2009,
2010) description of modal fixing and modal framing. Where modal fixing
concerns the applied modes used for representation (e.g. a written text represents an advertised product in one way, an image in another, etc.), modal
framing is about the ways pieces of information are connected to each other
so that we know what to put together with what, what to read in relation to
what (Kress, 2010: 149).
As pointed out by Kress and Van Leeuwen (2001: 67), modes can appear
in different media, and the materiality of the medium will affect how modes
appear. For instance, written text on the screen in a television commercial
and written text on a website have different qualities. It follows that connections can be established in many ways between the same mode across media,
but connections between different modes across media are also possible, for
example as a written factual piece of information on the website specifying a
dream-like audio-visual scenario from the television commercial.
Based on the linguistic works of MAK Halliday and JR Martin, Van
Leeuwen (2005) presents the linking of items of information as an example of
multimodal cohesion. He categorizes information linking as related either to
the concept of elaboration that repeats or restates information for the purposes
of clarification or to the concept of extension that adds new information, linking it to the existing information in a particular way for example, temporally,
or logically (p. 222). These basic categories are further sub-categorized into
systems of verbal, visual as well as visualverbal linking. Information linking
is considered as a link between two items of content appearing in one and the
same text. For instance, the linking between a text and an image can be realized as an explanation (a sub-category of elaboration) when the text makes the
image more specific (p. 230). Also hypertext links can be considered from the
perspective of information linking (p. 240).
The general categories of information linking seem useful in describing the links between television commercials and their announced websites
as regards the overall functions of the two connected texts. Even though the
announced web address in the television commercial is currently a nonclickable link, it seems reasonable to consider the two texts as linked to each
other (p. 219). The website can, for instance, be characterized as an extension
of the television commercial in that it provides additional information to the

444

Visual Communication 12(4)

Downloaded from vcj.sagepub.com at E-nformation SRL on January 24, 2016

short message of the television commercial. The presence of announcements


of the connected text can help to emphasize this kind of linking, for instance
in the shape of remediation (Bolter and Grusin, 1999) or by verbal statements.
In the following, we suggest adapting the idea of modal fixing and
modal framing to explore cross-media communication. Unlike the usual conception of framing as a common semiotic principle realized in a materially
connected whole for instance in a magazine advertisement (Van Leeuwen,
2005: 7) this study will focus on the connections between two texts not
simultaneously present. Our proposal is to consider multimodal connections
across media on a range of analytical dimensions considered relevant to the
genre of advertising. Hereby, we wish to pay attention to the ways in which
items of information appear similar or dissimilar both regarding the applied
modes of representation in the two media and regarding what is represented
in terms of participants. The concept of participants refers to Kress and Van
Leeuwens (2006: 48) understanding as
the participants who constitute the subject matter of the communication; that is, the people, places and things (including abstract things)
represented in and by the speech or writing or image, the participants
about whom or which we are speaking or writing or producing images.

By exploring what is represented in terms of participants (e.g. the advertised


product) and how it is represented as regards mode, we should be able to
describe cross-media communication in more detail.
E x ploring M ultimodal C onnections Between
T elevision C ommercials and W e b sites

To further explore the issues of cross-media communication presented by the


YouSee example, we will now present a sample of connected television commercials and websites. Similar to the procedure of Kim et al. (2005), we have
gathered a sample of ad pairs each comprised of a television commercial and
a website. From this sample, we induct configurations of multimodal connections inspired by the approach of Van Leeuwen (2005: 6). The ad pairs have
been determined by an explicit visual or auditory announcement of the website in the television commercial, and hereby the sampling procedure is apparently dissimilar to that of Kim et al. (2005) where no such announcement is
discussed. In this way, the sampling procedure arguably resembles characteristic patterns of reference and reception as shortly addressed in the beginning: television commercials often refer to websites (usually not the other way
around), and several studies have demonstrated that television commercials
can bring about increased traffic on the announced websites (e.g. Coffey and
Stipp, 1997; Zigmond and Stipp, 2010).
Encouraged by a recommendation for future research to use samples drawn at random from a specified universe (Kim et al., 2005: 56), we

Jessen and Graakjr

Downloaded from vcj.sagepub.com at E-nformation SRL on January 24, 2016

445

recorded all television commercials from the national Danish broadcaster TV


2 on a random evening, Wednesday 11 May 2011 from 5:45 pm to Thursday
12 May 00:05 am. The sample consists of 121 unique television commercials
(and 41 doublets in addition), and 80 of these include an announcement of a
website. For practical reasons, the 80 websites were recorded during the next
day, Thursday 12 May 2011. In this way, the recordings of television commercials and websites were not performed simultaneously. Even though this may
not accurately reflect typical reception patterns (the websites might have been
visited by some viewers during Wednesday night), we believe that the pairing
is appropriate in terms of production and, hence, the intended connection, i.e.
the website announced is the website recorded.
In analyzing the 80 ad pairs, we focused on the connections between
the television commercial and the announced website. The connections were
analyzed with reference to the whole television commercial (usually with a
duration of approximately 30 seconds) and to the front page of the website,
exclusively. While this procedure ignores potentially important content elsewhere on the website, the front page is the minimal unit that defines a website; it is the part that users are most likely to encounter, hence, arguably, the
most salient and important part to analyze (Herring, 2004: 6). Following this
procedure, we intend to map a range of analytical dimensions that present
pertinent ways in which ad pairs may differ or appear similar. The sample
provides us with empirical evidence for significant and recurring dimensions of multimodal connections, and the above theoretical perspectives have
helped to focus our attention and provided us with applicable terminology.
Accordingly, we have identified the following three dimensions of multimodal
connections: (1) announcements, (2) participants, and (3) address strategy. In
the next sections we will present how multimodal connections appear in the
sample according to these dimensions.

Announcements
As mentioned, the ad pairs in the sample were determined by the presence of
an explicit reference from the television commercial to the website. This kind of
reference to the connected text as such arguably establishes an important multimodal connection. The question is now how the television commercial refers
to the website, and how the website might refer to the television commercial.
The television commercial most typically refers to the website via
a spoken or written announcement of the web address, i.e. by verbal references. The web address can be presented more or less significantly (by means
of size, volume, duration and placement), and it can be further accompanied
by instructive co-texts that offer arguments and reasons for visiting the website (see Graakjr and Jessen, 2012, for a typology of arguments). In terms of
information linking (Van Leeuwen, 2005), the website is usually presented as
an extension of the television commercial. Visual references to specific web-

446

Visual Communication 12(4)

Downloaded from vcj.sagepub.com at E-nformation SRL on January 24, 2016

sites are not represented in the sample, solely the more conventional features
of the digital medium are remediated, e.g. by showing a moving cursor in the
television commercial.
References from websites to television commercials as presented on
television usually appear in a(n) (audio-)visual mode. (Audio-)visual references from websites to television commercials are identified as three kinds of
remediation: firstly, and most distinctively, remediation of television commercials on websites occurs as a repetition of the television commercial. The television commercial takes up nearly the whole space of the web page and loads automatically. Although the television commercial is used as a background on the
web page with other items of information in front of it, the television commercial
is highly foregrounded by the automatic playing of moving images and sound.
Secondly, remediation is identified when the television commercial appears side
by side with other items of information on the web page. The television commercial is not loaded or played automatically; instead it has to be initiated by
the user. Sometimes the television commercial appears as an item of information without further introduction, sometimes it is launched by a welcome or a
slogan. Thirdly, remediation occurs as stills from the television commercial, and
these stills function either as links to a subsidiary page from which the television
commercial can be played or as mood enhancement or product presentation.
Since television commercials appear unannounced during the flow of television
and with no specific destination to refer to, verbal references from the websites
to television commercials are more unusual and are characterized by unspecific
references such as the products from the television campaign.

Participants
In the following, we will examine the appearances of particular participants in
the television commercials and on the announced websites. Informed generally by the communicative characteristics of the genre of advertising, we have
selected four participants: logo, product, character and setting. While other
genres may prompt a different selection of participants, these present what
we consider to constitute the primary subject matter of the communication
(to paraphrase Kress and Van Leeuwen above) when it comes to advertising.
Logo A logo is a graphic mark used by commercial enterprises and organiza-

tions to promote recognition. Logos are often accompanied by a written


phrase in the form of a slogan, a brand name or, in the context of a television
commercial, a website address. Virtually every television commercial and
associated website present a logo, and therefore the logo presents itself as a
significant connecting element. However, the placement and functions of the
logo differ when the logos of television commercials are compared to the logos
of websites.
On websites the logo is typically reproduced in a small format placed in
the periphery, for example in the top left or right-hand corner of the website.

Jessen and Graakjr

Downloaded from vcj.sagepub.com at E-nformation SRL on January 24, 2016

447

The logo might also appear elsewhere as an integral part of a display of products.
In the periphery, the logo functions as an assurance that the user has arrived, as
a permanent reminder of where the user is (on the particular website), and as
an identification of what and where the user can come back to, namely the front
page of the website.
In television commercials, the logo might also appear small-scaled permanently in the periphery of the screen, for example in the lower left corner
of the screen. More often, though, the logo appears only sporadically during the commercial, for example as an integral part of a display of products
(diegetic or non-diegetic). However, most significantly, the logo in television
commercials appears large and in full screen at the end of the commercial as a
kind of exclamation mark. Here the logo functions to conclude the commercial by identifying and underlining the sender. Sometimes the underlining of
the sender is emphasized by animation of the logo, and often the visual logo is
anchored and enhanced by an accompanying sound signature. In television
commercials that link to websites, the logo at the end also functions as a kind
of colon. Thus, the logo presentation at the end signals that the preceding
information can in some way be extended when visiting a website (cf. information linking). Typically, this possibility is spelled out via an accompanying
verbal incitement to visit the website for example: see [website] for further
information. Whereas the logo on the website has a user-anchoring function
the user is informed and guided on the website, not away from it the logo
in the television commercial can function so as to lead the viewer away from
the television commercial (and television programming, more generally) and
to the website.
Product As expected, connections between television commercials and
websites are often established through the advertised product (or brand). The
product is either shown (in moving images or still pictures) or mentioned (in
speech or writing) in the television commercial as well as on the website. The
product is not necessarily represented in the same mode in the two media,
but there seems to be a common reference point to what is advertised. If we
take a closer look at how the product is shown in the sample, we find that the
product in the television commercial first and foremost is represented in an
audio-visual mode, sometimes followed by cut-out photographic stills of the
product (a so-called pack shot) at the end, whereas on the website the product most often is represented in a photographic mode as still pictures. In the
television commercial the product may appear as part of a narrative, for
instance showing the product in use and the values associated with it. On the
websites the product both appears as a detached or decontextualized object
on a neutral background (this seems to be the case on websites facilitating
purchase) and as implied in different kinds of settings, for instance in pictures showing the product in a use situation or in pictures embellishing the
product by means of, e.g. light and juxtaposition with pleasurable objects
such as flowers and fruit.

448

Visual Communication 12(4)

Downloaded from vcj.sagepub.com at E-nformation SRL on January 24, 2016

In television commercials that communicate through a dramatic composition, the product is sometimes hidden to begin with, only to be revealed
later as part of a (humorous) punch line. The hiding of the product serves
as a means to activate the viewer who is supposed to speculate about what is
advertised and how the product is related to the story. In contrast, the representation of the product on websites is often highlighted and pointed out
(if the user needs to search for the product on the website it is probably due
to navigational problems and not resulting from a calculated communicative
choice). The emphasis and pictorial concretization of the product also seems
to be a common trait on websites accompanying television commercials in
which the product is not shown, but only mentioned and implied.
It seems reasonable to assume that the same product will appear in
both the television commercial and on its announced website. However, this is
not always the case. For example, an announcement of a particular product in
the television commercial is sometimes followed by a presentation of another
product on the website. In such cases, either the logo or other kinds of markers
of the brand or the company serve to connect the television commercial and
the website.
As mentioned, the product is not only shown but also represented verbally in speech or writing. In the television commercials the product is mentioned by voice-overs, presenters, testimonials, or by fictional characters in
a dramatic universe. Except for websites on which the television commercial
is remediated in an audio-visual format, the websites do not include speech
(or other kinds of sound that may be argued to represent qualities related
to the product). The verbal representation of the product on the website is
mainly in writing, and the writing appears as slogans, headlines, show cards,
menu items, hyperlinks, and body text. In the television commercials the written representation of the product is limited to slogans, headlines and show
cards (and perhaps conditions of purchase in a small font size). The websites
thus include much more written information about the product than the television commercial; on the other hand, the writing on the websites seems to
be imbued with a distinct oral character (Janoschka, 2004: 125ff.) echoing the
verbal representation of the product by the voice-over.
Character In addition to the visual or verbal representation of the advertised

product, real people and human-like characters (e.g. personified objects or


animals) play an important part in establishing connections between the two
media. In general, there are more characters represented in the television
commercials than on the websites. This fact reflects the tendency that characteristic fictional characters from television commercial dramas (e.g. mascots
or the main characters of humorous serial commercials) as well as characters
represented by voice-overs are rare on the website. The visual representation
of presenters and testimonials (humorous as well as serious) in both the
television commercial and on the website is widespread in the sample. The
same applies to non-speaking characters represented in television

Jessen and Graakjr

Downloaded from vcj.sagepub.com at E-nformation SRL on January 24, 2016

449

commercials (e.g. models showing the product) who are depicted in stills on
the website, often with simulated eye contact to the user. However, we also
find examples in which the presenter function is shared between different
people in the television commercial and on the website respectively. In such
cases, the website presenter appears more unobtrusive and anonymous than
the presenter in the television commercial. The I/yourelation that is proposed and emphasized by the presenter function is thus framed differently in
the two media. In addition, the presenter function is often represented by
means of written text on the website.
Setting The settings represented in the television commercial and on the

website are usually very different. Except for when the television commercial
is remediated on the website, products are presented in dissimilar settings, for
instance a highly articulated product setting in the television commercial in
contrast to a minimalistic product setting on the website. The product can also
be shown in different stages of the life cycle of the product, e.g. a representation of the product in use in the television commercial as opposed to a representation of the product in the process of production on the website. Usually,
the settings are represented in different modes. Most common is the difference between an audio-visual and a graphic or photographic representation of
settings. It seems, however, that the television commercial in some way anticipates the different setting on the website. We have observed that television
commercials often involve breaks in the composition by way of shifts from an
audio-visual mode to a graphic mode, for example in the final still. In many
cases these stills seem to match the settings of the website. So, despite the obvious differences in the representation of settings in the two media, a variety of
connections can also be identified, for instance in the use of color schemes as
well as in the recurrence of distinctive qualities related to the setting, e.g.
typography or graphic style.

Address strategy
Address strategy refers to ways in which the addressee is positioned. Both television commercials and websites, of course, more or less explicitly position
their addressees as consumers. However, the specific modes of address tend
to differ when it comes to how the consumer is positioned in television commercials compared to websites.
The television commercials in our sample represent a wide spectrum of
address strategies. In general, the sample mirrors the heterogeneous distribution of address strategies in previous, comparable samples (e.g. Stigel, 2001).
While the present sample does not exactly correspond to the distribution of
predominantly transformative address strategies in the above sample of Kim
et al. (2005) e.g. transformative address strategies tend to be more characteristic of the highly profiled commercials broadcast during the Super Bowl
than of commercials during everyday broadcasts transformative address
strategies occur more often in the present samples television commercials
compared to the samples websites.

450

Visual Communication 12(4)

Downloaded from vcj.sagepub.com at E-nformation SRL on January 24, 2016

The websites in our sample are predominantly informational. Typically,


they provide information on browsing directions and links to product information that is, information on where to order, learn more, see full product
range, etc. Transformational address only appears sporadically in the form of,
for example, stills of characters from small-scale dramas of a television commercial. Compared to the informational and factual television commercials,
the informational websites differ remarkably by rarely encompassing sound
as part of the address. For example, speech in any form is usually absent on
websites, and generally the addressee is positioned as a reader more than a
listener. The speaking presenters and testimonials of particular television
commercials may be depicted on the website in the form of stills.
When looking at connections between websites and television commercials in regard to address strategy, we can identify a continuum. At one end of
the continuum we can identify similarity, when, for example, both television
commercial and website are predominantly informational. At the other end
of the continuum we can identify dissimilarity, when the address of the website fundamentally differs from the address of the television commercial (e.g.
a drama-based indirect address of a television commercial vs a factual, direct
address on the website). Falling between these two basic types, we can identify
a number of more nuanced connections. Given that most websites are informational, a nuanced connection most often shows between a primarily transformational address of a television commercial and a primarily informational address
of the announced website. Thus, the television commercial might include informational hints at the end of the sequence (e.g. by providing information on
where to buy the product and learn more about the brand), and the website
might include textual references to the transformational address of the television
commercial (e.g. by showing stills of fictional characters). In television commercials, the address strategy on the website may in fact be spelled out to the viewer
through, for example, a voice-over: For more information, see [the website].
A n A nal y tical F ramewor k

Based on the examination of the sample, we have described a range of multimodal connections between the television commercials and their announced
websites. We have identified how connections can be established by the presence of different kinds of references to the connected text. These kinds of references are located in the text as an announcement of the other. Moreover, we
have examined multimodal connections between texts in terms of different
degrees of similarity or dissimilarity between what is represented and how it
is represented. What is represented concerns the participants in the television
commercial and on the website respectively, and how it is represented concerns mode of representation of participants (visual, verbal, or auditory) and
address strategy (informational or transformational). Different participants
can be represented in the same mode or in different modes across media (and

Jessen and Graakjr

Downloaded from vcj.sagepub.com at E-nformation SRL on January 24, 2016

451

Figure 3. Framework for analyzing multimodal connections between television


commercials and websites.

sometimes also in different modes within the same media); moreover, different modes of representation can be related to the same or to different modes
of address. Evidently, how multimodal texts are related across media can be
a very complex matter to describe. During the sequence of a television commercial the product may be represented in different modes, for instance the
product can be represented by means of moving images in the first part of
the commercial and by still photography and speech at the end. Likewise, the
website may involve multiple modes in the representation of the product, and
may implement a mixed strategy as regards address strategy.
Figure 3 summarizes the identified possible multimodal connections
specified as three analytical dimensions that should be taken into consideration when analyzing cross-media communication between television commercials and websites.
The framework allows us to map how multimodal connections are
actualized in specific ad pairs. According to the three analytical dimensions,
we begin by noticing the direction of the announcement from the television
to the website and/or from the website to the television commercial if such
announcement is present. The bi-directional arrows in Figure 3 indicate the
possible kinds of announcements to the connected text. In regard to the participants, two kinds of observations should be made. Firstly, we should examine whether the participants (logo, product, character and setting) are more or
less similar or dissimilar with regard to what is represented (e.g. is it the same
product that is advertised?). Secondly, the represented participants should be
examined according to whether they can be characterized as more or less similar
or dissimilar in regard to how (in what modes) the participants are presented
(e.g. is the product presented in the same way?). As illustrated by the different

452

Visual Communication 12(4)

Downloaded from vcj.sagepub.com at E-nformation SRL on January 24, 2016

Figure 4. Multimodal connections between the YouSee television commercial and


website.

kinds of shadings between the boxes in Figure 3, multimodal connections can


be established by similarity or dissimilarity as well as by more complex organizations involving both similarity and dissimilarity. Address strategy only concerns
how the addressee is spoken to, but as mentioned in the discussion of address
strategy above this dimension is closely related to the represented characters.
Yousee R evisited

We will now revisit the introductory example of YouSee to illustrate how the
proposed framework can be applied to describe and analyze multimodal connections (see Figure 4). Considering the announcement of connected texts,
the uni-directional arrow indicates that the television commercial explicitly
announces the website by a verbal (written) reference (see Figure 1b). The
website is not explicitly referring to the television commercial, neither is
any kind of visual reference in terms of remediation registered. In order to
deal with the complexity of participants and address strategy, the television
commercial is divided into two units of analysis. First the drama part will be
addressed, then the final still.
In the analysis of multimodal connections between the main drama
sequence in the television commercial and the front page of the website, we
begin by noticing what is similar in regard to what and how participants are
represented. In the television commercial the product is described by means
of moving images, yet in a very implicit way as something we hear (we do
not see the screen, only the sound from the television set is heard) and as
something mentioned (by the father in his critical attack on teenage media
culture). In contrast, the same product is described in a more factual way on

Jessen and Graakjr

Downloaded from vcj.sagepub.com at E-nformation SRL on January 24, 2016

453

the web, partly in writing, partly by means of the still pictures in the zero in
which we see examples of what YouSee can offer on the screen (see Figure 2). It
follows, that neither the characters nor the settings appear similar. The fictive
characters as well as the home settings of the drama are absent on the website.
Instead, we are confronted with a graphic setting and an explicit identification of the sender (television from us). Accordingly, we notice a difference in
address strategy, namely a transformational address strategy in the television
commercial and an informational address strategy on the website.
When considering the final still of the television commercial and the
front page of the website, a different set of connections appear. First and foremost, the logo appears similar even though the .dk is added in the television
commercial. As regards mode of representation, the product is represented dissimilarly by verbal speech in the television commercial and in written text on
the web. Also the characters are dissimilar as we encounter a voice-over in the
television commercial and a written identification of sender or seller (us) on
the website, yet, as to what is represented (the sender or the seller), the characters appear dissimilar. Finally, the presentation of the setting and the characteristic stripes in a three dimensional room in the television commercial (see
Figure 1d) differ from the presentation of a two-dimensional wall on the website (see Figure 2). Yet, the colors and patterns of stripes are similar. The stripes
of the television commercial seem to allude to a path, while the stripes of the
websites seem to assure that the user has arrived. Unlike the drama part, what
is now becoming similar is the overall informational address strategy.
In Figure 4, the differences in the television commercial, i.e. the drama
part and the graphic still, are taken into account by acknowledging that the two
parts propose different multimodal connections (see the grids in the boxes of
settings and address strategy indicating both similarity and dissimilarity).
However, the framework does not provide consideration of how the
transition from dissimilarity to similarity as regards settings and address strategy is composed. In the YouSee television commercial, the setting of the final
still is, in fact, represented by hints in the drama part by way of two paintings
in which the characteristic stripes are displayed (see one example in Figure
1b). In addition, the colors of some of the house interior seem to match the
colors of the stripes, for instance, the yellow bookcase and the green lamp in
Figure 1a. In this way the drama part seems to tune in on the final still as
well as on the website a level of analytical detail which the proposed general
framework does not easily capture.
C oncluding R emar k s

Inspired by concepts from the field of multimodal analysis and advertising


research, this article has demonstrated how three dimensions announcements, participants and address strategy function to create different
kinds of multimodal connections between the television commercials and the

454

Visual Communication 12(4)

Downloaded from vcj.sagepub.com at E-nformation SRL on January 24, 2016

websites in our sample. The three dimensions have been presented in a general
framework (see Figure 3) and illustrated by way of an examination of a specific example (YouSee). Whereas multimodal connections established by the
announcements concern how the two texts are referred to as texts in the two
texts respectively, the participants and address strategies in both texts form
multimodal connections to varying degrees of similarity and dissimilarity.
In regard to participants and address strategy, the connection between
a television commercial and a website can rarely be described unambiguously
as either similarity or dissimilarity (as the YouSee example has shown). It is
also important to stress that similarity does not necessarily imply a stronger
or more coherent connection. Unlike similarity that seems to highlight that
parts belong together or are identical, dissimilarity seems to suggest that the
parts belong to different domains (Van Leeuwen, 2005: 12), yet still supplement each other. Dissimilarity can even be explained via the announcement
of the website which typically informs or indicates what the user should expect
to find on the website, e.g. acquire information about the product. Hence, in
some cases, dissimilarity can be interpreted as expressions of a communicative
intent involving a semiotic division of labor between the television commercial and the website (e.g. the television commercial showing the product in use
and the website facilitating transaction as in the YouSee example), arguably
mirroring characteristic features (i.e. push or pull) of different media involved.
While this article has presented ways of analyzing multimodal connections in relation to three analytical dimensions, it has not examined whether
a typology of multimodal connections between television commercials and
websites can be developed on a general level. Such an examination should
preferably involve an even larger sample of ad pairs, which could then be analyzed and categorized on the basis of the framework presented in Figure 3.
Also, diachronic and synchronic studies i.e. of possible developments and
differences between comparable samples could profitably be performed to
examine the representativeness of the present study and its results. It is beyond
the scope of this article to further examine these questions, but hopefully the
proposed framework can inspire future research on these and similar subjects.
At least, in relation to current research on cross-media effect between
television commercials and websites, the proposed analytical dimensions provide a framework for a more detailed description of the stimuli to which the
consumer is (potentially) responding. Moreover, the framework may serve to
generate specific hypotheses about the relation between types of connections
(a possible typology) and cross-media effects. In the realm of multimodal
analysis, the framework contributes with a particular focus on the complex
nature of multimodal connections by rethinking the concept of modal fixing and modal framing (Kress, 2009, 2010) in a cross-media perspective. The
framework allows us to explore and consider what to read in relation to what
(Kress, 2010: 149) and to make that consideration across media platforms.

Jessen and Graakjr

Downloaded from vcj.sagepub.com at E-nformation SRL on January 24, 2016

455

The extent to which the proposed framework can be applied to other


genres and media contexts remains to be examined. However, we suggest that
the framework presents dimensions of analysis that can be modified e.g.
supplemented, reduced, and/or emphasized differently according to the
particular media and genre characteristics in each case. For example, when
analyzing multimodal connections in the news genre, some dimensions may
prove less relevant (e.g. specific participants such as logo and product) compared to others (e.g. setting, announcement and address strategy). When analyzing, for example, multimodal connections between radio commercials and
websites, the verbal connections e.g. the announcement and possible slogan
should perhaps be foregrounded, whereas the logo might represent a less
relevant participant. Furthermore, the framework needs to be expanded considerably when analyzing cross-media connections in cases involving a vast
variety of media, e.g. printed ads, outdoor ads, apps, websites, television commercials, radio commercials, and physical settings.
In any event, due to the fact that more and more texts and media in
advertising campaigns and in other contexts seem to connect, it is increasingly important to consider the issue of multimodal connections in crossmedia communication.
F unding

This research received no specific grant from any funding agency in the public,
commercial, or not-for-profit sectors.
R eferences

Assael, H. (2011) From silos to synergy. Journal of Advertising Research 51:


4258.
Bolter, J.D. and Grusin, R. (1999) Remediation: Understanding New Media.
Cambridge, MA: MIT Press.
Chang, Y. and Thorson, E. (2004) Television and web advertising synergies.
Journal of Advertising 33(2): 7584.
Coffey, S. and Stipp, H. (1997) The interactions between computer and
television usage. Journal of Advertising Research 37(2): 6167.
Dijkstra, M., Buijtels, H. and Van Raaij, W.F. (2005) Separate and joint effects
of medium type on consumer responses: A comparison of television,
print, and the Internet. Journal of Business Research 58(3): 377386.
Engebretsen, M. (2012) Balancing cohesion and tension in multimodal
rhetoric. An interdisciplinary approach to the study of semiotic
complexity. Learning, Media and Technology 37(2): 145162.
Geertz, C. (1973) The Interpretation of Cultures. London: Hutchinson.
Graakjr, N.J. and Jessen, I.B. (2012) Fra tv til net Reklamer i et tvrmedialt
og stetisk perspektiv (From television to web Advertising in a crossmedia and aesthetic perspective). In: Stigel, J. (ed.) Reklame eller

456

Visual Communication 12(4)

Downloaded from vcj.sagepub.com at E-nformation SRL on January 24, 2016

hvordan stter man sving i bevidstheden (Advertising or how to affect


the mind). Aarhus: Systime Academic.
Herring, S.C. (2004) Content analysis for new media: Rethinking the paradigm.
In: New Research for New Media: Innovative Research Methodologies
Symposium, Working Papers and Readings. Minneapolis: University
of Minnesota School of Journalism and Mass Communication,47-66.
Available
at:
http://ella.slis.indiana.edu/~herring/newmedia.pdf
(consulted 5 August 2013).
Janoschka, A. (2004) Web Advertising: New Forms of Communication on the
Internet. Amsterdam: John Benjamins.
Kim, J., McMillan, S.J. and Hwang, J-S. (2005) Strategies for the Super Bowl of
advertising: An analysis of how the Web is integrated into campaigns.
Journal of Interactive Advertising 6(1): 4660.
Kress, G. (2009) What is mode? In: Jewitt, C. (ed.) The Routledge Handbook of
Multimodal Analysis. London: Routledge, 5467.
Kress, G. (2010) Multimodality: A Social Semiotic Approach to Contemporary
Communication. London: Routledge.
Kress, G. and Van Leeuwen, T. (2001) Multimodal Discourse: The Modes and
Media of Contemporary Communication. London: Arnold.
Kress, G. and Van Leeuwen, T. (2006) Reading Images: The Grammar of Visual
Design, 2nd edn. London: Routledge.
Laskey, H., Day, E. and Crask, M. (1989) Typology of main message strategies
for television commercials. Journal of Advertising 18(1): 3641.
Puto, C. and Wells, W. (1984) Informational and transformational advertising: The
differential effects of time. Advances in Consumer Research 11(1): 638643.
Stigel, J. (2001) The aesthetics of Danish TV-spot-commercials: A study of
Danish TV commercials in the 1990s. In: Hansen, F and Hansen,
LY (eds) Advertising Research in the Nordic Countries. Copenhagen:
Samfundslitteratur, 327350.
Van Leeuwen, T. (2005) Introducing Social Semiotics. London: Routledge.
Wang, A. (2011) Branding over internet and TV advertising. Journal of
Promotion Management 17(3): 275290.
Zigmond, D. and Stipp, H. (2010) Assessing a new advertising effect:
Measurement of the impact of television commercials on internet
search queries. Journal of Advertising Research 50(2): 162168.
Biograp h ical notes

IBEN BREDAHL JESSEN obtained her PhD on web advertising at Aalborg


University and is now an Assistant Professor. Her research interests include
advertising and aesthetics in a text analytical perspective. Among her publications are The Aesthetics of Web Advertising: Methodological Implications for
the Study of Genre Development (in N Brgger (ed.), Web History, Peter Lang,
2010) and Sounds of Web Advertising (with NJ Graakjr, in MS Eastin et al.

Jessen and Graakjr

Downloaded from vcj.sagepub.com at E-nformation SRL on January 24, 2016

457

(ed.) Handbook of Research on Digital Media and Advertising: User Generated


Content Consumption, Information Science Reference, 2010).
Address: Aalborg University, Department of Communication & Psychology,
Nyhavnsgade 14, 9000 Aalborg, Denmark. [email: iben@hum.aau.dk]
NICOLAI JRGENSGAARD GRAAKJR obtained his PhD on music in television commercials at Aalborg University and is now Professor (with special duties).
His research interests include musicology, advertising aesthetics and social psychology. Among his publications are Music In Advertising Commercials Sounds
in Media Communication and Other Settings (Aalborg University Press, 2009; coedited with C Jantzen), Sound and Genre in Film and TV (MedieKultur. Journal
of media and communication research 2010; co-edited with B Langkjr) and
Music in Television Commercials Theory and Analysis (Samfundslitteratur, 2011,
in Danish).
Address: same address as Iben Bredahl Jessen. [email: nicolaig@hum.aau.dk]

458

Visual Communication 12(4)

Downloaded from vcj.sagepub.com at E-nformation SRL on January 24, 2016

You might also like