MUSI 2559 / 4535

Audio-Visual Composition:
History, Analysis, and
Creative Application
June 12th – July 7th, 2017, MTWRF
11:30AM – 1:45PM, New Cabell Hall 268

Eli Stine
ems5te@virginia.edu
Office: Wilson Maker Space (first floor of Wilson hall)
Office Hours: Wednesdays 2:00PM to 3:00PM & by appointment

Course Overview

This course explores the ways in which sound interacts with video. We will
investigate this interaction by first discussing the history and contexts of
electronic sound, video, and their relationship, establishing (or refreshing) audio
and video editing skills, and then getting hands-on experience through creative
projects. Projects include composing sound design for film, creating video art
(that incorporates video-recorded and/or animated materials), and designing
real-time interactive media projects. The target student of this course is a
musician interested in expanding their relationship to video and multimedia. No
experience with audio or video technologies is required, although it is welcome.

Students exiting this course will have a broad, working knowledge of the history
of audio-visual media, tools to analyze and contextualize (both artistically and
technically) a variety of different audio-visual mediums, and the artistic and
technical skills to create their own audio-visual compositions.

Materials & Resources

This course requires no purchase of texts, recordings, software, or other

materials. That being said, having access to several programs and devices is
recommended, and outlined below.

1
Software

A non-exhaustive list of software for use in this class is outlined below. This
software was chosen for its access (free or inexpensive) and capabilities
(diverse and powerful), but a great deal of other software exists out there, and
there is no right set of tools with which to make audio-visual composition.

Audio Tools
Audacity, GarageBand, Reaper, Pure Data
(free/cheap, but full functionality)
Logic, Pro Tools, Ableton Live (and Max for Live), Max
(paid, but polished)

Video Tools
Lightworks (free, available for Mac/Windows, etc.)
Adobe Creative Suite (paid, available on classroom computers)
Final Cut Pro X (paid, Mac only)
Processing (free, for animation)
Cinema 4D (paid, 3D modeling)
Jitter (paid, part of Max)

Software Recommendations
Audacity (free) / GarageBand (free, Mac only) OR
Cockos Reaper ($60, has a flexible demo period)
Pure Data (free) OR Cycling ’74 Max ($59 a year, demo period)
Lightworks (free) OR Adobe Creative Suite (Premiere, others)
($19.99 a month or use on classroom computers)

Recording Devices

Although both audio recorders and video cameras will be available to check out
during class periods (and after, via the music department supply and the Digital
Media Lab) having access to an audio recorder, video camera (on a phone or
otherwise) could prove helpful.

Audio Recorder Recommendations: Zoom H1/H4/H5, Tascam DR-05

Video Camera Recommendations: Sony Handycam/Point & Shoot, Canon Vixia,
HDLR cameras with video support, Smartphone Camera (Samsung/Pixel/iPhone,
etc.)

2
Media Sets

Media sets are collections of audio and video files, a document that
contextualizes those files, and a reading, all of which should be consumed
together. The media sets provided are outlined below and are available in the
Resources/Media directory on the course site. Readings are available as .pdf
files and provide historical and analytical context within the fields of electronic
music, film, video art, and their intersections. The consumption of a media set
should take approximately 30 – 40 minutes (20 minutes of audio-visual material
and a short reading).

Set 1-Electronic Music

1.1-Early Electronic Music (40s-60s)
1.2-Computer + Synth + Tape Music (60s-80s)
1.3-Contemporary Electroacoustic Music (90s-Present)

Readings Selected From:

Thom Holmes-Electronic and Experimental Music
Nicholas Collins-Introduction to Computer Music
Christoph Cox, Daniel Warner-Audio Culture
Trevor Wishart-On Sonic Art
Kim Cascone-The Aesthetics of Failure
Caleb Kelly-Cracked Media: The Sound of Malfunction
Natasha Barrett-Trends in Electroacoustic Music
Antye Greie-Ripatti-Herstory of Electronic Music

Set 2-Film, Video Art, Music Video

2.1-Historical Film, Video Art
2.2-Historical Sound Design, Music Video
2.3-Contemporary Film, Video Art
2.4-Contemporary Sound Design, Music Video
2.5-Animation

Readings Selected From:

David A. Cook, Robert Sklar-History of the Motion Picture
Kristin Thompson, David Bordwell-Film History: An Introduction
Kate Horsfield-Busting the Tube: A Brief History of Video Art
Tom Sherman-The Nine Lives of Video Art
Pat Aufderheide-Music Videos: The Look of the Sound

3
 Michel Chion-Audio-Vision
Interviews from Designing Sound (online resource)
Maria Donata Napoli-Short Review of The Relationship Between
Video, Music, and Art
Holly Willis- New Digital Cinema: Reinventing the Moving Image
Stephen Cavalier-The World History of Animation
Casey Reas, Ben Fry-Getting Started with Processing

Set 3-Unfixed Media

3.1-Interactive Art Systems
3.2-Control Interface Contexts
3.3-Mediated Spaces

Readings Selected From:

Nicholas Cook-Analyzing Musical Multimedia
Eduardo Miranda, Marcelo Wanderley-New Digital Musical
Instruments
Kathleen Maloney-Sounding Images and Imaging Sounds
Claire Bishop-The Anachronic Time(s) of Installation Art
Michael J. Lyons, Alexander Jensenius-A NIME Reader
Steve Reich-Music as a Gradual Process
Erika Suderburg-Space, Site, Intervention: Situating Installation Art

Attendance, Participation, & Grading

Participation - 15%

Maintaining active engagement with the class materials is absolutely necessary

in order to do well on class assignments. Participation includes engaging with
the assigned media sets, listening and sharing during discussions, and focusing
during in-class work sessions.

Attendance - 15%

Timely class attendance is mandatory. Consistent lateness will be counted as

unexcused absences. 2 or more unexcused absences will result in a letter
grade deduction. Absences will be excused if, and only if, they are officially
recognized by the University of Virginia (medical absences, religious holidays,
etc.).

4
Quizzes - 15%

Short quizzes will be given at the end of the first three weeks to test your
comprehension and retention of historical context and technical information.
Active note-taking and briefly studying your notes after each class period will aid
you in passing these quizzes.

Assignments 55%

The majority of this class is focused on a series of assignments designed to

explore various intersections of audio and visual practices. Assignments
(outlined below) start as being worth little of the grade and progress to the final
project, which is worth a quarter of the total class grade. Start early on
assignments and do not be afraid to ask for help. More information about
assignment grading practices can be found below.

Daily Curriculum

Each day will consist of a discussion, lecture, and activity

Monday, June 12th

Introduction to Sound and Image
Introductions, syllabus discussion, goal discussion
What is sound? What is image? (Acoustics, Optics, Physics)
What is digital audio? What is digital video? (Computer
Representations)

Activity: Sound Walk + Discussion of Perception (a la Lopez)

HW: Media Set 1.1

Tuesday, June 13th

History and Context of Electronic Music
Discussion of Media Set 1
History of Electronic Music
How do microphones work? How do recorders work?
How can we capture real-world audio?

Activity: Field Recording, Sound Share

5
 HW: Media Set 1.2

Wednesday, June 14th

Electronic Music Technique + Composition I
Discussion of Media Set 1.2
Introduction to Electronic Music Environment (Reaper)
How do we edit audio? (Technique)
How do we organize sound over time? (Composition)

Activity: Begin Sound Collage Assignment

HW: Continue Sound Collage, Media Set 1.3

Thursday, June 15th

Electronic Music Technique + Composition II
Discuss Media Set 1.3
Introduction to Synthesis
Introduction to Audio Processing Techniques (outside of Reaper)
Automation, Intra-Track Interactivity (modulation, side-chaining)
Workflow, Extensibility, Personalization
Multi-Channel Audio and Ambisonics

Activity: Continue Sound Collage

HW: Finish Sound Collage

Friday, June 16h

Piece Presentations + Quiz + Introduction to Video Art
Piece Presentations
Short quiz on acoustics, history of electronic music, Reaper
Introduction to Video Art

Activity: Video Walk (Minute Zoom, a la Bussigel)

HW: Media Set 2.1
///

Monday, June 19th

History and Context of Film/Video I
Discuss Media Set 2.1
History of Film, Video Art
How does film work?

6
 How has film evolved over time?

Activity: Watch The Movies Begin (compilation of early films)

HW: Media Set 2.2

Tuesday, June 20th

History and Context of Film/Video II
Discuss Media Set 2.2
History of Sound Design, Music Video

Activity: Sound Design / Music Video Workshop

HW: Media Set 2.3

Wednesday, June 21st

Video Technique + Composition I
Discuss Media Set 2.3
Introduction to Video Environment (Lightworks / Premiere)
How do we edit video? (Technique)
How do we organize video over time? (Composition)

Activity: Begin Film Sound | Music Video (FS|MV) Assignment

HW: Continue FS|MV, Media Set 2.4

Thursday, June 22nd

Video Technique + Composition II
Discuss Media Set 2.4
More Video Editing Techniques (effects, blend modes, masks,
color correction, rendering options)

Activity: Continue FS|MV

HW: Finish FS|MV

Friday, June 23rd

Piece Presentations + Quiz + Introduction to Interactive Media
Piece Presentations
Short quiz on optics, History of Video, Lightworks
Introduction to Interactive Media

Activity: Audio-Visual Relationship Design Exercise (a la Stine)

7
 HW: Media Set 2.5
///

Monday, June 26th

Animation Workshop: Processing
Discuss Media Set 2.5
Introduction to Animation (History and Context)
How do we draw and animate synthesized image?

Activity: Processing Workshop

HW: Media Set 3.1

Tuesday, June 27th

Patching Workshop: Max, Pd, Jitter
Discuss Media Set 3.1
Introduction to Patch-based Software
How can we build dynamic musical/visual environments?

Activity: Max/Pure Data Workshop

HW: Media Set 3.2

Wednesday, June 28th

Mediating Action: Control Interfaces
Discuss Media Set 3.2
History and Context of Control Interfaces
How can we perform sound and image in real-time?

Activity: Interface Design Workshop

HW: Media Set 3.3

Thursday, June 29th

Mediating Space: Installation Art
Discuss Media Set 3.3
History and Context of Installation Art
How can we mediate space through light and sound?

Activity: Installation Design Workshop

HW: Prep. For Final Project

8
Friday, June 30th
Final Project Discussion + Interactive Media Quiz + Final
Project Design Meetings
Discuss Final Projects
Short quiz on animation, control interfaces, and installation art

Activity: Individual Meetings to Discuss Final Project

HW: Assigned Media, Final Project Proposals
///

Monday, July 3rd

Final Project In-Class Work
Work independently or in group on projects

Tuesday, July 4th

Final Project In-Class Work
Work independently or in group on projects

Wednesday, July 5th

Final Project In-Class Work + Mid-Week Update
Work independently or in group on projects

Activity: Mid-week Feedback Session

Thursday, July 6th

Final Project In-Class Work
Work independently or in group on projects

HW: Finish Final Project (due 9PM)

Friday, July 7th

Final Project Presentations + Showing
Project Presentations
Last Thoughts on Audio-Visual Composition

6PM: Setup for Showing at Venue TBD

8PM: Class Showing at Venue TBD

9
Assignments

Assignment grading will be based on artistic inventiveness and creativity, on the

effectiveness of the pairing of the audio and video, and quality (both technical
and aesthetic) of the audio and video portions separately.

Assignment 1 – Sound Collage – 5% – Due June 15th at 9PM

Using the techniques and resources discussed, compose a short
(approximately 2 minute) sound composition that makes use of responsibly
sourced* real-world and/or synthesized sounds and engages with the
parameters of electronic music composition: collage, rhythm, timbre, etc. During
your composition you must have at least one moment of silence.

This assignment must be completed individually

Assignment 2 – Film Sound / Music Video – 10% – Due June 22nd at 9PM
Pick ONE of the following:
Find a video (from a website like archive.org or use one of the provided
examples) and create sound for it (music and/or sound effects). NO sound from
the original video should be present. All sounds used in the video should be
recorded or sourced responsibly*.

Pick a piece of music (or use one of the provided examples) and create video
for it. NO extra sounds should be layered on to the music. All images should be
animated or shot footage made by the creator or sourced responsibly*.

You may work alone or in collaboration with one classmate on this

assignment

Assignment 3 – Live Media (In-Class) – 15% – Completed June 28th / 29th

Different than the other assignments, these projects will be completed in class
and will engage with 1) control interfaces and 2) installation art. Because of
the limited amount of time available project expectations will be different than for
the other assignments.

> For the first assignment, design and implement a program/physical system
that invites real-time input to engage with sound and image. This could be a
software instrument, a system built-around a physical controller, a collection of
responsive electronics, an Internet-activity controlled synthesizer, etc. as long as
the system facilitates the real-time manipulation of sound and image.

10
> For the second assignment, design and implement a mediation of space
through sound and image. This is necessarily active within the physical domain,
so while it may make use of a software program (although it does not have to), it
must ultimately have a significant, mediated impact on the physical world. This
could be a responsive light and sound environment, a curated collection of
sound and visual objects, etc. as long as the created project interacts with
physical space via sound and image.

Assignment 4 – Final Project – 25% of final grade – Due July 6th at 9PM

Spend the first day or two of the last week of class (2 to 4 hours in class)
designing a final project for this class. Build on one of the projects completed
earlier in the course or create a new, substantial (but achievable within the short
amount of time available), responsibly sourced* project.

Incorporate your own personal interests within the space of sound and video.
Integrating your other areas of interest (e.g. your major, skills, hobbies, interests,
social engagement, etc.) is encouraged but not necessary.

Making the project in collaboration with other classmates is also possible, but
will be graded accordingly (i.e. if 3 people work on a final project, the amount of
work and quality of work should be comparable to that of 3 people).

As an accompaniment to the final project, write an approximately 1000 word

essay that contextualizes your work, describing how it functions within the
historical and analytical topics discussed in class, engaging with specific artists,
artistic movements, and citing influential artistic examples, and also putting into
text your relationship to the work and what challenges the project posed and
how you overcame them.

Grading will be based on project design, implementation, how well the text
follows the guidelines and matches the project, and overall engagement with the
topics presented in the course.

This project will be presented in a public showing of student works, documented,

and made publicly accessible online (if desired).

*Responsible sourcing means only using media whose copyright allows for
reuse and giving proper credit to all materials used.

11
 Reading Citations

Aufderheide, Pat. "Music Videos: The Look of The Sound." Journal of

Communication 36.1 (1986): 57-78.

Barrett, Natasha. "Trends in Electroacoustic Music." The Cambridge Companion

to Electronic Music (2007): 232-255.

Bishop, Claire. Installation Art. Tate, 2005.

Cascone, Kim. "The Aesthetics of Failure: “Post-digital” Tendencies in

Contemporary Computer Music." Computer Music Journal 24.4 (2000):

12-18.

Cavalier, Stephen. The World History of Animation. University of California Press,

2011.

Chion, Michel, and Walter Murch. Audio-Vision: Sound on Screen. Columbia

University Press, 1994.

Collins, Nick. Introduction to Computer Music. Chichester, West Sussex, Wiley-

Blackwell, 2010.

Cook, David A., and Robert Sklar. “History of the Motion Picture.” Encyclopædia

Britannica, Encyclopædia Britannica, inc., 11 Mar. 2016,

www.britannica.com/art/history-of-the-motion-picture. Accessed 15 Mar.

2017.

Cook, Nicholas. Analysing Musical Multimedia. Oxford, Oxford University Press,

2004.

12
Cox, Christoph, and Daniel Warner. Audio Culture: Readings in Modern Music.

Continuum, 2005.

Gould, Richard, et al. Interviews from Designing Sound, designingsound.org/.

Accessed 15 Mar. 2017.

Greie-Ripatti, Antye. “herstory of electronic music.” NERDGIRLS Mash by

poemproducer AGF 8 March 2014 - for equality, diversity and world

peace, 8 Mar. 2014, nerdgirls.poemproducer.com/. Accessed 15 Mar.

2017.

Holmes, Thom. Electronic and Experimental Music: Technology, Music, and

Culture. New York, NY, Routledge, 2012.

Horsfield, Kate. "Busting the Tube: A Brief History of Video Art." Feedback: The

Video Data Bank Catalogue of Video Art an Artist Interviews (2006): 7-17.

Jensenius, Alexander Refsum, and Michael J. Lyons, eds. A NIME Reader:

Fifteen Years of New Interfaces for Musical Expression. Vol. 3. Springer,

2017.

Kelly, Caleb. Cracked Media: The Sound of Malfunction. Cambridge, MA, MIT

Press, 2009.

Maloney, Kathleen. "Sounding Images and Imaging Sounds-Audiovisual

Interactivity in Performance." Sight Lines Thesis Projects, Graduate

Program in Visual critisism, CCA (2005).

Miranda, E.R, and M. Wanderley. New Digital Musical Instruments: Control and

Interaction Beyond the Keyboard. Middleton, WI, A-R Editions, 2006.

13
Napoli, Maria Donata. "Short Review of the Relationship Between Videomusic

and Art." Journal of Science and Technology of the Arts 4.1 (2012): 33-40.

Reas, Casey, and Ben Fry. Getting Started with Processing: A Hands-On

Introduction to Making Interactive Graphics. Maker Media, Inc., 2015.

Reich, Steve. "Music as a Gradual Process." Writings on Music, 1965-

2000 (1968): 34-36.

Sherman, Tom. "The Nine Lives of Video Art: Technological Evolution, the

Repeated Near-Death of Video Art, and the Life Force of Vernacular

Video…." Responses to YouTube Video Vortex, Amsterdam 18 (2008).

Suderburg, Erika, ed. Space, Site, Intervention: Situating Installation Art. U of

Minnesota Press, 2000.

Thompson, Kristin, and David Bordwell. Film History: An Introduction. New York

(Etc.), McGraw Hill, 2002.

Willis, Holly. New Digital Cinema: Reinventing the Moving Image. Vol. 25.

Wallflower Press, 2005.

Wishart, Trevor, and Simon Emmerson. On Sonic Art. Harwood Academic, 1996.

14