Chapter1 IntroductiontoMultimedia I t d ti t M lti di

1.1WhatisMultimedia? 1.2MultimediaandHypermedia 1 2 Multimedia and Hypermedia 1.3WorldWideWeb 1.4OverviewofMultimediaSoftwareTools 1.5FurtherExploration 1 5 F th E l ti

1

1.1WhatisMultimedia?
Whendifferentpeople mentionthetermmultimedia,they oftenhavequitedifferent,orevenopposing,viewpoints. often have quite different, or even opposing, viewpoints.
APCvendor:aPCthathassoundcapability,aDVDROMdrive,and perhapsthesuperiorityofmultimediaenabledmicroprocessors thatunderstandadditionalmultimediainstructions. that understand additional multimedia instructions Aconsumerentertainmentvendor:interactivecableTVwith hundredsofdigitalchannelsavailable,oracableTVlikeservice deliveredoverahigh speedInternetconnection. delivered over a highspeed Internet connection AComputerScience(CS)student:applicationsthatusemultiple modalities,includingtext,images,drawings(graphics),animation, video,soundincludingspeech,andinteractivity. video sound including speech and interactivity

MultimediaandComputerScience: p
Graphics,HCI,visualization,computervision,datacompression, graphtheory,networking,databasesystems.

ComponentsofMultimedia p
Multimediainvolvesmultiplemodalitiesoftext, audio,images,drawings,animation,andvideo. di i d i i ti d id Examplesofhowthesemodalitiesareputto use: use:
Videoteleconferencing. Distributedlecturesforhighereducation. Di t ib t d l t f hi h d ti Telemedicine. Cooperativeworkenvironments. Searching in(very)largevideoandimagedatabases fortargetvisualobjects. f t t i l bj t "Augmented"reality:placingrealappearing computergraphicsandvideoobjectsintoscenes. computer graphics and video objects into scenes
3

Includingaudiocuesforwhere videoconference participantsarelocated. participants are located Buildingsearchablefeaturesintonewvideo,and enablingveryhightoverylowbitrateuseofnew, enabling very highto very lowbitrate use of new scalablemultimediaproducts. Makingmultimediacomponentseditable. Making multimedia components editable Building"inverseHollywood"applications thatcan recreatetheprocessbywhichavideowasmade. recreate the process by which a video was made.
Videounderstandinghasalsobeencalledaninverse Hollywoodproblem.

Usingvoicerecognition tobuildaninteractive environment,sayakitchenwallwebbrowser.

4

MultimediaResearchTopicsandProjects p j
Tothecomputerscienceresearcher,multimedia To the computer science researcher, multimedia consistsofawidevarietyoftopics:
Multimediaprocessingandcoding:multimediacontent analysis,contentbasedmultimediaretrieval,multimedia l i t tb d lti di ti l lti di security,audio/image/videoprocessing,compression,etc. Multimediasystemsupportandnetworking:network y pp g protocols,Internet, operatingsystems,serversandclients, qualityofservice(QoS),anddatabases. Multimedia tools endsystems and applications: Multimediatools,end systemsandapplications: hypermediasystems,userinterfaces,authoringsystems. Multimodalinteractionandintegration:"ubiquity" webeverywheredevices,multimediaeducationincluding b h d i lti di d ti i l di ComputerSupportedCollaborativeLearning,anddesign andapplicationsofvirtualenvironments.
5

CurrentMultimediaProjects j
Manyexcitingresearchprojectsarecurrently underway.Hereareafewofthem: d H f f th
1. Camerabasedobjecttrackingtechnology:trackingof thecontrolobjectsprovidesusercontroloftheprocess. th t l bj t id t l f th 2. 3Dmotioncapture:usedformultipleactorcaptureso thatmultiplerealactorsinavirtualstudiocanbeused that multiple real actors in a virtual studio can be used toautomaticallyproducerealisticanimatedmodels with naturalmovement. 3. Multipleviews:allowingphotorealistic(videoquality) synthesisofvirtualactorsfromseveralcamerasorfrom asinglecameraunderdiering li h i i l d di i lighting. 4. 3Dcapturetechnology:allowsynthesisofhighly realisticfacialanimationfromspeech. realistic facial animation from speech
6

1.2MultimediaandHypermedia yp
5. Specic multimediaapplications:aimedat handicappedpersonswithlowvisioncapability handicapped persons with low vision capability andtheelderly aricheld ofendeavor. 6. Digitalfashion:aimstodevelopsmartclothingthat 6. Digital fashion: aims to develop smart clothing that cancommunicatewithothersuchenhanced clothingusingwirelesscommunication,soasto articially enhance human interaction in a social enhancehumaninteractioninasocial setting. 7. ElectronicHousecall system:aninitiativefor 7 Electronic Housecall system: an initiative for providinginteractivehealthmonitoringservicesto patientsintheirhomes 8. AugmentedInteractionapplications:usedto developinterfacesbetweenrealandvirtual humansfortaskssuchasaugmentedstorytelling. humans for tasks such as augmented storytelling
7

HistoryofMultimedia:
1. Newspaper:perhapstherst masscommunication medium,usestext,graphics,andimages. 2. Motionpictures:conceivedofin1830'sinorderto observemotiontoorapidforperceptionbythe humaneye. 3. Wirelessradiotransmission:GuglielmoMarconi,at Pontecchio,Italy,in1895. 4. Television:thenewmediumforthe20thcentury, establishedvideoasacommonlyavailablemedium andhassincechangedtheworldofmass communications. communications
8

5. Theconnectionbetweencomputersandideasaboutmultimediacovers whatisactuallyonlyashortperiod: what is actually only a short period:

ComponentsofMultimedia MultimediaResearchTopicsandProjects CurrentMultimediaProjects

1985 NegroponteandWiesner cofoundedtheMITMediaLab. 1989 TimBernersLeeproposedtheWorldWideWeb 1990 KristinaHooperWoolseyheadedtheAppleMultimediaLab. 1991 MPEG1 1991 MPEG 1 wasapprovedasaninternationalstandardfordigital d i i l d d f di i l video ledtothenewerstandards,MPEG2,MPEG4,and furtherMPEGsinthe1990s. 1991 TheintroductionofPDAs in1991begananewperiodinthe h d f b d h useofcomputersinmultimedia. 1992 JPEG wasacceptedastheinternationalstandardfordigital imagecompression ledtothenewJPEG2000standard. 1992 Therst MBone audiomulticastontheNetwasmade. 1993 TheUniversityofIllinoisNationalCenterforSupercomputing 1993 The University of Illinois National Center for Supercomputing ApplicationsproducedNCSAMosaic therst fulledged browser.

1945 Vannevar Bushwrotealandmarkarticledescribingwhatamountsto ahypermediasystemcalledMemex.

LinktofullV.Bush1945Memex article,"AsWeMayThink"

1960 TedNelsoncoinedthetermhypertext. 1967 NicholasNegroponteformedtheArchitectureMachineGroup. g p p 1968 DouglasEngelbart demonstratedtheOnLineSystem(NLS),another veryearlyhypertextprogram. 1969 NelsonandvanDamatBrownUniversitycreatedanearlyhypertext 1969 Nelson and van Dam at Brown University created an early hypertext editorcalledFRESS. 1976 TheMITArchitectureMachineGroupproposedaprojectentitled MultipleMedia resultedintheAspenMovieMap,therst p p p, hypermediavideodisk,in1978.
9

10

HypermediaandMultimedia yp
1994 JimClarkandMarcAndreessencreatedthe Netscapeprogram. Netscape program 1995 TheJAVA languagewascreatedforplatform independentapplicationdevelopment. p pp p 1996 DVDvideowasintroduced;highqualityfulllength moviesweredistributedonasingledisk. 1998 XML 1.0wasannouncedasaW3C Recommendation. 1998 HandheldMP3devicesrst madeinroadsinto consumeristtastesinthefallof1998,withthe introductionofdevicesholding32MBofash introduction of devices holding 32MB of ash memory. 2000 WWWsizewasestimatedatover1billionpages.
11

Ahypertextsystem:meanttobereadnonlinearly, byfollowinglinks thatpointtootherpartsofthe b f ll i li k th t i t t th t f th document,ortootherdocuments(Fig.1.1) HyperMedia:notconstrainedtobetextbased, can includeothermedia,e.g.,graphics,images,and especiallythecontinuousmedia soundandvideo. especially the continuous media sound and video
TheWorldWideWeb(WWW) thebestexampleofa hypermediaapplication. hypermedia application

Multimediameansthatcomputerinformationcan berepresentedthroughaudio,graphics,images, be represented through audio graphics images video,andanimationinadditiontotraditional ed a media.

12

Examplesoftypicalpresentmultimediaapplications include: include:

13

Digitalvideoeditingandproduction systems. Electronicnewspapers/magazines. El t i / i WorldWideWeb. Onlinereferenceworks:e.g.encyclopedias,games,etc. O li f k l di Homeshopping. InteractiveTV. Multimediacourseware. Videoconferencing. Videoondemand. Interactivemovies.
14

1.3WorldWideWeb
TheW3Chaslistedthefollowinggoalsforthe WWW: WWW
1. Universalaccessofwebresources(byeveryone everywhere). h ) 2. Eectiveness ofnavigatingavailableinformation. 3. Responsibleuseofpostedmaterial.

1990 TimBernersLeeinventedtheHyperText MarkupLanguage(HTML),andtheHyperText M k L (HTML) d th H T t TransferProtocol(HTTP). 1993 NCSAreleasedanalphaversionofMosaic basedontheversionbyMarcAndreessenfor XWindows therst popularbrowser. X Windows the rst popular browser 1994 MarcAndreessenetal.formedMosaic CommunicationsCorporation l t th C i ti C ti laterthe NetscapeCommunicationsCorporation. 1998 TheW3CacceptedXML version1.0 1998 Th W3C d XML i 10 specications asaRecommendation the mainfocusoftheW3CandsupersedesHTML. main focus of the W3C and supersedes HTML
16

HistoryofWWW 1960s CharlesGoldfarbetal.developedthe p g g ( ) GeneralizedMarkupLanguage(GML)forIBM. 1986 TheISOreleasedanal versionofthe StandardGeneralizedMarkupLanguage Standard Generalized Markup Language (SGML).
15

HTTP(HyperTextTransferProtocol) ( yp )
HTTP:aprotocolthatwasoriginallydesignedfor transmittinghypermedia,butcanalsosupportthe transmitting hypermedia but can also support the transmissionofanyle type. HTTPisastateless request/responseprotocol:no HTTP is a stateless request/response protocol: no informationcarriedoverforthenextrequest. Thebasicrequestformat: The basic request format:
MethodURIVersion AdditionalHeaders: Messagebody

Twopopularmethods:GETandPOST. Thebasicresponseformat:
VersionStatusCodeStatusPhrase AdditionalHeaders Additional Headers Messagebody

Twocommonlyseenstatuscodes:
1 200 OK th 1.200OK therequestwasprocessed t d successfully. 2.404NotFound theURIdoesnot exist.
17 18

TheURI(UniformResourceIdentier):anidentier The URI (Uniform Resource Identier): an identier fortheresourceaccessed,e.g.thehostname, y p y p alwaysprecededbythetoken"http://".

HTML(HyperTextMarkupLanguage) ( yp p g g )
HTML:alanguageforpublishingHypermediaonthe WorldWideWeb dened i SGML W ld Wid W b d d usingSGML:
1.HTMLusesASCII,itisportabletoalldierent (possibly binaryincompatible)computerhardware. bi i tibl ) t h d 2.ThecurrentversionofHTMLisversion4.01. 3.ThenextgenerationofHTMLisXHTML a reformulationofHTMLusingXML.

AverysimpleHTMLpageisasfollows:

HTMLusestagstodescribedocumentelements:
<tokenparams> dening astartingpoint, </token> theendingpointoftheelement. g g Someelementshavenoendingtags.
19

Naturally,HTMLhasmorecomplexstructuresand canbemixedinwithotherstandards. b d h h d d
20

XML(ExtensibleMarkupLanguage) ( p g g )
XML:amarkuplanguagefortheWWWinwhich thereismodularityofdata,structureandviewso th i d l it f d t t t d i thatuserorapplicationcanbeabletodene the tags(structure). tags (structure) ExampleofusingXMLtoretrievestockinformation fromadatabaseaccordingtoauserquery: from a database according to a user query:
1. FirstuseaglobalDocumentTypeDenition (DTD) that isalreadydened. is already dened 2. TheserversidescriptwillabidebytheDTDrulesto generateanXMLdocumentaccordingtothequery generate an XML document according to the query usingdatafromyourdatabase. y y ( ) p g 3. FinallysendusertheXMLStyleSheet(XSL)depending onthetypeofdeviceusedtodisplaytheinformation.
21

ThecurrentXMLversionisXML1.0,approved bytheW3CinFeb.1998. y y , g XMLsyntaxlookslikeHTMLsyntax,althoughit ismuchmorestrict:

All tags are in lower case and a tag that has only Alltagsareinlowercase,andatagthathasonly inlinedatahastoterminateitself,i.e.,<token params />. UsesnamespacessothatmultipleDTDsdeclaring dierent elementsbutwithsimilartagnamescan havetheirelementsdistinguished. p DTDscanbeimportedfromURIsaswell.
22

AnexampleofanXMLdocument structure thedenitionforasmall XHTMLdocument: XHTML document:

ThefollowingXMLrelatedspecications are alsostandardized: l t d di d

XMLProtocol:usedtoexchangeXMLinformation betweenprocesses. between processes XMLSchema:amorestructuredandpowerful languagefordening XMLdatatypes(tags). language for dening XML data types (tags) XSL: basicallyCSSforXML. SMIL synchronized Multimedia Integration SMIL:synchronizedMultimediaIntegration Language,pronounced"smile" aparticular applicationofXML(globallypredened DTD)that allowsforspecication ofinteractionamongany mediatypesanduserinput,inatemporally scriptedmanner. scripted manner.
23 24

SMIL (SynchronizedMultimedia IntegrationLanguage) I t ti L )

PurposeofSMIL:itisalsodesirabletobeableto publishmultimediapresentationsusingamarkup publish multimedia presentations using a markup language. Amultimediamarkuplanguageneedstoenable A multimedia markup language needs to enable scheduling andsynchronization ofdierent multimediaelements,anddene theirinteractivity withtheuser. ih h TheW3CestablishedaWorkingGroupin1997to comeupwithspecications f ith i ti foramultimedia lti di synchronizationlanguage
SMIL 2 0 was accepted in August 2001 SMIL2.0wasacceptedinAugust2001.

1. AllSMILelementsaredividedintomodules sets ofXMLelements,attributesandvaluesthatdene f XML l t tt ib t d l th t d oneconceptualfunctionality. 2. Intheinterestofmodularization,notallavailable 2 I th i t t f d l i ti t ll il bl modulesneedtobeincludedforallapplications. 3. LanguageProles:species aparticulargrouping 3 L P l i i l i ofmodules,andparticularmodulesmayhave integrationrequirementsthataprole must integration requirements that a prole must follow.
SMIL2.0hasamainlanguageprole thatincludesalmost SMIL 2.0 has a main language prole that includes almost allSMILmodules.

SMIL2.0isspecied inXMLusingamodularization approachsimilartotheoneusedinxhtml: approach similar to the one used in xhtml:

25

BasicelementsofSMILasshowninthe Basic elements of SMIL as shown in the followingexample:

26

1.4OverviewofMultimediaSoftware Tools T l
Thecategoriesofsoftwaretoolsbriey examinedhereare:
1.MusicSequencingandNotation 1 Music Sequencing and Notation 2.DigitalAudio 3.GraphicsandImageEditing 4.VideoEditing 4 Vid Edi i 5.Animation 6.MultimediaAuthoring
27 28

MusicSequencingandNotation q g
Cakewalk: nowcalledProAudio.
Thetermsequencercomesfromolderdevicesthat storedsequencesofnotes("events",inMIDI). It is also possible to insert WAV les and Windows MCI ItisalsopossibletoinsertWAVles andWindowsMCI commands(foranimationandvideo)intomusictracks (MCIisaubiquitouscomponentoftheWindowsAPI.)

DigitalAudio g
DigitalAudiotoolsdealwithaccessingand editingtheactualsampledsoundsthatmake diti th t l l d d th t k upaudio:
C l Edit CoolEdit:avery powerfulandpopulardigitalaudio f l d l di it l di toolkit;emulatesaprofessionalaudiostudio multitrack productionsandsoundle editing p g includingdigitalsignalprocessingeects. SoundForge:asophisticatedPCbasedprogramfor editingaudioWAVles. di i di WAV l ProTools:ahighendintegratedaudioproduction andeditingenvironment MIDIcreationand and editing environment MIDI creation and manipulation;powerfulaudiomixing,recording, andeditingsoftware.
30

Cubase:anothersequencing/editingprogram,with capabilitiessimilartothoseofCakewalk.Itincludes somedigitalaudioeditingtools. some digital audio editing tools MacromediaSoundedit:matureprogramfor creatingaudioformultimediaprojectsandtheweb creating audio for multimedia projects and the web thatintegrateswellwithotherMacromedia productssuchasFlashandDirector.
29

GraphicsandImageEditing p g g
AdobeIllustrator:apowerfulpublishingtoolfromAdobe. Usesvectorgraphics;graphicscanbeexportedtoWeb. Uses vector graphics; graphics can be exported to Web AdobePhotoshop:thestandardinagraphics,image p processingandmanipulationtool. g p
Allowslayersofimages,graphics,andtextthatcanbe separatelymanipulatedformaximumexibility. Filterfactorypermitscreationofsophisticatedlightingeects lters.

VideoEditing g
AdobePremiere:anintuitive,simplevideoediting toolfornonlinearediting,i.e.,puttingvideoclips tool for nonlinear editing i e putting video clips intoanyorder:
Video and audio are arranged in "tracks". Videoandaudioarearrangedin tracks Providesalargenumberofvideoandaudiotracks, superimpositionsandvirtualclips.=>effective multimediaproductionswithlittleeffort.

MacromediaFireworks:softwareformakinggraphics Macromedia Fireworks: software for making graphics specificallyfortheweb. MacromediaFreehand:atextandwebgraphicsediting g p g toolthatsupportsmanybitmapformatssuchasGIF, PNG,andJPEG.
31

AdobeAfterEffects:apowerfulvideoeditingtool thatenablesuserstoaddandchangeexisting th t bl t dd d h i ti movies.Canaddmanyeects:lighting,shadows, g; y motionblurring;layers. FinalCutPro:avideoeditingtoolbyApple; Macintoshonly.

32

Animation
MultimediaAPIs:
Java3D:APIusedbyJavatoconstructandrender3D graphics,similartothewayinwhichtheJavaMedia graphics, similar to the way in which the Java Media Frameworkisusedforhandlingmediales.
1. Providesabasicsetofobjectprimitives(cube,splines, 1 Provides a basic set of object primitives (cube splines etc.)forbuildingscenes. y p p 2. ItisanabstractionlayerbuiltontopofOpenGLorDirectX (theusercanselectwhich).

RenderingTools:
3DStudioMax:renderingtoolthatincludesanumberof veryhighendprofessionaltoolsforcharacteranimation, g gamedevelopment,andvisualeects production. p , p SoftimageXSI:apowerfulmodeling,animation,and renderingpackageusedforanimationandspecial eects in lms and games inlms andgames. Maya:competingproducttoSoftimage;aswell,itisa completemodelingpackage. complete modeling package. RenderMan:renderingpackagecreatedbyPixar.

DirectX : Windows API that supports video, images, :WindowsAPIthatsupportsvideo,images, audioand3Danimation OpenGL: the highly portable most popular 3 D API OpenGL:thehighlyportable,mostpopular3DAPI.
33

GIFAnimationPackages:asimplerapproachto GIF Animation Packages: a simpler approach to animation,allowsveryquickdevelopmentof eective smallanimationsfortheweb.

34

MultimediaAuthoring g
MacromediaFlash:allowsuserstocreateinteractive moviesbyusingthescoremetaphor,i.e.,atimeline movies by using the score metaphor i e a timeline arrangedinparalleleventsequences. MacromediaDirector:usesamoviemetaphorto Macromedia Director: uses a movie metaphor to createinteractivepresentations verypowerful andincludesabuiltinscriptinglanguage,Lingo,that allowscreationofcomplexinteractivemovies. ll i f l i i i Authorware:amature,wellsupportedauthoring productbasedontheIconic/Flowcontrol metaphor. d t b d th I i /Fl t l t h Quest: similartoAuthorware inmanyways,usesa typeofowcharting metaphor.However,the type of owcharting metaphor However the owchart nodescanencapsulateinformationina moreabstractway(calledframes)thansimply subroutinelevels.
35

1.5FurtherExploration p
LinktoFurtherExplorationforChapter1. InChapter1oftheFurtherExplorationdirectory,the websiteprovideslinkstomuchofthehistoryof website provides links to much of the history of multimedia. Otherlinksinthetextwebsiteincludeinformationon: Other links in the text website include information on:
TedNelsonandtheXanadu project. NicholasNegroponte sworkattheMITMediaLab. Nicholas Negroponte's work at the MIT Media Lab. DouglasEngelbart,andthehistoryofthe"OnLineSystem". TheMITMediaLab Clientsideexecution.

36