A VALUE ADDED TELEPHONE SET .

y CHUNG BOON KUAN 1011123456 *ession +,/,0+,//

The project report is prepared for Faculty of Engineering Multimedia University in partial fulfilment for Bachelor of Engineering (Hons) Electronics majoring in Telecommunications

FA U!T" #F E$%&$EE'&$% MU!T&ME(&A U$&)E'*&T" April +,,-

The copyright of this report .elongs to the author under the terms of the opyright Act /123 as 4ualified .y 'egulation 5(/) of the Multimedia University &ntellectual 6roperty 'egulations7 (ue ac8no9ledgement shall al9ays .e made of the use of any material contained in: or derived from: this report7

ii

DECLARATION

& here.y declare that this 9or8 has .een done .y myself and no portion of the 9or8 contained in this report has .een su.mitted in support of any application for any other degree or 4ualification of this or any other university or institute of learning7 & also declare that pursuant to the provisions of the opyright Act /123: & have not engaged in any unauthorised act of copying or reproducing or attempt to copy 0 reproduce or cause to copy 0 reproduce or permit the copying 0 reproducing or the sharing and 0 or do9nloading of any copyrighted material or an attempt to do so 9hether .y use of the University;s facilities or outside net9or8s 0 facilities 9hether in hard copy or soft copy format: of any material protected under the provisions of sections < and 3 of the Act 9hether for payment or other9ise save as specifically provided for therein7 This shall include .ut not .e limited to any lecture notes: course pac8s: thesis: te=t .oo8s: e=am 4uestions: any 9or8s of authorship fi=ed in any tangi.le medium of e=pression 9hether provided .y the University or other9ise7 & here.y further declare that in the event of any infringement of the provisions of the Act 9hether 8no9ingly or un8no9ingly the University shall not .e lia.le for the same in any manner 9hatsoever and underta8es to indemnify and 8eep indemnified the University against all such claims and actions7

*ignature> ???????????????????????? $ame> *tudent &(> (ate>

iii

ACKNOWLEDGEMENT & 9ould li8e to than8 my parents: si.ling and cats for the support7

iv

ABSTRACT &n the near future: a ne9 telephone set 9ill .e developed7 The phone 9ill have <( capa.ilities of immersive communication7

TABLE OF CONTENTS

LIST OF ABBREVIATIONS................................................................................v LIST OF MATHEMATICAL S!MBOLS.............................................................." CHAPTER 1#INTRODUCTION..............................................................................1 CHAPTER 1#LITERATURE REVIEW..................................................................3 CHAPTER 2#DETAILED DESIGN........................................................................4 CHAPTER 3#DATA PRESENTATION .................................................................$ CHAPTER 4#DISCUSSIONS...................................................................................% CHAPTER 5#CONCLUSIONS ...............................................................................& REFERENCES.........................................................................................................11 APPENDI' A# UMTS SIMULATION .................................................................12

vi

LIST OF FIGURES F ()*+ 3.1# D+,-. /0-0 ,0*- - 12+/...........................................................................5 F ()*+ 3.2# B033112.....................................................................................................5

vii

LIST OF TABLES

LIST OF ABBREVIATIONS

+( <( <% <(T) A&' B@frame

T9o dimensions Three dimensions <rd %enerations <( Television Adaptive &ntra 'efresh Bidirectional predicted frame onvolutional ode

LIST OF MATHEMATICAL S!MBOLS

D Ec(n) Ecq(n) Ee(n) Eeq(n) F(n-1) Fc(n) Fc(n) fd(i,j) kfar m

)ie9ing distance 'esidual error for current frame oded residual error *ide information residual error oded side information residual error 6revious reconstructed odd frame urrent frame 6redicted current frame (o9n@sampled image 'ange of depth infront camera )alue of depth map

CHAPTER 1#INTRODUCTION

1.1

P*+0453+

ommunication using three dimensional (<() video is an enhanced representation method for current t9o dimensional (+() video7 <( video differs from +( in the sense that it also accounts for depth information7 &t 9ill allo9 users to feel the presence of the persons they are communicating 9ith or .e truly immersed in the event they are 9atching7

1.2

M1- v0- 12

This project is motivated .y applications that re4uire <( video: for e=ample in entertainment (such as movies and video games): training (such as military0non@ military pilot and car driving training): in amusement par8 (such as virtual roller coaster) and communication (such as <( video conferencing and mo.ile <( television)7 'esearch on <(@T) and <( audio and video communication such as in A/B@A5B: is .eing conducted to provide <( television contents to the users and to e=plore the possi.ilities of <( video communication7

1.3

A,,3 60- 12 02/ S6+20* 17 can .e .eneficial includes <(

E=ample of scenarios 9here <( video and M( colla.oration <( video conferencing A-B7 1.3.1 3DTV B*10/607-

television (<(T)) .roadcast: mo.ile <(T): <( video streaming and virtual

Cith <( content: soon consumers 9ill .e a.le to 9atch <(T) in the comfort of their home7 The <(T) may .e ena.led using (igital )ideo Broadcast (()B) technologies that include digital television signal .roadcast over satellite (()B@*: ()B@*+ and ()B@*H): ca.le (()B@ ) and terrestrial (()B@T)7

1.4

O58+6- v+7 02/ Ov+*033 P*18+6- D+76* ,- 12

The research in this thesis involves <( visual data compression for transmission over error prone net9or8s7 &ssues such as resilience and scala.ility have to .e ta8en into account to mitigate the effects of transmission errors and information loss on the compressed <( video stream7

1.5

S-*)6-)*+ 19 T.+7 7

C.0,-+* 2 presents .ac8ground information on <( video: specifically the stereoscopic video in terms of its production: source material: 4uality assessment and coding7 ompression of the +( video and depth information (and later reduced resolution depth information) using M6E%5@MA chapter7 &n C.0,-+* 3: the compressed <( video (+( plus depth video) are intended for transmission over communication net9or8s7 Error resilience tools are needed to suppress the effect of channel errors on the compressed <( video7 C.0,-+* 4 proposes a novel multiple description coding for <( video .ased on even and odd frames M( developed in hapter <7 &n C.0,-+* 5: the com.ination of scala.le coding and multiple description coding (scala.le M( ) is proposed to improve error ro.ustness: and at the same time provides adapta.ility to .and9idth variations and receiving device characteristics7 The last hapter presents the overall conclusion of the thesis leading to some is also investigated in this

suggestions for future 9or87

CHAPTER 1#LITERATURE REVIEW

1.6

I2-*1/)6- 12

This chapter presents .ac8ground information on <( video: specifically the stereoscopic video in terms of its production: source material: 4uality assessment and coding7 A classification of <( video is presented in *ection +7+7 *tereoscopic video generated from +( video and depth information is descri.ed in *ection +7<7 &t also descri.es 9ays to vie9 stereoscopic video7 %eneration of stereoscopic video from +( video and depth (ADB@A1B) is descri.ed in *ection +75 and e=amples of +( video and depth image se4uences are given in *ection +7-7 The 4uality evaluation of <( video is discussed in *ection +7D7 The coding of stereoscopic <( video using the availa.le video coding standard is presented in *ection +737 #ur contri.ution in term of compression of depth information at reduced resolution is investigated in *ection +727 Furthermore: our contri.ution in term of compression of +( video and depth using M6E%5@MA is e=plored in *ection +727 *ection +71 concludes the chapter and suggests the 9ay for9ard7

<

CHAPTER 2#DETAILED DESIGN

1.$

I2-*1/)6- 12

Chen the compressed <( video is transmitted over communication net9or8s: the associated pac8et loss can lead to poor visual 4uality A/,B7 Error control schemes: 8no9n as error resilience techni4ues are applied at the encoder to ma8e the compressed video more resilience to channel errors7 Many error resilience methods for +( video are availa.le in the literature7 &n *ection /72: the error resilience tools included .y the M6E%@5 standard for +( video are .riefly descri.ed7 The error resilience tools are then e=tended to the depth data in <( video7

1.%

E**1* R+7 3 +26+ T1137

Errors are introduced in the .it stream 9hen the compressed video data is transmitted over noisy communication channels7 M6E%@5 has adopted the follo9ing error resilience tools to provide .asic error ro.ustness: namely pac8et resynchronisation: data partitioning and reversi.le varia.le@length@codes (')! ) (A//B and A/+B)7 1.%.1 P06:+- R+7;26.*12 70- 12

A video decoder that is decoding an erroneous .it stream 9ill loose synchronisation 9ith the encoder if errors are encountered causing the 4uality of the decoded video degrades rapidly7 #ne remedial action is for the encoder to insert resynchronisation mar8ers in the .it stream at various locations7 Chen an error is detected: the decoders can then search for the ne=t resynchronisation mar8er and regain resynchronisation7 The M6E%@5 encoder has the option of dividing the image into video pac8ets: each made up of an integer num.er of consecutive macro@.loc8s7 At the .eginning of each video pac8et: the M6E%@5 encoder inserts a resynchronisation mar8er7 The siEe of the video pac8et can .e determined .y the user in terms of F .its7 This ensures that the decoder can effectively localise the error depending on the

content of the images7 &n addition to inserting the resynchronisation mar8ers at the .eginning of each video pac8et: the M6E%@5 encoder removes all data dependencies that e=ist .et9een the data .elonging to t9o different video pac8ets 9ithin the same image7 This is re4uired .ecause if one pac8et is in error: the other pac8et can still .e decoded .y the decoder7 This is sho9n in Figure < 7/7

'esynchronisation code

Motion0( Mar8er

( Mar8er

Administrative .its G *hape0Motion data

Te=ture data

(epth data

F ()*+ 3.1# D+,-. /0-0 ,0*- - 12+/

Frame interpolation is performed using the e4uation .elo9>

I ip (i: j ) = I prev (i: j ) + I +
E4uation <7/
fut

(i: j )

9here Iip(i,j) is the frame to .e interpolated at pi=el location (i,j): Iprev(i,j) is the previous frame and Ifut(i,j) is the future frame7 This average frame interpolation is used in the simulation 9hen there are errors in a frame7 Motion compensated frame interpolation can also .e used to o.tain improved performance .ut at the e=pense of decoder comple=ity7 Figure < 7+ sho9s a .alloon7

F ()*+ 3.2# B033112

&n this thesis: pac8et error trace from M * mode , as sho9n in is used to corrupt the compressed +( and depth video se4uences7 For *( : the +( and depth video se4uences are compressed using the original M6E%5 version + reference soft9are A/<B7 For M( @E#: modified M6E%5 version + reference soft9are is used to compress the se4uences7 At the reference soft9are decoder: pac8ets are dropped according to the pac8et error trace generated from the Cireless !A$ channel simulator A/5B7

T053+ 3.1# S;7-+4 7,+6 9 60- 12 19 IEEE %02.11

S;7-+4 P0*04+-+* arrier Modulation hannel oding Modulation

D+76* ,- 12 #F(M 6unctured onvolutional oding B6*F: H6*F: /DHAM: D5HAM

T053+ 3.2# M1/)30- 12 02/ C1/ 2( S6.+4+ 0 <MCS 0=

P0*04+-+* Modulation 6hysical layer .it rate oded .its per #F(M sym.ol (ata .its per #F(M sym.ol

V03)+ B6*F D M.ps 52 +5

CHAPTER 3#DATA PRESENTATION

1.&

I2-*1/)6- 12

This chapter proposes a novel multiple description coding techni4ue for <( video .ased on the even and odd frames M( developed in hapter <7 The even and odd frames .ased M( is improved .y adding varia.le redundancy in the form of side information (M( @E#*)7 The redundant side information consists of the difference .et9een the interpolated frame and the locally reconstructed frame that can .e 4uantised: hence: the redundancies can .e controlled .y the 4uantisation parameter7 This method is descri.ed in *ection 57+7 The method is improved in *ection 57< .y including the side information in the central prediction and using the concept of multiple predictions (M( @E#*6)7 The side information is found to .e 4uite large: 9hich affects the coding efficiency of the M( coder7 Hence: in *ection 575: an attempt is made to reduce the side information using B@frame interpolation (M( @E#*B)7 The simulation results for M( @E#*: M( @E#*6 and M( @E#*B are presented and discussed in *ection 57-7 The side information is also reduced .y ma8ing it adaptive according to the motion information and is applied to M( @E#* and M( @E#*6 in *ection 57D7 Cith the adaptive side information: M( @E#* is renamed to M( @E#A* and M( @E#*6 is renamed to M( @E#A*67 The simulation results for M( @E#A* and M( @E#A*6 are also presented and discussed in *ection 57D7 *ection 573 concludes this chapter7

CHAPTER 4#DISCUSSIONS

1.10

I2-*1/)6- 12

Modern video transmission and storage systems are typically characterised .y a 9ide range of access net9or8 connection 4ualities and receiving devices7 E=amples of applications include video conferencing in a virtual colla.oration system scenario: 9here a large: fi=ed@terminal acts as the main control point and serves a group of remote users7 The remotely located users possess small: fi=ed0mo.ile terminals connected via an error prone net9or87

CHAPTER 5#CONCLUSIONS

1.11

Ov+*v +>

This thesis involves <( visual data compression for transmission over error prone net9or8s7 &ssues such as resilience and scala.ility have .een ta8en into account to mitigate the effects of transmission errors and information loss on the compressed <( video stream7 The 9or8s in this thesis can .e concluded in three main areas as follo9s> <( video compression <( video error resilience <( video scala.ility

Each 9ill .e e=plain in the follo9ing three sections7

1.12

S)440*; 19 -.+ W1*: 12 3D V /+1 C14,*+77 12

#ne of the o.jectives of the research is to compress the stereoscopic <( video data for transmission purposes7 A format of stereoscopic video: namely +( video plus depth has .een chosen to represent the stereoscopic video7 Using the depth .ased image rendering techni4ue: a left and right image se4uence from the +( image and its associated depth information is produced 9hich can .e rendered to o.tain a stereoscopic <( video7 ompression of this type of data can .e accomplished .y using an availa.le video coding standard such as M6E%@57 Cith regard to the depth compression: do9n@sampling prior to encoding and up@sampling of the depth data after decoding is proposed7 The use of do9n@sampling prior to encoding and up@ sampling after decoding introduces up@sampling distortions .eside the 4uantisation errors7 Ho9ever: the simulation results on high resolution image se4uence demonstrated that if the resolution of the depth image se4uence is reduced prior to encoding and up@sampled .ac8 to its original resolution after decoding: far .etter

o.jective and su.jective 4uality could .e achieved compared to compressing using the original resolution of the depth image at lo9 .it rates7

1.13

O* ( 203 A6. +v+4+2-7

The 9or8 presented in this thesis involved e=tensive studies on the video coding standard: multiple description video coding and scala.le multiple description video coding for stereoscopic <( video application

1.14

A*+07 19 F)-)*+ R+7+0*6.

This section descri.es some of the issues: 9hich remain to .e tac8led in the provision of stereoscopic <( video 9ireless communication7 1.14.1 3D V /+1 C1/ 2( &n hapter +: a .asic depth image .ase rendering techni4ue is implemented to

convert the +( plus depth image se4uence to stereoscopic video7 Further research can .e carried out to improve the implemented depth image .ase rendering techni4ue ta8ing into consideration the occlusion pro.lems7

/,

REFERENCES

A/B

7 Fehn: E7 oo8e: #7 *chreer: and 67 Fauff: I<d analysis and image@.ased rendering for immersive tv applications:J Signal Pr c!"Image # mmunicati n: vol7 /3: pp7 3,-@ 3/-: +,,+7

A+B <( amera K <( )ideo *olutions L <() *ystems: http>009997<dvsystems7com: !ast visited> - Mune +,,27 A<B 7 Fehn: F7 *chuur: 67 Fauff: and A7 *molic: I oding results for EE5 in M6E% <(A):J IS$%IE# &'#1%S#()%*+11 ,PE+-(%,)./1: 6attaya: March +,,<7

A5B <(T) $oE: https>009997<dtv@research7org0: !ast visited> - Mune +,,27 A-B A7 *ullivan: I<@deep:J IEEE Spectrum: vol7 5+: no7 5: pp7 ++@+3: April +,,-7 ADB ((( L Bringing <( to Mainstream: https>009997ddd7com0: !ast visited> - Mune +,,27 A3B C7 Fnight: IThe factory of dreams:J IE' ,aga0ine: vol7 <: no7 2: pp7 /D@/1: /,@+< May +,,27 A2B A7 *molic and H7 Fimata: I'eport on <(A) e=ploration: 1 IS$%IE# &'#1%S#()%*+11 2.343: $or9ay: Muly +,,<7 A1B C7 &jsselsteijn: 67 M7 H7 *euntiens: and !7 M7 M7 Meesters: I*tate@of@the@art in human factors and 4uality issues of stereoscopic .roadcast television:J 5''ES' Delivera6le /: August +,,+7 A/,B A//B A/+B %7 M7 &ddan and %7 "ahav: I<d imaging in the studio (and else9here777):J in Pr ceeding7 f SPIE: vol7 5+12: pp 52@--: Manuary +,,57 7 Fehn: IA <d@tv approach using depth@image@.ased rendering (di.r):J in Pr ceeding7 f 8IIP(--9: pp7 52+@523: *pain: *eptem.er +,,<7 A77 'edert: M7 #7 (7 Beec8: 7 Fehn: C7 &jsselsteijn: M7 6ollefeys: !7 )7 %ool: E7 #fe8: &7 *e=ton: and 67 *urman: IAttest> Advanced three@dimensional television system technologies:J in Pr ceeding7 f Int! S:mp! n 9D Data Pr ce77ing 8i7uali0ati n and 'ran7mi77i n, 9DP8'-(: pp7 </<@</1: &taly: Mune +,,+7 A/<B 67 *chel8ens: A7 %avrilescu: A7 Munteanu: "7 (;Hondt: F7 )ermeirsch: '7 )7 d7 Calle: #7 %uye: *7 Corrall and A7 $avarro: IError@resilient transmission of H7+D5 *) streams over ()B@T0H and C&MAN channels 9ith multiple description coding techni4ues:J in Pr ceeding7 f 1.t; Eur pean Signal Pr ce77ing # nference (--4 (E<SIP#$ (--4): pp7 /11-@/111: 6oEnan: 6oland: <@3 *eptem.er +,,37 A/5B A7 H7 *ad8a: I ompressed video communications:J Mohn Ciley: +,,+: pp7 /+<@/<-7

//

APPENDI' A# UMTS SIMULATION

The UMT* error patterns used in this thesis are o.tained from the simulated UMT* channel7 The UMT* channel model simulates the UMT* air interface7

/+