Investigate of 3D realistic broadcast system

Outline
Introduction Video processing Transmission Rendering What are the most technical challenge? Conclusion References

I. Introduction
Trends on broadcasting technology

Realistic broadcast systems

Requirements for realistic

Requirements for realistic

Multi-modal information - 3D video with depth information at multiple viewpionts. - Multi-channel audio. - CG model. - Haptic Data. High quality data. User friendly interaction.

Main components

II. Video processing

Multi-view video

Multi-view camera system

Multi-view codec system

Coding of multi-view video

Multi video coding

Cu trc d on
D on gia cc min khng gian

Php d on da trn H264/VAC

3D video system

Depth information

Depth camera system

Hybird camera system

Image of hybird camera

Problems of depth map

High quality depth map generation

III. Transmission

Cu trc ca b m ha H264/AVC

NAL
Gip d dng nh x d liu H264/AVC ti tng vn chuyn nh:
RTP/IP cho bt k dch v internet thi gian thc. nh dng file thng dng lu tr, truyn dn. H.23x cho dch v m thoi. H thng MPEG-2 cho dch v qung b.

D liu video c m ha c sp xp vo NAL units.1 NAL unit gm s nguyn ln byte.

1 h thng dch v xut

3D contents generation

IV. Multiview Video Rendering

Belongs to the broad research field of imagebased rendering. Multi-stereoscopic video with depth maps:
A number of video sequences captured from diferent view points. A single depth map is available to facilitate the virtual view rendering.

Multi-stereoscopic video with depth map

An example multiview data set.

Rendering and weight map generation

The rendering process from multiview video

Rendering Process
Split the to be rendered view into light rays. For each light ray, trace the light ray to the surface of the depth map, obtain the intersection, and reprojected the intersection into nearby cameras (Cam 3 & Cam4). The intensity of the light ray is thus the weighted average of the projected light rays in Cam 3 and Cam 4. The weight is the angular difference between the light ray to be rendered and the light ray being projected.

Rendering Process (Cont)

When the virtual view point moves away from Cam4 (where the depth map is given), the will be occlusions and holes when computing the light ray/geometry intersection.

Rendering Process (Cont)

Convert the given depth map into 3D mesh surface, where each vertex corresponds to one pixel in the depth map. The mesh surface is then projected to the capturing cameras to compute any potential occlusions in the captured images. Finally, the mesh is projected to the virtual rendering point with multi-texture blending.

Rendering Process (Cont)

For each vertex being rendered, it is projected to the nearby captured images to locate the corresponding texture coordinate. This process takes into consideration the occlusion computed earlier: if a vertex is occluded in a nearby view, its weight for that camera will be set to zero.

The weight maps

The weight maps generated by the redering process

The weight maps (Cont)

The brighter pixels are the ones with larger weights. During compression of the multiview video, the pixels with high weights shall be encoded with high quality, while the rest pixels can be encoded with low quality Quantization Parameter map: portion of the image that have low weights (and thus high QP) are much more coarsely quantized

Quantization Parameter Maps

QP values for weight maps

Rendering Process (Cont)

A segment of enceded images from cameras 2, 3 and 4

V. Technically Challenging
Depth/Disparity Estimation
Sub-pixel accuracy Temporal enhancement to reduce flickering effects Depth map refinement for distorted depth map

Coding of Multiview Video + Depth Map

Coding structure Depth map coding scheme Bit allocation for depth map coding

Intermediate View Synthesis

View synthesis method for depth map distortion Filtering along object boundaries