Dongsoo Kim (dongsoo45.kim@samsung.

com)
Heungjun Kim(riverful.kim@samsung.com)
Samsung Electronics

Introducing mobile camera devices

- SoC camera subsystem
- Mobile camera module devices

Framework for digital camera

- Video4Linux2 and camera device
- New APIs for high-end mobile camera devices
- Dual camera framework for 3G handset

Porting issues in SoC camera subsystem

AP : Application Processor
Camera interface : interface device which is syncing image data from
external camera device and gives some commands to them
Camera module : external camera device packaged with lens, sensor
and ISP on it
ISP : Image Signal Processor. Also supports lots of additional
functions
V4L2 : Video for Linux 2

This document contains some animations.

Slideshow mode is recommended

Exposure control
- Aperture (Iris)
- Shutter
- ISO

Strobe support
- Various program

mode : slow sync,

red-eye reduction
and on.

Lens control
- Focus
- Zoom

Introducing mobile camera devices

History of camera phone

1.3MP
in 2003
First
Camera
Phone
5MP
3MP
7MPin2004
in 2004
2005
12MP
in
2009
2MP
in
in 10MP
2000
with
3.5MP
2006
7

Old and low-end mobile camera modules

Feature

Exposure

Lens
Strobe

Low end camera

Aperture

Not supported

Shutter

Just change frame rate

ISO

Low sensitivity & High noise level

Focus

Pan focus

Zoom

Digital zoom
Just LED light most of time

Brand new and high-end mobile camera modules

Feature

Exposure

Lens
Strobe

High end camera

Aperture

Supported with lens packaging

Shutter

Supported with lens packaging

ISO

High sensitivity & Low noise level

Focus

Movable focus point (Auto/manual)

Zoom

Optical zoom & Digital zoom

XENON flash is also supported

What is SoC camera subsystem?

ARM SoC solution
Peripherals
Camera interface
Peripheral3 (USB)

ARM
Core

External
Camera
module

Peripheral2 (MMC)
Peripheral1(Display)

10

Request for data

Image data in expected format

Camera interface
Command
interface

Resolution
handling

Data format
handling
(YUV/RAW/JPEG
)

Buffer
handling

I2C
RAW DATA

Camera module
Exposure
control

Lens
control

White
balance
control

Effect
control

Face
detect

Zoom
control

Strobe
control

.
.
.
11

Various ways of preview (Live view) in SoC camera subsystems

DATA

ARM
core

Camera
Module

DATA

Camera
Interface

User
Application
M
e
m
o
r
y

COPY!

Frame buffer

OK

Cancel

12

Various ways of preview (Live view) in SoC camera subsystems

User
Application

DATA

ARM
core

Camera
Module

DATA

Camera
Interface

M
e
m
o
r
y
D
M
A

COPY!
Save
as file

Frame buffer

Through DMA

OK

Cancel

13

How to capture still shots in SoC camera subsystem

- Your camera module has a dedicated JPEG encoder on it

Camera module

ARM
core

DATA

JPEG
PROCESS

MEMORY

DATA

RAW
DATA
PROCESS

DATA

Camera
Interface

User
Application
M
e
m
o
r
y

COPY!

Save as
JPEG file

14

How to capture still shots in SoC camera subsystem

- No JPEG encoder in your camera module
User
Application

DATA

ARM
core

Camera
Module

DATA

Camera
Interface

M
e
m
o
r
y

COPY!
RAW
DATA
Codec S/W

Save as
JPEG file
15

How to take motion pictures in SoC camera subsystem

COPY!

DATA

ARM
core

Camera
Module

DATA

Camera
Interface

M
e
m
o
r
y

Frame buffer

User
Application
OK

Codec
S/W

Cancel

Save
as

COPY!

16

In Kernel
V4L2 (AKA Video For Linux 2)
- The second version of the Video For Linux API
- Kernel interface for analog radio and video capture and output drivers

In user space
Multimedia middleware
Gstreamer
- Pipelinebasedmultimedia frameworkwritten in theC programming languagewith
the type system based onGObject

OpenMAX
- Cross-platform set ofC-language programming interfaces that provides abstractions
for routines especially useful for audio, video, and still images

17

Is V4L2 enough for

mobile digital
camera modules?

No way

18

Framework for digital camera

19

Camera subsystem in Linux kernel aspect

V4L2

videobuf
videobufcore

V4l2dev

videobufsg-dma

V4l2ioctl

videodev
V4l2-device

V4l2-intdevice
(old)

V4l2subdev
(new)

Camera interface
V4L2 driver
Command
interface

Resolution
handling

Data format handling

(YUV/RAW/JPEG)

Buffer
handling

Camera module
V4L2 driver
Exposure
control

Lens
control

White
balance
control

Effect
control

Face
detect

Zoom
control

Strobe
control

.
.
.
20

How does it work with V4L2?

Application or
Middleware

Copy memory to userspace

Open device

VIDIOC_QUERYCAP

VIDIOC_S_INPUT

VIDIOC_QBUF

VIDIOC_REQBUF
VIDIOC_DQBUF

VIDIOC_QBUF

VIDIOC_QUERYBUF

VIDIOC_STREAMON

21

New V4L2 APIs for digital camera

Exposure control

22

Need for detailed control in exposure

- Weve got camera devices with mechanical shutter
- Iris could be handled also
- ISO controlled output is somehow useful

Why not using enhanced exposure control like a

regular digital camera?

23

What weve got in mainline V4L2 right now?

#defineV4L2_CID_EXPOSURE_AUTO (V4L2_CID_CAMERA_CLASS_BASE+1)
enumv4l2_exposure_auto_type{
V4L2_EXPOSURE_AUTO=0,
V4L2_EXPOSURE_MANUAL=1,
V4L2_EXPOSURE_SHUTTER_PRIORITY=2,
V4L2_EXPOSURE_APERTURE_PRIORITY=3
};
#defineV4L2_CID_EXPOSURE_ABSOLUTE (V4L2_CID_CAMERA_CLASS_BASE+2)
#defineV4L2_CID_EXPOSURE_AUTO_PRIORITY (V4L2_CID_CAMERA_CLASS_BASE+3)

No way to control iris

No way to control exposure metering method

24

Additional exposure control

enum

v4l2_exposure_auto_type {
V4L2_EXPOSURE_AUTO = 0,
V4L2_EXPOSURE_MANUAL = 1,
V4L2_EXPOSURE_SHUTTER_PRIORITY = 2,
V4L2_EXPOSURE_APERTURE_PRIORITY = 3,
+
/* Additional features for digital camera */
+
V4L2_EXPOSURE_ISO_PRIORITY =4,
};
#define V4L2_CID_EXPOSURE_ABSOLUTE (V4L2_CID_CAMERA_CLASS_BASE+2)
#define V4L2_CID_EXPOSURE_AUTO_PRIORITY
(V4L2_CID_CAMERA_CLASS_BASE+3)
. . .
+#define V4L2_CID_EXPOSURE_LOCK (V4L2_CID_CAMERA_CLASS_BASE+18)

25

+#define V4L2_CID_CAM_APERTURE(V4L2_CID_CAMERA_CLASS_BASE+19)
+#define V4L2_CID_CAM_SHUTTER (V4L2_CID_CAMERA_CLASS_BASE+20)
+#define V4L2_CID_CAM_ISO
(V4L2_CID_CAMERA_CLASS_BASE+21)

Supported aperture stages, shutter speed, and ISO speed could be different
between every different camera module products

Supported values for aperture, shutter and ISO should be made in

V4L2_CTRL_TYPE_MENU
{
.id=V4L2_CID_CAM_APERTURE,
.type=V4L2_CTRL_TYPE_MENU,
.name="Aperture",
.minimum=0,
.maximum=4,
.step=1,
.default_value=0,
},

staticconstchar*camera_iris_stages[]={
/*This modulesupports5Irisstages
*onit'sown butFnumberdepends
* howyoupackage Lensmodule
*/
"2.8, "4,"5.6, "8, "11,NULL
};
26

New exposure metering control

+/* Exposure Methods */
+#define V4L2_CID_PHOTOMETRY (V4L2_CID_CAMERA_CLASS_BASE+17)
+enum v4l2_photometry_mode {
+
V4L2_PHOTOMETRY_MULTISEG = 0,
+
V4L2_PHOTOMETRY_CWA = 1,
+
V4L2_PHOTOMETRY_SPOT = 2,
+
V4L2_PHOTOMETRY_AFSPOT = 3,
+};

27

New V4L2 APIs for digital camera

Lens control

28

Need for enhanced Focus mode API

- Current V4L2 framework just slightly covers MANUAL &

AUTO FOCUS mode for usb webcam

- High-end camera modules support for various focus mode

presets like

AF Macro
AF-S / AF-C

For compatibility
Focus control API should be defined

29

Focus Mode & Enumeration values

#define V4L2_CID_FOCUS_ABSOLUTE
(V4L2_CID_CAMERA_CLASS_BASE+10)
#define V4L2_CID_FOCUS_RELATIVE
(V4L2_CID_CAMERA_CLASS_BASE+11)
+#define V4L2_CID_FOCUS_MODE (V4L2_CID_CAMERA_CLASS_BASE+13)
+/* Focus Methods */
+enum v4l2_focus_mode {
+
V4L2_FOCUS_AUTO
+
V4L2_FOCUS_MANUAL
+
V4L2_FOCUS_MACRO
+
V4L2_FOCUS_CONTINUOUS
+};

=
=
=
=

0,
1,
2,
3,

30

New V4L2 APIs for digital camera

Object recognition

31

Object detection

Gee!
It recognized my face!
Hello everyone

32

What do we set?
- Detect mode
- Type of object
- How many objects to detect
- Triggered action
What do we get?
- How many objects are detected
- Detected objects type
- Detected objects coordinates

33

New API for object recognition

- New capability
#define V4L2_CAP_ASYNCIO
#define V4L2_CAP_STREAMING

0x02000000 /* async I/O */

0x04000000 /* streaming I/O ioctls */

+#define V4L2_CAP_OBJ_RECOGNITION

0x10000000

- New IOCTL
+#define VIDIOC_S_RECOGNITION _IOWR ('V', 85, struct v4l2_detect)
+#define VIDIOC_G_RECOGNITION _IOR ('V', 86, struct v4l2_detect)

34

New API for object detection

+/* Object detection and triggered actions */

+enum v4l2_recog_mode {
+
V4L2_RECOGNITION_MODE_OFF
= 0,
+
V4L2_RECOGNITION_MODE_ON
= 1,
+
V4L2_RECOGNITION_MODE_LOCK
= 2,
+};
+
+enum v4l2_recog_action {
+
V4L2_RECOG_ACTION_NONE
= 0,/* only detection */
+
V4L2_RECOG_ACTION_BLINK
= 1,/* Capture on blinking */
+
V4L2_RECOG_ACTION_SMILE
= 2,/* Capture on smiling */
+};
+
+enum v4l2_recog_pattern {
+
V4L2_RECOG_PATTERN_FACE
= 0; /* Face */
+
V4L2_RECOG_PATTERN_HUMAN
= 1; /* Human */
+
V4L2_RECOG_PATTERN_CHAR
= 2; /* Character */
+}

35

New API for object detection

+struct v4l2_recog_rect {
+
enum
v4l2_recog_pattern p;
/* What pattern detected */
+
struct
v4l2_rect o;
/* detected area */
+
__u32
reserved[4];
+}
+
+struct v4l2_recognition_data {
+
__u32
detect_cnt; /* detected object counter */
+
struct
v4l2_recog_rect obj; /* detected area */
+
__u32
reserved[4];
+};
+
+struct v4l2_recognition {
+
enum v4l2_recog_mode mode;
+
enum v4l2_recog_pattern pattern;
/* What pattern to detect */
+
__u32
obj_num; /* How many object to detect */
+
__u32
detect_idx;
/* select detected object */
+
struct v4l2_recog_data
data; /* read only */
+
enum v4l2_recognition_action action;
+
__u32
reserved[4];
+};
36

What do we set?
- Detect mode
- How many objects to detect
- Triggered action
What do we get?
- How many objects are detected
- Detected objects coordinates

37

New API for object recognition

- New capability
#define V4L2_CAP_ASYNCIO
#define V4L2_CAP_STREAMING

0x02000000 /* async I/O */

0x04000000 /* streaming I/O ioctls */

+#define V4L2_CAP_OBJ_RECOGNITION

0x10000000

- New IOCTL
+#define VIDIOC_S_RECOGNITION _IOWR ('V', 85, struct v4l2_detect)
+#define VIDIOC_G_RECOGNITION _IOR ('V', 86, struct v4l2_detect)

38

New API for object detection

+/* Object detection and triggered actions */
+enum v4l2_recognition_mode {
+
V4L2_RECOGNITION_MODE_OFF
= 0,
+
V4L2_RECOGNITION_MODE_ON
= 1,
+
V4L2_RECOGNITION_MODE_LOCK
= 2,
+};
+
+enum v4l2_recognition_action {
+
V4L2_RECOG_ACTION_NONE
= 0,/* only detection */
+
V4L2_RECOG_ACTION_BLINK
= 1,/* Capture on blinking */
+
V4L2_RECOG_ACTION_SMILE
= 2,/* Capture on smiling */
+};
+
+struct v4l2_recognition_data {
+
__u8 detect_cnt; /* detected object counter */
+
struct
v4l2_rect
o; /* detected area */
+};
+
+struct v4l2_recognition {
+
enum v4l2_recognition_mode
mode;
+
__u8
obj_num; /* How many object to detect */
+
__u8
detect_idx;
/* select detected object */
+
struct v4l2_recognition_data data; /* read only */
+
enum v4l2_recognition_action action;
+};
39

New V4L2 APIs for digital camera

Dual camera

40

What is dual camera?

Flip side camera

: for still shots

Front side camera

: for special purpose

Dual camera applications

- Video telephony
- Camera applications
Taking self portrait shots

41

H/W restrictions in Application Processors camera peripheral interface

Mega
camera

VGA
camera

Data
path

42

H/W restrictions in Application Processors camera peripheral interface

SINGLE camera interface

on Application Processor !
Only one camera could be handled at a time
Only one camera could be using data pins at a time
Each camera is different in

Working MCLK

Working resolution
(SYNC)

Working PCLK

43

We already have API for dual camera

- Enumerate video inputs
VIDIOC_ENUMINPUT

- Query or select the current video input

VIDIOC_G_INPUT

VIDIOC_S_INPUT

44

What do we need to consider?

Let user space application get noticed about camera input devices
Dont forget VIDIOC_ENUMINPUT
Check out supported pixel format using VIDIOC_ENUM_FMT and
VIDIOC_ENUM_FRAMESIZES
Context of each camera module
Capability : Supported color spaces, resolutions, additional functions
Latest working status : resolution, color space, effect and so on
Context of camera interface
Working information of the camera module getting switched : Clock
information and so on

45

How to switch active camera

Condition 1
How to enable camera module:
Give power source
Enable expecting camera
Give MCLK

Condition 2
Shared things
Camera power enable pin
Camera data pins
MCLK

Mega camera

VGA camera

Mega
Enable

VGA
Enable

Mega
Reset

VGA
Reset

CAM
POWER

MCLK

DATA PINS

CPU

Condition 3
Dedicated things
Camera enable pins
Camera reset pins

46

Cam
power off

STREAMON

Camera (A)
initialize

Start
preview

Camera (A)
enable pin off

Camera (B) Enable

pin on

Cam
power on

Switch to camera
STREAMOFF
(B)
Stop
preview

G_INPUT/
S_INPUT

Camera (B)
initialize

Start
preview

Camera interface
reset & setting

Stop preview
Stop DMA

Reset camera
interfaces current
setting

Make a proper
MCLK for
camera (B)

Setup resolution,
color-space, DMA

47

Porting issues in SoC camera subsystem

48

I2C protocol has start and stop condition

Some camera modules generate unexpected noise until their power up

sequence
- Makes I2C bus in bus busy state

49

How to prevent this malfunctioning issue

Use a noise free on power up camera module

OR
Use a level shifter to block and ignore noise from camera module

OR
Change functionality of I2C pins while turning on camera module

50

All
-

about between camera interface and camera module

Expected resolution from camera interface
Served resolution from camera module
Symptoms depend on camera interface

Expecting resolution is bigger than served resolution

-

A select time out error in user space application or

Preview image is floating in some direction

Serving resolution is bigger than expected resolution

-

A select time out error in user space application or

Preview angle looks like to be zoomed in

51

Color order mismatched between camera interface and camera module

52

Preview freezing while checking lens response

- Using while() loop while it checks response, preview will freeze until
focusing job gets finished

- If the ISP driver use the kernel thread when it check response from
current lens status could make smooth preview on the LCD while
lens is on focusing.

53

Solving preview freezing while checking lens status

struct platform_isp {
const struct platform_data *pdata;
struct v4l2_int_device *v4l2_int_device;
struct i2c_client *i2c_client;
int reschk;
. . . . . . . . .
}
static int ioctl_s_ctrl(struct v4l2_int_device *s,
struct v4l2_control *vc)
{
struct platform_isp *isp = s->priv;
struct i2c_client *client = isp->i2c_client;
switch (vc->id) {
case V4L2_CID_FOCUS_AUTO:
isp->vfm = vc->value;
if (vc->value == V4L2_CID_FOCUS_MACRO) {
err = i2c_command_for_isp( client, focus_macro );
isp->reschk = 1;
}
If isp->reschk is Set,
break;
The kernel thread is active.
}
}
54

Solving preview freezing while checking lens status

int pseudo_reschk_thread (void *data) Platform Data Vairable for isp, or
static variable.
{
int err = 0;
This function return when the ISP is

matched the focus(+ return) or

do {
timeout(- return).
switch (isp->reschk) {
case 1:
err = i2c_command_for_isp(client, focus_check);
if (err < 0) {
isp->reschk = 1; /* i2c command function failed. */
} else {
isp->reschk = 0; /* if succeed, reschk ended. */
}
break;

This is able to re-check

}
to the next check time.
} while(1);
return 0;
}

55

JPEG has no synchronization protocol when it gets transferred

All about camera interface and camera modules data synchronization

- When camera interface supports JPEG sync

Follow user manual and make camera module to follow that protocol

- When camera interface does not support JPEG sync

Make synchronization protocol: make JPEG data to be synchronized with
VSYNC

56

Q&A
57

Thank you
58