Smart CCD Image Sensors for Optical Metrology and Machine Vision

T. Spirig, P. Seitz, 0. Vietze

Paul Schrerrer Institute Zurich
Badenerstrasse 569, CH-8048 Zurich
e-Mail: Thomas. Spirig @ psi.ch; Peter.Seitz@ psi.ch; Oliver.Vietze@ psi.ch
Abstract -Two types of CCD image sensors are described.
The first sensor is a two-dimensional, synchronous
detector/demodulator (Lmk-In CCD) of spatially
modulated light fields for applications in heterodyne
interferometry and time-of-flight range imaging.
Simultaneous measurements of amplitude, phase and
background level are carried out at each pixel site. This
is made possible by the principle of synchronized,
periodic multi-tap sampling and photo charge
accumulation. The second sensor, the Convolver CCD
is capable of performing image acquisition and real-time,
parallel convolution with an arbitrary kernel. Tap weight
accuracies of typically 2% of the largest tap values have
been obtained for a variety of linear filters that are
commonly used in machine vision. The sensors have been
realized by using a commercially available multi-project
wafer CMOS/CCD process. For both sensors, the
principle, design, operation and measurement results are
presented and discussed.
I. INTRODUCTION
Charge-coupled devices (CCDs) are used in most of
todays image sensing applications, although the primary
motivation for their development was charge storage [I].
Soon, it became apparent that the CCD principle is also well
suited for the processing of analog signals in the charge
domain [2]. Using modem semiconductor technologies, it is
possible to fabricate photosensors whose geometry and
functionality are adapted to specific sensing tasks of various
optical measurement techniques. In the present work, two
types of such sensors are presented. The first type, called
lock-in CCD, is a two dimensional array of pixels, each of
which is a synchronous detectoddemodulator for spatially
modulated light fields. This is made possible by a
synchronous photo-charge detection and storage as proposed
in an incomplete form with a modified CCD in Ref. [3]. The
Lock-In CCD is described in Section 2. The second type of
sensor, described in Section 3, is an image sensor capable of
carrying out convolutions with arbitrary kemels during the
exposure. The possibility of realizing such a device was
initially suggested by Beaudet [4]. Both sensors have been
realized using Orbits Foresight 2pm N-well CMOYCCD
process [ 5] . In the concluding Section 4 the obtained results
are discussed and possible applications are presented.
11. LOCK-IN CCD
In typical image sensing applications a scene is
optically projected onto the image sensor, which has to
detect the local light intensity of an essentially stationary
scene. There are circumstances, however, in which the
sought information is not the local light intensity, but is
encoded in a modulation parameter of an oscillating
waveform. One might be interested in the local phase of the
modulated light, the amplitude and the background
illumination. The well-established measurement technique
by which local phase, amplitude and offset of a sinusoidal
electrical signal are measured, is called lock-in technique.
Conventional synchronous detectors (lock-in detectors),
often based on phase-locked loop (PLL) technology, are
essentially restricted to the analysis of one temporal signal at
a time. If modulated light is detected with 3 or more samples
per period, its mean brightness level B (offset), phase cp and
amplitude A can be determined unambiguously. Figure 1
illustrates the measurement principle for the so called four-
bucket technique. The charge is integrated during 4 time
intervals I1 to I4 of equal length At within one modulation
period T. Four charge packets per pixel, ao to a3 are then
obtained, which have to be stored at four different spatial
locations. The charge signals a0 to a3 are converted into the
At T
Fig. 1 Measurement principle of the Lock-in CCD. During one
modulation period, four charge packets *a3 are created by
integrating thephoto-generated charge. With thesefour packets
thephase9, theoffset B and &heamplitude A can bedetermined.
(0-7803-2943-0) 11
sought amplitude A, phase cp. and mean brightness level B
by using the relationships, described in Reference [6].
The architecture of one pixel is illustrated
schematically in Figure 2. One pixel consists of a photo-gate
(PG), a dump-gate (DG), a dump-diffusion (DD) and 4
transfer gates (TGo-TG3). In our implementation the area of
the photogate is 10.5 pm x 41.5 pm. Two vertical four-phase
CCD lines are located adjacent to each pixel. The four
transfer-gates act as switchable connections between PG and
the four-phase CCD. The four-phase CCD is covered by a
metal layer in order to prevent charge smear during read out.
Tests have been carried out using an image sensor with 10 x
fo
Fig. 2. Schematic layout of onepixel. It consists of a photo-gate
(PG), four transfer-gates (TGo-TGs), a dump gate (DG) and a
Dump diffusion (DD). Thechargepackets arestored under the
gates fpB of thetwo adjacent CCD lines.
15 pixels, with a horizontal pitch of 92.75 pm and a vertical
pitch of 80 pm.
The operation occurs in two stages, signal integration
and storage, followed by read out. During the signal
integration phase, the modulated light intensity produces a
modulated photo-current, which is integrated on the MOS
capacitor of the photosite. Synchronously with the master
oscillator this integrated charge is transferred and
accumulated in one of the shielded storage sites by
appropriately clocking the transfer gates and the photogate.
The exposure time and the location where the signal charge
is stored can be chosen by programming an appropriate
clocking sequence. The pixel can be cleared, e.g. for
shuttering operations, by pulsing the dump-gate high and
the photo-gate low. The charge is then transported to the
dump diffusion.
In the read out phase, the electrodes of the line
transfer CCD are clocked sequentially, transporting the four
charge packets per pixel in shielded CCD lines to the output
stage. The signal charge is converted to a voltage by use of a
floating diffusion output stage with a single on-chip p-
channel MOSFET acting as a source follower.
The CCD has been tested using a light source with a
modulated intensity. The center wavelength is at 630 nm and
the beam was focused on the CCD as described in Reference
[7] in more detail. The modulation frequency was chosen to
be 100.4 kHz. This is a typical frequency for the intended
application of the lock-in CCD in heterodyne interferometry.
The charge was collected over 100 modulation periods.
Figure 3 shows the measured phase vs. the true master
oscillator phase for 12 different phases starting at (pmaster =0"
and increasing with a step size of 30". An individual offset
for all outputs per pixel was subtracted and the resulting
signals were normalized. The r.m.s deviation of measured
vs. true master oscillator phase was 3.1 degrees (relative
phase error of 0.009). The relative deviation from true
amplitude was 0.05.
C 0.81
v
- measured values
l - average
master oscillator phase [deg]
Fig. 3. Amplitudemeasurements obtained with theLock-in pixel
CCD, exposed to a sinusoidally oscillating LED light source. The
modulation frequency was 100.4 kHz. The relative amplitude
measurement deviation fromthe truemaster oscillator amplitude
was 0.05.
LI. CONVOLVER CCD
Many signal and image processing functions are
linear, i.e. one-or two dimensional convolutions with specific
kemels [8]. The two-dimensional result of the convolution is
determined as a linear combination of neighboring pixel
intensities with suitable weights, as described in Reference
An interpretation of the convolution is that the
picture is shifted laterally in both dimensions and for each
position the pixel values are multiplied with a different
weight and accumulated to the sum, resulting in the new
pixel value. As the CCD is capable of shifting charge
laterally in two dimensions, this interpretation leads to a
naturally parallel implementation of the convolution using a
CCD. Theweighting can be realized by varying the exposure
times for the different lateral shifts. This principle holds
only for kemels with positive weights. The problem with
negative weights can be solved by performing two
convolutions each with only positive weights and performing
the difference either on or off-chip. The convolution CCD
consists of bidirectional CCD columns and bi-directional
rows and at the intersections of rows and columns the
~71.
12
photosites are located. Each pixel has its own associated
storage area, realized as an additive CCD column with
separate gates in parallel to the main CCD column. In our
implementation, the area of the photo-gates is 30 x 30 pm2
and the pixel pitch is 63 pm horizontally and 65 pm
vertically leading to a fill factor of 22%. First tests have been
performed with 16 x 16 pixels.
The Point spread function (PSF) can be measured by
illuminating a single pixel with a 20 pm diameter light spot.
Four types of filters have been chosen, which are commonly
used in machine vision. These are: (I) anisotropic Gaussian
filters; (11) Canny filters (the first deviation of Gaussian
filters); (111) Laplacian of Gaussian filters (LOG-filters) and
(IV) quadrature pair filters (Gabor filters). A set of 7 x 7
filter coefficients was calculated for each of these four
continuos functions. They were used to set the exposure
times. As the image is a single spot, there is a direct
correspondence between the calculated coefficients and the
measured output voltage. To obtain a figure of accuracy, the
image was offset corrected and normalized to fit the filter
coefficients and an r.m.s. deviation is obtained. We found
excellent correspondence between measured values and true
filter values. Typical r.m.s. deviations from the ideal filter
characteristics are between 1-2% of the largest kernel tap
value. As an example the two-dimensional point spread
function of a Laplacian of Gaussian filter is illustrated in
Figure 4. The following three components contribute to the
error:
(1) The read-out noise of the output amplifier, (2) the
dark current noise, a shot noise component that depends on
the exposure time and (3) the non-ideal charge transfer
efficiency (CTE). In our case, most of the error is attributed
to the imperfections of our measurement technique, largely
due to the non-linearity of the output stage. In addition
filters without negative elements in their filter mask have
slightly smaller r.m.s. errors since only one convolution was
Fig. 4 Two-dimensional point spread function of a Laplacian
of Gaussian filter measured with the convolution CCD. The
r.m.s deviation of theobained point spread function fromthe
ideal filter was less than 2%.
performed and hence the exposure time for these filters was
smaller.
ILL DISCUSSION
Two image sensors have been presented with on-chip
signal processing capabilities. These sensors have been
fabricated with a commercially available CMOSKCD
process. The Lock-in CCD accurately measures the phase
and the amplitude of light modulated at 100 kHz. This is
ideal for the intended application in heterodyne
interferometry, higher modulation frequencies are possible,
since the charge coupled transfer principle has been
demonstrated at frequencies up to 325 MHz in silicon [9].
Interesting applications for the convolver CCD are in
real time object recognition using matched filter techniques.
By providing two or more storage sites per pixel, it would be
possible to perform two convolutions and hence to detect
keypoints in real-time [lo]. Either sensor could also find
application in motion detection due to its capability of
storing two (convolver CCD) or four (Lock-in CCD)
successive pictures before readout. It is noted that spatial
resolution of the CCDs, i.e. the modulation transfer function
(MTF) is degraded due to carrier diffusion in the
semiconductor sensor Ell], [12]. Light with a long
wavelength produces a large penetration depth of the
photons in the semiconductor, leading to a larger cross-talk
and hence a reduced spatial resolution for the convolver
CCD and a significantly reduced dynamic range for
measurements with the lock-in CCD.
The presented CCD structures offer an interesting
alternative to the active pixel sensors ( APS) , where the
signal processing is performed with a conventional CMOS
circuitry. The CCD approach has the advantage that the
signal processing is carried out in the charge domain,
without any noise contributions from the device other than
dark noise. The novel device structures presented here are
merely an example of possibilities that CCD processing has
to offer in the field of optical metrology and machine vision.
ACKNOWLEDGEMENT
We gratefully acknowledge the invaluable help from
many people at the Paul Scherrer Institute in Zurich. Special
thanks goes to M.Kuhn, who provided the design and the
realization of the optical setup, ro P.Metzler, who designed
and fabricated the trapezoidal clock drivers and to J.M.
Raynor for his invaluable help with analog and digital
electronic problems. We would also acknowledge
stimulating discussions with0.Kubler and F.Heitger at the
Federal Institute of Technology (ETH) Zurich. This work
was supported in part by the Swiss priority programme
Optique, project number 524.
13
REmRENCES
[41
[71
W.S. Boyle and G.E. Smith, Charge Coupled
Semiconductor Devices, Bell Syst. Tech. Jour., Vol.
C.H. Shui n and M.F. Tompsett, Charge Transfer
Devices, Academic Press, Inc., New York, 1975.
H.Pove1, H. Aebersold and J.O. Stenflo,
Chargcoupled device image sensor as a demodulator
in a 2-D polarimeter with a piezoelastic modulator,
Applied Optics, Vol. 29, pp. 1186-1190, 1990.
P.R.Beaudet, Time and Space Multiplexing Focal
Plane Convolvers, Proc. SPZE, Vol. 107 1, pp. 90-99,
1989.
Orbit Semiconductor Inc., 1230 Bordeaux Drive,
Sunnyvale, California, 94089, USA.
T.Spirig, P.Seitz, 0. Vietze and F.Heitger, The
Lock-In CCD - Two-Dimensional Synchronous
Detection of Light, submitted to IEEE J. Quantum
Electronics, in print (1995).
P. Seitz, T. Spirig, 0. Vietze and K. Engelhardt,
Smart sensing using custom photo-ASICs and CCD
technology, Optical Engineering, to appear in
August 1995.
W.K.Pratt, Digital Image Processing, 2nd Edition,
Wiley & Sons, New York, 1991.
A.L. Lattes, S.C. Munroe and M.M. Seaver,
Ultrafast Shallow-Buried-Channel CCDs with Built-
In Drift Fields, IEEE Electr. Dev. Lett., vol. 12, pp.
F. Heitger, L. Rosenthaler, R. von der Heydt, E.
Peterhans and 0. Kubler, Simulation of neural
contour mechanisms: From simple to end-stopped
cells, Vision Research, 32, pp. 963-981, 1992.
D.H.Seib, Carrier diffusion degradation of
modulation transfer function in charge coupled
imagers, IEEE Trans. Electron Devices, vol ED-21,
E.G. Stevens and J .P. Lavine, An analytical,
aperture, and two-layer carrier diffusion MTF and
quantum efficiency model for solid-state image
sensors, IEEE Trans. Electron Devices, vol. 41, pp.
49, pp. 587-593, 1970.
104-107, 1991.
pp. 210-217, Mar. 1974.
1753-1760, Oct. 1994
14