Professional Documents
Culture Documents
Image Registration in
Medical Imaging
Colin Studholme
Associate Professor in Residence
Biomedical Image Computing Group
http://radiology.ucsf.edu/bicg
colin.studholme@ieee.org
Image Segmentation
C.Studholme U.C.S.F.
C.Studholme U.C.S.F.
Registration
What is Registration?
the determination of a one-to-one
mapping between the coordinates
in one space and those in another,
such that points in the two spaces that
correspond to the same anatomical
point are mapped to each other.
Definitions
Classifications: Geometry, Transformations, Measures
Calvin Maurer 93
Voxel Based Methods:
Image Intensity Difference and Correlation
Multi-Modality Measures
C.Studholme U.C.S.F.
Image Registration
Establishing correspondence,
correspondence,
between features
in sets of images,
images,
and
using a transformation model
to
infer correspondence
away
from those features.
features.
Bill Crum 05
C.Studholme U.C.S.F.
Problem:
C.Studholme U.C.S.F.
MRI-CT
MRI-PET/SPECT
structural MRI- functional MRI
structural MRI to structural MRI
Problem:
Subject scanned multiple times -> Different scanners
We cannot easily fix/know patient location and orientation with
respect to different imaging systems
Need to remove differences in patient positioning to relate
information from different types of images
C.Studholme U.C.S.F.
C.Studholme U.C.S.F.
Components of Image
Registration Algorithms
PET(x)
MRI(y)
Optimization Method :
maximizing/minimizing criteria wrt T()
C.Studholme U.C.S.F.
C.Studholme U.C.S.F.
10
C.Studholme U.C.S.F.
12
Nuc. Medicine
11
y2
t1
y1
x1 t1=a13 t2=a23
y1=a11.x1+a12.x2+a13
y1
t2
y2=a21.x1+a22.x2+a23
a11 a12 a13
y1
x2 x
1
C.Studholme U.C.S.F.
y2
y=Ax
Matrix form:
extend to 3x3 Matrix
Homogeneous
coordinate
Transformation
x1
x2
1
13
C.Studholme U.C.S.F.
14
Image Scaling
Scaling with respect to a fixed point (x,y)
T ( x, y) R ( ) T ( x, y )
1 0 x cos sin 0 1 0 x
= 0 1 y sin cos 0 0 1 y
0 0 1 0
0
1 0 0 1
cos sin x(1 cos ) + y sin
0
0
1
1 0 x sx 0
= 0 1 y 0 sy
0 0 1 0 0
sx 0 (1 sx ) x
= 0 sy (1 sy ) y
0 0
0 1 0 x
0 0 1 y
1 0 0 1
R 1 ( ) S ( s x , s y ) R ( )
C.Studholme U.C.S.F.
15
R ( )
S (sx , s y )
R 1 ( )
C.Studholme U.C.S.F.
16
Composing Transformations:
To Apply:
We need to COMPOSE:
R ( 90 )
T ( x , 3)
i.e.:
C.Studholme U.C.S.F.
Translation
followed by
rotation
17
C.Studholme U.C.S.F.
18
Radiotherapy:
Treatment verification (Portal images)
Treatment planning (simulator images)
Treatment guidance (Cyberknife system)
Orthopaedic Surgery
2D Image Data
3D Image Data
Neurointerventions
Matching MRA to DSA
Surgical Microscope
MRI/CT neurosurgery guidance
Virtual Endoscopy
:
19
C.Studholme U.C.S.F.
20
Detector Array
4 Intrinsic Parameters:
Imaging System:
Mapping points in 3D space
to imaging plane
k1 0
0 k2
0 0
u RPP
v RPP
1
x, y, z
R
x
0
u
y
0 = v
z
1
0
1
x
u
y
P R T = v
z
1
Composed 1
Scaling/Resolution:
Xray source
Transformations
C.Studholme U.C.S.F.
21
C.Studholme U.C.S.F.
3D-3D Registration
3D to 3D image registration
X, Y, Z,
Image Visit 1
x, y, z
Image Visit 2
22
diagnosis
treatment planning
treatment guidance
clinical research: studying disease
Voxels
z
x
Transformation types:
Z y
Y
X
k1,k2,k3
scaling
C.Studholme U.C.S.F.
23
Z y
Y
X
Imaging
Process
(inc.planning)
Voxels
z
Z y
Y
X
x
Z y
Y
X
Differences in Patient Positioning
within scanner
Imaging
x Process
k1,k2,k3
scaling
C.Studholme U.C.S.F.
24
3D Rigid Transformations
3 Translations x,y,z
3 Rotations:
x cos
y sin
z = 0
1 0
sin
cos
0
0
0 0 x
0 0 y
1 0 z
0 1 1
0
x 1
y 0 cos
z = 0 sin
1 0
0
T31 p 31
1 1
M
T (p) = 33
0
sin
cos
0
x cos
y 0
z = sin
0 x 1 0
0 y
0 z
1 1
PET(x)
Transformation Type
Rigid/Affine/Non-Rigid
Correspondence Criteria/Measure
Feature Based Methods
Voxel Based/Dense Field Methods
Optimization Method :
y=T(x)
MRI(y)
1
3
25
CT
3
2
C.Studholme U.C.S.F.
26
C.Studholme U.C.S.F.
28
MR
(homologous)
Point set->Point set
(non homologous..
so need to find order)
1
3
?
3
27
MR-CT REGISTRATION
1
3
3
2
3
2
pr
We want to Find:
Transformation T(q) that Minimizes
squared distance between
corresponding points:
E=
||pr-T(qr )||2
qr
B
29
C.Studholme U.C.S.F.
q i=qi- 1/N i qi
pN
x1, y1 z1
x2, y2 z2
x3, y3 z3
xN, yN zN
Q TP T=R.(Q T Q)
p i=p i- 1/N i p i
P=
30
q1
q2
Q= q3
qN
decomposition (SVD):
(QT Q) -> (USV T)
x1, y1 z1
x2, y2 z2
x3, y3 z3
xN, yN zN
sin( ) cos( )
K. S. Arun, T. S. Huang, and S. D. Blostein. Least square fitting of two 3-d point sets. IEEE
Transactions on Pattern Analysis and Machine Intelligence, 9(5):698 - 700, 1987.
K. S. Arun, T. S. Huang, and S. D. Blostein. Least square fitting of two 3-d point sets. IEEE
Transactions on Pattern Analysis and Machine Intelligence, 9(5):698 - 700, 1987.
C.Studholme U.C.S.F.
31
C.Studholme U.C.S.F.
32
Orthonormal matrices:
B. K. P. Horn, H. M. Hilden, and Sh. Negahdaripour.
Closed-form solution of absolute orientation using orthonormal matrices.
Journal of the Optical Society of America A, 5(7):1127 - 1135, July 1988.
Dual quaternions:
M. W. Walker, L. Shao, and R. A. Volz.Estimating 3-d location parameters
using dual number quaternions. CVGIP: Image Understanding, 54:358 367, November 1991.
D.L.G. Hill, et al, Accurate Frameless Registration of MR and CT Images of the Head:
Applications in Surgery and Radiotherapy Planning, Radiology, 191, 1994, pp 447-454.
33
C.Studholme U.C.S.F.
34
Extrapolating Transformations:
THEORETICAL POINT BASED REGISTRATION ERROR
(points on circle 50mm radius selected with RMS error of 2mm)
1
2.5
RMSerror (mm)
2
1.5
1
4 points
8 points
16 points
0.5
0
-100 -90
-80
-70
-60
-50
-40
-30
-20
-10
10
20
30
40
50
60
70
80
90
100
mm
35
C.Studholme U.C.S.F.
36
Need to be true unique 3D points in intensity map: tip-like, saddle-like, and sphere-like
structures.
Need to be consistent in different subjects and image types
[Stefan Wrz, Karl Rohr Localization of anatomical point landmarks in 3D medical images
by fitting 3D parametric intensity models, Medical Image Analysis Volume 10, Issue 1,
Page 41-58, Feb 2006.]
Easier to automatically find general features: for example points on a surface using
edge detection.
But: Which point maps to which point?
Need to then find correspondence and alignment: Point Cloud Fitting
37
Alignment of non-Homologous
Feature Locations: fuzzy correspondence
C.Studholme U.C.S.F.
Q
P
General case of two lists of point locations
Take each point Pi=[p1, p2 pN] and search list Qi to find the
nearest point Qii to create a new list with N points.
E(R,t)= i j kij||pi-(Rqj+t)||2
k21
k23
and t to Q=
=[q1, q2 qM]
38
C.Studholme U.C.S.F.
39
C.Studholme U.C.S.F.
40
10
Improvements/Adaptations in ICP
Orientation driven ICP:
Matches location and local
boundary/surface
orientation
[Godin 2001,Schutz
1998]
Subsampling: Choosing only meaningful points
Eg points on curved parts of surface
C.Studholme U.C.S.F.
41
[S. Rusinkiewicz, M. Levoy, Efficient Variants of the ICP Algorithm, Proc 3rd
international conference on 3D digital Imaging 2001]
C.Studholme U.C.S.F. 42
This successfully used to evaluate MRI-CT, MRI-MRI and MRIPET registration using bone implanted markers
J. West, J.M. Fitzpatrick, M.Y. Wang, B.M. Dawant, C.R. Maurer, R.M. Kessler, R.J.
Maciunas, C. Barillot, D. Lemoine, A. Collignon, F. Maes, P. Suetens, D.
Vandermeulen, P.A. van den Elsen,S. Napel,T.S. Sumanaweera, B. Harkness, P.F.
Hemler, D.L.G. Hill, D.J. Hawkes, C. Studholme, J.B.A Maintz, M.A. Viergever, G.
Malandain, X. Pennec, M.E. Noz,G.Q. Maguire, M. Pollack, C.A. Pelizzari, R.A.
Robb,D. Hanson, R.P. Woods, Comparison and Evaluation of Retrospective
Intermodality Brain Image Registration Techniques, J. Comp. Assist. Tomog. Vol
21(4), 1997, pp 554-566.
C.Studholme U.C.S.F.
43
C.Studholme U.C.S.F.
44
11
PET(x)
Transformation Type
Rigid/Affine/Non-Rigid
Correspondence Criteria/Measure
y=T(x)
MRI(y)
Optimization Method :
45
C.Studholme U.C.S.F.
46
w(x)
f(x)
Image
intensity
w(x)
t
f(x)
Consider 1D case:
for no noise, assume
w(x) is some exact translation of f(x)
Location
w(x)= f(x+t)
and image mis match or error for given displacement t can simply be
Local Difference in intensity:
e(t)=f(x+t)-w(x)
47
[Lucas and Kanade, An Iterative Image Registration Technique with an Application to Stereo Vision
Proc Image Understanding Workshop, P121-130, 1981]
C.Studholme U.C.S.F. 48
12
w(x)
t
w(x) t
f(x)
f(x+t)
f(x)
xt(x)
Image
intensity
w(x)
t
f(x)
/nx
x
nx
t [w(x)-f(x)]/f(x)
xt(x)
/nx
[Lucas and Kanade, An Iterative Image Registration Technique with an Application to Stereo Vision
Proc Image Understanding Workshop, P121-130, 1981]
C.Studholme U.C.S.F. 49
Image
intensity
f(x)
t xt(x) /nx
x
Weighted average:
nx
Use contributions where linear approximation to f(x) is better.
i.e. weight to points where |f(x)| closer to zero.
Possible weight k(x) for where (w(x)-f(x))/t at point x is:
k(x)=1/|w(x)-f(x)|
Giving t=
xk(x)t(x)
x 2f(x)
t
C.Studholme U.C.S.F.
51
x f(x)
f(x)
x
nx
(f(x)+t f(x)-w(x))
Giving:
xk(x)t(x)
xk(x)
w(x)
t
2
2
0=
x [f(x+t)-w(x)] =
x[f(x)+t f(x)-w(x)]
t
t
xk(x)
50
E(t)= x [f(x+t)-w(x)]2
w(x)
t
C.Studholme U.C.S.F.
2
x f(x)
x f(x)
(w(x)-f(x))
x f(x)
2
C.Studholme U.C.S.F.
52
13
E(t)= xR [f(x+t)-w(x)]2
As for 1D, we can then use a linear approximation for small t so that
f(x+t)=f(x)+t f(x)
x
We can then set
0= E
t so:
x
R [
C.Studholme U.C.S.F.
53
C.Studholme U.C.S.F.
54
Template Image
target image
intensity
template intensity
target image
intensity
t=[
E(A,t)= xR [f(xA+t)-w(x)]2
template intensity
55
C.Studholme U.C.S.F.
56
14
Correlation
Template Image
target image
intensity
Overall brightness
increase in target
target image
intensity
template intensity
Overall brightness
decrease in target
target image
intensity
template intensity
Overall contrast
increase in target
template intensity
C.Studholme U.C.S.F.
57
C.Studholme U.C.S.F.
58
C.Studholme U.C.S.F.
59
C.Studholme U.C.S.F.
60
15
Variance?
Correlation Coefficient
Target image
intensity
Target image
intensity
template intensity
Target image
intensity
template intensity
template intensity
C.Studholme U.C.S.F.
61
62
C.Studholme U.C.S.F.
64
template intensity
Target image
intensity
Target image
intensity
Target image
intensity
C.Studholme U.C.S.F.
template intensity
template intensity
C.Studholme U.C.S.F.
63
16
Target f
Target f
Register
Template w
Register
template intensity w
C.Studholme U.C.S.F.
65
SPECT-MRI
registration:
Patient
orientated
differently
with respect
to scanner
coordinates:
Target image
intensity f
Target image
intensity f
Template w
template intensity w
C.Studholme U.C.S.F.
66
SPECT-MRI
registration
1.Head rest
design
2.Gantry angle
limitations
3. No easy to
find 3D
landmarks
C.Studholme U.C.S.F.
67
C.Studholme U.C.S.F.
68
17
Registered
Translated by 2mm
PET/SPECT intensity f
T1 Weighted MRI
Translated by 8mm
Translated by 5mm
Grey Matter
AIR/CSF
White Matter
MRI intensity w
C.Studholme U.C.S.F.
69
C.Studholme U.C.S.F.
70
Functional/
Structural
Fusion:
Target image
intensity f
b=1
MRI-SPECT
template intensity w
C.Studholme U.C.S.F.
71
C.Studholme U.C.S.F.
72
18
Correlation Ratio
[A. Roche et al MICCAI 98]
Target image
intensity f
b=1
template intensity w
Correlation Ratio:
C.Studholme U.C.S.F.
73
C.Studholme U.C.S.F.
74
Correlation Coefficient
MR and CT
m
h(m,c)
c
Assumes Linear Relationship between MR and CT intensity.
Improved by using only modified soft tissue or bone
intensities from CT. (Van den Elsen, 1994).
Manual Registration Estimate
(Using Corresponding Anatomical Landmarks)
C.Studholme U.C.S.F.
75
C.Studholme U.C.S.F.
76
19
template intensity w
MR
Target image
intensity f
CT
Target f
Template w
C.Studholme U.C.S.F.
77
C.Studholme U.C.S.F.
template intensity w
p(w,f)
registered
Target image
intensity f
Target image
intensity f
78
C.Studholme U.C.S.F.
template intensity w
mis-registered
Small number of
high-probability pairs
79
Larger number of
lower-probability pairs
Less Uncertainty
More Uncertainty
20
Mutual Information
are
p(w,f)
H(W,F) is Minimized
registered
I(F,W)=H(F)+H(W)-H(F,W)
to be Maximized
where
p(w,f)
mis-registered
H(F)=
H(W)=
C.Studholme U.C.S.F.
81
C.Studholme U.C.S.F.
82
83
C.Studholme U.C.S.F.
84
C.Studholme U.C.S.F.
21
larger
peaks
smaller
peaks
p(w,f)
MI big!
more info
in target
explained by
template
C.Studholme U.C.S.F.
85
C.Studholme U.C.S.F.
86
MI
NMI
Alignment can be driven by choosing overlap that has large H(F) and
H(W).
e.g. overlap with equal area of background and foreground intensities
87
C.Studholme U.C.S.F.
88
22
MRI-CT Registration
MRI-CT Registration
C.Studholme U.C.S.F.
89
C.Studholme U.C.S.F.
90
CT: Bone
MR Gadolinium: Tumor
MRA: Blood Vessels
C.Studholme U.C.S.F.
91
C.Studholme U.C.S.F.
92
23
Ictal
Change in Uptake
Registered
MR+PET
C.Studholme U.C.S.F.
93
C.Studholme U.C.S.F.
94
Bone Features
PET: F Uptake.
MR: Marrow White.
Soft Tissue
95
C.Studholme U.C.S.F.
96
24
C.Studholme U.C.S.F.
97
C.Studholme U.C.S.F.
Bibliography I
Summary
A range of medical alignment measures have been
developed in the last 15yrs
98
C.Studholme U.C.S.F.
99
100
25
Bibliography II
C.Studholme U.C.S.F.
101
26