Optimal Filtering

Optimal ltering is a means of adaptive extraction of a weak desired signal

in the presence of noise and interfering signals.
Mathematically: Given
x(n) = d(n) +v(n),
estimate and extract d(n) from the current and past values of x(n).
EE 524, # 10 1
Optimal Filtering (cont.)
Let the lter coecients be
w =
_

_
w
0
w
1
.
.
.
w
N1
_

_
.
Filter output:
y(n) =
N1

k=0
w

k
x(n k) = w
H
x(n) =

d(n),
EE 524, # 10 2
where
x(n) =
_

_
x(n)
x(n 1)
.
.
.
x(n N + 1)
_

_
.
Omitting index n, we can write

d = w
H
x, x =
_

_
x
0
x
1
.
.
.
x
N1
_

_
.
EE 524, # 10 3
Optimal Filtering (cont.)
Minimize the Mean Square Error (MSE)
MSE= E{|e|
2
} = E{|d w
H
x|
2
}
= E{(d w
H
x)(d

x
H
w)}
= E{|d|
2
} w
H
E{xd

} E{dx
H
}w +w
H
E{xx
H
}w.
min
w
MSE MSE/w = 0 E{xx
H
}w E{xd

} = 0.
R = E{xx
H
} correlation matrix
r = E{xd

}.
Wiener-Hopf equation (see Hayes 7.2):
Rw = r w
opt
= R
1
r.
EE 524, # 10 4
Wiener Filter
Three common lters:
1. general non-causal:
H(z) =

k=
h
k
z
k
,
2. general causal:
H(z) =

k=0
h
k
z
k
,
3. nite impulse response (FIR):
H(z) =
N1

k=0
h
k
z
k
.
EE 524, # 10 5
Wiener Filter
Case 1: Non-causal lter:
MSE = E[|e(n)|
2
] = E{[d(n)

k=
h
k
x(n k)][d(n)

l=
h
l
x(n l)]

}
= r
dd
(0)

l=
h

l
r
dx
(l)

k=
h
k
r
dx
(k)

k=

l=
r
xx
(l k)h
k
h

l
.
Remark: for causal and FIR lters, only limits of sums dier.
Let h
i
=
i
+j
i
. MSE/
i
= 0, MSE/
i
= 0 i =
r
dx
(i) =

k=
h
k
r
xx
(i k) i.
EE 524, # 10 6
In Z-domain
P
dx
(z) = H(z)P
xx
(z),
which is the optimal non-causal Wiener lter.
Example: x(n) = d(n) +e(n),
P
dd
(z) =
0.36
(1 0.8z
1
)(1 0.8z)
,
P
ee
(z) = 1,
d(n) and e(n) uncorrelated.
EE 524, # 10 7
Optimal lter?
P
xx
(z) = P
dd
(z) +P
ee
(z)
=
0.36
(1 0.8z
1
)(1 0.8z)
+ 1
= 1.6
(1 0.5z
1
)(1 0.5z)
(1 0.8z
1
)(1 0.8z)
,
r
dx
(z) = E[d(n +k)[d(n)

+e(n)

]] = r
dd
(k)
P
dx
(z) = P
dd
(z),
H(z) =
P
dx
(z)
P
xx
(z)
=
0.36
1.6(1 0.5z
1
)(1 0.5z)
,
h(k) = 0.3(
1
2
)
|k|
.
EE 524, # 10 8
Case 2: Causal lter (7.3.2 in Hayes):
MSE = E[|e(n)|
2
] = E{[d(n)

k=0
h
k
x(n k)][d(n)

l=0
h
l
x(n l)]

}
= r
dd
(0)

l=0
h

l
r
dx
(l)

k=0
h
k
r
dx
(k)

k=0

l=0
r
xx
(l k)h
k
h

l
=
r
dx
(i) =

k=0
h
k
r
xx
(i k) i.
Let
B(z)B

(
1
z

) =
1
P
xx
(z)
.
EE 524, # 10 9
Pick B(z) to be a stable, causal, minimum-phase system. Then
P
dx
(z) = H(z)B
1
(z)
. .
causal
B

(
1
z

) =
H(z)
B(z)
= [P
dx
(z)B

(
1
z

)]
+
,
where
[Y (z)]
+
= [

k=
y
k
z
k
]
+
=

k=0
y
k
z
k
.
= H(z) = B(z)[P
dx
(z)B

(
1
z

)]
+
.
EE 524, # 10 10
Causal Filter: Example
Same as before: x(n) = d(n) +e(n),
P
dd
(z) =
0.36
(1 0.8z
1
)(1 0.8z)
,
P
ee
(z) = 1,
d(n) and e(n) uncorrelated.
EE 524, # 10 11
Optimal lter?
P
dx
(z) = P
dd
(z),
P
xx
(z) = P
dd
(z) +P
ee
(z)
= 1.6
(1 0.5z
1
)(1 0.5z)
(1 0.8z
1
)(1 0.8z)
,
B(z) =
1

1.6
1 0.8z
1
1 0.5z
1
stable and causal,
P
dx
(z)B

(
1
z

) =
0.36
(1 0.8z
1
)(1 0.8z)
1

1.6
1 0.8z
1 0.5z
=
0.36

1.6
1
(1 0.8z
1
)(1 0.5z)
=
0.36

1.6
_ 5
3
1 0.8z
1
+
5
6
z
1 0.5z
_
EE 524, # 10 12
[P
dx
(z)B

(
1
z

)]
+
=
0.36

1.6
5
3
1 0.8z
1
H(z) = B(z)[P
dx
(z)B

(
1
z

)]
+
=
1

1.6
1 0.8z
1
1 0.5z
1
0.36

1.6
5
3
1 0.8z
1
= 0.375
1
1 0.5z
1
= h(k) =
3
8
(
1
2
)
k
, k = 0, 1, 2, . . .
Case 3: FIR lter (done before):
r
dx
(i) =
N1

k=0
h
k
r
xx
(i k) i.
EE 524, # 10 13
FIR Wiener Filter - Generalization
Theorem 1. Assume that
_
x
d
_
N
__

x

d
_ _
C
xx
C
xd
C
dx
C
dd
__
.
Then the posterior pdf of d given x, f
d|x
(d|x), is also Gaussian, with
moments given by
E[d|x] =
d
+
W
H
..
C
dx
C
1
xx
(x
x
)
C
d|x
= C
dd
C
dx
C
1
xx
C
xd
.
EE 524, # 10 14
FIR Wiener Filter - Generalization
Consider d = A +,
y = B +, where
E
__

__
= 0, cov
__

__
=
_
V
11
V
12
V
21
V
22
_
.
Also, assume E[] =
0
and cov() = . We wish to predict d given y.
EE 524, # 10 15
Theorem 2. Minimum Mean-Square Error (MMSE) Solution:

d = A
0
+C(y B
0
),
where C = (AB
H
+V
12
)(BB
H
+V
22
)
1
.
MMSE:
AA
H
+V
11
+C(BB
H
+V
22
)C
H
(AB
H
+V
12
)C
H
C(BA
H
+V
21
).
EE 524, # 10 16
Kalman Filter
Consider a state-space model:
x(t) = Ax(t 1) +(t),
y(t) = Bx(t) +(t),
where
{(t)} and (t) are independent sequences of zero-mean random vectors
with covariances

and

, respectively;
(t) and x(u) are independent for t > u and (t) and x(u) are
independent for t u.
We wish to predict x(t) given y(1), y(2), . . . y(t).
EE 524, # 10 17
Idea: Theorem 2 can be used to predict x(s) given y(t). We write such
predictor as x(s|t) and its (minimum) mean-square error as P(s|t).
How to construct a recursive procedure?
EE 524, # 10 18
Kalman Filter
Assume that we know x(t +1|t) and its MMSE P(t +1|t). We wish to nd
x(t + 1|t + 1). Then, we can write the following model:
x(t + 1) =
y(t + 1) = B +,
where
0
= x(t + 1|t) and = P(t + 1|t). This implies
x(t + 1|t + 1) = x(t + 1|t) +C(t + 1) [y(t + 1) B x(t + 1|t)],
where
C(t + 1) = P(t + 1|t)B
H
[BP(t + 1|t)B
H
+

]
1
.
Also
P(t +1|t +1) = P(t +1|t) C(t +1)[BP(t +1|t)B
H
+

]
1
C(t +1)
H
.
EE 524, # 10 19
Kalman Filter
How about nding x(t + 1|t) based on x(t|t)? Then, we can write the
following model:
x(t + 1) = A +(t + 1),
where
0
= x(t|t) and = P(t|t). This implies
x(t + 1|t) = A x(t|t)
P(t + 1|t) = AP(t|t)A
H
+

.
Now we have all the equations for recursion!
EE 524, # 10 20