Professional Documents
Culture Documents
1 b b 0 0
r 0 1 r 0
r 0 0 1 r
1 0 0 0
.9 0.1 0 0
.8 0 .2 0
.8 0 0 .2
1 0 0 0
P
4
=
P
10
=
P
10
(1, 4) .0036. We can interpret this as saying that .0036% of
total machines need to be replaced on the 10th day.
Markov Chains, Examples
Initial distribution
The vector
0
dened by
0
= |P(X
0
= 1), P(X
0
= 2), P(X
0
= 3), P(X
0
= 4) = 0|
= |.75, .2, .05, 0|
is called the initial distribution of the MC X
n
n0
Markov Chains, Examples
Initial distribution 2
If
0
= |.75, .2, .05, 0|, nd P(X
1
= j ) for j = 1, , 4.
P(X
1
= 1) =
4
i =1
P(X
0
= i )P(X
1
= 1|X
0
= i ) =
4
i =1
0
(i )p
i ,1
= |
0
(1), ,
K
(0)|
p
1,1
p
2,1
p
3,1
p
4,1
, P(X
1
= 2) =
0
p
1,2
p
2,2
p
3,2
p
4,2
P(X
1
= 3) =
0
p
1,3
p
2,3
p
3,3
p
4,3
, P(X
1
= 4) =
0
p
1,4
p
2,4
p
3,4
p
4,4
P(X
1
= j ) = product of the vector
0
and the j th column of P.
Markov Chains, Examples
Initial distribution
Dene
1
to be the distribution of X
1
, then
1
= |P(X
1
= 1), P(X
1
= 2), P(X
1
= 3), P(X
1
= 4)|
= |
0
p
1,1
p
2,1
p
3,1
p
4,1
,
0
p
1,2
p
2,2
p
3,2
p
4,2
p
1,3
p
2,3
p
3,3
p
4,3
,
0
p
1,4
p
2,4
p
3,4
p
4,4
|
=
0
P
then:
1
=
0
P
Similarly, if we dene
n
to be the distribution of X
n
then
n+1
=
n
P = (
n1
P)P = =
0
P
n+1
Markov Chains, Examples
Example
3
=
0
P
3
= (.6, .2, .15, .05)
Then
P(X
n+m
= j |X
0
= i ) =
l S
P(X
n+m
= j , X
m
= l |X
0
= i )
=
l S
P(X
n+m
= j |X
m
= l , X
0
= i )P(X
m
= l |X
0
= i )
=
l S
P(X
n+m
= j |X
m
= l )P(X
m
= l |X
0
= i )
=
l S
P(X
m
= l |X
0
= i )P(X
n
= j |X
0
= l ) =
l S
p
n
i ,l
p
m
l ,j
,
where p
n
i ,j
= P(X
n
= j |X
0
= i ) = P(X
m+n
= j |X
m
= i ), is the
probability of going from state i to the state j in n steps.
Markov Chains, Examples
Going from state i S to state j S in n + m steps
S = 1, , K
n
i
n
n
n
n
j
P
P
P
P
P
P
P
P
Pq
"
"
"
"
"
"
"
"
"
p
m
l ,j
-
P
P
P
P
P
P
P
P
Pq
*
1
l
K
time m time 0 time n + m
P(X
n+m
= j |X
0
= i ) = P
n+m
(i , j ) = (P
n
P
m
)(i , j )
=
K
l =1
P
n
(i , l )P
m
(l , j )
p
n
i
,
l
K
P(X
n
= l , X
n+m
= j |X
0
= j ) = p
n
i ,l
p
m
l ,j
Markov Chains, Examples
Large time behavior
n
=
0
P
n
suggests a dynamical perspective of the Markov
chain in which the distribution of X
n
evolves with time.
In 2 dimensions,
0
and
n
s are points on the line x + y = 1.
We choose and x a point (x
0
, y
0
) to be
0
, and study its
transformation under P.
We know that
n
= (x
0
, y
0
)P
n
are again on the same line
n 1 (prove this)
@
@
@
@
@
1
=
0
P
2
=
0
P
2
6
RRR
Markov Chains, Examples
Examples
Let P = _
3]4 1]4
1]6 5]6
_. Let
0
= (2]3, 1]3). Then
1
=
0
P = (5]9, 4]9).
Let = lim
n
P
n
, then rows of are identical. This means
that where ever the chain starts, for large n, going to state j
has the same probability. j S, Or we can say that the chain
forgets the starting point when n is large. (this is not true for
all chains)
n
= lim
n
u
0
P
n
= (.4, .6)
3]4 1]4 0
1]8 2]3 5]24
0 1]6 5]6
Assume
n
converge to some limit R
m
. Compute
1
if
0
= .
P = ( lim
n
u
0
P
n
)P = lim
n
u
0
P
n+1
=
i.e, if the limit exists, then starting with all the later
distributions are
n+1
= , n 1
Let P = _
1 p p
q 1 q
_, where 0 < p, q < 1, then
P = _
1 p
1 q
_ _
1 0
0 1 p q
_ _
q
p+q
p
p+q
1
p+q
1
p+q
_
P
n
= _
1 p
1 q
_ _
1 0
0 (1 p q)
n
_ _
q
p+q
p
p+q
1
p+q
1
p+q
_
q+p(1pq)
n
p+q
pp(1pq)
n
p+q
qq(1pq)
n
p+q
p+q(1pq)
n
p+q
_
q
p+q
p
p+q
q
p+q
p
p+q
_
What is ?
Markov Chains, Examples
3 Important Examples
0 1 2 3 4
1
.5
.5
.5
.5
.5
.5
.1
P =
0 1 0 0 0
.5 0 .5 0 0
0 .5 0 .5 0
0 0 .5 0 .5
0 0 0 1 0
, P
2n
=
0 0
0 0 0
0 0
0 0 0
0 0
n large
=
.25 0 .5 0 .25
0 .5 0 .5 0
.25 0 .5 0 .25
0 .5 0 .5 0
.25 0 .5 0 .25
, P
2n+1
0 .5 0 .5 0
.25 0 .5 0 .25
0 .5 0 .5 0
.25 0 .5 0 .25
0 .5 0 .5 0
0 1 2 3 4
1
.5
.5
.5
.5
.5
.5
1
P =
1 0 0 0 0
.5 0 .5 0 0
0 .5 0 .5 0
0 0 .5 0 .5
0 0 0 0 1
, P
n
1 0 0 0 0
.75 0 0 0 .25
.5 0 0 0 .5
.25 0 0 0 .75
0 0 0 0 1
the rows are not identical, i.e, the chain does not forget the
initial sate.
Markov Chains, Examples
reducible chains
1 2 3 4 5
1/2
1/6
1
2
5
6
3
4
1/4
1/8
7/8
5
6
1/6
Let X
n
be a Markov chain on the sate space S = 1, 2, 3, 4, 5,
P =
1]2 1]2 0 0 0
1]6 5]6 0 0 0
0 0 3]4 1]4 0
0 0 1]8 2]3 5]24
0 0 0 1]6 5]6
P
n
.25 .75 0 0 0
.25 .75 0 0 0
0 0 .182 .364 .455
0 0 .182 .364 .455
0 0 .182 .364 .455