EE 178 EE 178

Lecture notes 14 Discrete-State Process

Stochastic Processes II: Markov Chains
Markov chains are popular models in many applications areas, including
economic systems and networking. Their language is sometimes different
Lecture Outline than the common usage of stochastic processes, but essentially they are a
special kind of stochastic process.
• Discrete State Processes
For simplicity we consider only discrete time models with discrete outputs.
• Markov Chains
Suppose that {Xn; n = 0, 1, 2, . . .} is a stochastic process
• State Classification
Suppose also that Xn can only take on values in the set {1, 2, . . . , m}.
• Steady State Probabilities
Think of these values as “states” of a system. Examples include
• Birth and Death Processes
• # of unread email messages at the begining of each hour
Reading: Bertsekas & Tsitsiklis 6.1,6.2 ,6.3
• # of customers in a bank at each minute

Stochastic Processes II: Markov Chains 14–1 Stochastic Processes II: Markov Chains 14–2

EE 178 EE 178

• # of cars in a parking lot at the beginning of each hour Notation: In the Markov chain literature the abbreviation

• # of packets in a queue at each clock period pij = P (Xn = j | Xn−1 = i)

is often used. Arguably pj|i would be better, but we adopt common

A Markov chain is a stochastic process where the probability of the next convention. Be careful when dealing with Markov chains to not confuse
state given the current state and the entire past depends only on the current this conditional probability with a joint probability.
state:
This can be conveniently expressed using matrix notation as
P (Xn = xn | Xn−1 = xn−1, Xn−2 = xn−2, · · · , X0 = x0) = P (Xn = xn |  
Xn−1 = xn−1 ) p11 p12 · · · p1m
 p21 p22 · · · p2m 
The Markovian property is a constraint on the memory of the process: P =  .. .. .. .. 
knowing the immediate past means the earlier outcomes are no longer pm1 pm2 · · · pmm
relevant.
Note that here i is the conditioning so that
Alternatively: The future is conditionally independent of the past given the m
X
present. pij = 1
j=1

i.e., the sums of any row of the matrix is 1.

Stochastic Processes II: Markov Chains 14–3 Stochastic Processes II: Markov Chains 14–4
 EE 178 EE 178

Note that any iid process trivially satisfies the Markov condition, since both p

probabilities are simply P (Xn = xn).

1−p H T
1−p

p
As a simple nontrivial example, consider a sequence of dependent coin flips
where each coin has probability of 1 − p of having the same outcome as the
Graph shows memory structure.
previous coin flip, regardless of all previous flips.
“binary symmetric Markov process”
1 2
1 1−p p
2 p 1−p

pij

Graphs called transition diagrams are often used to depict Markov chains.

Nodes are states, arcs are state transitions (i, j) from state i to state j
(only draw transitions with pij > 0)

Stochastic Processes II: Markov Chains 14–5 Stochastic Processes II: Markov Chains 14–6

EE 178 EE 178

Can construct this process from an iid process: Suppose that {Zn} is a If you are not convinced, consider:
Bernoulli process with property p.
Note that from binary arithmetic a ⊕ b = c if and only if a = b ⊕ c and
hence
Define a new process Xn recursively as follows: Let X0 be a Bernoulli
random variable with some bias q.
P (Xn = xn | Xn−1 = xn−1, Xn−2 = xn−2 , · · · , X0 = x0 )
Define = P (Xn−1 ⊕ Zn = xn | Xn−1 = xn−1, Xn−2 = xn−2 , · · · , X0 = x0)
Xn = Xn−1 ⊕ Zn; n = 1, 2, . . .
= P (Zn = Xn−1 ⊕ xn | Xn−1 = xn−1, Xn−2 = xn−2 , · · · , X0 = x0)
⊕ is exclusive-or, addition modulo 2 (0 ⊕ 0 = 1 ⊕ 1 = 0, 0 ⊕ 1 = 1 ⊕ 0 = 1) = P (Zn = xn−1 ⊕ xn | Xn−1 = xn−1, Xn−2 = xn−2 , · · · , X0 = x0)
You should be able to convince yourself that P (Xn = xn | Xn−1 = = P (Zn = xn−1 ⊕ xn)
xn−1 , Xn−2 = xn−2, · · · , X0 = x0) = pZn (xn−1 ⊕ xn)

p 6 xn−1
xn =
=
1−p xn = xn−1 The conditioning disappears because the conditioning event involves only
Xk for k ≤ n − 1 and hence Zk for k ≤ n − 1, but the Zn are by assumption
iid and hence mutually independent.
Hence {Xn} is a Markov chain.

Stochastic Processes II: Markov Chains 14–7 Stochastic Processes II: Markov Chains 14–8
 EE 178 EE 178

Can also have asymmetric binary Markov process. Another example: Two spiders and a fly

State 1 = You are up-to-date in EE178 A fly’s possible positions are represented by four states.

State 2 = You are behind in EE178

States 2, 3 : safely flying in the left or right half of a room
1 2
1 0.8 0.2
2 0.6 0.4 State 1: A spider’s web on the left wall
pij
State 4: A spider’s web on the right wall
0.2

1 2 3 4
0.8 1 2 0.4
1 1.0 0 0 0
2 0.3 0.4 0.3 0
0.6 3 0 0.3 0.4 0.3
4 0 0 0 1.0

pij

Stochastic Processes II: Markov Chains 14–9 Stochastic Processes II: Markov Chains 14–10

EE 178 EE 178
0.4 0.4
Some Questions of Interest
0.3

0.3 0.3
1 1 4 1
2 3
• Steady-state occupancy probabilities of each state (long term fraction of
time spent in each state)?
0.3
• Probability of hitting certain absorbing states? (e.g., spider web)

For example, in binary symmetric Markov chain, could use known pX0 and
pX1|X0 to compute pX1 :

1
X
pX1 (1) = pX1|X0 (1 | x0)pX0 (x0)
x0 =0

= q(1 − p) + (1 − q)p

This can then be combined with pX2|X1 = pX1|X0 to compute pX2 , and so
on.

Stochastic Processes II: Markov Chains 14–11 Stochastic Processes II: Markov Chains 14–12
 EE 178 EE 178

But is there a limit of pXn as n → ∞? In particular, if repeat the computation of pX1 from pX0 to find pX2 from
a steady state probability of the states. pX1 , then repeat again to find pX3 from pX2 , and so on to find pXn from
pXn−1 , then each time we are solving an equation of the form

1
X
pXn (1) = pXn|Xn−1 (1 | x)pXn−1 (x)
x=0
= pXn−1 (1)(1 − p) + (1 − pXn−1 (1))p ?

Suppose that indeed pXn (1) has a limit as n → ∞, say pX (1) = α, so that
n
α x=1
pX (x) =
1−α x=0

Substituting this into ?:

α = α(1 − p) + (1 − α)p

Stochastic Processes II: Markov Chains 14–13 Stochastic Processes II: Markov Chains 14–14

EE 178 EE 178
or A similar strategy yields the steady state probability for the asymmetric
α(1 − 1 + p + p) = p Markov chain. Suppose that asymptotically state 1 has probability α and
so that hence state 2 has probability 1 − α.
1
α=
2 In the long run the fraction of transitions from right to left in the graph
is the steady state probability that Xn = 1. must equal that from the left to the right (one cannot enter a state more
often than one leaves it) which implies that
shortcut and interpretation: If assume the steady state probabilities exist,
can argue that on the average number of transitions from state 1 to state 0.6(1 − α) = 0.2α
2 must equal the average of transitions in the reverse direction. If further
assume that the average frequency of events is given by their probability ⇒
or α = 3/4.
P( in state 1 and go to state two ) = P( in state 2 and go to state 1) or
In both cases write equations for steady state behavior (if it holds) and
αp = (1 − α)p
solve them.

or α = 1/2.
In general this is a question of writing matrix equations and solving them
(Later will do this carefully) (but solutions do not always exist).

Stochastic Processes II: Markov Chains 14–15 Stochastic Processes II: Markov Chains 14–16
 EE 178 EE 178

n-step transition probabilities

rij (n + 1) = P (Xn+1 = j|X0 = i)
m states m
X
= P (Xn+1 = j and Xn = k|X0 = i)
k=1
• rij (n) = P ( state = j after n transitions, starting from state i)
Xm
= P (Xn+1 = j | Xn = k, X0 = i)P (Xn = k|X1 = i)
• Recursive formula, starting with rij (1) = pij : k=1
Xm
m
X = P (Xn+1 = j | Xn = k)P (Xn = k|X0 = i)
rij (n + 1) = rik (n)pkj ; 1 ≤ i, j ≤ m ? k=1
k=1 m
X
= pkj rik (n)
(Special case of a “Chapman-Kolmogorov” equation.) k=1

Note: this is just total probability:

Stochastic Processes II: Markov Chains 14–17 Stochastic Processes II: Markov Chains 14–18

EE 178 EE 178

• rij (n) can be viewed as the “nth power” of the matrix pij . i.e., 1 2
1 0.8 0.2
Define matrix P = {pij ; i = 1, . . . m, j = 1, . . . m} and define the matrices 2 0.6 0.4
R(n) = {rij (n); i = 1, . . . m, j = 1, . . . m}; n = 1, 2, . . . Then the rij (1)
recursion and initial value can be expressed as
1 2
R(1) = P 1 .76 .24
2 .72 .28
R(2) = R(1)P = P 2
rij (2)
R(3) = R(2)P = P 3
.. 1 2
1 .752 .248
R(n) = R(n − 1)P = P n 2 .744 .256
rij (3)
1 2
1 .7504 .2496
2 .7488 0.2512
rij (4)

Stochastic Processes II: Markov Chains 14–19 Stochastic Processes II: Markov Chains 14–20
 EE 178 EE 178

1 2 Spider-and-Fly example
1 .7501 .2499
2 .7498 .2502
1 2 3 4
rij (5) 1 1.0 0 0 0
2 0.3 0.4 0.3 0
As n → ∞, rij (n) converges to a limit which does not depend on the initial 3 0 0.3 0.4 0.3
state i. 4 0 0 0 1.0

Part of the theory of Markov chains is the demonstration of general pij

conditions under which this behavior occurs, i.e., rij (n) converges to a limit
which does not depend on the initial state
1 2 3 4
1 1.0 0 0 0
2 .42 .25 .24 .09
3 .09 .24 .25 .42
4 0 0 0 1.0

rij (2)

Stochastic Processes II: Markov Chains 14–21 Stochastic Processes II: Markov Chains 14–22

EE 178 EE 178

1 2 3 4 rij (∞)
1 1.0 0 0 0
2 .50 .17 .17 .16 Note that
3 .16 .17 .17 .50
4 0 0 0 1.0 • States 2 and 3 have steady-state probability 0
rij (3)
• The steady-state probabilities of states 1 and 4 depend on the initial
1 2 3 4 state.
1 1.0 0 0 0
2 .55 .12 .12 .21
3 .21 .12 .12 .55
4 0 0 0 1.0
rij (4)

1 2 3 4
1 1.0 0 0 0
2 2/3 0 0 1/3
3 1/3 0 0 2/3
4 0 0 0 1.0

Stochastic Processes II: Markov Chains 14–23 Stochastic Processes II: Markov Chains 14–24
 EE 178 EE 178

Recurrent and Transient States • Recurrence/transcience is determined by the arcs (transitions with
nonzero probability), not be actual values of probabilities.

• A recurrent class = set of accessible states A(i) from some recurrent

• Let A(i) be the set of states that are accessible from state i.
state i.

• State i is recurrent if starting from i, than any accessible state j must

Note: Suppose that i is recurrent and hence A(i) is a recurrent class. Then
be such that i is accessible from j, i.e., j is in A(i) iff i is in A(j).
if k ∈ A(i), k is also recurrent and A(k) = A(i).

• Note: if i is recurrent, then necessarily i ∈ A(i). Proof:

Suppose i is recurrent and k ∈ A(i). Then k is accessible from i and hence,

• A state is transient if it is not recurrent.
since i is recurrent, k can access i and hence, through i, any state in A(i).
Thus A(i) ⊂ A(k). Since i can access k and hence any state in A(k),
• Example: A(k) ⊂ A(i). Thus A(i) = A(k).

k must be recurrent since it can access any state in A(k) and hence any
state in A(i), which can access i (since i is recurrent), which in turn can
1 2 3
4 access k.

Note: Recurrent classes contain no transient states.

Stochastic Processes II: Markov Chains 14–25 Stochastic Processes II: Markov Chains 14–26

EE 178 EE 178

Note: Two recurrent classes must be identical or disjoint. Periodic Classes

proof: Suppose have A(i) and A(j) where i and j are distinct and each is A recurrent class is called periodic if the states can be grouped in d > 1
recurrent. subsets such that all transitions from one subset lead to the next subset.
1 2 3
If j ∈ A(i), then have seen that j is recurrent and A(j) = A(i).
1 0 1.0 0
If j 6∈ A(i), then no state in A(j) can be also be in A(i) so that 2 1.0 0 0
A(i) ∩ A(j) = ∅. To see this, assume the contrary. Suppose that l ∈ A(j) 3 0.5 0.5 0
is also in A(i). Then j can access l which in turn can be accessed by i. pij
Since i is recurrent, l can access i and hence j ∈ A(i), a contradiction.
3
Note: A Markov chain can be decomposed into one or more recurrent 0.5
0.5
classes plus possibly some transient states. 1

A recurrent state is accessible from all states in its class, but it is not
1 2
accessible from states in other recurrent classes.
1

n
1 if n is even
rii(n) =
0 if n is odd
Stochastic Processes II: Markov Chains 14–27 Stochastic Processes II: Markov Chains 14–28
 EE 178 EE 178

Steady-State Probabilities

S1 3
Steady-State Convergence Theorem
1

4 If there is only one recurrent class and it is not periodic, then rij (n) tends
2
S2
to a steady-state value πj that is independent of i:
5

lim rij (n) = πj , for all i

n→∞

Steady-state equations
S3
6
Take limit as n → ∞ of
m
X
rij (n + 1) = rik (n)pkj ; 1 ≤ i, j ≤ m
If the class is periodic, rii(n) never converges to a steady-state value as n k=1
increases.

get

Stochastic Processes II: Markov Chains 14–29 Stochastic Processes II: Markov Chains 14–30

EE 178 EE 178

m
X Computation of Steady State Probabilities
πj = πk pkj , j = 1, 2, . . . , m
k=1
m
X Solve the steady state probabilities
πj = 1
j=1 m
X
πj = πk pkj , j = 1, 2, . . . , m
k=1

m
X
πj = 1
j=1
by elimination or by computer.

Stochastic Processes II: Markov Chains 14–31 Stochastic Processes II: Markov Chains 14–32
 EE 178 EE 178

Example Model of class mobility Expected Relative Frequency Interpretation

0.1
0.7
0.45 0.25
Interpretation: Under suitable conditions should have the long-term
.45 1 2 3
0.45 expected frequency of visits to state j

0.05 0.5
Upper Middle Lower E[# of visits to j in n transitions]
0.05 πj = lim
n→∞ n

Reminder: “suitable conditions” above means that the steady state

convergence theorem holds.
π1 = 0.45π1 + 0.05π2 + 0.05π3 This is an interpretation of the steady state “balance” equations
π2 = 0.45π1 + 0.7π2 + 0.5π3
m
X
π3 = 0.1π1 + 0.25π2 + 0.45π3 πj = πk pkj , j = 1, 2, . . . , m
1 = π1 + π2 + π3 k=1

which followed from combining the steady-state convergence theorem with

the Chapman-Kolmogorov equation.

Stochastic Processes II: Markov Chains 14–33 Stochastic Processes II: Markov Chains 14–34

EE 178 EE 178

We proved the Chapman-Kolmogorov equation, we did not prove the steady- The left hand side can be written as
state convergence theorem. (Just stated it with some handwaving, B&T
m
X
sketch a proof. The usual careful proofs involve the limiting properties of
powers of stochastic matrices.) πj = πj pjk
k=1
This “interpretation” is actually a theorem, a variation on the law of large m
X
numbers relating probabilities to relative frequencies (to be proved in the = pjk πj
next lecture notes) k=1
m
X
Another “stability” result follows for transitions into and out of a fixed = pjj πj + pjk πj
state: k=1,k6=j

Rewrite the steady state equations as The right hand side can be written as

m m m
X m
X
X X
πk pkj = πj pjk , j = 1, 2, . . . , m πk pkj = pjj πj + πk pkj
k=1,k6=j k=1,k6=j k=1 k=1,k6=j

Pm and subtracting the pjj πj from both sides yields the desired form.
i.e., original form: πj = k=1 πk pkj , j = 1, 2, . . . , m

Stochastic Processes II: Markov Chains 14–35 Stochastic Processes II: Markov Chains 14–36
 EE 178 EE 178

1
Although we have not proved the connection between probabilities and
1
relative frequencies for Markov chains with one recurrent class that is not
π1 p1j πj pj 1
periodic, the result is very useful for quick derivations and will be used.
2 2

π2 p2j πj pj 2
j
.. ..
. .

πj pjm
m πm pmj m

E[ frequency of trans. into j] = E[ frequency of trans. out of j]

As when looking at frequency of visits to states, the frequency of transitions

between states is an interpretation of the steady state equations in terms
of relative frequencies. Here the interpretation is that πipij is the expected
frequency of transitions from state i into state j. As before, this is actually
a theorem, another variation on the law of large numbers.

Stochastic Processes II: Markov Chains 14–37 Stochastic Processes II: Markov Chains 14–38

EE 178 EE 178

Birth-death Markov Chains Here by “frequency interpretation” we mean specifically that the expected
frequency (under “suitable conditions”) of making a transition from state
i into state j is πipij , i.e., one small piece of the transition frequency
• State space= {1, 2, . . . , m} interpretation.

• Only three transitions possible: go left, go right, stay where you are The “balance equations” or limiting probabilities simplify greatly for birth-
death Markov chains because each state has only limited transitions possible.

If we interpret the probabilities of transitions, we must have as many

1 2
• Cut the chain between states i and i + 1
• Apply frequency intepretation:
πipi,i+1 = πi+1pi+1,i
transtitions going left at cut as we have going right, e.g.,
··· i i+1
···
If we go left through a cut, we will not be able to go left again through it
until we have gone right through it, there is no other route possible. Hence
at any time we can have at most a difference of 1 between the number of
left passages and right passages (and 1/n → 0)
This implies that πipi,i+1 = πi+1pi+1,i
In fact, you do not need the expected frequency interpretation to come to
this conclusion, it is just the easiest way to do so.

Stochastic Processes II: Markov Chains 14–39 Stochastic Processes II: Markov Chains 14–40
 EE 178 EE 178

To do this, recall the steady state (or balance) equation.

m
X
m
X πi = πk pki
πj = πk pkj , j = 1, 2, . . . , m k=1
k=1 = πi−1pi−1,i + πipii + πi+1pi+1,i
For state i=1 (the leftmost), this reduces to = πi−1pi−1,i + πi(1 − pi,i−1 − pi,i+1) + πi+1pi+1,i
= πipi,i−1 + πi(1 − pi,i−1 − pi,i+1) + πi+1p(i+1)i
π1 = π1p11 + π2p21
= πi(1 − pi,i+1) + πi+1pi+1,i

and hence since p11 = 1 − p12, this is

so that
πipi,i+1 = πi+1pi+1,i
π1p12 = π2p21
as claimed.

Next use mathematical induction: Assume the result is true for i − 1, i.e.,
that πi−1pi−1,i = πipi,i−1, and prove that it holds for i. (We know it holds
for i = 1, so this will complete the proof.)

Stochastic Processes II: Markov Chains 14–41 Stochastic Processes II: Markov Chains 14–42

EE 178 EE 178

Queueing Example
 
p
π0p = π1q ⇒ π1 = π0
q
Packets arrive at a communication node with storage capacity m. Time is    2
discretized in very small periods. At each period: p p
π1p = π2q ⇒ π2 = π1 = π0
q q
   3
p p
• Prob. of 1 arrival = p π2p = π3q ⇒ π3 = π2 = π0
q q
..
• Prob. of 1 transmission = q    m
p p
πm−1p = πmq ⇒ πm = πm−1 = π0
q q
• Prob. of 0 arrivals and 0 transmissions = 1 − p − q

1−p 1−p−q 1−p−q 1−q or defining ρ = p/q

πi = ρiπ0
p p p

0 1 ··· m−1 m Since π0 + π1 + · · · + πm = 1,

q q q q
π0(1 + ρ + ρ2 + · · · + ρm) = 1

Stochastic Processes II: Markov Chains 14–43 Stochastic Processes II: Markov Chains 14–44
 EE 178

hence for i = 1, . . . , m

ρi
πi =
1 + ρ + + · · · + ρm
ρ2

Using geometric progression formula,

1−ρ ρi
πi = ρi →m→∞
1 − ρm+1 1−ρ
if ρ < 1

Stochastic Processes II: Markov Chains 14–45