Tutorial On Markov Chains For Power Engineering

Industrial
IndustrialEngineering
Engineeringconcepts
conceptsfor
forPower
PowerEngineers
Engineers
Tutorial on Markov Chains for Power

Engineering
Esma S. Gel and Norma F. Hubele

Department of Industrial Engineering,
Arizona State University
Arizona
ArizonaState
StateUniversity
University 1
Industrial
Engineeringconcepts
conceptsfor
forPower
PowerEngineers
Engineers
Stochastic Processes
! A stochastic process is an indexed collection of

random variables {Xt}, where the index t runs through
a given set T
! Stochastic processes are of interest for describing the

behavior of a system operating over some period of
time
Arizona
ArizonaState
StateUniversity
University 2
Industrial
Engineeringconcepts
conceptsfor
forPower
PowerEngineers
Engineers
Stochastic Processes
! Discrete time stochastic process with finite state

space
X t ∈ {0,1,2,K , M }
– The system is observed at particular points in time, t=0, 1, 2, …
– The random variable Xt represents the state of the system at

time t
– Xt ={X0 , X1 , X2 , …} shows how the system evolves over time
Arizona
ArizonaState
StateUniversity
University 3
Industrial
Engineeringconcepts
conceptsfor
forPower
PowerEngineers
Engineers
Markov Chains
! A stochastic process {Xt} is said to have the Markovian
property if
P{ X t +1 = j | X 0 = k0 , X 1 = k1 ,K, X t −1 = kt −1 , X t = i}
= P{ X t +1 = j | X t = i}
for t = 0,1,K, and every sequence i, j , k0 , k1 ,K , kt −1 .
! The conditional probabilities

P{ X t +1 = j | X t = i}
are called one-step transition probabilities
Arizona
ArizonaState
StateUniversity
University 4
Industrial
Engineeringconcepts
conceptsfor
forPower
PowerEngineers
Engineers
Transition Probabilities
! If for each i and j
P{ X t +1 = j | X t = i} = P{ X 1 = j | X 0 = i}
then the one-step transition probabilities are said to be
stationary.
! That is, the transition probabilities do not change over
time.
! If stationary, then
P{ X t + n = j | X t = i} = P{ X n = j | X 0 = i}
for all t = 0,1, K
These are called n-step transition probabilities.
Arizona
ArizonaState
StateUniversity
University 5
Industrial
Engineeringconcepts
conceptsfor
forPower
PowerEngineers
Engineers
Transition Probabilities
! The n-step transition probability
p ij( n ) = P { X t + n = j | X t = i}
is just the conditional probability that the system will be
in state j after exactly n steps, given that it starts in
state i at any time t.
! Properties
pij( n ) ≥ 0 for all i, j; n = 0,1,2, K
and
M
∑ ij = 1 for all i; n = 0,1,2,K

p (n)
j =0
Arizona
ArizonaState
StateUniversity
University 6
Industrial
Engineeringconcepts
conceptsfor
forPower
PowerEngineers
Engineers
Transition Probability Matrix
! The n-step transition probabilities can be expressed in

matrix form
 p00
(n)
p (n)
01 L p (n)
0M

 (n) (n) (n)
 p10 p L p 
P (n) = 11 1M
 M M M M 
 (n) (n) 
 pM 0 pM( n1) L pMM 
Arizona
ArizonaState
StateUniversity
University 7
Industrial
Engineeringconcepts
conceptsfor
forPower
PowerEngineers
Engineers
Chapman-Kolmogorov Equations
! Chapman-Kolmogorov equations provide a method for

computing the n-step transition probabilities
M
pij( n ) = ∑ pik( m ) pkj( n − m )
k =0
for all i = 0,1, K, M ;

j = 0,1, K , M ,
and any m = 1,2 ,K ,n − 1;
n = m + 1, m + 2, K
Arizona
ArizonaState
StateUniversity
University 8
Industrial
Engineeringconcepts
conceptsfor
forPower
PowerEngineers
Engineers
Chapman-Kolmogorov Equations
! The special cases of m=1 and m=n -1 yield
M
p (n)
ij = ∑ pik pkj( n −1) and
k =0
M
p (n)
ij = ∑ pikn −1 pkj for all states i and j.
k =0
! Then, n-step transition probabilities can be calculated

recursively
P (2) = P ⋅ P = P 2
P (n) = PP ( n −1) = P ( n −1) P
= PP n −1 = P n −1P = P n
Arizona
ArizonaState
StateUniversity
University 9
Industrial
Engineeringconcepts
conceptsfor
forPower
PowerEngineers
Engineers
Classification of States
! State j is said to be accessible from state i if
p ( n )
ij > 0 for some n ≥ 0
! If state j is accessible from state i and state i is

accessible from state j, then states i and j are said to
communicate.
! In general,
– Any state communicates with itself
– If state i communicates with state j, then state j communicates
with state i .
– If state i communicates with state j and state j communicates
with state k, then state i communicates with state k.
Arizona
ArizonaState
StateUniversity
University 10
Industrial
Engineeringconcepts
conceptsfor
forPower
PowerEngineers
Engineers
Recurrent vs. Transient States

! A state is said to be a transient state if, upon entering
this state, the process may never return to this state
again.
– A state i is transient iff there exists a state j (j ≠ i) that is
accessible from state i but not vice versa, that is, state i is not
accessible from state j.
! A state is said to be recurrent if, upon entering this

state, the process definitely will return to this state
again.
– A state is transient iff it is not transient.
Arizona
ArizonaState
StateUniversity
University 11
Industrial
Engineeringconcepts
conceptsfor
forPower
PowerEngineers
Engineers
Absorbing States
! A state is said to be absorbing if, upon entering this

state, the process never will leave this state again.
– State i is an absorbing state iff pii =1.
Arizona
ArizonaState
StateUniversity
University 12
Industrial
Engineeringconcepts
conceptsfor
forPower
PowerEngineers
Engineers
Irreducible MC & Periodicities
! All states in an irreducible finite state Markov chain

are recurrent.
! The period of state i is defined to be the integer t

(t >1) such that
pii( n ) = 0
for all values of n other than t, 2t, 3t,… and t is the

largest integer with this property.
Arizona
ArizonaState
StateUniversity
University 13
Industrial
Engineeringconcepts
conceptsfor
forPower
PowerEngineers
Engineers
Irreducible MC & Periodicities
! If there are two consecutive numbers s and s+1 such

that the process can be in state i at times s and s+1,
the state is said to have period 1 and is called an
aperiodic state.
! In a finite state MC, recurrent states that are aperiodic

are called ergodic.
Arizona
ArizonaState
StateUniversity
University 14
Industrial
Engineeringconcepts
conceptsfor
forPower
PowerEngineers
Engineers
Long-run Properties of MC
! There is a limiting probability that the system will be in

each state j after a large number of transitions, and this
probability is independent of the initial state.
Arizona
ArizonaState
StateUniversity
University 15
Industrial
Engineeringconcepts
conceptsfor
forPower
PowerEngineers
Engineers
Steady State Probabilities

(n)
! For any irreducible ergodic Markov chain, lim p ij
n →∞
exists and is independent of i. Furthermore,
lim pij( n ) = π j > 0,

n →∞
where the π j uniquely satisfy the following steady state equations

M
π j = ∑ π i pij , for j = 0,1, K , M ,
i =0
M
∑π
j =0
j = 1.
Arizona
ArizonaState
StateUniversity
University 16
Industrial
Engineeringconcepts
conceptsfor
forPower
PowerEngineers
Engineers
Steady State Probabilities

! πj represents the probability of finding the process in a
certain state, say j, after a large number of transitions
! The steady state equations consist of M+2 equations in

M+1 unknowns.
– One of the equations is redundant
! If i and j are recurrent states belonging to different

classes, then
pij( n ) = 0, for all n.
! Similarly, if j is a transient state, then
lim pij( n ) = 0 for all i.
n →∞
Arizona
ArizonaState
StateUniversity
University 17
Industrial
Engineeringconcepts
conceptsfor
forPower
PowerEngineers
Engineers
Expected Average Cost/Time

! If the states are not aperiodic, then the limit lim pij( n )
n →∞
may not exist.
! The following limit always exists
 1 n (k ) 
lim ∑ pij  = π j
n →∞ n
 k =1 
! This helps us calculate the long run average cost per

unit time associates with a Markov chain.
Arizona
ArizonaState
StateUniversity
University 18
Industrial
Engineeringconcepts
conceptsfor
forPower
PowerEngineers
Engineers

! Suppose that a cost C(Xt) is incurred when the process
is in state Xt at time t.
! The expected average cost incurred over the first n
periods is given by
1 n 
E  ∑ C ( X t )
 n k =1 
! Since we know that
 1 n (k ) 
lim ∑ pij  = π j
n →∞ n
 k =1 
we can show that
1 n  M
lim E  ∑ C ( X t )  = ∑ π j C ( j ).
n →∞
 n t =1  j =0
Arizona
ArizonaState
StateUniversity
University 19
Industrial
Engineeringconcepts
conceptsfor
forPower
PowerEngineers
Engineers

! What if the cost function depends on other random
variable(s), in addition to the state of the system?
! Then, we have a similar relationship:
1 n  M
lim E  ∑ C ( X t , Dt + m )  = ∑ π j k ( j ),
n →∞
 n t =1  j =0
where k ( j ) = E[C ( j , Dt + m )]
Conditional expectation
with respect to Dt + m ,
given the state j
Arizona
ArizonaState
StateUniversity
University 20
Industrial
Engineeringconcepts
conceptsfor
forPower
PowerEngineers
Engineers
First Passage Times

! First passage time is the number of transitions made
in going from state i to state j for the first time.
! Called recurrence time for j=i.
! The first passage times are random.
! Let
f ij( n ) := the probability that the FPT is equal to n
Arizona
ArizonaState
StateUniversity
University 21
Industrial
Engineeringconcepts
conceptsfor
forPower
PowerEngineers
Engineers
First Passage Times

! In general the first passage time distribution has to
satisfy the following recursive relationships:
f (1)
ij =p (1)
ij = pij ,
f ij( 2 ) = ∑ pik f kj(1) ,
k≠ j
f ij( n ) = ∑ pik f kj( n −1) .

k≠ j
! Then, we can calculate the first passage time

distribution recursively from one-step transition
probabilities.
Arizona
ArizonaState
StateUniversity
University 22
Industrial
Engineeringconcepts
conceptsfor
forPower
PowerEngineers
Engineers
Expected First Passage Times

! May be much simpler to calculate the expected first
passage times
 ∞


∞ if ∑
n =1
f (n)
ij <1
µ ij =  ∞ ∞
 nf ( n ) if
∑ ∑
ij f (n)
ij =1
n =1 n =1
∞
! Whenever
∑
n =1
f ij( n ) = 1
µij uniquely satisfies the equation µ ij = 1 + ∑ pik µ kj

k≠ j
Arizona
ArizonaState
StateUniversity
University 23
Industrial
Engineeringconcepts
conceptsfor
forPower
PowerEngineers
Engineers
Absorbing States
! State k is ab absorbing state if pkk=1.
! The probability of absorption is the probability of

ever going into the absorbing state.
! Absorption probabilities satisfy

M
f ik = ∑ pij f jk , for i = 0,1,K , M .
j =0
Subject to
f kk = 1 and f ik = 0
if state i is recurrent and i ≠ k .
Arizona
ArizonaState
StateUniversity
University 24
Industrial
Engineeringconcepts
conceptsfor
forPower
PowerEngineers
Engineers
Random Walks
! A random walk is a Markov chain with the property
that if the system is in state i, then in a single transition
the system either remains at i or moves to one of the
states immediately adjacent to i.
Arizona
ArizonaState
StateUniversity
University 25
Industrial
Engineeringconcepts
conceptsfor
forPower
PowerEngineers
Engineers
Continuous Time MC
! In some systems, the evolution of process is observed
continuously over time.
! Let X (t ' ) := the state of the system at time t '

! A continuous time stochastic process has the
Markovian property if
P{ X (t + s ) = j | X ( s ) = i and X (r ) = x(r )}
= P{ X (t + s ) = j | X ( s ) = i},
for all i, j = 0,1, K , M and
for all r ≥ 0, s > r , and t > 0.
Arizona
ArizonaState
StateUniversity
University 26
Industrial
Engineeringconcepts
conceptsfor
forPower
PowerEngineers
Engineers
A Markov Chain in Genetics

Consider a large population of individuals each of
whom possesses a particular pair of genes, of which
each individual gene is classified as being of type A or
type a. Assume that the percentages of individuals
whose gene pairs are AA, aa, or Aa are respectively
p0, q0, and r0 (p0+q0+r0=1). When two individuals mate,
each contributes one of his/her genes, chosen at
random, to the resultant offspring. Assuming that the
mating occurs at random, in that each individual is
equally likely to mate with any other individual, we are
interested in determining the percentages of individuals
in the next generation whose genes are AA, aa, or Aa.
Arizona
ArizonaState
StateUniversity
University 27
Industrial
Engineeringconcepts
conceptsfor
forPower
PowerEngineers
Engineers
Homer’s Mood Example

! On any given day Homer is either cheerful (C), so-so
(S), or glum (G).
– If he is cheerful today, then he will be C, S, or G tomorrow with
respective probabilities 0.5, 0.4, 0.1.
– If he is so-so today, then he will be C, S, or G tomorrow with
– If he is glum today, then he will be C, S, or G tomorrow with
Model Homer’s mood as a Markov chain and find the

long-run fraction of time that he is cheerful, so-so and
glum.
Arizona
ArizonaState
StateUniversity
University 28
Industrial
Engineeringconcepts
conceptsfor
forPower
PowerEngineers
Engineers
Ball Drawing Example

Three white and three black balls are distributed in two
urns in such a way that each contains three balls. We
say that the system is in state i, i=0,1,2,3, if the first urn
contains i white balls. At each step, we draw one ball
from each urn and place the ball drawn from the first
urn into the second, and conversely with the ball drawn
from the second urn.
Let Xn denote the state of the system after the nth step.
! Model the stochastic process {Xn,n≥0} as a Markov

chain.
Arizona
ArizonaState
StateUniversity
University 29

Tutorial On Markov Chains For Power Engineering

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Tutorial On Markov Chains For Power Engineering

Uploaded by

Copyright:

Available Formats

Industrial

Tutorial on Markov Chains for Power

Esma S. Gel and Norma F. Hubele

! A stochastic process is an indexed collection of

! Stochastic processes are of interest for describing the

! Discrete time stochastic process with finite state

– The system is observed at particular points in time, t=0, 1, 2, …

– The random variable Xt represents the state of the system at

– Xt ={X0 , X1 , X2 , …} shows how the system evolves over time

! The conditional probabilities

are called one-step transition probabilities

∑ ij = 1 for all i; n = 0,1,2,K

Transition Probability Matrix

! The n-step transition probabilities can be expressed in

! Chapman-Kolmogorov equations provide a method for

for all i = 0,1, K, M ;

! Then, n-step transition probabilities can be calculated

! If state j is accessible from state i and state i is

Recurrent vs. Transient States

! A state is said to be recurrent if, upon entering this

! A state is said to be absorbing if, upon entering this

Irreducible MC & Periodicities

! All states in an irreducible finite state Markov chain

! The period of state i is defined to be the integer t

for all values of n other than t, 2t, 3t,… and t is the

Irreducible MC & Periodicities

! If there are two consecutive numbers s and s+1 such

! In a finite state MC, recurrent states that are aperiodic

! There is a limiting probability that the system will be in

Steady State Probabilities

lim pij( n ) = π j > 0,

where the π j uniquely satisfy the following steady state equations

Steady State Probabilities

! The steady state equations consist of M+2 equations in

! If i and j are recurrent states belonging to different

Expected Average Cost/Time

! The following limit always exists

! This helps us calculate the long run average cost per

Expected Average Cost/Time

Expected Average Cost/Time

! Then, we have a similar relationship:

First Passage Times

! Called recurrence time for j=i.

! The first passage times are random.

First Passage Times

f ij( n ) = ∑ pik f kj( n −1) .

! Then, we can calculate the first passage time

Expected First Passage Times

µij uniquely satisfies the equation µ ij = 1 + ∑ pik µ kj

! The probability of absorption is the probability of

! Absorption probabilities satisfy

! Let X (t ' ) := the state of the system at time t '

A Markov Chain in Genetics

Homer’s Mood Example

Model Homer’s mood as a Markov chain and find the

Ball Drawing Example

! Model the stochastic process {Xn,n≥0} as a Markov

You might also like