6.

Stochastic processes (2)

Lect06.ppt S-38.145 - Introduction to Teletraffic Theory – Spring 2005 1

6. Stochastic processes (2)

Contents

• Markov processes
• Birth-death processes

2
6. Stochastic processes (2)

Markov process

• Consider a continuous-time and discrete-state stochastic process X(t)

– with state space S = {0,1,…,N} or S = {0,1,...}
• Definition: The process X(t) is a Markov process if

•
P{ X (t n +1 ) = xn +1 | X (t1 ) = x1, K, X (tn ) = xn } =
P{ X (t n +1 ) = xn +1 | X (tn ) = xn }

for all n, t1< … < tn+1 and x1,…, xn +1

• This is called the Markov property
– Given the current state, the future of the process does not depend on its past
(that is, how the process has evolved to the current state)
– As regards the future of the process, the current state contains all the
required information
3
6. Stochastic processes (2)

Example

• Process X(t) with independent increments is always a Markov process:

X (tn ) = X (tn −1 ) + ( X (tn ) − X (t n −1 ))

• Consequence: Poisson process A(t) is a Markov process:

– according to Definition 3, the increments of a Poisson process are
independent

4
6. Stochastic processes (2)

Time-homogeneity

• Definition: Markov process X(t) is time-homogeneous if

P{ X (t + ∆ ) = y | X (t ) = x} = P{ X (∆ ) = y | X (0) = x}
for all t, ∆ ≥ 0 and x, y ∈ S
– In other words,
probabilities P{X(t + ∆) = y | X(t) = x} are independent of t

5
6. Stochastic processes (2)

State transition rates

• Consider a time-homogeneous Markov process X(t)

• The state transition rates qij, where i, j ∈ S, are defined as follows:

qij := lim 1h P{ X ( h) = j | X (0) = i}

h↓0
• The initial distribution P{X(0) = i}, i ∈ S, and the state transition rates
qij together determine the state probabilities P{X(t) = i}, i ∈ S, by the
Kolmogorov (backwards/forwards) equations
• Note that on this course we will consider only time-homogeneous
Markov processes

6
6. Stochastic processes (2)

Exponential holding times

• Assume that a Markov process is in state i

• During a short time interval (t, t+h] , the conditional probability that there
is a transition from state i to state j is qijh + o(h) (independently of the
other time intervals)
• Let qi denote the total transition rate out of state i, that is:

qi := ∑ qij
j ≠i
• Then, during a short time interval (t, t+h] , the conditional probability that
there is a transition from state i to any other state is qih + o(h)
(independently of the other time intervals)
• This is clearly a memoryless property
• Thus, the holding time in (any) state i is exponentially distributed with
intensity qi
7
6. Stochastic processes (2)

State transition probabilities

• Let Ti denote the holding time in state i and Tij denote the (potential)
holding time in state i that ends to a transition to state j

Ti ∼ Exp(qi ), Tij ∼ Exp(qij )

• Ti can be seen as the minimum of independent and exponentially
distributed holding times Tij (see lecture 5, slide 44)
Ti = min Tij
j ≠i

• Let then pij denote the conditional probability that, when in state i, there
is a transition from state i to state j (the state transition probabilities);
qij
pij = P{Ti = Tij } =
qi
8
6. Stochastic processes (2)

State transition diagram

• A time-homogeneous Markov process can be represented by a state

transition diagram, which is a directed graph where
– nodes correspond to states and
– one-way links correspond to potential state transitions
link from state i to state j ⇔ qij > 0
• Example: Markov process with three states, S = {0,1,2}

0
− + 0
  q20 q01
Q =  0 − +
q21
 + + − 2 1
  q12

9
6. Stochastic processes (2)

Irreducibility

• Definition: There is a path from state i to state j (i → j) if there is a

directed path from state i to state j in the state transition diagram.
– In this case, starting from state i, the process visits state j with positive
probability (sometimes in the future)
• Definition: States i and j communicate (i ↔ j) if i → j and j → i.
• Definition: Markov process is irreducible if all states i ∈ S
communicate with each other
– Example: The Markov process presented in the previous slide is irreducible

10
6. Stochastic processes (2)

Global balance equations and equilibrium distributions

• Consider an irreducible Markov process X(t), with state transition rates qij
• Definition: Let π = (πi | πi ≥ 0, i ∈ S) be a distribution defined on the
state space S, that is:
∑i∈S π i = 1 (N)
It is the equilibrium distribution of the process if the following global
balance equations (GBE) are satisfied for each i ∈ S:

∑ j ≠i π i qij = ∑ j ≠i π j q ji (GBE)
– It is possible that no equilibrium distribution exists, but if the state space is
finite, a unique equilibrium distribution does exist
– By choosing the equilibrium distribution (if it exists) as the initial distribution,
the Markov process X(t) becomes stationary (with stationary distribution π)

11
6. Stochastic processes (2)

Example

− 1 0 0
  1 1
Q = 0 − 1
µ
1 µ − 
 2
1
1

π 0 + π1 + π 2 = 1 (N)

π 0 ⋅1 = π 2 ⋅1
π 1 ⋅1 = π 0 ⋅1 + π 2 ⋅ µ (GBE)
π 2 ⋅ (1 + µ ) = π 1 ⋅1
1+ µ
⇒ π 0 = 3+1µ , π 1 = 3+ µ , π 2 = 3+1µ
12
6. Stochastic processes (2)

Local balance equations

• Consider still an irreducible Markov process X(t).with state transition rates qij
• Proposition: Let π = (πi | πi ≥ 0, i ∈ S) be a distribution defined on the state
space S, that is:

• ∑i∈S π i = 1 (N)
If the following local balance equations (LBE) are satisfied for each i,j ∈ S:

π i qij = π j q ji (LBE)
•
then π is the equilibrium distribution of the process.
• Proof: (GBE) follows from (LBE) by summing over all j ≠ i
• In this case the Markov process X(t) is called reversible (looking stochastically
the same in either direction of time)

13
6. Stochastic processes (2)

Contents

• Markov processes
• Birth-death processes

14
6. Stochastic processes (2)

Birth-death process

• Consider a continuous-time and discrete-state Markov process X(t)

– with state space S = {0,1,…,N} or S = {0,1,...}
• Definition: The process X(t) is a birth-death process (BD) if state
transitions are possible only between neighbouring states, that is:
| i − j |> 1 ⇒ qij = 0
• In this case, we denote
µi := qi,i −1 ≥ 0
λi := qi ,i +1 ≥ 0
– In particular, we define µ0 = 0 and λN = 0 (if N < ∞)

15
6. Stochastic processes (2)

Irreducibility

• Proposition: A birth-death process is irreducible if and only if

λi > 0 for all i ∈ S\{N} and µi > 0 for all i ∈ S\{0}

• State transition diagram of an infinite-state irreducible BD process:

λ0 λ1 λ2
0 1 2
µ1 µ2 µ3

• State transition diagram of a finite-state irreducible BD process:

λ0 λ1 λN−2 λN−1
0 1 N-1 N
µ1 µ2 µN−1 µN

16
6. Stochastic processes (2)

Equilibrium distribution (1)

• Consider an irreducible birth-death process X(t)

• We aim is to derive the equilibrium distribution π = (πi | i ∈ S) (if it
exists)
• Local balance equations (LBE):

π i λi = π i +1µ i +1 (LBE)
• Thus we get the following recursive formula:
λi i λ
j −1
π i +1 = π
µ i +1 i
⇒ πi = π0 ∏ µ
j =1 j
• Normalizing condition (N):
i λ
j −1
∑ i 0 ∑ ∏ µj
π = π =1 (N)
i∈S i∈S j =1 17
17
6. Stochastic processes (2)

Equilibrium distribution (2)

• Thus, the equilibrium distribution exists if and only if

i λ
j −1
∑ ∏ µj <∞
i∈S j =1
• Finite state space:
The sum above is always finite, and the equilibrium distribution is
−1
i λ
j −1  N i λ j −1 
πi = π0 ∏ µ , π 0 = 1 + ∑ ∏ µ 
j =1 j
 i =1 j =1 j 
 
• Infinite state space:
If the sum above is finite, the equilibrium distribution is
−1
i λ
j −1  ∞ i λ j −1 
πi = π0 ∏ µ , π 0 = 1 + ∑ ∏ µ 
j =1 j
 i =1 j =1 j  18
18
 
6. Stochastic processes (2)

Example

− λ 0
  λ λ
Q = µ − λ 0 1 2
µ µ
0 µ − 

π i λ = π i +1µ
⇒ π i +1 = ρπ i ( ρ := λ / µ ) (LBE)
⇒ π i = π 0ρi
π 0 + π 1 + π 2 = π 0 (1 + ρ + ρ 2 ) = 1 (N)
ρi
⇒ πi =
1+ ρ + ρ 2 19
19
6. Stochastic processes (2)

Pure birth process

• Definition: A birth-death process is a pure birth process if

µi = 0 for all i ∈ S
• State transition diagram of an infinite-state pure birth process:
λ0 λ1 λ2
0 1 2

• State transition diagram of a finite-state pure birth process:

λ0 λ1 λN−2 λN−1
0 1 N-1 N

• Example: Poisson process is a pure birth process (with constant birth

rate λi = λ for all i ∈ S = {0,1,…})
• Note: Pure birth process is never irreducible (nor stationary)!

20