Lecture 11 Stochastic

Processes
Topics
Definitions
Review of probability
Realization of a stochastic process
Continuous vs. discrete systems
Examples
 Basic Definitions

Stochastic process: System that changes over time in

an uncertain manner
Examples
Automated teller machine (ATM)
Printed circuit board assembly operation
Runway activity at airport
State: Snapshot of the system at some fixed point in
time
Transition: Movement from one state to another
 Elements of Probability Theory
Experiment: Any situation where the outcome is uncertain.
Sample Space, S: All possible outcomes of an experiment (we
will call them the state space).

Event: Any collection of outcomes (points) in the sample

space. A collection of events E1, E2,,En is said to be
mutually exclusiveif EiEj =for all i j = 1,,n.
Random Variable (RV): Function or procedure that
assigns a real number to each outcome in the sample
space.
Cumulative Distribution Function (CDF), F(): Probability
distribution function for the random variable X such that
F(a) Pr{X a}
 Components of Stochastic Model
Time: Either continuous or discrete parameter.
t0 t1 t2 t3 t4 time

State: Describes the attributes of a system at some point in time.

s = (s1, s2, . . . , sv); for ATM example s = (n)

Convenient to assign a unique nonnegative integer index

to each possible value of the state vector. We call this X
and require that for each s X.
For ATM example, X = n.
In general, Xt is a random variable.
 Model Components (continued)
Activity: Takes some amount of time duration.
Culminates in an event.
For ATM example service completion.
Transition: Caused by an event and results in movement
from one state to another. For ATM example,
a a a a

0 1 2 3

d d d d

# = state, a = arrival, d = departure

Stochastic Process: A collection of random variables {Xt},

where t T = {0, 1, 2, . . .}.
 Realization of the Process
Deterministic Process
Time between Pr{ ta } = 0, < 1 min Arrivals occur
arrivals every minute.
= 1, 1 min

Time for Pr{ ts } = 0, < 0.75 min Processing takes

servicing exactly 0.75
customer = 1, 0.75 min minutes.

n
2
Number in
system, n
1
(no transient
0
0 1 2 3 4 5 6 7 8 9 10 time
response)
Realization of the Process (continued)
Stochastic Process

Time for servicing a Pr{ ts } = 0, < 0.75 min

customer
= 0.6, 0.75 1.5 min

= 1, 1.5 min

n
a a a
3

Number in
system, n
a a a a d a a d d
2

a d d d d d d
1

d
0
0 2 4 6 8 10 12 time
 Markovian Property
Given that the present state is known, the conditional probability of
the next state is independent of the states prior to the present state.

Present state at time t is i: Xt = i

Next state at time t + 1 is j: Xt+1 = j
Conditional Probability Statement of Markovian Property:
Pr{Xt+1 = j | X0 = k0, X1 = k1,, Xt = i } = Pr{Xt+1 = j | Xt = i }

for t = 0, 1,, and all possible sequences i, j, k0, k1, . . . , kt1

Interpretation: Given the present, the past is irrelevant in

determining the future.
 Transitions for Markov Processes
State space: S = {1, 2, . . . , m}
Probability of going from state i to state j in one move: pij

State-transition matrix
1 p11 p12 L p1m

2 p21

p22 L p2m
P =
M M M M

m pm1 pm2 L pmm

P = pij

Theoretical requirements: 0 pij 1, j pij = 1, i = 1,,m

 Discrete-Time Markov Chain
A discrete state space
Markovian property for transitions
One-step transition probabilities, pij, remain constant over
time (stationary)
Simple Example
State-transition matrix State-transition diagram
(0.6)
0 1 2
0
0 0.6 0.3 0.1 (0.1) (0.3)

P= 1 0.8 0.2 0 (1) (0.8)

2 1
2 1 0 0
(0.2)
 Game of Craps
Roll 2 dice
Outcomes
Win = 7 or 11
Loose = 2, 3, 12
Point = 4, 5, 6, 8, 9, 10
If point, then roll again.
Win if point
Loose if 7
Otherwise roll again, and so on

(There are other possible bets not included here.)

State-Transition Network for Craps
not(4,7) not(5,7) not(6,7) not(8,7) not(9,7) not(10,7)

P4 P5 P6 P8 P9 P10

6 8
5
4
9 7 7 7
7
5 6 8 9 7
10 7
Win 4 10 Lose
(7,11) Start (2,3,12)
 Transition Matrix for Game of Craps
Sum 2 3 4 5 6 7 8 9 10 11 12

Prob. 0.028 0.056 0.083 0.111 0.139 0.167 0.139 0.111 0.083 0.056 0.028

Probability of win = Pr{ 7 or 11 } = 0.167 + 0.056 = 0.223

Probability of loss = Pr{ 2, 3, 12 } = 0.028 + 0.056 + 0.028 = 0.112

Start Win Lose P4 P5 P6 P8 P9 P10

Start 0 0.222 0.111 0.083 0.111 0.139 0.139 0.111 0.083
Win 0 1 0 0 0 0 0 0 0
Lose 0 0 1 0 0 0 0 0 0
P4 0 0.083 0.167 0.75 0 0 0 0 0
P= P5 0 0.111 0.167 0 0.722 0 0 0 0
P6 0 0.139 0.167 0 0 0.694 0 0 0
P8 0 0.139 0.167 0 0 0 0.694 0 0
P9 0 0.111 0.167 0 0 0 0 0.722 0
P10 0 0.083 0.167 0 0 0 0 0 0.75

 Examples of Stochastic Processes

Single stage assembly process with single worker, no queue
a
State = 0, worker is idle
0 1

State = 1, worker is busy d

Multistage assembly process with single worker, no queue

a

State = 0, worker is idle

State = k, worker is performing operation k = 1, . . . , 5
 Examples (continued)
Multistage assembly process with single worker and queue
(Assume 3 stages only; i.e., 3 operations)
s1 number of parts in the system
s = (s1, s2) where
s2 current operation being performed

(1,3)
a
(2,3)
a
(3,3)
Operations d3 d3 d3
d2 d2 d2
k = 1, 2, 3
(1,2)
a
(2,2) (3,2)
a

d1 d1 d1

(0,0)
a
(1,1) (2,1)
a
(3,1)
a
Single Stage Process with Two Servers and Queue

0, if server i is idle
s = (s1, s2 , s3) where si i = 1, 2
1, if server i is busy
s3 number in the queue

d1
1
Arrivals

d2
2

(1,0,0)

State- d1 1 a
transition d2
d1 , d2 d1 ,d2
a
network 0
d1
3 4 5
a a
(0,0,0) (1,1,0) (1,1,1) (1,1,2)
d2 2 a

(0,1,0)
 Series System with No Queues
Arrivals Transfer Transfer Finished
1 2 3

Component Notation Definition

State s = (s1, s2 , s3)
0, if server i is idle
si
1, if server i is busy
for i 1, 2,3
State space S = { (0,0,0), (1,0,0), . . . , The state space consists of all
(0,1,1), (1,1,1) } possible binary vectors of 3
components.

Events Y = {a, d1, d2 , d3} a = arrival at operation 1

dj = completion of operation j
for j = 1, 2, 3
 What You Should Know
About Stochastic Processes
Definition of a state and an event.
Meaning of realization of a system
(stationary vs. transient).
Definition of the state-transition matrix.
How to draw a state-transition network.
Difference between a continuous and
discrete-time system.
Common applications with multiple stages
and servers.