ISYE 3232C Spring 2015 Midterm 2 Solution

Total # of
Exam Takers
61

Min

Max

Mean

Median

44

124

94.34

96

16

25%
quantile
80.75

75 %
quantile
109.875

std

CV

18.89

Total

14
12
10
8
6
4
2
0
40

50

60

70

80

90

100

110

120

130

1. Performance (88.65%):
35

Q1

30
25
20
15
10
5
0
4

10

12

14

16

18

20

22

Question and Solution:

(20) Let X0 , X1 , . . . be a Markov chain with state space {0, 1, 2}, initial distribution a =
( , 0.4, 0.5), and transition matrix

.4
.4
0.34 0.26 0.40
.2 , P2 = 0.33 0.35 0.32
P = .3
0.33 0.26 0.41
.2 .5
(a) (2) Fill in the entries for P and a (0.5 for each).
a = (0.1, 0.4, 0.5)

.4 .2 .4
P = .3 .5 .2
.3 .2 .5
(b) (1) P(X9 = 2 | X8 = 2) = 0.5
(c) (4) P(X8 = 2, X10 = 1 | X7 = 0)
P(X8 = 2, X10 = 1 | X7 = 0) = P(X10 = 1 | X8 = 2)P(X8 = 2 | X7 = 0))
= 0.26 0.4 = 0.104

(2)
(2)

(d) (4)P(X1 = 0)
2

P(X1 = 0) = P(X1 = 0 | X0 = i)P(X0 = i)

(2)

i=0

= 0.4 0.1 + 0.3 0.4 + 0.3 0.5 = 0.31

(2)

(e) (4)P(X0 = 2 | X1 = 2)
P(X0 = 2, X1 = 2) P(X1 = 2 | X0 = 2)P(X0 = 2)
=
P(X1 = 2)
P(X1 = 2)
0.5 0.5
= 0.676
=
0.4 0.1 + 0.2 0.4 + 0.5 0.5

P(X0 = 2 | X1 = 2) =

P(X1 = 2) = 0.4 0.1 + 0.2 0.4 + 0.5 0.5 = 0.37

(f) (5)E[X2 | X0 = 2]
E[X2 | X0 = 2] = 0.26 + 2 0.41 = 1.08

(5)

(1)
(1)
(1)
(1)

2. Performance (88.20%):
35

Q2
30
25
20
15
10
5
0
8

10

11

12

13

14

15

16

17

18

19

20

Question and Solution:

(20) Let (Xn )n1 be a Markov chain with state space {1, 2, 3, 4, 5} with transition matrix

0.25 0.75 0
0
0
0.5 0.5
0
0
0

0.25 0 0.75 0
P= 0
.
0
0
0.5
0
0.5
0
0
0
0
1
(a) (3) Draw transition diagram.
0.25

0.75

0.5

2
0.5
0.25
0.75

0.5

4
0.5

(b) (6) Specify the classes and determine whether they are transient or recurrent.
{1, 2}: Recurrent (2)
{5}: Recurrent (absorbing state) (2)
{3, 4}: Transient (2)
(c) (1)Identify the period of state 4.
As it is a transient state, we cannot determine the period. (Everyone got 1 point)
(d) (10) Calculate limn Pn and fill the matrix.

2/5
3/5
0 0 0

2/5
3/5
0 0 0

P = f3,{1,2} 1 f3,{1,2} 2 0 0 f3,5

f4,{1,2} 1 f4,{1,2} 2 0 0 f4,5
0
0
0 0 1

2/5 3/5 0 0 0
2/5 3/5 0 0 0

4/25 6/25 0 0 3/5

.
2/25 3/25 0 0 4/5
0
0
0 0 1

f3,{1,2} = P(X = {1, 2} | X0 = 3)

5

= P(X = {1, 2}, X1 = i | X0 = 3)

i=1

= 0.25 + 0.75 f4,{1,2}

(2)

f4,{1,2} = P(X = {1, 2} | X0 = 4)

5

= P(X = {1, 2}, X1 = i | X0 = 4)

i=1

= 0.5 f3,{1,2}

(2)

f3,{1,2} + f3,5 = 1

(2)

f4,{1,2} + f4,5 = 1

(2)

Solving the above equations:

f3,{1,2} = 2 f4,{1,2} = 0.25 + 0.75 f4,{1,2} f4,{1,2} = 0.2, f3,{1,2} = 0.4, f3,5 = 0.8, f4,5 = 0.6

3. Performance (91.4%):
30

Q3

25

20

15

10

0
6

10

11

12

13

14

Question and Solution:

(14) Suppose you are working as an independent consultant for a company. Your performance is
evaluated at the end of each month and the evaluation falls into one of three categories: 1, 2, and
3. Your evaluation can be modeled as a discrete-time Markov chain and its transition diagram is
as follows.

Denote your evaluation at the end of nth month by Xn and assume that X0 = 2. Your monthly
salary is determined by the evaluation of each month in the following way:
Salary is
$ 1000 when your evaluation is 1
$ 4000 when your evaluation is 2
$ 8000 when your evaluation is 3

(a) (4) What are state space, transition probability matrix and initial distribution of Xn ?
S = {1, 2, 3} (1)
a = (0, 1, 0) (1)

.1 .9
0
.5
P = .5 0
0 .15 .85

(2)

(b) (4) Does the stationary distribution exist? If so, what is the stationary distribution? (Setup the
equations, you dont need to solve them)
(1) Yes. It exists, since it is finite state space and irreducible, positive recurrent.

.1 .9
0
.5 = (1 2 3 )
(1 2 3 ) .5 0
0 .15 .85

= 1
0.11 + 0.52
(2) 0.91 + 0.153 = 2

0.52 + 0.853 = 3
n
(1) 1 + 2 + 3 = 1
(c) (2) What is the long-run fraction of time when your evaluation is either 2 or 3?
2 + 3

(2)

(d) (4) What is the long-run average monthly salary?

10001 + 40002 + 80003

(4)

4. Performance (76.7%):
14

Q4

12
10
8
6
4
2
0
2

10

12

14

16

18

20

22

Question and Solution:

(20) Consider the reflected random walk on state space {0, 1, 2, . . . } with the following transition
probabilities: p00 = q + r, p01 = p and pi,i1 = q, pii = r, pi,i+1 = p for i 0, where p + q + r = 1
and p, q, r > 0
(a) (2) Is the Markov chain periodic or aperiodic. Explain and if it is periodic also give the
period.
Aperiodic (1), since r > 0 (1).
(b) (2) Is the Markov chain irreducible? Explain.
Irreducible (1), since all states communicate (1).
(c) (8) Find the stationary distribution when p = 0.2, q = 0.4, r = 0.4. Is the stationary distribution unique?
Yes, it is unique (1).

p
0
q
p
p
1 p = 2 q 1 = 1 = ( )2 0
q
q
...
p
p
n1 p = n q n = n1 = ( )n 0
q
q
0 p = 1 q 1 =

(1)
(1)

(4)

i = 1

(1)

i=1

0 (1 +

p
p
+ ( )2 + ...) = 1
q
q
1
0 (
)=1
1 qp

p
1
0 = (1 ) =
1
2
p n
p
1
n = ( ) (1 ) = ( )n+1
q
q
2
(d) (4) Is the Markov chain positive recurrent when p = 0.5, q = 0.2, r = 0.3? If so, why? If
not, why not? (You dont need to prove it)
As p > q, it tends to move forward, and so to need an infinite number of steps to turn back
(2). Thus, it is not positive recurrent (2).
(e) (4) For the probabilities given in part (c), is the Markov chain positive recurrent? If so, why?
If not, why not?
Yes, it is positive recurrent (2), because p < q in that case (2).

5. Performance:
12

Q5

10

0
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

Question and Solution:

(12 + Bonus 10) A call center is staffed by two agents, Mary and John. Marys service times are
iid, exponentially distributed with mean 4 minutes and Johns service time are iid exponentially
distributed with mean 6 minutes. Suppose that when you arrive, both Mary and John are busy,
and you are the only one waiting in the waiting line.
(a) (4) What is the probability that John will serve you?
1 = 1/4 (1)
2 = 1/6 (1)
P(X2 < X1 ) =

1/6
2
=
1/6 + 1/4 5

(2)

(b) (4) What is the expected waiting time until you can be answered?
(1) E[min(X1 , X2 )] =

1
= 2.4 (2)
1/4 + 1/6

1 1
min(X1 , X2 ) exp( + ) (1)
4 6

10

(c) (4) What is the probability that you will be answered in 4 minutes?
(2) P[min(X1 , X2 ) 4] = 1 e(1/4+1/6)4 = 1 e5/3

(2)

(d) (Bonus 10) What is your expected time in the system (waiting plus service)?
Note: Bonus question has no partial credit.
1
E[W ] = 1/4+1/6
time to be served.
If served by Mary: 4 min (1) w.p.
If served by John: 6 min (1) w.p.
Thus, the total is:

1/4
1/4+1/6
1/6
1/4+1/6

(3.5)
(3.5)

1
1
1
12
6
4
3 = 7.2 mins
+
4
+
6
=
1
1
1
1
1
1
3+2
4+6
4+6
4+6

11

6. Performance:
10

Q6

9
8
7
6
5
4
3
2
1
0
2

10

12

14

16

18

20

Question and Solution:

(14+Bonus 5) True or False.
(1)(True/False) A transient state is accessible from any recurrent state.
(1)(True/False) A recurrent state is accessible from all states in its class, and it is also accessible from recurrent states in other classes.
(1)(True/False) If DTMC is periodic with period d 2, then the limiting probability exist.
(1)(True/False) At least one, possibly more, recurrent states are accessible from a given
transient state.
(1)(True/False) In a MC with finite state space, all states can be transient.
(1)(True/False) An irreducible DTMC with finite state space can have infinite number of
stationary distributions.
(1)(True/False) A system can be in state 0,1 or 2. Let Xn be the state of the system at time
n. We know P(Xn+1 = 1|Xn = 0, Xn1 = 2) = 0.4 and P(Xn+1 = 1|Xn = 0, Xn1 = 1) = 0.5.
Then {Xn , n 0} cannot be a Markov chain.
(2)(True/False) Assume a random variable X always has finite value, i.e., P(X < ) = 1,
then E[X] < .
(2)(True/False) A discrete-time Markov chain with state space S = {1, 2, 3, 4} has the
steady-state probabilities (0.2, 0.4, 0.3, 0.1). In steady-state, on average it would take 2.5 to
12

return to state 2 given that the system is currently in state 2.

Suppose that whether or not it rains today depends on previous weather conditions through
the last two days. Specifically, suppose that if it has rained for the past two days, then it
will rain tomorrow with probability 0.7; if it rained today but not yesterday, then it will rain
tomorrow with probability 0.5; if it rained yesterday but not today, then it will rain tomorrow
with probability 0.4; if it has not rained in the past two days, then it will rain tomorrow with
probability 0.2.
(2)(True/False) If we let Xn be the state at time n which depends only on whether or not it is
raining at time n, then {Xn } is a Markov chain.
(1)(True/False) This problem cannot be modeled as a Markov chain.
(Bonus 5) If you think you could model this problem as a Markov chain, please indicate the
state space and transition probability. (Note: Bonus question has no partial credit.)
State space S = {(i, j)} where (i,j) represents (yesterday, today).
Then, S =
{RR, NR, RN, NN} (1)

RR NR RN NN
RR 0.7 0 0.3 0

P=
NR 0.5 0 0.5 0
RN 0 0.4 0 0.6
NN 0 0.2 0 0.8

13

(4)

7. Performance:
25

Q7

20

15

10

0
0

10

Question and Solution:

(Bonus 10) A professor continually gives exams to her students. She can give three possible types
of exams, and her class is graded as either having done well or badly. Let pi denote the probability
that the class does well on a type i exam, and suppose that p1 = 0.3, p2 = 0.6, p3 = 0.9. If the
class does well on an exam, then the next exam is equally likely to be any of the three types. If the
class does badly, then the next exam is always type 1. Identify a state space, model it as a DTMC
and show how to determine the proportion of exams that are type i, i = 1, 2, 3.
(Note: Bonus question has no partial credit. )
S = {1, 2, 3}

(1)

14

1
P(Xn+1 = 1 | Xn = 1) = P(Xn is good) + P(Xn is bad)1
3
1
= 0.3 + 0.7 1 = 0.8
3
1
P(Xn+1 = 2 | Xn = 1) = P(Xn is good)
3
1
= 0.3 = 0.1
3
1
P(Xn+1 = 3 | Xn = 1) = P(Xn is good)
3
1
= 0.3 = 0.1
3

1
P(Xn+1 = 1 | Xn = 2) = P(Xn is good) + P(Xn is bad)1
3
1
= 0.6 + 0.4 1 = 0.6
3
1
P(Xn+1 = 2 | Xn = 2) = P(Xn is good)
3
1
= 0.6 = 0.2
3
1
P(Xn+1 = 3 | Xn = 2) = P(Xn is good)
3
1
= 0.6 = 0.2
3
1
P(Xn+1 = 1 | Xn = 3) = P(Xn is good) + P(Xn is bad)1
3
1
= 0.9 + 0.1 1 = 0.4
3
1
P(Xn+1 = 2 | Xn = 3) = P(Xn is good)
3
1
= 0.9 = 0.3
3
1
P(Xn+1 = 3 | Xn = 3) = P(Xn is good)
3
1
= 0.9 = 0.3
3

15

1
2
3
1 0.8 0.1 0.1

P=
2 0.6 0.2 0.2
3 0.4 0.3 0.3

(6)

In order to find the proportion of exams, we solve the folowing system:

P =

i = 1

to find i s (3).

16