Homework 6

PSTAT 160A FALL 2014

D UE T HURSDAY, D EC 4, 2014

(1) Let (Xn ) be a random walk on {0, 1, . . . , M } with two reflecting barrier at 0 and at M > 1. That is,
the transition matrix of (Xn ) is

Pi,j

1 p
=

if 0 < i < M and j = i + 1,

if 0 < i < M and j = i 1,
if i = M and j = M 1 or i = 0 and j = 1,
otherwise.

(a) Draw the transition graph for M = 3.

(b) Compute the invariant distribution for arbitrary M using the weighted graph method.
(c) Let T = inf {n 1 : Xn = 0} be the first return time to 0. Compute E [T | X0 = 0] and
E [T | X0 = 1].
(2) Let Z1 , Z2 , . . . be independent identically distributed, positive integer valued random variables.
These are the interarrival times of a bus service at a given bus stop. So, there is a bus at time 0,
the next bus arrives at Z1 , the next one at Z1 + Z2 and so on. Denote Un the time elapsed since the
last bus at time n and Vn the waiting time for the next bus at time n. The aim of this exercise is the
computation of the PMF Un + Vn for large n.
(a) What is the state space of the Markov chain Xn = (Un , Vn ), n 1?
(b) What are the transition probabilities of the Markov chain (Xn )n1 ?
(c) Is (Xn ) irreducible?
(d) What is the invariant distribution of (Xn ) in terms of the PMF of Z1 ?
(e) Compute
n
1X
P [Uk + Vk = j ] .
lim
n n
k=1
(f) If the law of Z1 is a so called non-lattice distribution, that is, the gratest common divisor of
{n : P [Z1 = n] > 0} is 1, then Xn is aperiodic. What is
lim P [Un + Vn = j ]

in this case?
(3) A particle performs a random walk on the non-negative integers {0, 1, . . .} as follows. At each step,
it moves right with probability p and left with probability q = 1 p, except that, if it is at 0 now,
it moves right with probability p and remains at 0 with probability q. This called a simple random
walk on D = {0, 1, 2, . . .} with a (not instanteneously) reflecting barrier at 0.
Let Xn be the particles position after the nth step. Then, (Xn )n0 is a Markov chain with state space
D.
(a) Write down the transition probabilities Pi,j = P [Xn+1 = j | Xn = i].
(b) Classify the states when p > q.
(c) Classify the states when p < q.
(d) Find the invariant distribution = (i )i0 when p < q.
(e) Let T = min {n 0 : Xn = 0}. Compute E [T | X0 = i] for i 1, when p < q.
Hint. What is the relationship between E [T | X0 = i] and E [T | X0 = 1]?

(4) Let (Xn )n0 be a Markov chain with state space D = {a, b, c, d, e} and transition matrix

0 0 0 0 1
1 0 0 0 0

1
1
P =
/2 /2 0 0 0
1/3 1/3 1/3 0 0
1/4 1/4 1/4 1/4 0
(a) Classify the states.
(b) Compute the invariant distribution = (a , b , c , d , e ).
(c) Let T = min {n 1 : Xn is b or c}. Compute

" T 1
#

X

E
f (Xn )  X0 = a .

n=0

for f with f (a) = 2, f (d) = 3, f (e) = 1.

(d) Define the reward function g by

2 if i = a and j = e,
g(i, j) = 3 if i = e and j = a,

0 otherwise.
Compute
n

1X
lim
g(Xk1 , Xk ).
n n
k=1

Each problem is worth 25 points. Write your name at the top right corner of each page you submit.
2