MS&E 221 Ramesh Johari

Midterm Examination February 14, 2007

Instructions
1. Take alternate seating. 2. Answer all questions in the spaces provided on these sheets. If needed, additional paper will be available at the front of the room. Answers given on any other paper will not be counted. 3. The examination begins at 1:20 pm, and ends at 2:30 pm. 4. Show your work! Partial credit will be given for correct reasoning.

Honor Code
In taking this examination, I acknowledge and accept the Stanford University Honor Code. NAME NAME (signed) (printed)

Problem 1 (40 points). Answer each of the following short answer questions. (10 points per question) (a) True or false: In a chain on a nite state space that has two communicating classes and a unique invariant distribution, at least one of the two classes is transient. (Justify your answer.)

(b) Suppose the irreducible, aperiodic transition matrix P has a greater second largest eigenvalue modulus (SLEM) than the irreducible, aperiodic transition matrix Q (both on the same nite state space). Which Markov chain converges faster to its equilibrium distribution, and why?

(c) Let X0 , X1 , X2 , . . . be a Markov chain on the state space {0, 1, 2, 3}, and suppose a new process Y0 , Y1 , Y2 , . . . is dened according to: Yn = 0, 1, if Xn is even; if Xn is odd.

Under exactly what conditions is the process Yn also a Markov chain? Explain your answer.

(d) Suppose a Markov chain is irreducible and has invariant distribution . What is the mean number of visits to i before the rst return to j ?

Problem 2 (30 points). Consider a Markov chain with the following transition matrix. 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 1/6 0 1/3 0 1/2 0 0 0 P = 0 0 0 1/6 0 0 1/3 1/2 0 0 0 0 0 1 / 2 1/2 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1/3 2/3 0

(a) (6 points) What are the communicating classes of this chain, and which are closed?

(b) (6 points) Which communicating classes are positive recurrent, null recurrent, and/or transient?

(c) (6 points) Find all invariant distributions of this chain.

(d) (6 points) For what values of i does the following limit exist? Give the limit for those values of i where it exists. lim P(Xn = i|X0 = 4).
n

(e) (6 points) Now suppose the seventh row of the matrix is changed to: 0 1/2 0 0 0 0 0 1/2 Does this change your answer to (d)? Why or why not? .

Problem 3 (30 points). A salesman travels to four different cities that are located at the vertices of the unit square (i.e., the square with vertices (0, 0), (0, 1), (1, 0), (0, 1)). At each time step, the salesman jumps to one of the two adjacent vertices. He jumps vertically with probability p, and horizontally with probability q . In all cities except (0, 0), with probability r the salesman stays in the same city in the next time step. In (0, 0) (his home ofce), with probability r the salesman takes a vacation in the next time step. (Here p + q + r = 1.) Once on vacation, the salesman stays on vacation each period with probability a, and otherwise returns to work at his home ofce. (a) (6 points) Describe the movement of the salesman as a Markov chain.

(b) (8 points) Assuming r = 0, nd the long run fraction of time the salesman spends in each city.

(c) (8 points) Suppose the salesman is currently on vacation. Find the probability that, once the salesman returns to work, he goes on vacation again without ever visiting (1, 1).

(d) (8 points) Suppose the salesman is currently in (1, 1). Suppose also that he earns a reward of one dollar for each horizontal sales trip, and two dollars for each vertical sales trip. Give a set of equations that can be used to compute the expected reward he will earn once he leaves (1, 1), until he returns to (1, 1). You do not need to explicitly compute the answer.