Professional Documents
Culture Documents
Probability measure, Review of probability theory, Markov chains, Recurrence transition matrices, Stationary distributions, Hitting times, Poisson processes, Renewal theory, Branching processes, Branching and point processes, Martingales in discrete time, Brownian motion, Martingales in continuous time
1.XERCISeX
Show that if denition of
A and B -algebra,
belongs to the
-algebra F i = 1, 2, . . .
and since Since
Ai F
for
Bc F, Bc A F
Ac F i=1
it follows that
2.
Though a fair die once. Assume that we only can observe if the number obtained is small, by
A = {1, 2, 3} B
and
insert diagram,
{5}
and
can be below.
F ={, , A, B, Ac , B c , A B, A B c , Ac B, Ac B c , A B, A B c , Ac B, Ac B c , (A B c ) (Ac B), (A B) (Ac B c )} ={, , {1, 2, 3}, {1, 3, 5}, {4, 5, 6}, {2, 4, 6}, {1, 2, 3, 5}, {1, 2, 3, 4, 6}, {1, 3, 4, 5, 6}, {2, 4, 5, 6}, {1, 3}, {2}, {5}, {4, 6}, {2, 5}, {1, 3, 4, 6}}.
The probability measure on each of the sets in of the partitions,
the additivity of the probability measure and the probability measure of each
3.
Given a probability space Dene
(, F, P)
and functions
X : R , Y : R.
are
Z = max{X, Y }. Z
is
1. Show that
F -measurable Z
if both
and
F -measurable. X
nor
is
is.
Proof. 1)
where the last conclusion is based on the result of proposition 3.2. 2) Suppose you threw two coins and our single set Let
-algebra F was generated by the A = {HT, T H}, i.e. F = {, , A, Ac } = {, {HH, T T, HT, T H}, {HT, T H}, {HH, T T }}. 1 0
if if
X=
Then
{HT } Y = {HH, T T, T H}
1 0
if if
{T H} . {HH, T T, HT }
Z=
hence
1 0
if if
{HT, T H} {HH, T T },
while
and
are not
F -measurable
is.
4.
Toss a fair soin functions
we want to consider
(, F1 , P1 )
How many
F1 -measurable
functions
E[X] = 0
are there?
(, F2 , P2 ) = i},
Ai = { :
are there?
Number of heads
i = {0, 1, 2, 3, 4} X
with
F2 -measurable n.
functions
E[X] = 0
Proof. 1) Each coin that is tossed has to possible outcomes, and since there
24 = 16 possible outcomes, or partitions. So the sample space is = {HHHH, HHHT, HHT H, etc.} with 16 members that describes all the information from that four tosses. F1 is the power set of consisting of all combinations of sets of (don't forget that allways is included in a -algebra ), hence F1 = (). P1 is deduced by rst observing that each has P() = 1/16 and then using the additivity och the probability
are four coins to be tossed, there are measure. For any function there is a set
X : {1, 1}
such that
F1
A F1
X() =
Hence
1 1
if if
A Ac .
and 16 8 ways of combining the partitions so that there are equally many of them in 16 A and Ac , hence there are 8 possible nctions X : {1, 1} such that
E[X] = P1 (A) P1 (Ac ) so if E[X] = 0 we must have P1 (A) = P1 (Ac ), since there are 16 partitions each with probability measure 1/16, there are
E[X] = 0.
2) The sets
A0 = {T T T T } A1 = {T T T H, T T HT, T HT T, HT T T } A2 = {T T HH, T HHT, HHT T, HT T H, T HT H, HT HT } A3 = {HHHT, HHT H, HT HH, T HHH} A4 = {HHHH}, . The -algebra F2 is the -algebra generated by this Pi = P(Ai ), then P0 = 1/16, P1 = 4/16, P2 = 6/16, P3 = 4/16 and P4 = 1/16. For any function X : {1, 1} that is measurable with respect to F2 there is a a set A F2 such that
denes a partition of partition. Dene
X() =
Hence
1 1
if if
A Ac .
E[X] = P2 (A) P2 (Ac ) so if E[X] = 0 we must have P1 (A) = P1 (Ac ). We may write E[X] = k0 P0 + k1 P1 + k2 P2 + k3 P3 + k4 P4 = k0 /160 + 4k1 /16 + 6k2 /16 + 4k3 /16 + k4 /16 which is zero either if ki = (1)i or if ki = (1)i . Hence, there are two possible F2 -measurable functions X : {1, 1} such that E[X] = 0.
3) Using the same notation as in 2), we conclude that, for any given integer
n > 0, A0 , A1 , . . . , An
is a partition of
Pn (Ai ) =
A Fn
X() =
Hence
1 1
if if
A Ac .
we must have
so if
E[X] = 0
Pn (A) = Pn (Ac ) =
1/2.
For even
n i=odd
n i=even
so
E[X] = 0
if
ki = (1)i
or
ki = (1)i .
(n1)/2 n
n,
we have
i=0 n n
i=(n1)/2+1
i = 0, 1, . . . , (n 1)/2 and there are 2(n1)/2 ways (n1)/2 of combining these. Hence, there are 2 possible Fn -measurable functions X : {1, 1} such that E[X] = 0.
so
E[X] = 0 if kni = ki
for
5.
Consider the probability space algebra on
(, F, P)
where
and
X() = P(X = Y ) = 0.
and
= [0, 1], F is the Borel (, F). Show that Y () = 2| 1/2| have the same X
is
x (, 0] x [0, 1] x [1, )
we get, with
x [0, 1],
symmetry}
P(X x) = P(Y x) = x
for
x [0, 1].
But since
P(X = Y ) = P(X = 2|X 1/2|) = P({ : X = 2X 1/2} { : X = 2X + 1/2}) = P({ : X = 1} { : X = 3}) = P() = 0,
we conclude that
P (X = Y ) = 0.
6.
Show that the smallest
-algebras -algebras
Proof. Let
containing is not
-algebra containing a set A is that intersection of all A. Also, show by counter example that the union of two necessarily a -algebra .
GA be the set of all -algebras containing A, we want to show that GGA G = { : G for every G GA } = F is a -algebra ; if it is, then it is the smallest -algebra since otherwise, there would be a -algebra containing A that was smaller than F , this would be a contradiction since then this -algebra would be included in G and therefore F would not be minimal. To check that F is a -algebra , we simply use Denition 1.3.
1.
G G GA
hence
F. c G G GA
then hence
2. If 3. If so
G G GA
then
c F .
hence
1 , 2 , . . . G G GA -algebra
.
i G G GA i=1
1 , 2 , . . . F .
is a
take
-algebras is not necessarily a -algebra , F1 = {, , A, Ac } and F2 = {, , B, B c } where A B . Then F1 F2 = { : F1 or F2 } = {, , A, Ac , B, B c } is not a -algebra since, e.g. B c A F1 F2 . /
To show that the union of two
7.
given a probability space algebra of
F,
and consider
variable
X.
Let
be a sub- -
1. Show that
E[(X X)Y ] = 0
is an
A-measurable
random variable.
that minimize
E[(X Y )2 ]
E[(X Y )2 ] = E[X 2 ] 2E[XY ] + E[Y 2 ] = E[X 2 ] 2E[E[XY |A]] + E[Y 2 ] = E[X 2 ] 2E[Y E[X|A]] + E[Y 2 ] = E[X 2 ] E[E[X|A]2 ] + E[E[X|A]2 ] 2E[Y E[X|A]] + E[Y 2 ] = E[X 2 ] E[E[X|A]2 ] + E[(Y E[X|A])2 ],
0
hence
E[(X Y )2 ]
is minimized when
Y = E[X|A].
8.
Let
space
and Y be two integrable random variables dened on the same probability (, F, P). Let A be a sub- -algebra such that X is A-measurable.
1. Show that
E[Y |A] = X
implies that
E[Y |X] = X .
does not necessarily imply
E[Y |X] = X
E[Y |A] = X .
Proof. (For denition of conditional expectation see Denition 4.6, for proper-
ties of the conditional expectation, see Proposition 4.8.) 1) Since 4.8, we get
is
=X
X and Z be independent and integrable random variables and assume E[Z] = 0 and dene Y = X + Z . Let A = (X, Z), then E[Y |(X)] = E[X + Z|(X)] = X + 0 = X,
while
Exercise 1 For each given p let X have a binomial distribution with parameters p and N. Suppose N is itself binomially distributed with parameters q and M, M N. (a) Show analytically that X has a binomial distribution with parameters pq and M. (b) Give a probabilistic argument for this result. Exercise 2 Using the central limit theorem for suitable Poisson random variables, prove that
n n
lim e
n k=0
nk 1 = . k! 2
Exercise 3 Let X and Y be independent, identically distributed, positive random variables with continuous density function f (x) satisfying f (x) > 0 for x > 0. Assume, further, that U = X Y and V = min(X, Y) are independent random variables. Prove that f (x) = ex for x 0; 0 elsewhere,
for some > 0. Hint: Show rst that the joint density function of U and V is fU,V (u, v) = f (v) f (v + |u|). Next, equate this with the product of the marginal densities for U, V. Exercise 4 Let X be a nonnegative integer-valued random variable with probability generating function f (s) = an sn . After observing X, then conduct X binomial trials with probability n=0 p of success. Let Y denote the resulting number of successes. (a) Determine the probability generating function of Y. (b) Determine the probability generating function of X given that Y = X. (c) Suppose that for every p (0 < p < 1) the probability generating functions of (a) and (b) coincide. Prove that the distribution of X is Poisson, i.e., f (s) = e(s1) for some > 0.
Solutions
Exercise 1 (a) Since the distribution of N is binomial(M, q), we have fN (s) = E[sN ] = (sq + 1 q)M . Since the conditional distribution of X given N = n is Binomial(n, p), we have fX|N=n (s) = E[sX |N = n] = (sp + 1 p)n . We can compute the generating function of X, say fX (s), by conditioning on N. We obtain
M
fX (s) = E[s ] =
n=0 M
E[sX | N = n] P[N = n]
=
n=0
(sp + 1 p)q + 1 q
= s(pq) + 1 pq
Here we recognize the generating function of the binomial(M, pq) distribution. Heres another approach. For k {0, 1, 2, ..., M}, we get
M
P[X = k] =
n=0 M
=
n=k
= =
This is the binomial(M, pq) distribution. (b) Imagine M kids. Each one of them will toss a silver coin and a gold coin. With a silver coin the probability of heads is q. With a gold coin, it is p. Let N be the number of kids who will get heads with the silver coin and let X be the number of kids who will get heads on both tosses. Clearly the distribution of N is binomial(M, q), the conditional distribution of X given N = n is binomial(n, p) and the distribution of X is binomial(M, pq).
Exercise 2 Let X1 , X2 , X3 , ... be independent random variables, each with distribution Poisson(1), and let S n = n k=1 Xk . Since the mean and the variance of the Poisson(1) distribution are both equal to 1, the central limit theorem gives us lim P Sn n x = n
x
2 1 eu /2 du 2
lim P
Sn n 1 0 = . 2 n
This last equation can be written as 1 lim P [S n n] = . n 2 In view of the lemma below, the distribution of S n is Poisson(n). Therefore, the last equation can be written as n nk 1 n = . lim e n k! 2 k=0 Lemma. If U Poisson() and V Poisson() and if U and V are independent, then U + V Poisson( + ). The elementary proof is left as an exercise. Exercise 3 Let fX,Y (x, y) denote the joint density of X and Y. Let fU,V (u, v) denote the joint density of U and V. For the moment we do not make any independence assumptions. Fix u > 0 and v > 0. The following computation will be easy to follow if you look at the rst quadrant of R2 and visualize the set of points (x, y) for which min(x, y) v and x y u. fU,V (u, v) = = = 2 P[(U u) (V v)] uv 2 P[(X Y u) (min(X, Y) v)] uv 2 (P[(X, Y) A] + P[(X, Y) B]) uv
where A = {(x, y) R2 : v < y < and 0 < x < v} and B = {(x, y) R2 : 0 < y v and 0 < x u + y}. Since A depends only on v, we have 2 P[(X, Y) A] = 0. uv Thus we have fU,V (u, v) = = = = 2 P[(X, Y) B] uv v u+y 2 fX,Y (x, y) dx dy uv 0 0 u+v fX,Y (x, v) dx u 0 fX,Y (u + v, v).
A similar calculation gives us fU,V (u, v) = fX,Y (v u, v) for the case where u > 0 and v < 0. (You should do the computation; be careful with the domain of integration in the (x, y) plane). In summary, we have shown that (1) fU,V (u, v) = fX,Y (v + |u|, v) 0 if u R and v > 0, otherwise.
If we assume that X and Y are independent and identically distributed with density f , then equation (1) can be written as f (v + |u|) f (v) if u R and v > 0, fU,V (u, v) = 0 otherwise. If we assume furthermore that U and V are independent, i.e., fU,V (u, v) = fU (u) fV (v) for all v > 0 and u R, then the above result gives us f (v) f (v + |u|) = fU (u) fV (v) In particular we have (2) f (u + v) = g(u) h(v) u > 0, v > 0 v > 0, < u < .
with g(u) = fU (u) and h(v) = fV (v)/ f (v). From (2) we get, for all x > 0 and y > 0, P[X > x + y] = P[X > x]
y 0
f (x + v)dv f (x + v)dv =
g(x) g(x)
h(v)dv y h(v)dv 0
h(v)dv y h(v)dv 0
Thus the ratio P[X > x + y]/P[X > x] does not depend on x. Taking the limit as x 0, we see that this ratio is equal to P[X > y]. Thus we have (3) P[X > x + y] = P[X > x] P[X > y] pour tout x > 0, y > 0.
The lemma below allows us to conclude that X exponential(). Lemma. Let X be a non negative random variable. (a) If X exponential() for some > 0, then equation (3) is true. (b) If equation (3) is true, then X exponential() (for some > 0). The proof is left to the reader. Exercise 4 (a) The distribution of Y given X = n is Binomial(n, p). Thus, E(sY |X = n) = (sp + 1 p)n . Using this, we have fY (s) = E(sY ) = (b) We have P(X = Y) = and then P(X = n | X = Y) = P(X = n, X = Y) P(X = n)P(Y = n | X = n) pn P(X = n) = = . P(X = Y) f (p) f (p)
n=0 n=0
E(sY |X = n) P(X = n) =
n=0
P(X = n, Y = n) =
pn P(X = n) = f (p)
sn P(X = n | X = Y) =
1 f (p)
n=0
sn pn P(X = n) =
f (sp) . f (p)
(c) From parts (a) and (b) we have f (1 p + ps) = f (ps) f (p) 0 < p < 1 and 1 < s < 1.
If we x p and we take derivative w.r.t. s, we obtain f (1 p + ps) p = f (ps) p f (p) 0 < p < 1 and 1 < s < 1.
Now we divide by p on both sides and we evaluate at s = 1. We obtain f (1) = i.e. (4) E[X] = f (p) f (p) 0< p<1 f (p) f (p) 0<p<1
When we solve equation (4), subject to the boundary condition f (1) = 1, we get f (s) = e(s1) (where = E[X]). We conclude that the distribution of X is Poisson.
10
Exercise 1 Determine the classes and the periodicity of the various states for a Markov chain with transition probability matrix 0 1
1 2 1 3
(a)
0 1 0 0 0 0 1 2 0 0 1 1 3 3 0
(b)
0 1 0 0 0 0 0 1 0 1 0 0 2 1 3 0 3 0
Exercise 2 Consider repeated independent trials of two outcomes S (sucess) or F (failure) with probabilities p and q, respectively. Determine the mean of the number of trials required for the rst occurence of the event SF (i.e., sucess followed by failure). Do the same for the event SSF and SFS. Exercise 3 Let a Markov chain contain r states. Prove the following: (a) If a state k can be reached from j, then it can be reached in r 1 steps or less. (b) If j is a recurrent state, there exists (0 < < 1) such that for n > r the probability that the rst return to state j occurs after n transitions is n . Exercise 4 In this exercise, consider a random walk on the integers such that Pi,i+1 = p, Pi,i1 = q for all integer i (0 < p < 1, p + q = 1). (a) Determine Pn . 0,0 (b) For any real a and natural n, dene a a(a 1) (a n + 1) . = n! n Prove that 2n 1/2 2n = (1)n 2 . n n
(c) Let P be the generating function associated to the sequence un = Pn , i.e., P(x) = un xn . Show n=0 0,0 that P(x) = (1 4pqx2 )1/2 . (d) Prove that the generating function of the recurrence time from state 0 to state 0 is F(x) = 1 1 4pqx2 . (e) What is the probability of eventual return to the origin?
11
Exercise 5 A player has k coins. At each round independently, he gains one coin with probability p, or loses one coin with probability 1 p = q. He stops to play if he reaches 0 (he loses) or N (he wins). What is the probability to win ? Exercise 6 Suppose X1 , X2 , . . . are independent with (Xk = +1) = p, (Xk = 1) = q = 1 p, where p With S 0 = 0, set S n = X1 + + Xn , Mn = max{S k : 0 k n} and Yn = Mn S n . If T (a) = min{n : S n = a}, show y a 1+y if p = q = 1/2; 0 max(a) Yk < y = qp qp y+1 a k T if p q 1 p y+1
q
Hint: The bivariate process (Mn , Yn ) is a random walk on the positive lattice. What is the probability that this random walk, started at (0, 0), leaves the rectangle in the picture from the top?
12
Solutions 3
Solutions
Exercise 1 (a) We have 1 3, 3 2 and 2 1, so 1, 2 and 3 are in the same class. Also, for states i = 1, 2, 3 we have P(i, 4) = 0, so class {1, 2, 3} is recurrent. Since P(4, 4) = 0, 4 itself forms a transient class. The classes are thus {1, 2, 3}, {4}. Periodicity is only dened for recurrent states, so it suces to determine the period of 1, 2 and 3. Since they are in the same class, they have the same period, so it suces to determine the period of 1. We have P2 (1, 1) > P(1, 3)P(3, 1) > 0 and P3 (1, 1) = P(1, 3)P(3, 2)P(2, 1) > 0, so the period of 1 must divide 2 and 3, hence it is equal to 1. (b) Since 1 2, 2 4, 4 3, 4 1 and 3 2, there is a single recurrent class {1, 2, 3, 4}. Let us determine the period of state 4. We have P(4, 4)P2 (4, 4) = 0. Also, P3 (4, 4) = 1, so P3n (4, 4) = 1 for all n. Fix N N, N > 3 and N not divisible by 3; we can then write N = 3n + k for some n N and k = 1 or 2. Then,
4
3n+k
(4, 4) =
i=1
> 0 if and only if N is a multiple of 3, so the period of 4 (and every other state) is
Exercise 2 Let 1 , 2 , . . . be a sequence of letters chosen independently at random so that for each i, P(i = S ) = p = 1 q = 1 P(i = F). Time until SF. Let X0 = , X1 = 1 and Xn = n1 n for n 2, where in each case we are simply concatenating the symbols. (Xn ) is then a Markov chain on the state space {, S , S , S S , S F, FS , FF}. Dene, for n 0, an = P (min{k : Xk = S F} = n), bn = P (Xn = S F), cn = PS F (Xn = S F);
F,S F (s) =
n=0
an s , P,S F (s) =
n=0
bn s , PS F,S F (s) =
n=0
cn sn ,
n n
2; 2,
c0 = 1, c1 = 0, cn = pq
pqs2 , pqs = 1s
n
PS F,S F (s) = 1 +
n=2
pqsn = 1 + y, we have
pqs2 . 1s
Using the formula F x,y (s)Py,y (s) = P x,y (s) (which applies when x F,S F (s) = Then,
F,S F (s) = pqs2 1s 2 1 + pqs 1s
pqs2 . 1 s + pqs2
13
and F,S F (1) = 1/pq. We are now done: E(min{n : Xn = S F}) = F,S F (1) = 1/pq.
3. Dene
an s , P,S S F (s) =
n=0
bn s , PS S F,S S F (s) =
n=0
cn sn .
We have
b0 = b1 = b2 = 0, bn = p2 q n c0 = 1, c1 = c2 = 0, cn = p q
2
3; 3,
so that
P,S S F (s) =
n=3
p2 qs3 , p qs = 1s
2 3
PS S F,S S F (s) = 1 +
n=3
p2 qs3 = 1 +
p2 qs3 1s
p2 qs3 . 1 s + p2 qs3
and F,S S F (1) = 1/p2 q = E(min{n : Xn = S S F}). Time until SFS. Similarly to before, put
F,S FS (s) =
n=0
an s , P,S FS (s) =
n=0
bn s , PS FS ,S FS (s) =
n=0
cn sn .
We now have
b0 = b1 = b2 = 0, bn = p2 q n
2
3; 3,
c0 = 1, c1 = 0, c2 = pq, cn = p q n so that
P,S FS (s) =
n=3
p2 qs3 =
p2 qs3 , 1s
PS FS ,S FS (s) = 1 + pqs2 +
n=3 p2 qs3 1s
p2 qs3 = 1 + pqs2 +
p2 qs3 1s
1 + pqs2 +
p2 qs3 1s
3p2 qs2 (1 s + pqs2 + (p2 q pq)s3 ) p2 qs3 (1 + 2pqs + 3(p2 q pq)s2 )) ; (1 s + pqs2 + (p2 q pq)s3 )2 3p2 q(pq + p2 q pq) p2 q(1 + 2pq + 3p2 q 3pq) 1 1 F,S FS (1) = = 2 + , 2 q pq)2 (pq + p p q p
F,S FS (s) =
so E(min{n : Yn = S FS }) =
1 p2 q
+ 1. p
14
Exercise 3 (a) We assume that k j. The fact that k can be reached from j means that there exists a sequence of states x0 , x1 , . . . , xN such that x0 = j, xN = k and P(xi , xi+1 ) > 0 for all i. Dene = {i {1, . . . , N 1} : j1 {1, . . . , i}, j2 {i + 1, . . . , N} : x j1 = x j2 } (the set of indices i such that xi is between two repeated entries in the sequence x1 , . . . , xN ). By enumerating {0, 1, . . . , N}\ = {a0 , . . . , aM } with a0 = 0 < a1 < . . . < aM = N, it is easy to see that P(xai , xai+1 ) > 0 for each i and that xa0 , xa1 , xa2 , . . . xaM has no repetitions. This implies that this sequence has at most r elements, so the path xa0 xa1 . . . xaM1 xaM shows that k can be reached from j in at most r 1 steps. (b) Dene j = min{n 1 : Xn = j}. Assume that j k. Since j is recurrent, we must also have k j, and by part (a), j can be reached from k in at most r 1 steps. This implies that Pk ( j or, equivalently, that Pk ( j r) < 1. Since the state space is nite, we thus conclude that the number = min Pk ( j r) : k can be reached from j r 1) > 0
is smaller than 1. Now, x n N. We have n/rr 1 < n, so the event { j > n} is contained in { j > n/rr 1}. Using this, we have P j ( j > n) =
k: jk
P j ( j > n/rr 1) = P j ( j
{X1 , . . . , Xn/rr1 }) r)
P j( j P j( j
k: jk
= P j ( j > (n/r 1)r 1). Proceeding inductively, we get P j ( j > n) n/r , n/r completing the proof. Exercise 4 (a) Let 1 , 2 , . . . denote a sequence of independent and identically distributed random variables with P(i = 1) = q, P(i = 1) = p for each i. Putting X0 0 and Xn = n n , we see that Xn is a Markov i=1 chain with transitions given by P. Since for any n 1 we have Xn Xn+1 {1, 1}, Xn has the same parity as n. As a consequence, Pn (0, 0) = 0 when n is odd. If n is even, then n n = 0 = (1 = y1, . . . , n = yn ) = n/2 pn/2 qn/2 . P(Xn = 0) = P i i=1 y ,...,y {1,1}n : y =0
1 n i
n>r
(b)
3 2 1 n 2 1 n 2
(2n 1) 2
15
P(x) =
n=0
u2n x2n =
n=0
2n (pqx2 )n . n
P(x) =
n=0
1/2 (4pqx2 )n n
= (1 4pqx2 )1/2 . (d) Using the formula F(x)P(x) = P(x) 1 and part (c), we immediately get F(x) = 1 (e) Writing F(x) = fk xk , we have k=0
1 4pqx2 .
P(n : Xn = 0) =
k=0
P(min{n : Xn = 0} = k) =
k=0
fk = F(1) = 1
1 4pq.
Exercise 5 Let (Xn )n be the Markov chain representing the fortune of the player, and let be the time when the game ends, that is, = inf{n 0 : Xn = 0 or Xn = N}. Let uk be the probability that starting from k coins, the player wins, that is, uk = Pk [X = N], where Pk = P[ | X0 = k]. For 0 < k < N, we have Pk [X = N] = Pk [X1 = k 1, X = N] + Pk [X1 = k + 1, X = N]. Note that this is equal to Pk [X1 = k 1] [X = N | X1 = k 1] + Pk [X1 = k + 1] Pk [X = N | X1 = k + 1] = q Pk1 [X = N] + p Pk+1 [X = N], where we used the Markov property in the last step. We have proved that uk = quk1 + puk+1 . Moreover, it is clear that u0 = 0 and uN = 1. We solve this recurrence equation as in Karlin-Taylor, (3.6) in chapter 3. For p = 1/2, we obtain k uk = , N and otherwise, 1 (q/p)k uk = . 1 (q/p)N Exercise 6 From the fact that (S n ) is a Markov chain, it follows that ((Yn , Mn )) is also a Markov chain. To justify this properly, x a sequence (y0 , m0 ) = (0, 0), (y1 , m1 ), . . . , (yn , mn ); noticing the equality {(Y0 , M0 ) = (0, 0), (Y1 , M1 ) = (y1 , m1 ), . . . , (Yn , Mn ) = (yn , mn )} = {S 0 = 0, S 1 = m1 y1 , . . . , S n = mn yn },
16
we have P((Yn+1 , Mn+1 ) = (y, m) | (Y0 , M0 ) = (0, 0), . . . , (Yn , Mn ) = (yn , mn )) =P max0 i n+1 S i = m, (max0 i n+1 S i ) S n+1 = y S 0 = 0, S 1 = m1 y1 , . . . , S n = mn yn
In words, (Yn , Mn ) takes values in the positive quadrant of Z2 (see the picture in the statement of the exercise) and moves according to the rules: if y > 0, we jump to the right with probability q and to the left with probability p; if y = 0, we jump to the right with probability q and up with probability p. Notice that we never jump down and we can only jump up on the vertical axis. For a > 0, we have T (a) = min{n : S n = a} = min{n : Mn = a}, so max Yk < y = (Yn , Mn ) reaches (a, 0) before reaching {y}
+
0 k T (a)
0 k T (a)
max Yk < y =
i=0
=P
The probability in the above expression is the probability that the chain in the picture reaches the upper point (0, a) before reaching the rightmost point (y, a 1). From the gamblers ruin problem, we see that this is equal to y 1+y if p = q = 1/2; p p y+1 q q if p q, p y+1
1
q
17
Exercise 1 Let P be the transition matrix of a Markov chain. If j is a transient state, prove that for all i,
+
Pn j < +. i,
n=1
Exercise 2 Consider a sequence of Bernoulli trials X1 , X2 , . . ., where Xn = 1 or 0. Assume that there exists > 0 such that, for any n, P [Xn = 1 | X1 , . . . , Xn1 ] . Show that (1) P[Xn = 1 for some n] = 1, (2) P[Xn = 1 innitely often] = 1. Exercise 3 Players I, with fortune a, and player II, with fortune b, play together (a > 10, b > 10). At each round, player I has probability p to win one unit from player II, else he gives one unit to player II. What is the probability that player I will achieve a fortune a + b 3 before his fortune dwindles to 5? Exercise 4 A car can be in one of three states: working (state 1), being repared (state 2), or denitely out of order (state 3). If the car is working on some day, then the probability it will be working the day after is 0.995, and the probability it will need repair is 0.005. If it is being repaired, then the probability it will be working the day after is 0.9, the probability it will still need repair is 0.05, the probability it will be denitely out of order is 0.05. (1) Write the transition matrix for this Markov chain. (2) Let us dene the reduced matrix R= 0.995 0.005 . 0.9 0.05
Show that Q = (I R)1 is well dened, and that Qi j is the expected total time spent at state j starting from state i (Hint: study the series + Rn . To show that Q and this series are well dened, you may n=0 study the spectrum of the transpose of R.) (3) Starting from a car that is working, what is the expected time before it gets denitely out of order ? (4) How can the result be extended to more general Markov chains ? (5) Use this method to solve exercise 2 of the previous series.
18
Solutions
Exercise 1 Let us write (Xn ) for the Markov chain with transition matrix P, Pi for P[ | X0 = i], and i = inf{n 1 : Xn = i}. Let Finj = Pi [ j = k], and let Fi j (s) be its generating function, Fi j (s) = show that P j j (s) =
+ n=0
+ n=0
1 , 1 F j j (s)
see Karlin-Taylor, Eq. (5.8) of Chapter 2 for a proof. By denition, if the state j is transient, then P j [ j < +] < 1. This amounts to saying that
+
F nj < 1, j
n=0
Pn j < +. j
n=0
For i
(see Karlin-Taylor, Eq. (5.10) of Chapter 2), and thus we also have lim Pi j (s) < +,
s1
or equivalently,
+
Pnj < +. i
n=0
Exercise 2 (1) Note that P[X1 = = Xn = 0] = P[Xn = 0 | Xn1 = = X1 = 0] P[Xn1 = = X1 = 0] (1 ) P[Xn1 = = X1 = 0]. By induction, we thus get P[X1 = = Xn = 0] (1 )n ,
19
and as a consequence, P[n {1, 2, . . .}, Xn = 0] = 0. (2) If it is not true that Xn = 1 innitely often, then there exists N such that for all n N, Xn = 0. We have
+
P[N : n N, Xn = 0]
N=0
P[n N, Xn = 0],
and we proceed as in (1) to show that each summand is equal to 0. Exercise 3 Let (Xn )n 0 be a Markov chain on Z such that, for each n, the transitions n n + 1 and n n 1 occur with probability p and 1 p, respectively. We want to nd P(Xn reaches a + b 3 before reaching 5 | X0 = a), which is the same as P(Xn reaches a + b 8 before reaching 0 | X0 = a 5). By Gamblers ruin, this is equal to Exercise 4 (1) The transition matrix is 0 0.995 0.005 0.05 0.05 . P = 0.9 0 0 1
q a5 1 p q a+b8 1 p
if p
q;
a5 a+b8
if p = q = 1/2.
(2) Remark: for the purpose of this question, computing explicitely the eigenvalues of this 2 2 matrix and checking that their modulus are strictly smaller than 1 would be ne. Here, we study the spectrum using arguments that are easier to extend to more general situations. Note that for any i, one has Ri j 1,
j
2,
RT x
=
j i
Ri j xi
j i
Ri j |xi | x 1 . : || 1}.
As a consequence, the spectrum of RT (which is the same as that of R) is contained in { Moreover, since R2 j < 1,
j
the inequality in (1) is strict if x2 0. Assume that there exists of modulus 1, and x 2 a non-zero vector such that RT x = x. Then it is easy to see that x2 0, and thus RT x 1 < x 1 , a contradiction with the assumption that has modulus one. We have proved that the eigenvalues of RT , which are also those of R, have modulus strictly smaller than 1.
20
Rn
n=0
(I R)
n=0
Rn =
n=0
Rn
n=1
Rn = I,
so in fact Q = + Rn . n=0 Finally, one has to see that Rnj is the probability that starting from i, the walk is at j at time n. By i denition, this probability is Pi0 i1 Pin1 in ,
i=i0 ,i1 ,...,in1 ,in = j
where i1 , . . . , in1 take all possible values in {1, 2, 3}. But since state 3 is absorbing (that is, P3 j = 0 if j 3), the sum above is zero if one ik is equal to 3. So the sum is equal to Ri0 i1 Rin1 in ,
i=i0 ,i1 ,...,in1 ,in = j
where i1 , . . . , in1 take all possible values in {1, 2}. This last expression is precisely Rnj . i (3) A computation shows that 3800 20 Q= . 3600 20 Starting from a car that is working, the expected time before it goes out of order is the sum of expected times spent in states 1 and 2, which is thus 3820 days (and on average, the total number of days during which the car is working before it gets out of order is 3800 days). (4) A careful examination of the proof of part (2) shows that the only requirement is that the out of order state can be reached from any other state in a nite number of steps. (5) For instance, to get the expected time before getting SF, we may consider the (3 3) reduced transition matrix R given by FF FS S S FF q p 0 . FS 0 0 p 0 0 p SS For the inverse, we obtain (I R)1 1/p 1 p/q = 0 1 p/q . 0 0 1/q
It takes two tosses to get the chain started. But the moment of appearance of SF is not changed if instead we start in the state FF. We get that the expected time before SF appears is 1 p 1 1 +1+ = + , p q p q as expected.
21
where 0 < pi < 1 for each i and p0 + p1 + + pm = 1. Determine lim Pnj , the stationary distribution. i Exercise 2 An airline reservation system has two computers only one of which is in operation at any given time. A computer may break down any given day with probability p. There is a single repair facility which takes 2 days to restore a computer to normal. The facilities are such that only one computer at a time can be dealt with. Form a Markov chain by taking as states the pairs (x, y) where x is the number of machines in operating conditions at the end of a day and y is 1 if a days labor has been expended on a machine not yet repaired and 0 otherwise. Enumerating the states in the order (2, 0), (1, 0), (1, 1), (0, 1), the transition matrix is q p 0 0 0 q q p 0 0 1 0 0 p 0 0
where p, q > 0 and p + q = 1. Find the stationary distribution in terms of p and q. Exercise 3 Sociologists often assume that the social classes of successive generations in a family can be regarded as a Markov chain. Thus, the occupation of a son is assumed to depend only on his fathers occupation and not on his grandfathers. Suppose that such a model is appropriate and that the transition probability matrix is given by Lower .40 .05 .05 Sons class Middle Upper .50 .10 .70 .25 .50 .45
Fathers class
For such a population, what fraction of people are middle class in the long run? Exercise 4 Suppose that the weather on any day depends on the weather conditions for the previous two days. To be exact, suppose that if it was sunny today and yesterday, then it will be sunny tomorrow with probability .8; if it was sunny today but cloudy yesterday, then it will be sunny tomorrow with probability .6; if it was cloudy today but sunny yesterday, then it will be sunny tomorrow with probability .4; if it was cloudy for the last two days, then it will be sunny tomorrow with probability .1.
22
Such a model can be transformed into a Markov chain provided we say that the state at any time is determined by the weather conditions during both that day and the previous day. We say the process is in State (S,S) if it was sunny both today and yesterday; State (S,C) if it was sunny yesterday but cloudy today; State (C,S) if it was cloudy yesterday but sunny today; State (C,C) if it was cloudy both today and yesterday. Enumerating the states in the order (S,S), (S,C), (C,S), (C,C), the transition matrix is .8 .2 .4 .6 .6 .4 .1 .9 Find the stationary distribution for the Markov chain. Exercise 5 (Chapter 3, Problem 4) Consider a discrete time Markov chain with states 0, 1, . . . , N whose matrix has elements i , j = i 1, i , j = i + 1, i, j = 0, 1, . . . , N; Pi j = 1 i i , j = i; 0, | j i| > 1. Suppose that 0 = 0 = N = N = 0, and all other i s and i s are strictly positive, and that the initial state of the process is k. Determine the absorption probabilities at 0 and N.
23
Solutions
Exercise 1 Since all entries in the transition matrix are strictly positive, the chain is irreducible and positive recurrent, so there exists a unique invariant measure. By the symmetric structure of the matrix, we can guess that 1 1 the uniform distribution = ( m+1 , . . . , m+1 ) is invariant, and indeed it is trivial to check that P = . Exercise 2 Following the enumeration in the exercise, let (1 , 2 , 3 , 4 ) denote the stationary distribution. It must satisfy: (1) (2) (3) (4) (5) Applying (4) in (2), we get q1 + q2 2 = 1 = 1 = Using this, (4) and (5) in (1), we get q2 p . 2 + 2 + p2 + q2 = 1 = 2 = p 1 + p2 We thus get 1 =
q2 , 1+p2
1 + 2 + 3 + 4 = 1; q1 + q3 = 1 ; p1 + p3 + 4 = 2 ; q2 = 3 ; p2 = 4 . q2 2 . p
2 =
p , 1+p2
3 =
qp , 1+p2
4 =
p2 . 1+p2
Exercise 3 Since all the entries of the transition matrix are positive, the chain is irreducible and aperiodic, so there exists a unique stationary distribution. Denoting this distribution by = (x, y, z), the quantity we are looking for namely, the fraction of people that are middle class in the long run is equal to y. We know that x + y + z = 1 and also that .40 .50 .10 (x y z) .05 .70 .25 = (x y z); .05 .50 .45 solving the corresponding system of linear equations, we get x =
1 13 ,
y=
5 8
and z =
31 104 .
Exercise 4 Let x, y, z, w denote the states of the chain in the order that they appear in the transition matrix. The invariant measure is obtained by solving the system .8 .2 .4 .6 = (x y z w); (x y z w) .6 .4 .1 .9 x + y + z + w = 1.
24
1 1 6 11 , 11 , 11 ).
Exercise 5 For k {0, . . . , N}, dene f (k) = P(n 0 : Xn = N | X0 = k). Since 0 and N are absorbing, we have f (0) = 0, f (N) = 1. For k {1, . . . , N 1}, we have f (k) = k f (k 1) + k f (k + 1) = f (k + 1) f (k) = k ( f (k) f (k 1)). k
For k {1, . . . , N}, dene k = f (k) f (k 1). We can thus rewrite what we have obtained above as k k+1 = k k ; iterating this we get k = We can now nd 1 : 1 = f (N) f (0) = 1 + + N = 1 + 1 k1 1 1 + + 1 1 k1
1
1 k1 1 1 k1
25
Exercise 1 Let (Xn ) be an irreducible Markov chain with stationary distribution . Find E (min{n : Xn = X0 }). Exercise 2 Let P be the transition matrix of an irreducible Markov chain in state space S . A distribution on S is called reversible for P if, for each x, y S , we have (x)P(x, y) = (y)P(y, x). (a) Show that, if is reversible for P, then is stationary for P. (b) Assuming that is reversible for P, show that, for any n and any x0 , x1 , . . . xn S , P (X0 = x0 , . . . , Xn = xn ) = P (Xn = x0 , . . . , X0 = xn ). (c) Random walk on a graph. Let G = (V, E) be a nite connected graph. For each x V, let d(x) denote the degree of x. Let P be the Markov transition matrix for the Markov chain on V given by P(x, y) =
1 d(x)
if (x, y) E; otherwise.
z d(z)
is reversible for P.
Exercise 3 We identify the square grid = {1, . . . , 8}2 with a chessboard. In chess, a knight is allowed to move on the board as follows. If the knight is in position (x, y), it can go to any of the positions in (x + 1, y + 2), (x 1, y + 2), (x 2, y + 1), (x 2, y 1), (x 1, y 2), (x + 1, y 2), (x + 2, y 1), (x + 2, y + 1) .
Let (Xn ) denote the sequence of positions of a knight that at time zero is in position (1, 1) and at each step chooses uniformly among its allowable displacements. Find the expected time until the knight returns to (1, 1). (Hint: describe the chain as in Exercise 2(c).) Exercise 4 Consider a nite population (of xed size N) of individuals of possible types A and a undergoing the following growth process. At instants of time t1 < t2 < t3 < . . ., one individual dies and is replaced by another of type A or a. If just before a replacement time tn there are j As and N j as present, we postulate that the probability that an A individual dies is j1 /B j and that an a individual dies is (N j)2 /B j where B j = 1 j + 2 (N j). The rationale of this model is predicated on the following structure: Generally a type A individual has chance 1 /(1 + 2 ) of dying at each epoch tn and an a individual has chance 2 /(1 + 2 ) of dying at time tn . (1 /2 can be interpreted as the selective advantage of A types over a types). Taking account of the sizes of the population it is plausible to assign the probabilities 1 j/B j and (2 (N j)/B j) to the events that the replaced individual is of type A and type a, respectively. We assume no dierence in the birth pattern of the two types and so the new individual is taken to be A with probability j/N and a with probability (N j)/N. (1) Describe the transition probabilities of the Markov chain (Xn ) representing the number of individuals of type A at times tn (n = 1, 2, . . .). (2) Find the probability that the population is eventually all of type a, given k As and (N k) as initially.
26
Exercise 5 Let (Xk ) be a Markov chain for which i is a recurrent state. Show that
N
lim P(Xk
i for n + 1
n + N | X0 = i) = 0.
If i is a positive recurrent state prove that the convergence in the above equation is uniform with respect to n.
27
Solutions
Exercise 1 1 = #S . (x)
E (min{n : Xn = X0 }) =
xS
(x) E x (min{n : Xn = X0 }) =
xS
(x)
(x)P(x, y) = (x)
y
P(x, y) = (x),
so is stationary. (b) P (X0 = x0 , . . . , Xn = xn ) = (x0 )P(x0 , x1 )P(x1 , x2 ) P(xn1 , xn ) = P(x1 , x0 )(x1 )P(x1 , x2 ) P(xn1 , xn ) = P(x1 , x0 )P(x2 , x1 )(x2 )P(x2 , x3 ) P(xn1 , xn ) = = P(x1 , x0 )P(x2 , x1 )P(x3 , x2 ) P(xn , xn1 )(xn ) = P (Xn = x0 , . . . , X0 = xn ). (c) Let x, y V. If x and y are not adjacent, we have (x)P(x, y) = (y)P(y, x) = 0. Otherwise, (x)P(x, y) = 1 d(x) = d(z) d(x) z 1 = z d(z) d(y) 1 = (y)P(y, x). d(z) d(y) z
Exercise 3 We give a graph structure to the chessboard by taking the vertex set to be and setting edges x y if and only if the knight can go from x to y (this implies that it can also go from y to x, so the relation is symmetric). The stochastic sequence (Xn ) is then a random walk on this graph. We claim that the graph is connected (equivalently, that the chain is irreducible). To see this, consider the reduced board = {1, 2, 3}2 . Without leaving , it is possible to go from (1, 1) to (1, 2): performing the moves (1, 1) (3, 2) (1, 3) (2, 1); of course, reversing these moves allows us to go from (2, 1) to (1, 1). Now, consider two adjacent positions x, y (x and y dier by 1 in one coordinate and coincide in the other). For any such pair x, y, it is possible to nd a 3x3 subset of such that either x or y is one of the corners of the square. Then, except possibly for a rotation, the moves given for can be repeated so that we can go from x to y and from y to x. Now, for arbitrary points x, y , we can construct a path from x to y such that at each step we go to an adjacent position, so all cases are now covered. By Exercise 2(c), the stationary distribution is given by (x) = d(x)/ z d(z). The degrees of the vertices
28
The expected return time to (1, 1) is given by 1/((1, 1)) = ( Exercise 4 (1) The transition probabilities are given by P j, j1 = 1 j(N j) , B jN
= 336/2 = 168.
P j, j+1 =
2 (N j) j , B jN
P j j = 1 P j, j1 P j, j+1 ,
Pi j = 0,
for |i j| > 1.
(2) We are exactly in the context of last weeks Exercise 5. Using what was obtained there, we have P(n : Xn = 0 | X0 = 0) = 1, P(n : Xn = 0 | X0 = N) = 0 and P(n : Xn = N | X0 = k) = Since in our present case we have
Pk,k1 Pk,k+1
1+ 1+
P1,0 P2,1 P1,0 P1,2 + P1,2 P2,3 P1,0 P2,1 P1,0 P1,2 + P1,2 P2,3
+ +
1 2
P(n : Xn = 0 | X0 = k) = 1 P(n : Xn = N | X0 = k) = 1
+ + + +
1 k1 2 1 N1 2
1 N 2
1 k 2
1 N 2
lim Pi (i > K) = 0.
Pi (i > l) = Ei (i ) < ,
l=0
so that
lim
Pi (i > l) = 0.
l=K
n1
Pi (k [n, n + N] : Xk = i) =
t=0 n1
i)
n
=
t=0
Pi (Xt = i) Pi (i > N + n t)
t=0
Pi (i > N + n t) =
l=1
Pi (i > N + l).
29
It follows from (1) that, if we keep n xed and take N to innity, the last expression converges to zero. In addition, if i is positive recurrent, we can use (2) to obtain
n N+n
Pi (i > N + l) =
l=1 l=N+1
Pi (i > l)
l=N+1
Pi (i > l) 0
30
Exercise 1 Let S = {0, 1, 2, 3}. We consider the Markov chain on S whose transition matrix is given by 1/4 1/4 1/2 0 1/6 1/6 1/3 1/3 . P= 0 0 1/3 2/3 0 0 1/2 1/2 1) What are the communication classes, the transient states ? Is the chain aperiodic ? If not, give the period. 2) What is P2 ? What approximately is P100 ? 00 3) Starting from 2, what is the expected number of visits to state 3 before returning to 2 ? Exercise 2 Consider the Markov chain on the state space below, where at each step a neighbour is uniformly chosen among sites that are linked by an edge to the current position.
1) What are the communication classes, the transient states ? Is the chain aperiodic ? If not, give the period. 2) Is there an invariant measure ? Is it unique ? In case it is, give it. 3) Starting from x, what is the probability to hit S before F ? 4) Starting from x, what is the expected number of visits to F before returning to x ? Exercise 3 We have n coins C1 , . . . , Cn . For each k, Ck is biased so that it has probability 1/(2k + 1) of falling heads. We toss the n coins. What is the probability of obtaining an odd number of heads ? (Hint: letting pk = 1/(2k + 1) and qk = 1 pk , you may study the function f (x) = (q1 + p1 x) (qn + pn x) and its power series.)
31
Solutions
Exercise 1 The communication classes are {0, 1} and {2, 3}. To see that state 0 is transient, note that for the chain starting at 0, there is a non-zero probability that the chain goes to state 2. Once there, there is no path going back to 0, so indeed, the probability that the return time to 0 is innite is strictly positive. The same holds for state 1. The chain is aperiodic on its recurrent class {2, 3}. 2) We have 1 5 1 + = . P2 = (P00 )2 + P01 P10 = 00 16 24 48 To approximate P100 (that is, to nd what Pn looks like when n tends to innity), we rst compute the stationary distribution on the communication class {2, 3}, which has transition matrix 1/3 2/3 . 1/2 1/2 We nd a stationary distribution with weights (3/7 4/7). On the set {2, 3}, the chain is irreducible and aperiodic, so starting from any point, the distribution of Xn converges to the stationary distribution. Hence, for i, j {2, 3}, one has Pnj ( j) as n tends to innity. Clearly, if i {2, 3} and j {0, 1}, then i Pnj = 0. i If j {0, 1}, then it is a transient state, so for any i, we have Pnj 0 (in fact, we proved a stronger i statement in exercise 1 of series 3: that n Pnj is nite). There remains to determine Pnj when i {0, 1} i i and j {2, 3}. Intuitively, the chain enters the class {2, 3} at some point, and then converges to equilibrium in the class {2, 3}, so we should have (1) Here is a rigorous proof of this fact. Let = inf{n : Xn {2, 3}}. Since i is transient, Pi [ = +] = 0. For any > 0, we can nd N large enough such that Pi [ N] . For j {2, 3} and n N, we have
N1 n+
(2)
Pnj = Pi [Xn = j] = i
k=0
Pi [Xn = j, = k] + Pi [ N] .
We decompose Pi [Xn = j, = k] into Pi [Xn = j, = k, X = 2] + Pi [Xn = j, = k, X = 3], and we apply the strong Markov property at time on each term. The rst term becomes Pi [ = k, X = 2] P2 [Xnk = j] Pi [ = k, X = 2] ( j),
n+
where we used the fact that the Markov chain on {2, 3} is aperiodic. Similarly, Pi [Xn = j, = k, X = 3] Pi [ = k, X = 3] ( j),
n+
32
3) We can use the fact that this expected value is exactly (3)/(2) = 4/3. We can also take a more explicit approach and decompose the possible movements of the chain: with probability 1/3 the walk stays at 2 and therefore the number of visits to 3 is zero. Else it goes to 3, and then stays for a time geometric(1/2), and then goes back to state 2. The expectation we are looking for is thus 2 1 0+ 3 3
+
k
k=1
1 2
4 . 3
Exercise 2 1) The Markov chain is a random walk on a graph (see exercise 2 of series 5). Let G = (V, E) denote the graph. G is connected, that is, for any x, y V, there exists a path x0 = x, x1 , x2 , . . . , xN = y such that (xi , xi+1 ) E for each i. We then have P x (y is ever reached) P(x, x1 )P(x1 , x2 ) P(xN1 , y) > 0, so the chain is irreducible, and there are no transient states. The chain is also aperiodic. To see this, x any vertex that is the extreme point of one of the diagonal segments. Starting from this vertex, we can return to it in 3 steps (using the diagonal) and also in 4 steps (not using the diagonal), so the period of this vertex must divide 3 and 4, so it is 1. Since the chain is irreducible, we conclude that all vertices have period 1, so the chain is aperiodic. 2) since the chain is irreducible and the state space is nite, there exists a unique invariant measure. We can look for a reversible measure as in exercise 2 of series 5, and we nd that the equilibrium distribution evaluated at a vertex x is equal to the degree of x divided by the sum of the degrees of all vertices. In our case, this gives the following invariant measure. 2/52 3/52 3/52 2/52 3/52 5/52 5/52 3/52 3/52 5/52 5/52 3/52 2/52 3/52 3/52 2/52
(Remark: the invariant measure is represented here as a matrix just for ease of notation, following the positions of the graph; we can of course enumerate the 16 states of the chain and present the invariant measure as a row vector as we usually do.)
33
3) Consider the diagram 1/2 1 b 1 a 0 b 1/2 1 c 1 a a c 1/2 1 b 1 a b 1/2 The value at each site represents the probability, starting from this site, to arrive at S before F. Notice that we have used symmetry several times. Using the relation P x (S is reached before F) = valid for any x 1 deg(x) Py (S is reached before F),
yx
f (x) =
k=0
ak xk .
By developping, we observe that ak is the probability of getting exactly k heads. We want to know what is a1 + a3 + a5 + . Note that
n
f (1) =
k=0
ak ,
while f (1) =
ak (1)k =
k=0 k even
ak
k odd
ak .
So the quantity we are interested in is f (1) f (1) . 2 Clearly, f (1) = 1, while an easy computation gives f (1) = 1/(2n + 1). The probability we are interested in is thus n/(2n + 1).
34
Exercise 1 Let X1 and X2 be independent exponentially distributed random variables with parameters 1 and 2 so that P(Xi > t) = exp(i t) Let N= 1 if X1 < X2 ; 2 if X2 X1 , for t 0.
U = min{X1 , X2 } = XN , V = max{X1 , X2 }, W = V U = |X1 X2 |. Show a) P(N = 1) = 1 /(1 + 2 ) and P(N = 2) = 2 /(1 + 2 ). b) P(U > t) = exp{(1 + 2 )t} for t 0.
c) N and U are independent random variables. d) P(W > t|N = 1) = exp{2 t} and P(W > t|N = 2) = exp{1 t} for t e) U and W = V U are independent random variables. Exercise 2 Assume a device fails when a cumulative eect of k shocks occur. If the shocks happen according to a Poisson process with parameter , nd the density function for the life T of the device Answer: k k1 t t e if t > 0; f (t) = (k) 0, if t 0. Exercise 3 Let {X(t), t 0} be a Poisson process with intensity parameter . Suppose each arrival is registered with probability p, independently of other arrivals. Let {Y(t), t be the process of registered arrivals. Prove that Y(t) is a Poisson process with parameter p. 0.
0}
Exercise 4 At time zero, a single bacterium in a dish divides into two bacteria. This species of bacteria has the following property: after a bacterium B divides into two new bacteria B1 and B2 , the subsequent length of time until each Bi divides is an exponential random variable of rate = 1, independently of everything else happening in the dish. a) Compute the expectation of the time T n at which the number of bacteria reaches n. b) Compute the variance of T n .
35
Solutions
2 e2 y 1 e1 x dy dx =
1 ; 1 + 2
2 . P(N = 2) = 1 P(N = 1) = 1 + 2 b) P(U > t) = P(X1 , X2 > t) = P(X1 > t) P(X2 > t) = e(1 +2 )t . c) 1 = P(U > t) P(N = 1). 1 + 2 t x P(U > t, N = 2) = P(U > t) P(U > t, N = 1) = P(U > t)(1 P(N = 1)) = P(U > t) P(N = 2). P(U > t, N = 1) = P(t < X1 < X2 ) = 2 e2 y 1 e1 x dy dx = d) P(W > t | N = 1) = P(W > t, N = 1) P(X2 > X1 + t) = P(N = 1) P(N = 1) 1 1 1 + 2 e2 t = e2 t . = 2 e2 y 1 e1 x dydx = 1 1 1 + 2 0 x+t
1 +2
Exchanging the roles of X1 and X2 , we get P(W > t | N = 2) = e1 t . e) For all s, t 0, P(U > s, W > t) = P(s < X1 < X2 t) + P(s < X2 < X1 t)
=
s x+t
2 e2 y 2 e1 x dy dx +
s y+t
1 e1 x 2 e2 y dx dy
= Also,
1 2 e2 t e(1 +2 )s + e1 t e(1 +2 )s . 1 + 2 1 + 2
1 2 e2 t + e1 t . 1 + 2 1 + 2
We thus have P(U > s, W > t) = P(U > s) P(W > t), so they are independent. Exercise 2 If T 1 , T 2 , T 3 , . . . are the arrival times for the Poisson process, then T 1 , T 2 T 1 , T 3 T 2 , . . . are independent and all have exponential() distribution. We are looking for the law of T k = T 1 +(T 2 T 1 )+ +(T k T k1 ). This is given by the k-convolution of exponential . We claim that the density f (k) of this convolution is given by k k1 t et if t 0; (k) f (x) = (k) 0, otherwise.
36
This is easy to prove by induction: when k = 1, it is trivial; assuming it is true for k, we get
x
e(xy)
since k(k) = (k + 1). Exercise 3 Let T 1 , T 2 , . . . denote the sequence of arrival times of X(t). One rigorous way of stating what is said in the exercise is as follows. Let Z1 , Z2 , . . . be a sequence of independent Bernoulli(p) random variables. X(t) Assume that they are also independent of the process X. Then, put Y(t) = i=1 Zi . With this construction, Zi is 1 if arrival i is registered and 0 otherwise. Given an interval A [0, ), we will denote respectively by NA and NA the number of arrivals of X(t) and Y(t) in A. We prove that Y(t) is a Poisson process of parameter p by verifying: If A, B are disjoint intervals, then NA and NB are independent. P(NA = m NB = n) =
m1 m n1 n
P(NA = m, NB = n, NA = m1 , NB = n1 )
=
r,s: m1 m, n1 n r+m1 s
=
r,s: m1 m, n1 n r+m1 s
=
m1 m, n1 n
P(Bin(m1 , p) = m) P(Bin(n1 , p) = n)
r,s: r+m1 s
NA = m1 , T r , . . . , T r+m1 1 A, NB = n1 , T s , . . . , T s+m2 1 B
=
m1 m, n1 n
Repeating the above computation in each of the last big parenthesis, we get P(NA = m, NB = n) = P(NA = m) P(NB = n) as required. P(N[t,t+h] 2) = o(h). This is true because {N[t,t+h] 2} {N[t,t+h]
m1 m
n1 n
2} and P(N[t,t+h]
2) = o(h).
2)
= =p
37
Exercise 4 Let R0 = 0 and let R1 < R2 < . . . be the times at which the population size increases. Since we already start with two bacteria at time 0, we have T n = Rn2 . Consider the following alternate experiment. Start at time S 0 = 0 a population of 2 bacteria and wait until one of them reproduces; call this time S 1 . At this instant, we forget about these bacteria and start watching another population with three new-born bacteria and wait until one of them reproduces; call this time S 2 . Then yet again we forget about this population and start a new one with four new-born bacteira, and wait until one of them reproduces, time S 3 , and so on. By the lack of memory of the exponential distribution, the law of R1 , R2 R1 , R3 R2 , . . . is equal to that of S 1 , S 2 S 1 , S 3 S 2 , . . . In the alternate experiment it is easy to see that S m S m1 is equal to the minimum of m + 1 independent exponential(1) random variables (the times until the reproduction of each bacterium in the mth population) and using Exercise 1b we conclude that S m S m1 has exponential(m+1) distribution. Thus, E(T n ) = E(Rn2 ) = E(R1 + (R2 R1 ) + + (Rn2 Rn3 )) = Using independence, Var(T n ) = Var(Rn2 ) = Var(R1 ) + Var(R2 R1 ) + + Var(Rn2 Rn3 )) = 1 1 1 + 2 + ... + . 2 2 3 (n 1)2 1 1 1 + + ... + . 2 3 n1
38
Exercise 1 Messages arrive at a telegraph oce in accordance with the laws of a Poisson process with mean rate of 3 messages per hour. (a) What is the probability that no message will have arrived during the morning hours (8 to 12)? (b) What is the distribution of the time at which the rst afternoon message arrives? Exercise 2 A continuous time Markov chain has two states labeled 0 and 1. The waiting time in state 0 is exponentially distributed with parameter > 0. The waiting time in state 1 follows an exponential distribution with parameters > 0. Compute the probability P00 (t) of being in state 0 at time t starting at time 0 in state 0. Solution: P00 (t) = + + + e(+)t . Exercise 3 In the above problem, let = and dene N(t) to be the number of times the system has changed states in time t 0. Find the probability distribution of N(t). n Solution: P(N(t) = n) = et (t) . n! Exercise 4 Let X(t) be a pure birth continuous time Markov chain. Assume that P(an event happens in (t, t + h) | X(t) is odd) = 1 h + o(h); P(an event happens in (t, t + h)) | X(t) is even) = 2 h + o(h), where o(h)/h 0 as h 0. Take X(0) = 0. Find the probabilities: P1 (t) = P(X(t) is odd), P2 (t) = P(X(t) is even).
Exercise 5 Under the conditions of the above problem, show that EX(t) = (1 2 )2 21 2 [exp{(1 + 2 )t} 1]. t+ 1 + 2 (1 + 2 )2
39
Solutions
Exercise 1 The distribution of the number of messages that arrive during a four-hour period is Poisson(4 ) = Poisson(12). Thus, P(No messages from hour 8 to 12) = P(Poi(12) = 0) = e12 . By the lack of memory of the Poisson process, the arrival time T of the rst message after hour 12 is given by T = 12 + X, where X is a random variable with exponential(3) distribution. Exercise 2 The forward Kolmogorov equation tells us that P (t) = P00 (t) + P01 (t) 00 = P00 (t) + (1 P00 (t)) = ( + )P00 (t) + . We can then observe that d P00 (t)e(+)t = e(+)t . dt (+)t e + c. +
such that
P00 (t)e(+)t =
Observing further that P00 (t) = 1, we arrive at the conclusion. Exercise 3 Let T 0 = 0 and T 1 < T 2 < be the times at which the chain changes state. Then, the hypothesis implies that T 1 , T 2 T 1 , T 3 T 2 , . . . is a sequence of independent random variables with the exponential distribution. Thus, N(t) = max{n : T n t} is a Poisson process of intensity . In particular, for xed t the law of N(t) is Poisson(t), that is, P(N(t) = n) = ((t)n /n!)et . Exercise 4 For t 0, h > 0, P1 (t) = P(X(t + h) is odd) = P (X(t) is odd, no arrivals in [t, t + h]) + P (X(t) is even, one arrival in [t, t + h]) + P (X(t + h) is odd, two or more arrivals in [t, t + h]) = P(X(t) is odd) P(No arrivals in [t, t + h]] | X(t) is odd) + P(X(t) is even) P(One arrival in [t, t + h] | X(t) is even) + o(h) = P1 (t)(1 1 h o(h)) + P2 (t)(2 h + o(h)) + o(h). Dividing by 1/h and taking h to zero, we conclude that P1 is the solution of the dierential equation P (t) = 1 P1 (t) + 2 P2 (t). 1
40
The initial condition is P1 (0) = 0, since we start at zero which is even. The solution of this equation is P1 (t) = We then immediately get P2 (t) = 1 P1 (t) = 2 1 + et(1 +2 ) 1 + 2 1 + 2 2 2 et(1 +2 ) . 1 + 2 1 + 2
Exercise 5 To begin with, note that, for t, h 0, one has E[X(t + h) X(t)] = P[X(t + h) X(t) = 1] + E[(X(t + h) X(t)) 1X(t+h)X(t)2 ]. We would like to let h tend to 0 in the equality above, and to show that the last term is negligible, that is, E[(X(t + h) X(t)) 1X(t+h)X(t)2 ] = o(h). Let = min(1 , 2 ). It is possible to nd a coupling of X(t + h) X(t) with a Poisson random variable N of parameter h, such that X(t + h) X(t) N. So
+
+ k=2
ke h As a consequence,
( h)k = O(h2 ). k!
E[X(t + h) X(t)] = P[X(t + h) X(t) = 1] + o(h) = P[X(t + h) X(t) = 1 | X(t) is odd]P1 (t) + P[X(t + h) X(t) = 1 | X(t) is even] + o(h), where P1 (t) = P[X(t) is odd] and P2 (t) = P[X(t) is even]. Using the assumptions of the exercise, we get E[X(t + h) X(t)] = 1 hP1 (t) + 2 hP2 (t) + o(h). Hence, the function t E[X(t)] is (right-) dierentiable, and d E[X(t)] = 1 P1 (t) + 2 P2 (t). dt Using the results of the previous exercise, together with the fact that E[X(0)] = 0, we get
t
E(X(t)) =
41
Exercise 1 A patient arrives at a doctors oce. With probability 1/5 he receives service immediately, while with probability 4/5 his service is deferred an hour. After an hours wait again with probability 1/5 his needs are serviced instantly or another delay of an hour is imposed and so on. (a) What is the patients waiting time distribution? (b) What is the distribution of the number of patients who receive service over an 8-hour period assuming the same procedure is followed independently for every arrival and the arrival pattern is that of a Poisson process with parameter 1? Exercise 2 The random lifetime of an item has distribution function F(x). Show that the mean remaining life of an item of age x is E(X x | X > x) =
{1 x
F(t)}dt
1 F(x)
Hint. Recall the derivation, that applies to any positive integrable random variable Z,
x
E(Z) =
0
x FZ (dx) =
0 0
1 dt FZ (dx) =
0 t
FZ (dx)dt =
0
(1 FZ (t))dt.
Try to do something similar. Exercise 3 Find P(N(t) k) in a renewal process having lifetime density f (x) = where > 0 is xed. Exercise 4 For a renewal process with distribution F(t) =
t 0
n=0
42
Solutions 10
Solutions
Exercise 1 (a) P(Waiting time = k) = (4/5)k1 (1/5), k = 0, 1, 2, . . .. (b) We will need the following fact. Let X be a Binomial(N, p) random variable, where N is itself a Poi() random variable. Then, the distribution of X is Poi(p). This can be proved with probability generating functions: gBin(n,p) (s) = (sp + 1 p)n ,
gX =
n=0
E(sX |N = n) P(N = n) =
n=0
(1 + sp p)n
Now, for i = 0, . . . 7, let Ni denote the number of patients that arrive in the time interval (i, i + 1] and Xi the number of these patients that receive service before hour 8. Since patients arrive as a Poisson process with parameter 1 and (i, i + 1] has length 1, Ni has Poisson(1) distribution. For xed i, the probability that a given patient that arrives in (i, i + 1] receives service before hour 8 is equal to the probability that he waits 7 i or less hours for service; this is equal to 1 (4/5)7i+1 = 1 (4/5)8i . Thus, Xi has Binomial(Ni , 1 (4/5)8i ) = Poi(1 (4/5)81 ) distribution. Notice that X0 , . . . X7 are all independent, because they are each determined from arrivals of a Poisson process in disjoint intervals and patients waiting times, which are assumed to be independent. Since for the sum of independent Poisson random variables has Poisson distribution with parameter equal to the sum of the parameters, we have 8 7 1 1 4 + 1 4 + + 1 4 . X0 , . . . X7 Poi 5 5 5 Exercise 2 E(X x | X > x) = x + 1 P(X > x) 1 = x + P(X > x)
x 0 y
y F(dy) = x +
x x 0 x
1 P(X > x)
x t
1 dt F(dy)
F(dy) dt +
F(dy) dt
(1 x
(1 F(t))dt =
x
F(t))
1 F(x)
dt.
Exercise 3 Let the sequence X1 , X2 , . . . of independent random variables with the assigned density be the sequence of inter-arrival times of the renewal process. The density in the statement of the exercise is the density for a random variable obtained by adding to a exponential() random variable. We can thus write Xi = Yi + , where Yi exp() and i 0. Then, P(N(t) k) = P(X1 + + Xk t) = P(Y1 + + Yk t k) = 0, P(N(t k) if t k 0; k), otherwise, k) = P(Poi() k).
43
Exercise 4 x Let F be the distribution function of the renewal times; we have F(x) = 0 yey dy = 1 ex xex for x 0. Also let f (x) = xex I{x 0} be the density function of the renewal times. Let f (n) denote the n-convolution of f ; let us show by induction that f (n) (x) = 1 x2n1 ex I{x (2n 1)!
0} .
Indeed, this trivially holds for n = 1 and, assuming it holds for xed n and x
x
0,
f (n+1) (x) =
0 x
f (x y) f (n) (y) dy 1 y2n1 ey dy (2n 1)! x2n+1 x2n+1 1 x2(n+1)1 ex . = 2n 2n + 1 (2(n + 1) 1)!
=
0
Of course, if x < 0 we have f n+1 (x) = 0 and so the induction is complete. Now, let N(x) denote the number of renewals in [0, x]; we have
x x
P(N(x) = 2n) =
0
y4n1 dy + x
0 0
y4n1 dy
0
y4n dy
44
Exercise 1 The weather in a certain locale consists of rainy spells alternating with spells when the sun shines. Suppose that the number of days of each rainy spell is Poisson distributed with parameter 2 and a sunny spell is distributed according to an exponential distribution with mean 7 days. Assume that the successive random durations of rainy and sunny spells are statistically independent variables. In the long run, what is the probability on a given day that it will be raining? Exercise 2 Determine the distribution of the total life t of the Poisson process. Exercise 3 Consider a renewal process with non-arithmetic, nite mean distribution of renewals, and suppose that the excess life t and current life t are independent random variables for all t. Establish that the process is Poisson. Hint: use limit theorems on the identity P[t x, t > y] = P[t x] P[t > y], to derive a functional equation for v(x) = 1
+
(1 F(t)) dt.
x
Exercise 4 Show that the renewal function corresponding to the lifetime density f (x) = 2 xex , is x 0
1 1 M(t) = t (1 e2t ). 2 4 Hint. Use the uniqueness of the solution of the renewal equation. Here are some shortcuts for the computations: T 1 et dt = (1 eT ); 0 T 1 tet dt = 2 (1 eT T eT ); 0 T 1 2 T 2 eT ; t2 et dt = 3 1 eT T eT 2 0
T
tet dt =
0
1 (1 eT + T eT ). 2
45
Solutions
Exercise 1 Since we are only interested in the long run result, we can assume that at time 0 we are starting a rainy spell. For t > 0, dene A(t) = P(It is raining at time t); a(t) = P(The initial rainy spell is still taking place at time t). Also let F denote the distribution of the duration of two successive seasons (i.e., the distribution of a sum of two independent random variables, one with law Poisson(2) and the other with law exponential(1/7)). We then have the renewal equation
t
A(t) = a(t) +
0
A(t s) dF(s).
(Notice that F is neither purely discrete nor purely continuous, so the integral has to be seen as a sum plus an integral). From the renewal theorem, we have lim A(t) = 1
a(s) ds,
0
where is the expectation associated with F, equal to 2 + 7. Denoting by Y the duration of the initial 2 rainy spell, we have 0 a(s) ds = P(Y > s) ds = E(Y) = 2. So the nal solution is 2+7 . s=0 Exercise 2 Let denote the parameter of the Poisson process. We have t = t + t , where t = S N(t)t is the residual life and t = t S N(t) is the current life. Moreover, since the process is Poisson(), t and t are independent, with distributions given by Ft (x) = 0, if x < 0; 1 ex if x 0, if x < 0; x , if 0 x < t; Ft (x) = 1 e 1 if x t.
0.
Notice that the distribution of t does not depend on t, and is the exponential() distribution. The distribution of t is given by the convolution of the two above distributions. If x < t, we have
x
Ft (x) =
0 x
If x t, we have to take into account that the distribution of t has a mass of et in point {t}; the convolution is then given by
x
Ft (x) =
0 t
46
Putting together the two cases, the solution can be expressed as Ft (x) = 0, if x < 0; 1 ex min(t, x) ex if x 0.
Note that P[t x, t > y] = P[tx > y + x], and using the previous result, we get that this converges to v(x + y) as t tends to innity. Similarly, observing the case y = 0, we infer that
t+
We thus obtain that v(x + y) = v(x)v(y). Since v is monotone, it is a classical exercise to check that there must exist c, such that v(x) = ex (if g = log v, then g(x + y) = g(x) + g(y). . . ) By dierentiating, we nd 1 F(t) = x e ,
so F is the repartition function of an exponential random variable of parameter (> 0) (and = ). Exercise 4 Let M(t) = E(#Arrivals until time t); 1 1 M(t) = t (1 e2t ). 2 4 = M(t). Our objective is to show that M(t) We know that the renewal equation
T
A(T ) = F(T ) +
0
A(T t) F(dt)
has a unique solution and this solution is M (see course notes or page 183 of the textbook). So, we only T need to check that M(T ) = F(T ) + 0 M(T t) F(dt). First note that, for T 0,
T
F(T ) =
0
2 tet dt = 1 eT T eT .
T
Next,
0
M(T t) F(dt) = = 3 2
T
T
0
tet dt
0 3
1 1 1 (T t) + e2(T t) 2 tet dt 2 4 4
T
t2 et dt
0
2 4
tet dt +
0
2 2T e 4
tet dt.
0 T 0
Using the shortcuts in the statement of the exercise and simplifying, we get F(T ) + M(T ) as required.
M(T t) F(dt) =
47
Exercise 1 Sir Galton is worried about the survival of his family name. He himself has three sons, and estimates that each of them has probability 1/8 to have no boy, probability 1/2 to have 1 boy, probability 1/4 to have 2 boys, and probability 1/8 to have 3 boys. He thinks that these probabilities will be constant in time, that the numbers of children his descendants will have are independent random variables, and since he lives in the xixth century, he believes that only men can pass on their name. According to these assumptions, what is the probability that his name will become extinct ? Exercise 2 A server takes 1 minute to serve each client. During the n-th minute, the number Zn of clients that arrive and get in line for service is a random variable. We assume that these variables are independent and that P(Zn = 0) = 0.2, P(Zn = 1) = 0.2, P(Zn = 2) = 0.6.
The attendant can leave to have a coee only if there are no clients to serve. What is the probability that he can ever leave for a coee? Exercise 3 (i) Show that, for a positive random variable Y with i2 P(Y = i) < , we have
Var(Y) = g (1) + E(Y) E(Y)2 , Y where gY is the probability generating function of Y. (ii) Let (Xn )n 0 be a branching process (as is usual, assume X0 = 1) and let Z be a random variable whose distribution is equal to the distribution of the number of descendents in (Xn ). Let m = E(Z). Using the relation gXn+1 = gXn gZ , show that E(Xn ) = mn . (iii) Let 2 = Var(Z). Again using gXn+1 = gXn gZ , show that gn+1 = (gn gZ )(g )2 + (g n gZ )g . X X X Z Z (iv) Show by induction that Var(Xn ) =
mn (mn 1) 2 m2 m n2
if m 1; if m = 1.
48
Solutions
Exercise 1 Let p be the probability that the descendence of one of Galtons sons dies out. Since the average number of boys is strictly larger than 1, we know that p > 0. The xed point equation in our case reads p= 1 1 1 1 + p + p2 + p3 . 8 2 4 8
As always, p = 1 is a solution. It is not the one we are looking for, so we can simplify the equation into p2 + 3p 1 = 0. The solution we are looking for is p= 13 3 . 2 Now, the descendences of Galtons three sons are assumed to be independent. The probability that they all become extinct is thus 3 13 3 , 2 and eternal survival is the complementary event. Exercise 2 We use a branching process as a model to describe the problem. Let us suppose that there are X0 clients in line. They constitute generation 0. The direct descendents of a client are those that arrive while that client is being served. Generation n + 1 is formed of direct descendents of clients of generation n. Thus, each one of the Xn clients of generation n will be served during a minute during which a certain number Zi (i = 1, . . . , Xn ) of clients arrive: his direct descendents. Once all clients of generation n have been served, we nd Xn+1 = Z1 + + ZXn new clients in line Xn is thus a branching process. The probability that the server can have a pause is equal to the probability of extinction of the branching process, which in turn is given by the smallest solution of the equation = namely = 1/3. Exercise 3 (i) For |s| < 1, we have g (s) Y When
2 n=1 n
3 1 1 + + 2 , 5 5 5
d2 = 2 ds
n=0
s P(Y = n) =
n=0
d2 n s P(Y = n) = ds2
n=1
n(n 1)P(Y = n) =
n=1
n2 P(Y = n)
n=1
(ii) We know that the generating function of Xn is given by the recursion formula gXn+1 = gXn gZ , so that we get (1) and then E(Xn+1 ) = g n+1 (1) = g n (gZ (1)) g (1) = E(Xn )E(Z), Z X X since gZ (1) = 1. Thus, E(Xn ) = mn for each n 1. (iii) (iv) We obviously have Var(X0 ) = 0. Deriving equation (1), we get gn+1 = (g n gZ ) g + (g n gZ )g = (gn gZ )(g )2 + (g n gZ )g . z Z Z X X X X Z X We then have Var(Xn+1 ) = gn+1 (1) + E(Xn+1 ) E(Xn+1 )2 X g n+1 = (g n gZ )g Z X X
2 = (Var(Xn ) + E(Xn ) E(Xn ))E(Z)2 + E(Xn )(Var(Z) + E(Z)2 E(Z)) + E(Xn+1 ) E(Xn+1 )2
= mn 2 + m2 Var(Xn ). Thus,
Var(Xn+1 ) = mn 2 + m2 Var(Xn ) = (mn + mn+1 )2 + m4 Var(Xn1 ) (n + 1)2 n n+1 2n 2 2n+2 = = (m + m + + m ) + m Var(X0 ) = mn (mn+1 1) 2
m1
if m = 1 if m 1
50
Exercise 1 Suppose that in a branching process, the number of ospring of an initial particle has a distribution whose generating function is f (s). Each member of the rst generation has a number of ospring whose distribution has generating function g(s). The next generation has generating function f , the next g, and the functions continue to alternate this way from generation to generation. What is the probability of extinction ? Does this probability change if we start the process with the function g, and then continue to alternate ? Can the process go extinct with probability 1 in one case, but not in the other ? Exercise 2 Consider a branching process with initial size N and probability generating function (s) = q + ps, p, q > 0, p + q = 1.
Determine the probability distribution of the time T when the population rst becomes extinct. Exercise 3 Points are thrown on 2 according to a Poisson point process of rate . What is the distribution of the distance from the origin to the closest point of the point process ?
Exercise 4 Let t X(t) be a Poisson process of rate . What is the distribution of the set of jumps of the process t X(et ) ?
51
Solutions
Exercise 1 Considering only even generations, one obtains a usual branching process with generating function f (g(s)). The probability of extinction is thus the smallest p > 0 such that f (g(p)) = p. If we start with the generating function g instead of f , then the probability of extinction is the smallest p such that g( f (p)) = p. These two probabilities are dierent in general. Choose f (s) = 1 3 + s, 4 4
g(s) = s2 . Solving f (g(p)) = p for 0 < p < 1 leads to p = 1/3, while solving g( f (p)) = p for 0 < p < 1 leads to p = 1/9. It is however not possible to nd an example where one has a non-zero probability of survival in one case, but not on the other. Recall that this question is decided by comparing the expected number of descendants to the value 1. Here, if m1 is the expected number of descendants associated to the generating function f , and m2 is the expected number of descendants associated to the generating function g, then the expected number of descendants associated to the generating function f (g(s)) is m1 m2 , which is the same as the value for g( f (s)). This can be proved by representing the process using random variables, but can also be seen directly on the characteristic function, since the value is d f (g(s)) ds = g (1) f (g(1)) = g (1) f (1) = m1 m2 =
s=1
d g( f (s)) ds
.
s=1
Exercise 2 Let us consider the process started with one individual, and let Xn be the size of the population at time n. The generating function of Xn is (n) (s), and one can show by induction that (n) (s) = q + pq + + pn1 q + pn s. The probability that Xn = 0 is thus (n) (0) = q + pq + + pn1 q = q(1 pk ) = 1 pk . 1 p
When the population is started with N individuals, the ospring of these individuals are independent, and thus P[T N] = P[the ospring of the N individuals are all extinct] = (1 pk )N . Exercise 3 The distance d from the origin to the closest point is larger than r if and only if there is not point of the Poisson process that falls within the ball of radius r centred at the origin. The number of such points follows a Poisson distribution with parameter r2 . Hence, P[d > r] = exp(r2 ).
52
It thus follows that d has a probability density 2r exp(r2 )dr. Exercise 4 For any interval I, let N(I) be the number of jumps of X occuring during I, and N (I) be the number of jumps of X that occur during I. One can check that N([eb , ea ]) = N ([a, b]). As a consequence, it is easy to see that for two disjoint intervals I1 and I2 , the random variables N (I1 ) and N (I2 ) are independent. Moreover, N ([a, b]) follows a Poisson distribution with parameter
b
(ea eb ) =
a
ex dx.
The set of jumps of the process t X(et ) thus forms a Poisson point process with intensity measure ex dx.
53
1.
let Xi be a sequence of independent 2 V (Xi ) = E[(Xi E[Xi ])2 ] = i . Show random variables with that the sequence
E[Xi ] = 0
and
Sn =
i=1
is a martingale with respect to
2 2 (Xi i )
Proof. (For denition of martingale see Denition 5.2) We rst check the inte-
grability os
Sn .
n n 2 2 Xi i |] E[ i=1 n i=1 n 2 2i < i=1 2 2 |Xi i |] i=1 n 2 2 E[|Xi i |]
E[|Sn |] = E[|
i=1
hence
2 2 E[Xi ] + E[i ] =
Sn
is integrable.
To check that
Sn
is
Fn
Xi
for
i =
1, 2, . . . , n
are
Fn
2 2 2 2 E[Sn+1 |Fn ] = E[Sn + Xn+1 n+1 |Fn ] = E[Sn |Fn ] + E[Xn+1 |Fn ] n+1 2 2 2 2 = Sn + E[Xn+1 ] n+1 = E[Sn |Fn ] + n+1 n+1 = Sn .
Hence
Sn
is an
F -martingale.
2.
Let
Xi
be IID with
Xi N (0, 1)
for each
and put
Yn =
n i=1
Xi .
Show that
Sn = exp{Yn n2 /2}
is an
Fn -martingale Sn .
for every
R.
and by knowing that
Proof. (For denition of martingale see Denition 5.2) We rst check the in-
tegrability os
Since
Sn 0
n
E[ecX ] = ec
/2
for
X N (0, 1)
Xi n2 /2}] = en
/2 i=1
E[eXi ] = 1 < .
Sn
is integrable.
To check that
Sn
is
Fn
Xi
for
Fn
i = 1, 2, . . . , n Sn is
measurable as well.
54
= Sn E[exp{Xn+1 2 /2}|Fn ] = Sn .
=1 (as in 7.1)
Hence
Sn
is an
F -martingale.
3.
Let
Xi
Sn =
i=1
is an
Xi =0
for
F -martingale.
i = j. i > 1,
hence for
E[Xi ] = 0
for
1n
n+m
we have
Cov(Xn , Xn+m )
Sn
is an
F -martingale.
Hence
Xi
4.
Let
{Mn }
and
{Nn }
be square integrable
F -martingales.
n+1
M, N
Proof. (For denition of square integrability see Denition 8.1, for denition of
quadratic variatio and covariation see page 54) The right hand side of equality (1) yields
M, N
n
n+1
M, N
=
n1
=
i=0
55
{Mn }
and
{Nn },
and the
Mn
and
Nn
with respect to
Fn ,
we get
E[Mn+1 Nn+1 Mn+1 Nn Mn Nn+1 + Mn Nn |Fn ] = E[Mn+1 Nn+1 |Fn ] E[Mn+1 Nn |Fn ] E[Mn Nn+1 |Fn ] + E[Mn Nn |Fn ] = E[Mn+1 Nn+1 |Fn ] Nn E[Mn+1 |Fn ] Mn E[Nn+1 |Fn ] + Mn Nn = E[Mn+1 Nn+1 |Fn ] Nn Mn ,
and the proof is done.
5.
Let
{Mn }
and
{Nn }
be square integrable
F -martingales. n 0,
n.
1. Let
and
M + N
= 2 M
+ 2 M, N
+ 2 N
| M, N
n|
n,
n 0.
Proof. (For denition of square integrability see Denition 8.1, for denition of
quadratic variatio and covariation see page 54) 1) By Denition 8.3 we get
n1
M + N
n1
=
i=0
=
i=0 n1
E[((Mi+1 Mi ) + (Ni+1 Ni ))2 |Fi ] E[2 (Mi+1 Mi )2 + 2(Mi+1 Mi )(Ni+1 Ni ) + 2 (Ni+1 Ni )2 |Fi ]
i=0 n
= 2 M
+ 2 M, N
+ 2 N
n,
which is what we wherev set out to prove. 2) It is easily seen that the quadratic variation is alway positive and by using this observation, combined with the result from the rst part of this exercise, we get, for any
R, 0 M N
n
= M
2 M, N
+ 2 N
n.
Let
= M, N 0 M = M
n n
n/
n, n
2 M, N 2 M, N
+ 2 N N
n
n 2 n/
2 n/
+ M, N
= M
M, N
2 n/
hence
M, N
and the proof is done.
2 n/ n
M M
n
M, N
56
6.
Let
{Mn }
and
{Nn }
be square integrable
F -martingales.
parallellogram equality,
M, N
1 ( M +N 4
M N
n ).
M +N = M
hence
M N
n
+ 2 M, N
n
+ N
n
+ 2 M, N
n ).
= 4 M, N
n,
M, N
= 1( M + N 4
M N
7.
Let
{Mn }
and
be bounded
{Nn } be two square integrable F -martingales and let and F -adapted processes. Derive the Cauchy-Schwarz inequality | IM (), IN () n | IM ()
n
IN () n , n 0. IM ()
and
IN ()
tegrable
F -martingales,
problem 8.2.
8.
In this problem we look at a simple market with only two assets; a bond and a stock. The bond price is modelled according to
Bn B0
where
= (1 + r)Bn1 =1
for
n = 1, 2, . . . , N
r > 1
is the constant rate of return for the bond. The stock price is
Sn S0
where
= (1 + Rn )Sn1 =s
for
n = 1, 2, . . . , N
s>0
and
{Rn } is a sequence of IID random variables on (, F, P). Fur{Fn } be the ltration given by Fn = (R1 , . . . , Rn ) n = 1, . . . , N
a martingale with respect to the ltration
Sn /Bn
{Fn }?
For every
n = 0, 1, 2, . . . , N let xn and yn be the number of stocks and bonds respectively bought at time n and held over the period [n, n + 1). furthermore, let Vn = xn Sn + yn Bn
57
[n, n + 1),
and let
V0
n 1,
which at time
is
xn1 Sn + yn1 Bn .
n,
i.e.
withdraw or add any money to the portfolio. A portfolio with this restriction is as a pair
xn1 Sn + yn1 Bn = xn Sn + yn Bn , n = 1, . . . , N.
b) Show that if so is
Vn /Bn ,
where
Sn /Bn is a martingale with respect to the ltration {Fn }, then Vn is the portfolio value of any self-nancing portfolio.
{xn , yn }
with probability one, i.e. your are certain to make money on your
strategy. We say that a model is arbitrage free if the model permits arbitrage portfolios. c) Show that if arbitrage free. Let
Sn /Bn
Qn
{Fn }
such that
Qn > 0 a.s.
and
Q0 = 1 a.s..
d) Show that even if Sn /Bn is not a martingale with respect to the ltration {Fn }, nding a process Qn as dened above such that Sn Qn /Bn , will give that Vn Qn /Bn is a martingale with respect to the ltration {Fn }, and furthermore, that Vn is arbitrage free. Even though the multiplication of the positive martingale which gives us the ability to change measure.
Qn
portant, we will later in the course see that this is in fact a very special action In nancial applications, this is important since the portfolio pricing theory say that a portfolio should be priced under a risk neutral measure, a measure where all portfolios, divided by the bank process
Bn
should be a martingale.
theory is based on a no-arbitrage assumption, which hold if is a martingale as proven in this exercise. that
So the existence of
Vn
Vn
consisting of
Sn
and
Bn
in a consistent way.
Sn /Bn
is an
the process is integrable, measurable and have the martingale property, i.e. that
58
E[Sn+1 /Bn+1 |Fn ] = Sn /Bn . and the Rn 's are IID we get
Since
Sn /Bn 0
n i=1 E[1 n i=1 (1
Since is
for every
n, Bn
is deterministic
(2)
Sn /Bn
is integrable if
Rn
is.
Sn /Bn
Sn E[1 + Rn |Fn ] Sn 1 + E[Rn |Fn ] Sn E[1 + Rn+1 |Fn ] = = . Bn (1 + r) Bn 1+r Bn 1+r E[Sn+1 /Bn+1 |Fn ] = Sn /Bn
we must have
E[Rn+1 |Fn ] = r.
Hence
Sn /Bn
is an
{Fn }-martingale
if
E[Rn+1 |Fn ] = r.
b) Since the denition of a self-nancing portfolio is that
xn1 Sn + yn1 Bn = xn Sn + yn Bn , n = 1, . . . , N.
we get, by the denition of
Vn ,
E[
xn+1 Sn+1 + yn+1 Bn+1 xn Sn+1 + yn Bn+1 Vn+1 |Fn ] = E[ |Fn ] = E[ |Fn ] Bn+1 Bn+1 Bn+1 Bn+1 Sn+1 |Fn ] + yn = xn E[ Bn+1 Bn+1 {xn , yn }
are
since
Fn -measurable.
Sn /Bn
is an
Fn -
martingale we get
E[
so
Vn /Bn
Fn -martingale
if
Sn /Bn
is.
Vn = xn Sn + yn Bn is Vn /Bn is a martingale if Sn /Bn is. To check that any self-nancing Vn is arbitrage free, we must have V0 = x0 S0 + y0 B0 = 0. Let Sn /Bn E[Vn+1 /Bn+1 ] = V0 /B0 = V0 = 0.
Assume that
Bn <
for any
P (Vn 0) = 1 n we get E[
and
Since
E[Vn /Bn ] = 0
and
59
where
I{}
E[Vn /Bn ] = 0,
Vn . Sn Qn /Bn
is a martingale
d) Following the same lines as in b) we get that if with respect to the ltration
{Fn }
and
Vn
is self nancing,
E[
so
Vn+1 Qn+1 xn Sn+1 Qn+1 + yn Bn+1 Qn+1 xn Sn Qn + yn Bn Qn Vn Qn |Fn ] = E[ |Fn ] = = , Bn+1 Bn+1 Bn Bn Vn Qn /Bn
is an
Fn -martingale
if
Sn Qn /Bn
E[
Assume that P (Vn 0) = 1 and P (Vn > 0) > 0. Qn > 0, Bn < for any n we get
E[Vn Qn /Bn ] = 0
and
E[
where
I{}
E[Vn Qn /Bn ] = 0,
Vn .
9.
A coin is tossed
invested in a coin toss gives the net protof and the net prot of
with probability
nth
P(Xn = 1) = p
and
P(Xn = 1) = 1 p,
and the Xn 's are independent of each other. Let Fn = (X1 , . . . , Xn )and let Sn , n = 1, 2, . . . , N ne the wealth of the investor at time n. Assume further that the initial wealth S0 is a given constant. Any non-nergative amount Cn can be invested in coin toss n + 1, n = 1, . . . , n 1, but we assume that borrowing money is not allowed so Cn [0, Sn ]. Thus we have
Sn+1 = Sn + Cn Xn+1 , n = 1, . . . , N 1
rate of return a) Show that
and
Cn [0, Sn ].
Finally assume that the objective of the investor is to maximize the expected
{F}.
Ln = log(Sn ) n
where
Cn the investor use in the investment game, = p log(p) + (1 p) log(1 p) + log(2) is a superto the ltration {Fn }.
for
x [0, 1]
and
p (1/2, 1).
60
log(Sn ) n
d) Show that if
Cn = Sn (2p 1), Ln
is an
{F}-martingale. {F}
we want
Proof. (For denition of submartingale and supermartingale see the text follow-
Sn
is a submartingale w.r.t.
E[Sn+1 |Fn ] Sn ,
and
Sn
are
Fn -measurable}
Fn } = Sn + Cn E[Xn+1 ] = Sn + Cn (1 p 1 (1 p)) Sn ,
independent of
>0
hence
Sn
is a submartingale w.r.t.
{Fn }.
E[Ln+1 |Fn ] Ln ,
E[Ln+1 |Fn ] = E[log(Sn+1 ) (n + 1)|Fn ] = E[log(Sn + Cn Xn+1 )|Fn ] (n + 1) = E[log(Sn (1 + Cn Xn+1 /Sn ))|Fn ] (n + 1) = E[log(Sn ) + log(1 + Cn Xn+1 /Sn ))|Fn ] (n + 1) = log(Sn ) n +E[log(1 + Cn Xn+1 /Sn ))|Fn ]
=Ln
g (x) = p/(1+x2 )(1p)/(1x2 ) < 0 for x [0, 1), g is concave in that region, and the maximum is x = 2p1 since g () = p/(1+)(1p)/(1) = 0 x x x so g(x) g() for all x [0, 1]. Since Cn /Sn [0, 1], x
Since
Ln
is a supermartingale w.r.t.
{Fn }.
supermartingale w.r.t.
{Fn }.
Ln = log(Sn ) n is a E[Ln ] L0 so
d) For
Cn = SN (2p 1)
we get
Ln
is a
{Fn }-martingale
Cn = Sn (2p 1).
61
10.
Xn n = 0, 1, 2, . . . is that price of a stock at time n and assume that {Fn }. This means that if we buy one unit of stock at time n, paying Xn , the expected price of the stock tomorrow (represented by the time n + 1) given the information Fn is lower
Assume that
Xn
than today's price. in other words, we expect the price to go down. Investing in the stock does not seem to be a good idea, but is it possible to nd a strategy that performs better? The answer is no, and the objective of this exercise is to show that. Let after
Cn
be a process adapted to
{Fn }
with
0 Cn n = 0, 1, 2, . . ., C
with respect to
days is given by
IX (C)n ,
X.
Xn
bounded process
i.e. that
IX (C)n
is also a supermartingale.
IX (C)n =
Ci (Xi+1 Xi )|Fn ]
= E[
i=0
= E[IX (C)n + Cn (Xn+1 Xn )|Fn ] = {IX (C)n = IX (C)n + E[Cn (Xn+1 Xn )|Fn ] = {Cn = IX (C)n + Cn (E[Xn+1 |Fn ] Xn ) {Xn IX (C)n + Cn (Xn Xn ) = IX (C)n .
Hence and
Fn
-measurable}
Xn
is supermartingale and
Cn 0}
IX (C)n
{Fn }
if
Xn
is.
62
Bn , n = 0, 1, 2, . . .
Bn P 0 B n
that is for every
as
n ,
>0 P Bn > B n 0
as
n . X ,the
Chebyshev's
inequality is
P(|X| > )
Since
E[X 2 ]
2
B P
= n,
Bn > B n
=P =
Bn > n
as
= P (|Bn | > n ) n .
2 n E[Bn ] = (n )2 (n )2
1 1 0 2 n
Hence
Bn P 0 B n
as
n .
2.
Assume the value of the bond's rate of return is
r = e2 1
for some constant . What should be the distribution of the (1+Rn ) in order to model Sn = Sn /Bn as a geometric Brownian
2 Sn = seWn 2n , S0 = s,
where
Wn
Sn
we get
Sn
which holds if
1 + Rn+1 = se(Wn+1 Wn ) .
63
1.
Let
{Mt }
and
{Nt }
be square integrable
Ft -martingales. t 0,
1. Let
and
M + N
= 2 M
+ 2 M, N
+ 2 N t.
| M, N t |
N t , t 0.
Proof. (For denition of square integrability see Denition 11.3, for denition
of quadratic variation and covariation see pages 74-75) 1) We use the denition of the covariation process to get
n1
M + N
= lim
(Mi+1 + Ni+1 Mi Ni )2
i=0
n1
= lim
= lim
= lim
2 (Mi+1 Mi )2 + 2 lim
i=0 n1
(Mi+1 Mi )(Ni+1 Ni )
i=0
+ lim 2 M
0 t
2 (Ni+1 Ni )2
i=0 t
+ 2 M, N
+ 2 N t,
which is what we wherev set out to prove. 2) Recall the Cauchy-Schwarz inequality for
n-dimensional
euclidean space
ai bi
i=1
We have
a2 i
i=1 i=1
b2 . i
n1
M, N
= lim
(Mi+1 Mi )(Ni+1 Ni )
i=0 n1 n1
lim
(Mi+1 Mi )2
i=0 i=0
(Ni+1 Ni )2
64
and since
n1
n1
M, N
lim
(Mi+1 Mi )2
i=0 t i=0
(Ni+1 Ni )2
2.
Let
{Mt }
and
{Nt }
be square integrable
Ft -martingales.
parallellogram equality,
M, N
1 ( M +N 4
M N t ), t 0.
M +N = M
hence
M N
t t
t + 2 M, N
t+ N
+ 2 M, N
= 4 M, N t ,
M, N
= 1( M + N 4
M N t ).
65
St
of a risky asset (i.e. an asset that has no deterministic payo )at time
is
St = se(r 2
where time
)t+Bt
Bt
initial value of the asset. The value of a European Call option, with maturity
is
(ST K)+
T.
If
T > t,
compute explicitly
of a function on a time
h of the Brownian motion evaluated at time T , h(BT ), conditioned t < T is only dependent on the value Bt and the time to maturity
2 2
T t.
)(T t)+(BT Bt )
2 2
)(T t)+(BT Bt )
66
BT Bt =
T tX
where
N (0, 1),
so
2 2
)(T t)+(BT Bt )
K)+ ]
= EBt [(St e(r 2 )(T t)+ T tX K)+ ] 2 x2 1 (St e(r 2 )(T t)+ T tx K)+ e 2 dx. = 2 R
Since
()+
St e(r
2 2
)(T t)+ T tx
which may be
rewritten to get
separated as
log x
K St
(r )(T t) 2 . T t d1 ,
the integral may be written as
x2 2 1 E[(ST K)+ |Ft ] = (St e(r 2 )(T t)+ T tx K)e 2 dx 2 d1 x2 2 x2 1 1 = St e(r 2 )(T t)+ T tx e 2 dx K e 2 dx 2 d1 2 d1 P(Xd1 )
1 = St er(T t) 2
d1
2 2
(T t)+
2 T tx x 2
1 dx K 2
e
d1 P(Xd1 )
x2 2
dx
=P(Xd1 T t)
2.
Let
Bt
Ft
be the ltration
generated by
Bt .
Show that
respect to
Fs
Ft ,
namely
3 E[Bt |Ft ] = E[(Bt Bs + Bs )3 |Ft ] = E[(Bt Bs )3 + 3(Bt Bs )2 Bs 3 2 + 3(Bt Bs )Bs + Bs |Ft ] = E[(Bt Bs )3 ] + 3Bs E[(Bt Bs )2 ] 3 2 + 3Bs E[Bt Bs ] + Bs .
67
Bt Bs N (0, t s)
moments is zero, so
3.
Show that the following processes are martingales with respect to tion generated by theone dimensional Brownian motion 1. 2.
Ft ,the
ltra-
Bt
Bt
is an
Ft -martingale
we get
3 Bt 3tBt with respect to Ft . 4 2) We start by computing E[Bt |Fs ], and as in Exercise 12.2 we do this by separating Bt in a part that is measurable with respect to Fs and part that is
independent of
Fs
4 E[Bt |Fs ] = E[(Bt Bs + Bs )4 |Fs ] = E[(Bt Bs )4 + 4(Bt Bs )3 Bs 4 3 2 + 6(Bt Bs )2 Bs + 4(Bt Bs )Bs + Bs |Fs ] = E[(Bt Bs )4 ] 4 3 2 + 4Bs E[(Bt Bs )3 ] + 6Bs E[(Bt Bs )2 ] + 4Bs E[Bt Bs ] + Bs . Recall that Bt Bs N (0, t s) so we may write Bt Bs = T sX X N (0, 1) so we may rewrite our expression as
where
3 4 2 E[Bt |Fs ] = (t s)2 E[X 4 ] + 4Bs t s E[X 3 ] + 6Bs (t s)E[X 2 ] 3 4 + 4Bs t sE[X] + Bs
and since all odd moments of the standard normal distribution is zero and the second moment is one we have
E[X 4 ]
mal distribution
X (u) = E[euX ] = eu
and use the result that the
/2
n'th
derivative of
n'th
moment of
X.
in
u=0
is the
(n)
68
and since
X (0) = 1
we get
X (0) = E[X 4 ] = 3.
(4)
2 that Bt
2 4 Bt 6tBt + 3t2 ,
in an
Ft -martingale,
4 2 2 4 2 E[Bt 6tBt + 3t2 |Fs ] = 3(t s)2 + 6Bs (t s) + Bs 6tE[Bt |Fs ] + 3t2 2 4 2 = 3(t s)2 + 6Bs (t s) + Bs 6tE[Bt t + t|Fs ] + 3t2 2 4 2 = 3(t s)2 + 6Bs (t s) + Bs 6t(Bs s + t) + 3t2 2 2 4 2 = 3t2 6ts + 3s2 + 6Bs t 6sBs + Bs 6tBs + 6ts 6t2 + 3t2 4 2 = Bs 6sBs + 3s2 ,
hence
2 4 Bt 6tBt + 3t2
is an
Ft -martingale
4.
Let
t i
t i
such that
ti <
n1
Sn =
i=0
where
Bt (Bti+1 Bti ), i
Bti
Sk , 0 k n B.
Sk .
k1
E[Sk |Fk1 ] = E[
i=0
are
Fk1 -measurable
i k 2} =
i=0
Bt (Bti+1 Bti ) i
=Sk1
+ E[Bt (Btk Btk1 )|Fk1 ] = {E[Btk1 (Btk Btk1 )|Fk1 ] = 0} k1 = Sk1 + E[Bt (Btk Btk1 )|Fk1 ] E[Btk1 (Btk Btk1 )|Fk1 ] k1 = Sk1 + E[(Bt Btk1 )(Btk Btk1 )|Fk1 ] = Sk1 + (t tk1 ) k1 k1 = Sk1
for any
tk1 < t k1 tk
hence
Sk
B.
69
5.
Let
Bt
Xt
t
Xt =
0
1. Determine the expectation
est dBs
and the variance
E[Xt ]
V (Xt )
of
Xt .
Wt =
has distribution
Proof. 1) By part
2(t + 1)Xlog(t+1)/2
Wt N (0, t).
of Proposition 13.11, that denes properties of then Ito
(vi)
est ,
Xt N 0,
0
Hence
(est )2 ds = N 0,
0
1 e2(st) ds = N 0, (1 e2t ) . 2
.
Xt
1 Xt N 0, 2 (1 e2t )
2) From the rst part of the exercise, we know that For a normally distrbuted random variable
Xt N 0, 1 (1 e2t ) 2
it holds that
Y N (0, )
cY
N (0, c2 2 )
hence
Wt N 0,
2(t + 1)
21
= N 0, (t + 1)(1
1 t ) = N 0, (t + 1) = N 0, t . t+1 t+1
6.
Let
zR
T
and
(s, ) V
such that
F () = z +
0
in the following cases 1. 2. 3.
(s, )dBs
3 F () = BT ().
F () =
T 0
3 Bs ds.
70
Proof. 1) We get
d(tBt ) = tdBt + Bt dt
we have to get
tBt = 0 sdBs + 0 Bs ds which may be rewritten t t 3 Bs ds = 0 (t s)dBs . We may now write the BT 0
T 3 BT = z + 0 2 3(Bt + (T t))dBt
using as
tBt =
t 0
tdBs
where 2)
3 z = E[BT ] = 0.
Hence
T 3 T BT = z + 0
for some
T 2 s3Bs dBs + 0
3sBs ds +
0
3 Bs ds,
z R.
T 3 3 Bs ds = z + T BT 0
which by problem 1) gives
T 2 s3Bs dBs 0 0
3sBs ds
T 3 Bs ds = z + 0
We need to rewrite
3sBs ds
ds.
Study
d(T 2 BT ) = 2T BT dT + T 2 dBT ,
hence
T 2 BT =
T 0
2sBs ds +
T
T 0
s2 dBs 1 2
and by writing
T 2 BT =
T 0
T 2 dBs
we get
sBs ds =
0
hence
(T 2 s2 )dBs ,
0
T 3 Bs ds = z + 0 0
3 2
(T 2 s2 )dBs .
0
as
71
Hence
d(eBt t/2 ) = eBt t/2 dBt d(eBt +t/2 ) = eBt +t/2 dBt ,
we may write
1 1 eT /2 (eBT () + eBT () ) = eT /2 (eT /2 eBT ()T /2 + eT /2 eBT ()+T /2 ) 2 2 T 1 T /2 T Bs s/2 eBs +s/2 dBs ) e dBs eT /2 = z + eT /2 (e 2 0 0 1 = z + (eT 2
T T T
=z+
0
where
eT eBs s/2 e 2
is
+ eT /2 E[eBT +T /2 ]
=1
1 1 = eT /2 eT /2 + eT /2 = (eT + 1). 2 2
7.
Let
Xt
dXt = t Xt dt + t Xt dBt
where
(3)
and
is a Brownian
Xt
and compute
E[Xt ].
t 0
zR
and
(t, ) V
t 0 t 0
such that
X(T, ) = z + Xt
is
Proof. 1) Let
Xt = e
s ds s ds
Yt ,
the dierential of
t 0
dXt = t e
Yt dt + e
s ds
dYt = t Xt dt + e
t 0
s ds
dYt
t 0
s ds
dYt = t e Yt
t 0
t 0
s ds
Yt dBt
=Xt
which holds if
dYt = Yt dBt
hence
Yt = y0 e
t dBt 1 2
t 0
2 t dt
72
where
y0 = 1
since
X0 = 1.
Xt
Xt = e
t dBt +
t (t 1 t )2 dt 2 0
Yt
gives
t 0
E[Xt ] = E[e
t 0
s ds
Yt ] = e
s ds
E[Yt ] = e
=Y0
t 0
s ds
T 0
T s ds
XT = e
and that
s ds
YT = e
T 0
(1 +
0
s Ys dBs ) = e
T 0
T s ds
+
0
T 0
s ds
s Ys dBs
=z
(s, ) = e s
s ds
s Ys ().
(s, ) V ,
the criterias
are given in Denition 13.1. Part 1 and 2 of Deniton 13.1 is showed by noticing is the product of the processes therefore universally measurable, and does also fullll criterias 1) and 2). by using that
T 0
u du
therefore fullll the conditions 1) and 2). The product of these two processes
0<K <
such that
|t | < K E
for every
t [0, T ],
T 0
(t e
0
T 0
s ds
Yt )2 dt = e
T
T 0
T s ds
E
0
(4)
e
t 0
s ds =
K2 E
0
2 s ds
Yt2 dt = {Fubini} =
0
E Yt2 dt.
(5)
Yt2 = e
2s dBs
t 0
Yt2 = e
T
t 0
t 0
2s dBs 1 2
t (2)2 ds s 0
t 0
2 s ds
so that
E
0
(t e
T
T 0
s ds
Yt )2 dt
0
t 0 1 2s dBs 2
E e
1 2s dBs 2
t (2s )2 ds 0
t 0
2 s ds
dt
=
0 T
e e
0
t 0
2 s ds
E e
t (2s )2 ds 0
dt =
0 T
t 0
2 s ds
dt 1 < .
Hence
t 0
K 2 ds
dt
0
K2t
dt =
0
K2t
dt =
eK
K2
s V .
8.
Let
f (t) = et /2 1 and let B be a brownian motion on the probability space (, F, {Ft , t t}, P). Show that there exists another Brownian motion B such
f (t)
that
Xt =
0
1 dBs = 1+s
t 0
sdBs
73
Xt
is
f (t)
=
0 t
1 ds = 1+s sds.
et 0
2 /2
= t2 /2 =
0
t}, P)
Xt =
0
which solves the problem at hand.
sdBs
9.
Let
Xt
constants and
1 St .
Xt .
then the SDE of
f (t, x) = e(
/2)tx
1 St
is
1 dSt =
= ( 2 /2)e(
/2)tx
dt e(
/2)tx
2 1 dBt + 2 e( /2)tx dt 2
1 1 = ( 2 )St dt St dBt .
2)
1 1 1 1 d(Xt St ) = Xt dSt + s1 dXt + X, S 1 = Xt (( 2 )St dt St dBt ) t 1 1 + St ((Xt + )dt + (Xt + )dBt ) + (St )(Xt + )dt 1 1 1 = ( 2 )Xt St dt Xt St dBt + Xt St dt 1 1 1 1 1 + St dt + Xt St dBt + St dBt 2 Xt St dt St dt 1 1 1 = ( 2 ) + 2 Xt St dt + + Xt St dBt + St dt 1 1 1 + St dBt = ( 2 ) + 2 Xt St dt + + Xt St dBt 1 1 1 1 + St dt + St dBt = St dt + St dBt .
3) From 2) we get that
t 1 Xt St = 0
multiplying both sides by
t 1 Ss ds + 0 1 Ss dBs
St
and express
St
Xt = e(
t
t /2)t+Bt
( )
0
e(
/2)sBs
ds
+
0
e(
/2)sBs
dBs .
74
1
Compute the stochastic dierential 1. 2. 3. 4. 5.
dz
when
Zt = e , R, t [0, ). Zt =
t 0
g(s)dBs
where
g(s)
Zt = eBt . Zt = eXt
2 Zt = Xt
where
dXt = dt + dBt .
where
dXt = Xt dt + Xt dBt .
Proof. 1) Since
2) 3)
t is of rst variation, dZt = et dt = Zt dt. dZt = g(t)dBt since it is the dierential of an integral. x Let f (x) = e then, using the Ito formula, since Bt 1 dZt = f (Bt )dBt + f (Bt )d B 2 2 = Zt dBt + Zt dt. 2 = eBt dBt +
has quadratic
variation, we get
2 Bt dt e 2
2 Xt 2 1 dt dZt = f (Xt )dXt + f (Xt )d X t = eXt (dt + dBt ) + e 2 2 2 () = Zt dBt + ( + )Zt dt. 2 2 5) Let f (x) = x , we get 1 1 dZt = f (Xt )dXt + f (Xt )d X t = 2Xt dXt + 2d X t 2 2 2 2 2 2 2 = 2Xt (Xt dt + Xt dBt ) + Xt dt = 2Xt dt + 2Xt dBt + 2 Xt dt = (2 + 2 )Zt dt + 2Zt dBt .
75
2.
Compute te stochastic dierential for dierential
when
Zt = 1/Xt
where
Xt
has the
dXt = Xt dt + Xt dBt .
Proof. Let
f (x) = 1/x
and hence
f (x) = 1/x2
and
f (x) = 2/x3 .
By the Ito
formula we get
3.
Let
{Ft }
B.
Show
2 Bt t .
eBt
2 t 2
dBt -part,
the form
0dt + (. . .)dBt ,
dened
in Denition 13.1, then we may be certain that the process is a martingale by Proposition 13.11. 1)
E[
0
sds =
t2 < , 2
2Bt
2)
2 V , and hence Bt
is a martingale.
d(eBt
2 t 2
)=
2 t 2 Bt 2 t 1 2 Bt 2 t 2 dt + eBt 2 dB + 2 dt e e t 2 2 2 t 2
= eBt
And since
dBt .
t 0 t 0
E[
0 t
eBs
2 s 2
ds] = E[
0
2
E[e2Bs s ]ds
=
0
hence
e4 s e s ds < , 0 t <
is a matingale.
76
4.
Check whether the following processes are martingales with respect to the ltration generated by 1. 2. 3. 4.
Bt .
Xt = Bt + 4t
2 Xt = Bt
Xt = t2 Bt 2 Xt = Bt Bt
(1) (2)
t 0
rBr dr (Bt , Bt )
(1) (2)
is a two dimensional Brownian motion.
where
Proof. The proofs may (at least) be done in two ways, one by checking the
stochastic dierential of the process and the use the Martingale Representation Theorem (Theorem 15.3) to conclude whether the process is a martingale or not. We will do both. When using the Martingale Representation Theorem we should also prove that the integrand is a member of the class in the solutions. 1) E[Xt |Fs ] = E[Bt + 4t|Fs ] = Bs + 4t = Bs + 4s hence Bt + 4t is not an Ft -martingale. dXt = dBt + 4dt = g(s, )dBt hence Bt + 4t is not an Ft -martingale. 2 2 2 2) E[Xt |Fs ] = E[Bt |Fs ] = E[(Bt Bs + Bs ) |Fs ] = E[(Bt Bs ) |Fs ] + 2 2 2 2 E[Bs |Fs ] = t s + Bs = Bs hence Bt is not an Ft -martingale. 2 dXt = 2Bt dBt + dt = g(s, )dBt hence Bt is not an Ft -martingale. 3)
V,
which we omit
rBr dr|Fs ]
t
= t 2 Bs 2
0 s
r E[Br |Fs ] dr 2
s =Br s
r E[Br |Fs ] dr
=Bs
= t2 Bs 2
0
hence
rBr dr 2Bs
s t
rdr = s2 Bs 2
0
rBr dr = Xs
t 0
rBr dr
is an
Ft -
martingale.
(1) (2) (1) (2) 4) E[Xt |Fs ] = E[Bt Bt |Fs ] = {independent} = E[Bt |Fs ]E[Bt |Fs ] = (1) (2) (1) (2) Bs Bs hence Bt Bt is an Ft -martingale. (1) (2) (2) (1) (2) (1) (2) (2) (1) dXt = Bt dBt + Bt dBt + d B (1) , Bt = Bt dBt + Bt dBt hence (1) (2) Bt Bt is an Ft -martingale.
5.
Let
dXt = Xt dt + dBt , X0 = x0
where
, , x0
is a brownian motion.
77
1. determine 2. Determine
E[Xt ]. V (Xt ).
E[Xt ] = E[x0 +
0 t
Xs ds +
0
dBs ] = x0 + E[
0
Xs ds] + E[
0
dBs ]
=0
= x0 + E[
0
Xs ds]
E[Xt ] = E[Xt ]
E[X0 ] = x0
which has the solution
E[Xt ] = x0 et 2 2 2 t 2) V (Xt ) = E[Xt ] (E[Xt ]) = E[Xt ] x0 e so we need to nd an explicit 2 2 expression for E[Xt ]. Looking at the stochastic dierential of Xt we get
2 dXt = 2Xt dXt + d X t
t 2 Xt = x2 + 0 0 2 E[Xt ]
becomes
t 2 (2Xt + 2 )dt + 2 0
Xt dBt .
Xt dBt ] = x2 + E[ 0
0
2 2Xt dt] + 2 t
(6)
+ 2 E[
0
Xt dBt ] = {Fubini} = x2 + 0
0 =0 (Mg)
2 2E[Xt ]dt + 2 t.
Let
2 g(t) = E[Xt ],
g(t) = 2g(t) + 2 .
Dierentiating
e2t g(t).
e2t g(t)
d dt
we get
g(t) = e2t 2 .
78
e2t
we get
g(t) = e2t c + 2
0
e2s ds .
as
g(0) = x2 0
c = x2 . 0
for
E[Xt ]
is
E[Xt ] = e2t x2 + 2 0
0
so the variance is 3) Taking the
e2s ds = x2 e2t + 2 0
2t
e2t 1 2
1 2 .
d et Xt = et Xt dt + et dXt = et dBt .
Integrating both sides of (7) and multiplying both sides with
(7) gives
et
Xt = et c +
0
where
et dBt X0 = x0
to be
c = x0 .
The
Xt = et x0 +
0
et dBt
Xt
6.
Let
h(t)
Xt
as
Xt =
0
Show that
h(s)dBs .
Xt N (0,
t 0
h2 (s)ds)
by showing that
u2 2 t 0
E[eiuXt ] = e
h2 (s)ds
. N (0,
(8)
t 2 h (s)ds) which 0 is a unique transformation, therefore proving (8) is equal to proving that Xt t N (0, 0 h2 (s)ds). iuXt Let Yt = e . Since dXt = h(t)dBt we get
Proof. Recall that (8) is the characteristic function of
u2 Yt d X 2
u2 2 t 0
= iuYt h(t)dBt
since
u2 Yt h2 (t)dt. 2
The expected value
Yt = 1 + iu Yt is
t 0
Yt h(s)dBs
t
Ys h2 (s)ds
t
Y0 = 1.
E[Yt ] = E[1 + iu
0
Yt h(s)dBs
u2 2
Ys h2 (s)ds] = 1 + iu E[
0 0
Yt h(s)dBs ]
=0
(9)
u E[ 2
Ys h2 (s)ds] = {Fubini} = 1
0
u 2
2 0
(10)
79
Let
g(t) = E[Yt ]
g(t) = e
u2 2
t 0
h2 (s)ds
u2 2
hence
t 0
E[eiuXt ] = e
from which we conclude that
h2 (s)ds
Xt N (0,
t 0
h2 (s)ds).
7.
Let
X, Y
X 0 = x0 Y 0 = y0
2 Rt = Xt + Yt2
is deterministic.
E[Xt ], E[Yt ]
and Cov(Xt , Yt ).
2 Xt
and
Yt2 .
Rt
as
t 2 Rt = Xt + Yt2 = x0 + 2 0 t t 2 Xt ds + 2 0
Xs Ys dBs +
0 t
Ys2 ds
(11)
+ y0 + 2
0
Ys2 ds 2
0 t
Ys Xs dBs +
0 t
2 Xs ds = x0 + y0 t
+ (1 + 2)
0
Let
2 Xt ds + (1 + 2) 0
Ys2 ds = (1 + 2)
0
Rs ds.
g(t) = Rt ,
Rt = (x0 +
y0 )e
(1+2)t
2) rewriting
Xt
Yt
Xt = x0 +
0 t
Xs ds +
0 t
Ys dBs Xs dBs .
0
Yt = y0 +
0
Ys ds
80
we get
E[Xt ] = E[x0 +
0 t
Xs ds +
0
Ys dBs ] = x0 + E[
0
Xs ds] + E[
0
Ys dBs ]
=0
= x0 +
0
E[Xs ]ds
t t t t
E[Yt ] = E[y0 +
0 t
Ys ds
0
Xs dBs ] = y0 + E[
0
Ys ds] E[
0
Xs dBs ]
=0
= y0 +
0
E[Ys ]ds.
This gives two ordinary derential equations solved in the same manner as in part 1) and gives the solutions
E[Xt ] = x0 et E[Yt ] = y0 et .
Since the covariance is Cov(Xt , Yt )
E[Xt Yt ].
Yt2 dBt
t 0
Xt Yt = x0 y0 + (2 1)
Xs Ys ds +
and we get
E[Xt Yt ] = E[x0 y0 + (2 1)
0 t
Xs Ys ds +
0 t
2 Ys2 Xs dBs ]
(12)
= x0 y0 + (2 1)E[
0 t
Xs Ys ds] + E[
0
(13)
= x0 y0 + (2 1)
0
Let
E[Xs Ys ]ds.
(14)
g(t) = E[Xt Yt ], then solving (12) amounts to solving the ordinary dierential d g(t) = (2 1)g(t) dt g(0) = x0 y0
equation
g(t) = x0 y0 e(21)t
hence
E[Xt Yt ] = x0 y0 e(21)t
and
81
8.
Let
and
(1)
+ Xt dBt ,
(2)
X 0 = x0
(1) Yt dBt ,
Y0 = y0
are independent Brownian mo-
and
B (1) , B (2)
to get
Xt Yt
t
= Xt (Yt dt + Yt dBt )
(1)
(1)
+ Xt Yt dBt .
Using the initial condition
X0 Y0 = x0 y0
t
we have
Xt Yt = x0 y0 + ( + )
0
so
Xs Ys ds + ( + )
0
E[Xt Yt ]
becomes
E[Xt Yt ] = E[x0 y0 + ( + )
0 t
Xs Ys ds + ( + )
0 t
(1) Xs Ys dBs s
(15)
+
0
Xs Ys ds] + ( + ) E[
0
(1) Xs Ys dBs ] =0
(16)
+ E[
0
E[Xs Ys ]ds.
0
(17)
=0
Let
g(t) = E[Xt Yt ],
g(t) = x0 y0 e(+)t
hence
E[Xt Yt ] = x0 y0 e(+)t .
82
References General Probability: 1. 2. 3. 4. 5. 6. 7. 8. Ross, S.M. (1980). Introduction to Probability Models, Second Edition. Academic Press, New York. (Chapters 1, 2 and 3) Breiman, L. (1968). Probability. Addison-Wesley, Reading, Massachusetts. Billingsley, P. (1979). Probability and Measure. John Wiley, New York. Feller, W. (1957). An Introduction to Probability Theory and Its Applications, Vol. 1. John Wiley & Sons, New York. Feller, W. (1971). An Introduction to Probability Theory and Its Applications, Vol. 2. John Wiley & Sons, New York. Renyi, A. (1970). Foundations of Probability. Holden-Day, San Francisco, California. Chung, K.L. (1968). A Course in Probability. Harcourt, Brace and World, New York. Gnedenko, B.V. and A.N. Kolmogorov (1954). Limit Distributions for Sums of Independent Random Variables. Addison-Wesley, Reading, Massachusetts (Translation from Russian).
General Stochastic Processes: 9. 10. 11. 12. 13. 14. 15. Cinlar, E. (1975). Introduction to Stochastic Processes. Prentice-Hall, Englewood Cliffs, New Jersey. Cox, D.R. and H.D. Miller (1965). The Theory of Stochastic Processes. John Wiley & Sons, New York. Doob, J.L. (1953). Stochastic Processes. John Wiley & Sons, New York. Taylor, H.M. and S. Karlin (1984). An Introduction to Stochastic Modeling. Academic Press, New York. Parzen, E. (1962). Stochastic Processes. Holden-Day, San Francisco, California. Prabhu, N.U. (1965). Stochastic Processes. Macmillan, New York. Spitzer, F. (1964). Random Walks. Van Nostrand. Princeton, New Jersey.
Discrete-time Markov Chains: 16. Ross, S.M. (1980). Introduction to Probability Models, Second Edition. Academic Press, New York. (Chapter 4) 17. Karlin, S. and H. Taylor (1975). A First Course in Stochastic Processes, Second Edition. Academic Press, New York. (Chapters 2 and 3) 18. Taylor, H.M. and S. Karlin (1984). An Introduction to Stochastic Modeling. Academic Press, New York. Continuous-time Markov Chains : 19. Ross, S.M. (1980). Introduction to Probability Models, Second Edition. Academic Press, New York. (Chapter 6) 20. Ross, S.M. (1983). Stochastic Processes. John Wiley & Sons, New York. 21. Taylor, H.M. and S. Karlin (1984). An Introduction to Stochastic Modeling. Academic Press, New York. Stability for General State Space Markov Chains: 22. S.P. Meyn and R.L. Tweedie (1993). Markov Chains and Stochastic Stability. Springer-Verlag, New York. Stochastic Control 23. D. Bertsekas (1995). Dynamic Programming and Optimal Control. Volume 2. Athena, MA. 24. W. Fleming and R.W. Rishel (1975). Deterministic and Stochastic Optimal Control. Springer-Verlag. 25. W. Fleming and H.M. Soner (1992 ). Controlled Markov Processes and Viscosity Solutions. SpringerVerlag, 26. M. L. Puterman (1994). Markov Decision Processes. Wiley. Diffusion Approximations and Brownian motion: 27. Karlin, S.and H. Taylor (1975). A Second Course in Stochastic Processes, Second Edition. Academic Press, New York. 28. Taylor, H.M. and S. Karlin (1984). An Introduction to Stochastic Modeling. Academic Press, New York. 29. Bharucha-Reid, A.T. (1960). Elements of the Theory of Markov Processes and their Applications. McGraw-Hill, New York
83
Renewal Theory: 30. Cox, D.R. (1962). Renewal Theory. Methuen, London. 31. Feller, W. (1971). An Introduction to Probability Theory and Its Applications, Vol. 2. John Wiley & Sons, New York. 32. Asmussen, S. (2003). Applied Probability and Queues. 2nd ed, Springer. 33. Taylor, H.M. and S. Karlin (1984). An Introduction to Stochastic Modeling. Academic Press, New York. Queueing Theory: 34. 35. 36. 37. 38. Asmussen, S. (2003). Applied Probability and Queues. 2nd ed, Springer. Gross, D. and C. Harris (1985). Fundamentals of Queueing Theory, Second Edition. Wiley, New York. Cox, D.R. and W.L. Smith (1961). Queues.? Methuen, London. Gross, D. and C. Harris (1985). Fundamentals of Queueing Theory, Second Edition. Wiley, New York. Tijms, H. (1986). Stochastic Modeling and Analysis-A Computational Approach. John Wiley & Sons, New York.
Algorithms 39. Tijms, H. (1986). Stochastic Modeling and Analysis-A Computational Approach. John Wiley & Sons, New York. 40. Bremaud, P. Markov Chains: Gibbs Fields, Monte Carlo Simulation, and Queues. Springer.
84