A. K. Basu - Introduction To Stochastic Process

Introduction to
STOCHASTIC PROCESS
Introduction to
STOCHASTIC PROCESS
A.K. Basu
0
Alpha Science International Ltd.
Harrow, U.K.
Professor A.K. Basu
Editor of Stochastic Modelling and Applications
Department of Statistics, Calcutta University
Kolkata-700 019, India
Copyright © 2003 Narosa Publishing House Pvt. Ltd.

First Reprint 2005
Second Reprint 2007

7200 The Quorum, Oxford Business Park North
Garsington Road, Oxford OX4 2JZ, U.K.
www.alphasci.com
ISBN 978-1-84265-105-6
Printed in India
Preface
From my thirty years experience of teaching the subject of stochastic process to

graduate and undergraduate students I reached the conclusion that there are
plenty of books in the market either of an elementary level or of a highly
specialized sophisticated level which are very much technical in nature. The
adequate demand for intermediate level texts sound in both rigor and application.
The present book is an attempt to meet this demand. This text grew out of my
lectures on advanced undergraduate courses and graduate level courses at Canadian
and U.S. Universities and a postgraduate course at Calcutta University.
Moreover, most of the textbooks available lack enough worked out problems and
exercises to illustrate the theory. The present book containing many relevant exercises
at the end of each chapter is written in genuine class notebook fashion. All the
theorems are proved in sufficient details unlike others which prove only the main
steps and leave the details to the readers. The difficult theorems are motivated by
applications and supplemented with exercises. For inquisitive readers further reading
materials, an appendix and a bibliography have been appended at the end. Large
number of worked out exercises, ranging from easy to difficult, make this textbook
self-taught for students with some background. For the benefit of students all the
exercises are solved and appear as a separate section at the end of the book to help
weaker students. The text contain a good number of examples and counter examples
illuminating the subtleties and finer points of the general theory. A novel feature is
a chapter on ‘Time Series and Stochastic Integrals’ in the appendix. The book
consists of 10 chapters, an appendix and a section of more than 250 exercises.
Of course, in preparing my lectures, I borrowed heavily from the existing
books in the field and the final text reflects this. In particular, the books by
Kerlin and McGregor, D.R. Cox, Parzen,^ Chung, Prabhu were significant
contributors. The only pre-requisites for the book are a sound knowledge in
advanced calculus and elementary probability of the level of Feller Vol. I or my
book on “Measure Theorety and Probability” (Prentice-Hall, India).
This book is suitable for one year (or two semesters) courses offered to
postgraduate students of Indian universities and advanced undergraduate and
first year graduate students of US universities and colleges in statistics, mathematics,
operations research, engineering and business administration. The content of
this book is based on UGC (University Grant Commission) of India recommended
syllabus for postgraduate students in Indian universities.
My warm thanks and appreciation go to M/s DYZICOM, Kolkata, for their
expert typing of the manuscript. I am also indebted to my friends and students
lor their enthusiasm and encouragement while writing this book.
A.K. B asu
Abbreviations and Notations
P{A) Probability of the event A

r.v. Random variable
p.d.f. Probability density function (or density function)
p.g.f. Probability generating function
p.f. Probability mass function
d.f.(c.d.f.) Distribution (cumulative) function
iff If and only if
Ch.f. Characteristic function
u.i. Uniformly integrable
1.0. Infinitely often
a.s. Almost surely
a.e. Almost everywhere
1.1.d. Independent, identically distributed
E(X) Expectation of the r.v. X
L. T. Laplace Transform
w.r.t. With respect to
l.h.s.(r.h.s.) Left (right) hand side
Asymptotic to
F*G Convolution of two distributions F and G
Fn* Convolution of n i.i.d. r.vs. having common d.f. F
0 (.) Distribution function of standard normal r.v.
CLT Central limit theorem
SLLN Strong law of large numbers
M. C. Markov chain
M.P. Markov process
LIL Law of the iterated logarithm
® (J ) o-algebra generated by the class ^
or a ( J )
m (X) a-algebra generated by the r.v. X
Convergence almost surely or almost everywhere
P or d, or £
Convergence in probability or in distribution or in law
Convergence in mean of order p
®n Class of ^-dimensional Borel sets
Minimum of
Maximum of
P(s) Probability generating function (p.g.f.) of a r.v.
177/ Abbreviations and Notations
Sn Sum of N i.i.d. r.v.s Xh where /V is a r.v.

G(s) p.g.f. O f {£„}
F*(s) f e~stclF
(x)L.T. of r.v. X or d.f.
Jo
[ x m { x ,} Stochastic process in continuous time
C(.,; C(.s, 0 Covariance function between X(t) and X(s)
Pij Transition probability from state i to j
m Number of occurrances of an event by time t (upto epoch /)
C(z) Autocavariance generating function of {X,}
P(k)\ pk Correlation function of X, at lag k
P Utilization factor
A => B A implies B
Contents
Preface v
Abbreviations and Notations Vll
1. Introduction 1
1.1 Notion of Stochastic Processes 1
1.2 Different Types of Stochastic Processes 2
1.3 An Introduction to Stationary Processes 2
Exercises and Complements 5
2. Discrete Time Markov Chains 8

2.1 Definition and Transition Probabilities 8
2.2 A Few More Examples 10
2.3 Classification of States 12
2.4 Limit Theorems for Markov Chains 23
2.5 Stationary Distribution 27
2.6 Special Chains and Foster Type Theorems 32
2.7 Theorems Regarding Finite Markov Chains 37
2.8 Methods of Evaluation of the n-Step Transition Probability 39
3. Random Walks 51
3.1 Different Types of Random Walks 51
3.2 Sequential Analysis 58
3.3 Wald’s Equation and Wald’s Identity 59
3.4 Fluctuation Theory 62
4. Renewal Theory 65
4.1 Introduction 65
4.2 Renewal Equation 66
4.3 Renewal Theorems 70
4.4 Central Limit Theorem for Renewal Theory 74
4.5 Delayed and Equilibrium Renewal Processes 75
4.6 Residual and Excess Life Times 78
4.7 Renewal Reward Process 80
4.8 Replacement Policies Connected with Renewal Theory 82
x Contents
5. Branching Process 87
5.1 Introduction and History of Branching Process 87
5.2 Properties of Generating Functions 88
5.3 Probability of Extinction 91
5.4 Distribution of Total Number of Progeny 96
5.5 Continuous Parameter (Markov) Branching Process 98
6. Continuous Time Discrete State Markov Processes 107

6.0 Axiomatic Definition and Transition Function 107
6.1 Differentiability of Transition Function 108
6.2 Kolmogorov Differential Difference Equation 112
6.3 Infinitesimal Generators and Examples 114
6.4 Birth and Death Processes 117
6.5 The Yule Process 118
6.6 Some Population Models 121
7. Poisson Process 133

7.1 Different Definitions and Their Equivalence 133
7.2 Poisson Process and Renewal Theory 136
7.3 Properties of Poisson Process 138
1A Characterization of Poisson Process 140
7.5 Generalization of Poisson Process 142
7.6 Order Statistics and Poisson Process 146
8. Continuous Time and Continuous State Markov Process 150

8.1 Diffusion Processes 150
8.2 Kolmogorov’s Forward and Backward Diffusion
Equations 155
8.3 Wiener Process (Brownian Motion) 156
8.4 Simple Properties of Wiener Process and First Passage
Time Distribution 158
8.5 Ornstein-Uhlenbeck Process and Gaussian Processes 162
9. Time Series Analysis 175

9.1 Introduction and Stationary Time Series 175
9.2 Different Approaches of Time Series 184
9.3 Correlogram Analysis 187
9.4 Wold’s Decomposition and Wiener-Khintchine and Bochner’s
Theorem for Stationary Time Series 194
9.5 Spectral Densities of some Standard Processes and
Autocovariance Generating Function 199
9.6 Estimation of Spectrum 204
Contents xi
9.7 Forecasting 2/2

9.8 Estimation of Parameter in the First Order Autoregressive
Model AR(1) 215
10. Queueing Theory 227
10.1 Introduction 227
10.2 Standard Queuing Models 230
10.3 The Model MIMIC 239
10.4 The Concept of Imbeded Markov Chains in Queuing
Models 243
10.5 The Queueing Models M/Gll and G1IMI1 246
10.6 System Where Customers Arrive in Stages: EkIMI\ Model 251
10.7 Machine Interference Problem 254
10.8 Bulk Queues: Elementary Results 256
Appendices
I. Sample Function of Brownian Motion 259
II. Second Order Stochastic Analysis 27/
III. Stochastic Integral (Ito Integral) 287
IV. Some Important Martingale Theorems 299
Solution to Exercises 303

References 411
Subject Index 415
1
Introduction
1.1 Notion of Stochastic Processes

Loosely speaking, the mathematical description of a random phenomenon as it
changes in time is a stochastic process. Since the last century there has been
greater realisation that stochastic (or non-deterministic) models are more realistic
than deterministic models in many situations. Observations taken at different
time points rather than those taken at a fixed period of time began to draw the
attention of scientists. The physicists and communication engineers played a
leading role in the development of dynamic indeterminism. Many a phenomenon
occurring in physical and life sciences are studied not only as a random phenomenon
but also as one changing with time or space. Similar considerations are also
made in other areas such as social sciences, economics and management sciences,
and so on. The scope of applications of stochastic processes which are functions
of time or space or both is ever increasing.
Definition
A stochastic process is a family of random variables {Xt}, where t takes values
in the index set T (sometimes called a parameter set or a time set).
The values of Xt are called the state space and will be denoted by S.
If T is countable then the stochastic process is called a stochastic sequence (or
discrete parameter stochastic process). If S is countable then the stochastic
process is called a discrete state {space) process.
If S is a subset of the real line the stochastic process is called a real valued
process.
If T takes continuously uncountable number
of values like (0, °o) or (-°o, «?) the stochastic
process is called a continuous time process. To
emphasize its dependence on t and sample point
w, we shall denote the stochastic process by
X(t, w), te 7\ w e Q. i.e. for each we Q, Xt = X(r,
w) is a function of t.
This graph is known as the “typical sample function” or “realization of the
stochastic process” X (t, w).
2 Introduction to Stochastic Process
1.2 Different Types of Stochastic Processes

Following are the most important types of stochastic processes we come across:
1. Independent stochastic sequence (Discrete time process)
T = [1, 2, 3, . . .] and {X(, t E T} are independent random variables.
2. Renewal process (Discrete time process)
Here T = [0, 1, 2, 3, . . .], 5 = [0, «>].
If Xn are i.i.d. non-negative random variables and Sn = X j + . . . + Xn then {Sn}
forms a discrete time (renewal process).
3. Independent increment process (Continuous time process)
T = [r(), °o], where r0 be any real number (+ or -). For every
t0 < r, < . . . < tn, tj e T, i = I, 2, . . . , n (1.1)
if X, , X, - XtQ, Xt2 - Xt] , . . . , Xt - Xtn_] are independent for all possible

choices of ( 1. 1), then the stochastic process {Xn t E 7 } is called independent
increment stochastic process.
4. Markov process
h P[X,n+l e A\Xtn = an , Xtn ] = an_j , . . . , XtQ= a0 ]
= P[Xtn+] E A\Xtn = an] holds for all choices of
/o < r, < t2 < ■• • < C+1, tj G 7 • / = 0, 1, 2, . . . , n + 1 (1.2)

andA e.98, the Borel field of the state space S, then {Xn t e T } is called a Markov
process.
5. Martingale or fair game process
If E[Xlfj+] \Xtn —an , Xtn ^ —an_\, . . . , XtQ—Oq] —an
i.e. E[Xtn+l \Xtn ,. . . , XtQ] = Xtn a.s. for all choices of the partition (1.1), then
{Xt, t E T) is called a Martingale process.
6. Stationary process
If the joint distribution of (X,1+th, . . . , Xt +h) are the same for all h > 0 and
t\ < t2 < . . . < tn, tj e T, ti + h E T

then {X(, t E T } is called a stationary process (strictly stationary process).
7. Point processes
When a countable set or sets of points randomly distributed on the real line or
any arbitrary sets we call the family of random variables governed by the distribution
of those random points as point processes. They represent number of occurrences
of a particular event as time progresses. Poisson process and Renewal processes
are particular cases of point processes. Cox processes (Doubly Poisson processes)
are other examples of Point processes.
1.3 An Introduction to Stationary Processes

A stochastic process (Xr t E T} with EXf < <*> for all t E T is called covariance
stationary or stationary in the wide-sense or weakly stationary if its covariance
Introduction 3
function Cs f = E(XtXs) depends only on the difference | t-s | for all r, s e T. Note
that in our definition we have taken a zero mean stochastic process.
Examples of stationary processes

(a) Electrical pulses in communication theory are often postulated to describe a
stationary process. Of course, in any physical system there is a transient period
at the beginning of a signal. Since typically this has a short duration compared
to the signal length, a stationary model may be appropriate. In electrical
communication theory, often both the electrical potential and the current are
represented as complex variables. Here we may encounter complex-valued
stationary processes.
(b) The spatial and/or planar distributions of stars of galaxies, plants and
animals, are often stationary. Time parameter set T might be Euclidean space,
the surface of a sphere or the plane.
A stationary distribution may be postulated for the height of a wave and T is
taken to be a set of longitudes and latitudes, again two dimensional.
(c) Economic time series, such as unemployment, gross national product,
national income etc., are often assumed to correspond to a stationary process, at
least after some correction for long-term growth has been made.
Ergodicity
The behavior in which sample averages formed from a process converge to some
underlying parameter of the process is termed ergodic. To make inference about
the underlying laws governing an ergodic process, one need not observe separate
independent replications of entire processes or sample paths. Instead, one need
only observe a single realization of the process, but over a sufficiently long span
of time. Thus, it is an important practical problem to determine conditions that
lead to a stationary process being ergodic. The theory of stationary processes has
a prime goal the clarification of ergodic behavior and the prediction problem for
processes falling in the wide range of extremeties.
In covariance stationary process usually the added condition that E(Xr) does
not depend on t is imposed. But it should be noted that in order for a stochastic
process with E{X}) < °° to be covariance stationary it is not necessary that
its mean function m(t) = E(Xt) be a constant. Consider the example: X(t) =
cos + T(r), where Y(t) = N(t + L) - N(t), {N(t), t > 0} be a Poisson process
with intensity parameter A (to be defined in Chapter 7) and L is a positive
constant. Its mean function m(t) - E(Xt) = X{t + L) - X{t) + cos

functionally dependent on t. But
Cov (X(t\ X(s)) = Cov (K(r), Y(s))
[X{L -\t - s\) if|r-^|<L
0 if \t - s | > L
depends on t - s only.
Example 1 (Gaussian process). If the joint distribution of (X lx, . . . , Xtn) for all
t.\, ... ,tnE T is multivariate normal, then {Xt, t E T] is a Gaussian process. Such
type of process appears in e.g. physics and atmospheric turbulence theory. If a
Gaussian process is wide sense stationary then it is strictly stationary. Now the
multivariate normal distribution of (X t],. . . , X,^) is completely determined by
its mean vector (EXtx, . . . , EXtn) and covarience-matrix (Ctj ti) = (EXtj Xt j )
/, j = I, 2, , n. If a Gaussian process is covariance stationary then
E(X~ ) < oo, E(X~. ) < oo and Ctj tj = Cf/ is a function of | t, - t/ |.
Example 2 Let [Xlv n > 1} be uncorrelated random variables with mean 0 and
variance 1.
Then C„.m = Cov (X„ X J = E(X„ X J
0 if n * m
1 if n = m
So {Xn, n> 1} is covariance stationary.
If moreover, Xns are normally identically distributed then {Xn, n > 1} is strictly
stationary.
A process which is not stationary is called evolutionary.
Example 3 Poisson process
P[Xt = k\ - exp (-Xt)(Xt)kl k\, X > 0, k = 0, 1, . . . with the mean function
m{t) = Xt, Var(X,) = Xt are functions of t and is evolutionary. Also note that
Cov(X,, Xs) = min (r, s) in not a function of {t - s) only.
Example 4 Let X(t) = A cos (wt) + B sin(wr), t > 0, where A and B are uncorrelated
random variables each with mean 0 and variance 1 and w > 0 (constant). Here
EX(t) = 0,
E(X(t) ■X(s)) = cos (wf) • cos (ws) + sin (ws) • sin (wr)
= cos ((s - t) • w), and E{X\t)) = 1.
Therefore the process is covariance stationary.
N I—
Example 5 Let Zt = Z Ak el6kt, / = V -l , where 02, . . . , 0N are real
constants, Ak's are uncorrelated with mean 0 and Var (A,-) = a ] . Here E(Zt) =
m(t) = 0 and
= S 2 E(Aj Ak ) ei{0kS~6jt)
7=1k=1
yV
= Z crA
2 eiGk {s~n
k=1
Introduction 5
N
Var ( Z t ) = X err.
k= 1
Therefore, the process is stationary in the wide-sense.
Example 6 Let Xn = X a* £n_k, where £,’s are uncorrelated with mean zero
k=0
and variance a 2 < and | a \ < 1.
Cm,„ = E( XmX„)= £ £ a * k £(£„_*

,/= .0 * = 0
(72 £ a 2i+»-m if in - « < 0

I * =0
(72 £ a 2i+"-m if m - n >0

A
:=w-/?
= if | a | < l.
1- Cl“
Therefore, the process is covariance stationary.

Stationary processes are appropriate for describing many phenomena that
occur in communication theory, astronomy, biology and economics.
A stochastic process {X,, t e T) is called regular if EX 2 < °° for all t e T. A
regular stochastic process is called orthogonal if E(Xt, Xs) = 0 for t, s E T, t ^ s.
Exercises and Complements

Exercise 1.1 Let a, b, c be independent r.v.’s uniformly distributed on [0, 1]. What is the
probability that ax2 + bx + c has real roots?
Exercise 1.2 Let A be a Poisson r.v. with parameter A > 0. Suppose A itself is a r.v.
following a gamma distribution with density /(A) = ~ rA " " 1e~A, A> 0. Show that
dn
P(X = k) = + - (l/2)l+" , >0 (note that when n is a positive integer A is negative
4nx k + 1
binomial with p =1/2).
Exercise 1.3 The following experiment is performed. An observation is made of a Poisson
r.v. A with parameter A. Then a binomial event Y with probability p of success is repeated
A number of times and Y successes are observed. What is the distribution of K?
Exercise 1.4 Let {A„ t > 0} be a continuous time stochastic process with independent
increments. Also P(A0 = 0) = 1. If 0(0, t - u) is the characteristic function of a single
increment i.e.
0(0, t - u) = £[exp(/0(A; - Am))],
prove that the joint characteristic function of A,, , A,,, . . . , Xln where /, < t2< . . . < tn is
® i ’ L “ 11 J • • • 0 ( 0/1 ’ K - 1n- 1) •
Exercise 1.5 Prove that every continuous parameter Stochastic process with independent
increments is a Markov process.
Exercise 1.6 Let T be a nonnegative discrete random variable. Prove that T has a
geometric distribution iff
P[T > x + y\T > a] - P[T > y] for all integers a, y > 0.
Exercise 1.7 Let T be a non-negative continuous random variable. Prove that T has an
exponential distribution iff P[T > x + y|7’ > x] = P[T > y].
Exercise 1.8 Let 7' be a nonnegative random variable such that X(T) is a stochastic
process and for a fixed value of T, say /, X(t) has a gamma density
(Au)a'-' e - x\ a > 0, A > 0 .
Assume that the distribution of T is F(t) = P[T < /].

(a) Derive an expression for E\e~<X(T)]
(b) Prove that T has a negative binomial distribution, i.e.
P + k fp + k- r
T= p p (1 - p ) ,kp > 0
a V k ,
1 >p= 1 - q > 0 , k = 0,1,2___
then X{T) also has a gamma density. Derive its parameters,
(c) Prove that if X(T) has a gamma distribution, i.e.
E[e -vviT) 1_ where ft > 0, A > fj. > 0, then conversely, T has a negative
n+ s
binomial distribution. Determine its parameters.
Exercise 1.9 The random variables X and Y have the following properties: X is positive
i.e. P(X > 0) = 1, with continuous density function/(x) and Y \ X has a uniform distribution
on [0, X].
Prove that if Y and X - Y are independently distributed, then
/ (a ) = a2xe~a\ x > 0, a > 0.
Exercise 1.10 Let U and Vbe independent AÔ, 1) r.vs. Let T be the circumference of
the unit circle T = [0, 2k] and Y(t) = U sin t + V cos tZ{t) = - U cos / + V sin /. Then show
that X{t) = (Y(t), Z(/)), t e T is a strictly stationary process.
Exercise 1.11 Let X(t) = A cos A t + B sin A/, where A and B are uncorrelated random
variables with common mean zero and common variance o-2. Decide whether X(t) is
covariance stationary.
Exercise 1.12 Let {X,} and {Y,} be two sequences of r.vs. on a probability space with
E(Xj) = E{Yj) = 0, Var(X,) = Var(T,) = cr2 < E(X,Xj) = E(Yft) = E(X,Yj) = 0 for all i *
j and EX\Yx = 0 for all /. Let {Z(/), t > 0} be a stochastic process defined by
n
Z(t) = S {Xj cos (Ay /) + Yj sin (Ay f)}
Determine its covariance function. Is the process second order stationary? Is the
process strictly stationary? Justify your answer.
Introduction 7
Exercise 1.13 Let X(/) = sin wt, where w is uniformly distributed on [0, 2k ] (i) Show
that {X(/), / = 1, 2,. . .) is covarience stationary but is not strictly stationary (ii) show that
(X(/), 1 > 0} is neither covariance stationary nor strictly stationary.
Exercise 1.14 Let Xn = cos n U (n > 1), where U is uniformly distributed on \-n, n).
Show that {X,,} is covariance stationary but not strictly stationary.
Exercise 1.15 (Random Binary Noise) A stochastic process X(t) is called random binary
noise if X(t) = Y(t - u), where Y is a process taking value ± 1 with probability 1/2 on
successive intervals of fixed length T and such that the r.v.’s Y(t) are independent for
values of t lying in non overlapping intervals and U is the random shift of time such that
U is uniformly distributed on (0, T) and is independent of Y(t). Show that X{t) is wide
sense stationary.
Exercise 1.16 (Random Telegraph Signal Process)
Let X(t) be a stochastic process defined by X(t) = Y(-\)N(t\ t > 0 where N(t), t > 0 is a
Poisson process with parameter Aand Y is a r.v. independent of the process N{t), i > 0 with
probabilities specified by P(Y = ± 1) = 1/2. Show that X(t), t > 0 is a covariance stationary
process.
Exercise 1.17 (Model of a Germination Process)
Let 7 / T2, . . . , Tn be the germination times of N seeds planted at time t = 0 and they are
assumed to be i.i.d. with common unknown d.f. H(t) = P(T> t) with H{0) = 0 and //(°°)
= 1 - p, where 0 0. Find the probability distribution of Xt, the number of seeds that will germinate in
(0, /).
Exercise 1.18 (FKG inequality)
Let Xh X2, • • • , Xn be i.i.d. r.v.’s having Bemouli distribution with parameter p. Let/be
increasing such that /(x) < / ( y) whenever < y, for each i. Let x = (xu . . . xn).
(a) Show that EPx (/(x)) < Epi (/(x ) if p x 0.
Exercise 1.19 (Antithetic variables)
Let #(*,, . . . , xn) be an increasing function in all its variables, and let {Ur} be
i.i.d. uniform r.v.’s. on [0, 1] showthat
cov {g(f/1, . . . , U J , g ( l - U 1, . . . , l - U J } < 0.
[Hint. Use FKG inequality] Explain how this can help in the efficient estimation
2
Discrete Time Markov Chain
2.1 Definition and Transition Probabilities

Here S - a countable set, T = {0, 1, 2, . . .}, {Xn, n > 0} is a stochastic process
satisfying P[Xn+l = j \ X 0 = i0, X x = . . . , Xn = in\ = P[Xn+l = j \X n = in], the
Markov property. Then the stochastic process {Xn, n > 0} is called a Markov
chain (M.C.). We shall assume that the M.C. is stationary i.e. P[Xn+x = j | Xn =
/] = ptj is independent of n for all i, j e , S. Let P = (PtJ)\ i, j e S be a finite or
countably infinite dimensional matrix with elements ptj.
The matrix P is called the one step transition matrix of the M.C. or simply the
Transition matrix or the Probability matrix of the M.C.
Example (Random Walk) A random walk on the (real) line is a Markov chain
such that
pjk = 0 if k * j - 1 or j + 1.
Transition is possible only to neighbouring states (from j toy - 1 and j + 1). Here
state space is
5= 3 , - 2 , - 1, 0, 1, 2, 3, . . .}.
Theorem 2.1 The Markov chain {Xn, n > 0} is completely determined by the
transition matrix P and the initial distribution {/?*}, defined as P[X0 = k] - p k > 0,
K € zS Pk = L
Proof
P[Xq = i0>X\ —i» • • • »Xn = in]
- P[Xn = in I Xn-\ - Xn-2 - in-2» • • • ♦X ] = i‘i . . . X0 = i0]
P\Xn- 1 - *n-b Xn-2 = in-2* • . • , X\ = i\, Xq = io\
—P\Xn —in| Xjj.j — —f/2-b • • •» *o —i0]
= P i n-\in P i n- j i n-\ ^ X n-2 ~ in-2> • • • » ^0 = *()]
= /Vi/„ Pin- 2in-\ • • •P1V2 Pio*i (by induction).
Definition 2.1 A vector u = (ui, u2, . • • , un) is called a probability vector if the
components are non-negative and their sum is one.
Discrete Time Markov Chain 9
Definition 2.2 A square matrix P = (ptj) is called a stochastic matrix if each of

its row is a probability vector i.e. if each element of P is non-negative and the
sum of the elements in each row is one, e.g.
' 0 1 0
1/2 1/6 1/3 is a stochastic matrix.
vl/3 2/3 0
A transition matrix P of a M.C. is a
Definition 2.3 A stochastic matrix A is said to be regular if all the entries of
some power Am is positive (m is a positive integer), e.g.
r 1/2 1/2")
A= ff i °l l is regular, since A2 =
,1/4
) 1/4,
Problems
1. Suppose P is a stochastic matrix, then show that Pn is also a stochastic
matrix for all n> 1.
2. If P n is stochastic, is P stochastic?
3. Show that 1 is an eigenvalue if A is a stochastic matrix, i.e.
| A / - A | = 0=> A= 1.
Consider a sequence of trials with possible outcomes E u E * . . . , Ek . . . To
the pairs of outcomes (Ep Ek) we can associate some numbers (i.e. conditional
probabilities) p]k. The {Ek} are referred to as the possible states of the system.
Instead of saying that the nth trial results in Ek one says that the nth step leads
to Ek or that Ek is entered at the nth step.
We shall denote by P j ^ the probability of transition from Ej to Ek in exactly
n steps i.e. the conditional probability of entering Ek at the nth step from Ey This
is the sum of all the probabilities of all possible paths Ej —> Ejx—» . . . Ejnl —» Ek
of length n starting at Ej. and ending at _Ek
In particular, p ^ =p jk.
Theorem 2.2 (Chapman-Kolmogorov equation)
„(*)_ V n{n~X)n
P ‘J ~ kes Plk Pkj
Proof P<p=P[Xa = j \ X 0 = i]
= 2 P[Xn =J I = k, X0 =—k |
k ES
= Z P[Xn =j I X„_, = k}P[Xn_, (by Markov property)
= kes
2 <v )P//,-
* | ) = kes
2 KJ ik
PkjP t l
Corollary For
Note Actually P-"} are the elements of Pn, where P = (ptj) and the corollary
express the matrix multiplication rule Pm+n - PmPn.
To be consistant we define
p.^0) = P[X() =j \ X0 = i] = Sij (Kronecker’s delta).
Transition probabilities of a non-homogeneous (non-stationary) M.C. {X„} also
satisfies Chapman-Kolmogorov equation given by
P<m'n) = 2 p l r u)Ptt'n) for m < u < n

lJ k e s ^ lk kj
and for all values of /, j in S, where p (tjn'n)= P(Xn = j \ Xm = i ).

In terms of transition probability matrices Chapman-Kolmogorov equation
may be written as
p{m,n) = p(m,u) p(u,n) f o r a jj n > u > m > 0.
where P<m'n) = ( p ^ n)).
We also have P{m'n) = p ^ n^ p ^ ~

_ p(m,n-2)p(n-2,n-\)p(n-\,n). . .
_ p(m,m+\)p(m+l,m+2) p {n -l,n )
Proof P ^ ’n) =ZP(Xn = j | Xn= k , X m = i)P(Xn_t
= 2 P(Xn = j | = k) P(X„_, = 11 x m = i )
- T n (w,n-l)
kes Pik Pk j
2.2 A Few More Examples

Examples
(a) Independent trials
Pn = P for all n > 1, where ptj = pj i.e. all the rows are same.
(b) Success runs

Consider an infinite sequence of Bernoulli trials and at the nth trial the system
is in the state Ej if the last failure occurred at the trial number n = 0, 1,
2, . . . and zero-th trial counts as failure. In other words, the index j equals the
length of uninterrupted run of successes ending at nth trial.
qp i for7 = 0, 1, 2,. . . , i + n - 1
Here PJ for; =7 + n
0 otherwise
This follows either directly or from Chapman-Kolmogorov’s equation. It can

be shown that Pn converges to a matrix whose all elements in the column j

equals qpJ, where the transition matrix P is given by
P if j = i + l
PiJ = P(Xn = j \ X n_x = i) = q if; = 0
0 otherwise.
(c) Two state M.C.

There are two possible states Ex and E2 in which the matrix of transition probability
is of the form
i -p p } , 0 < p < 1 and 0 < a < 1.
The system is said to be in state Ex if a particle moves in the positive direction

and in E2 if the direction is negative.
(d) Random walk with absorbing barriers

Let the possible states be E0, Ex, E2, . . . , Ek. Consider the matrix of transition
probabilities
f\ 0 0 0 ... 0 0 0
q 0 p 0 ... 0 0 0
0 <7 0 p ... 0 0 0
0 0 0 0 ... q 0 p
^0 0 0 0 ... 0 0 1
From each of the ‘interior’ states Ex, E2, . . . , Ek_x transitions are possible to
the right and left neighbour with pii+x = p, pLi_x - q. However, no transition is
possible from either E0 or Ek to any other state.
The system may move from one state to another, but once E0 or Ek is reached
the system stays there permanently.
(e) Random walk with reflecting barriers
q p 0 0 ... 0 0
q 0 p 0 ... 0 0 0
0 q 0 p ... 0 0 0
0 0 0 0 ... q 0 p
0 0 0 0 ... 0 q pj
This chain may be interpreted in Gambling language by considering two

players playing for unit stakes with the agreement that every time a player loses
all his money his adversary returns it, so that the game can continue forever i.e.
a random walk in a finite interval such that whenever the particle is at point 1 it
has probability p of moving to position 2 and probability q to stay. The system
is in state Ek, if the capitals are k and R - k + 1 respectively.
2.3 Classification of States
2.3.1 Decomposition of state space

It may be possible that p tJ = 0, p -f = 0 but pf^ > 0. We say that the state j is
accessible from state i if p \p > 0 for some n > 0. In notation i —>y, i.e. i leads
to j. If i —>j and j —» /, then i and j communicate and we denote this by i <-» j.
Definition 2.4 The state i is essential if i —>j implies i <—y, i.e. if any state j
is accessible from /, then i is accessible from that state. We shall let 3 denote the
set of all essential states. States that are not essential are called inessential.
Lemma 2.1 i <-> j defines an equivalence relation on 3 , the class of essential
states.
Proof i -> i (reflexivity)
(i) Since for each /, Z p u = 1 there exists at least one j for which p u > 0. But
jes J J
if i is essential then there exists m > 1 such that > 0 . So by Chapmen-
Kolmogrov equation p[m+l) > p q p ^ > 0.
(ii) i j <=>j i (symmetry)
(iii) i <r->j and j <r* k => i <-> k (transitivity)
Proof of (iii)
To prove i —» k, since i j p ^ > 0 for some n > 1 and j -> k, p ^ > 0 for
some m > 1.
Claim: p \1^ > 0 for some / > 1
0 / <-» k.
By Lemma 2.1, i.e. 3 = u (C(/), where C(i) = {j e 3 | i <-» y} is called a
communicating class, i.e. the class of essential states is partitioned into disjoint
equivalent classes (communicating classes).
Definition 2.5
A Markov chain is called irreducible (or ergodic) if there is only one communicating
class, i.e. if all states communicate with each other or every state can be reached
from every other state.
Definition 2.6
A subset C of S is called closed (or transient) if it is impossible to leave C in one
step i.e. p^ = 0 for all i e C and all j £ C.
Lemma 2.2 If C is closed then

(i) for all n > 1, p {J ] = 0 if i e C , j £ C i.e. no state outside C can be reached
from any state in C.
(ii) X - 1 for i e C and all n > 1.
keCrik
Proof (i) : n = 2
p?i = S c PikPki = S c PikPkj + = 0 fory 2 C

Since Plk = 0 for k £ C and / e C
Pkj ~ 0 for k E C.
By induction p - ^ = 0 if i e C and j £ C.
<ii) i = ( v M= ~ p :;" * ^ c
Corollary 2.1 A Markov chain is irreducible if the only closed set in S is S

itself, i.e. there is no closed set other than the set of all states.
Theorem 2.3 (plk) i , k e C is Markov matrix if C is closed, i.e. if in the matrix
P n all rows and all columns corresponding to states outside the closed set C is
deleted, there remains stochastic matrices for which the fundamental relations
(Chapman-Kolmogorov equation)
P T =ZPJ ' PJ ? ^ pT '=^ p T pV h«'d

(This means that we have a M.C. defined on C and this subchain can be studied
independently of all other states). Proof follows from Lemma 2.2 and is omitted.
Definition 2.7 A single-state {k} forming a closed set is called absorbing
(trapping) i.e. a state such that the system remains in that state once it enters
there. Thus a state k is absorbing if the A:th row of the transition matrix P has 1
on the main diagonal and 0 elsewhere.
Example 2 . 1
1 2 3 4 5 6 7 8 9
ro 0 0 1/2 0 0 0 0 l/2>
0 1/4 1/4 0 1/4 0 0 1/4 0
0 0 0 0 0 0 0 1 0
1 0 0 0 0 0 0 0 0
0 0 0 0 1 0 0 0 0
0 1/3 0 0 0 0 1/3 0 1/3
0 1/3 0 0 0 1/3 1/3 0 0
0 0 1 0 0 0 0 0 0
^0 0 0 1/2 0 0 0 0 1/2 j
In this example p55 = 1, i.e. the state 5 is absorbing.

{3, 8} is a closed set since p83 = p 38 = 1. From 1, passages are possible into
4 and 9 and from there only to 1, 4 and 9. So {1, 4, 9} is a closed set.
From 2, transition is possible to 2, 3, 5 and since {3, 8} is closed and 5 is
absorbing, the closure of 2 is {2, 3, 5, 8}. But the closure of 6 and 7 are the
whole state space {1, 2, 3, 4, 5, 6, 7, 8, 9}. The determination of the closed sets
can be simplified by rearranging or renumbering the states in the order 5, 3, 8,
1, 4, 9, 2, 6, 7 and the rearranged matrix is:
5 3 8 1 4 9 2 6 7
' 1 0 0 0 0 0 0 0 0
0 0 1 0 0 0 0 0 0
0 1 0 0 0 0 0 0 0
0 0 0 0 1/2 1/2 0 0 0
0 0 0 1 0 0 0 0 0
0 0 0 1/2 1/2 0 0 0 0
1/4 1/4 1/4 0 0 0 1/4 0 0
0 0 0 0 0 1/3 1/3 0 1/3
v0 0 0 0 0 0 1/3 1/3 1/3
Hence, the communicating classes are {5}, {3, 8}, {1, 4, 9}, {2} and {6, 7}.
Example 2.2 A man tosses a fair coin until 3 heads occur in a row. Let Xn
denote the largest string of heads ending at the nth trial.
Then with positive probabilities Xn = 0,1, 2, 3. This is a Markov chain with
state space 0, 1, 2, 3.
The transition matrix is given by
1/2 1/2 0
0 1
1/2 0 1/2 0
1/2 0 0 1/2
0 0 0 1,
3 is an absorbing state. Therefore, M.C. is not irreducible. The communicating
classes are {0, 1, 2} and {3}.
2.3.2 Classification of states

Let P[Xn =j, X„_, *j, X„_2 * j , .
arriving at j at time n for the first time, given that the process starts at /.
Define /^ 0) = 0. Note that f - p = P[Tjj = n], where
Tij = min {n: X„= j \ X0 = i]

f ^ n) are called the first entrance probability at nth step if i * j and recurrence
probability at the nth step.
Note = p jj gives the diagonal of the transition matrix.
Theorem 2.4 p ' f 1= 2^ f ' f ' ' for all = 1, 2, . . . n.
Proof p O0 =P[Xn = j \ X 0 = i]
V [X
P n= j x m= j , XmA ± j .X, * y |X0 = i]
m=1 A Bm c
=i P[ABm\C]
m= 1
where B,„ are disjoint (mutually exclusive) and Bm z>

m -1
p (n) v P(ABmC)P(BmC
)
Hence
" «-i P(C)P(BmC)
= 2 F(i4 | BmQP(Bm | C)
m= l
= S F[X„ =y | Xm =y, Xm_, *y, . . . . X, *y, X0 =

m=l
P[Xm = y\Xm_, *y, . . . , X , * y | X 0 = i]
= m
i P(Xn = y | X m= y ) 4 m)
n
_ y D(n~m) fin)
m=\ FJl
Corollary 2.2 LetPpis) = 2^ P - " )s'1(for |1) be
and Fij (s) = (for | 5 | < 1) be the G.F. of {///">}.
Then P,j(s) = Su + Ff s) Pjj (s)
Proof P , j ( s ) = i Py s " + pl?
^ oo \
_ X . V n(n~m) „n~m Am) m
- °>j + n% Pa s hj s
V /
= *</ " 1 , P u M p - 's -
= i,v + ,/ • „ « f £ / '" > » "

V J
= SiJ+ P jjis W ijis )

Write fn = E / (n) = the probability that starting from i the system will ever
■ n- 1 •'
pass through j. Hence, 0 <f tj < 1.
Definition 2.8 A state i e 5 of a Markov chain is said to be transient if f u < 1
and recurrent (Persistent) iff u = 1. If f l} = 1, then {/^n)} is a proper probability
distribution and we shall refer to it as the first passage time distribution of j
(starting from i). Let Pi = 2

If = 1, then Pi < oo is the mea/i recurrence time for i.
The first question we can ask concerning any state i is ‘whether a return to it
is certain or not.’ The next question would be ‘whether the mean recurrence time
Pi is finite or not.’
Definition 2.9 A state i is called positive (or ergodic) if i is recurrent and
Pi < °°. A state i is called null if i is recurrent and p t = °o.
Theorem 2.5 Let / e S of a Markov chain. Then
(i) i is recurrent iff E p[n) = °°

/i=(T h
(ii) i is transient iff E z?},n) < °°

n=(T "
(iii) if i is recurrent then p , = E°° w/,-n) = lim —------1

n=l n .v-^1- (1 - s)Plt(s)
Before proving the theorem let us state the following important theorems
from Analysis which will be quoted frequently.
2.3.3 A few important theorems (for countable space)

Let {an} be a sequence of real numbers such that 0 < an < 1, n = 0, I, 2, . . .
Let A(z) - Z anzn be the generating function of {an}

n= 0
1. Pringsheim : A(z) converges in a circle | z | < r, where r < 1. Z = r is a
singularity of A(z).
2. Abel : If E then lim A{Z) = a.

n=0 c—>1-
3. Tauber : If lim A(Z) = a < <*>, then E an = a.

z—
>1— «=0
1 °°
4. Cesaro-Abel : If lim — E ak = L < then
/I £=1
lim (1 - z) A(z) = L.
z-> 1-
5. Cesaro-Tauber : If lim (1 - z) A(z) = L < <», then
H i-
1 n
lim — X ak = L.
z-+\^ n k=o
For proofs of 2 and 3 see the book of Karlin (page 46, Introduction of Stochastic
process) and proofs of 1,4 and 5 see the book of Tichmarsh-Theory of Functions.
Let n - 0, 1, 2, . . . .
1. (Lebesgue) Dominated Convergence Theorem
If (i) lim anm exists for every m
n—
(ii) | anm | < bm (independent of n) for all m > 0
(iii) m=0
£ bm
< °°
then lim £ anm= £ lim bnm.

m=0 m -0
2. Fatou’s Lemma
If (i) anm > 0 for all ra, n,
(ii) lim anm exists for all m,
then lim 2. a„ > X [ lim a

m=0 m=0 n—
3. Fubinis Theorem
In order that X X = X X
n=0 m=0 m=0 n=0
it is sufficient that at least one of the following conditions is satisfied:
(i) anm > 0 for all n. m
(ii) X X | a „ < oo
n=0 m=0 1
(iii) X X < oo
m - 0 n=0
Proof of Theorem 2.5

(i) Suppose i is recurrent and that f ti _ L
By Abel’s theorem,
lim Fa(s) = 1 => lim 1 - Fu(s) = 0
.v—
>1- s—
>1—
By Corollary to Theorem 2.4. Pf s ) = 5tJ + Ff s ) Pf s ) and in particular
Puis)= 1 + Fu(s)Pu(s) => Pu(s) = - p r — y (
Then lim Pu (s) =

s—>1—
By Tauber’s Theorem, X P-n) = oo.

n=0 11
Conversely, suppose Xq /^z(n) = oo and also suppose i is not recurrent i.e. it is
transient i.e./-, < 1. By Abel’s theorem,
lim Fu(s) < 1 => lim 1 - Flt(s) > 0.

s— >1— s—>1—
Again by Corollary to Theorem 2.4,
lim Pu(s) < oo => X p[n) < 00 (By Tauber’s theorem).

S_>1_ n=0
Hence, i must be recurrent.
(ii) i is transient i.e. /„ < 1 iff X^ p (un) < °o

The result follows from the proof of part (i).
(iii) f« (l) = since F„(s) = X s nf ii(n).

n- 1 n=0
1 - Fu(s)
Also F'f 1) = lim —-— -— (if i is recurrent, Fu{1) =f it = 1)
= lim —---- . (by Corollary to Theorem 2.4).

(1 - *)/>(*) w
Theorem 2.6 {Solidarity Theorem) All states in a communicating class of a

Markov chain are of the same type, i.e. states are all positive or all null or all
transient.
In particular, in an irreducible Markov chain all states are of the same type.
Proof Suppose i e S and j <-> /, i.e. p \ p > 0, for some N and p^f*) > 0 for some
M. Then 6 = p (" )PJ(iM)> 0.
1. Let i be transient
By repeated applications of Chapman-Kolmogorov equation
P\ r +M^ p \ P p ^ p T = e p ^ \ e > 0
(N+n+M) > 0 £
I p<w+"+M) > I /><"> and 00 > s /4 n) > 2
n= 0 n=0r j j n=0r " 0 “ n=0
Therefore, X PîV < 00 •

By Theorem 2.5 part(ii), 7 is transient.
2. Lef i be recurrent
Then as before p ^ +,7+M) > 0p[n) and hence
( N+n+M)
> 0 I p[n) = 00 (/ is recurrent by Theorem 2.5 part (ii))
Jo'7"
NOW X > X = oo.

n=0 JJ n=0r j j
By Theorem 2.5 part (i), j is recurrent.

3. Let i be recurrent
We had proved earlier
p (N+M+n) > 0 p (n)
p(N+M+n)> e p (n)
and
Also I
' n+m ^ P)nj)sn- n=0
SN+M p ]"+M+n)sn>e Z p[n)s n for 0 < « < 1.
JJ n=0r "
Therefore — - Pjj(s) > 6Pu(s) for 0 < 5 < 1.,
where P;j is) is the G.F. of [p]]') and Pjj( s ) is the G.F. of
Pa (s)< £ — Pjj( s ) for 0 < 5 < 1 (2.1a)
Similarly, vj+^ Pu(s) > 9Pjj(s) for 0 < < 1 (2.1b)
and hence (1 - s)dsM+NPjj(s)< (1 - s)P„(s) for 0 < 5 < 1(2

Let s —> 1 Then by Theorem 2.5 (iii) and (2.1a) and (2.1b), we get
 2 r c / - < 00

n—
0 Ju n=0
and 2 < oo => 2 n f j w) < °°

n=0 n=0
Thus, i positive <=>j is positive and i null <=>j null.

Exercise 2.1. Homogeneous Random Walk
Here state space is given by
5 = { . . . . , -2, -1 ,0 , 1, 2 . . .}
Pi = P> Vi ~ Q f°r all i - 1-
This is an irreducible M.C. Hence by solidarity theorem it is enough to consider
the state {0} only. The n-step recurrence probability is
q P
-4 -3-2-1 0 1 2 3 4
0 if n is odd
pin) _ r2m\
r 00 “ p mq m if n is even (n = 2m)
{0} is transient iff P0(0'?) < °°
and recurrent iff 2^ P0(0") = °° (by Theorem 2.5)
' 2m\ ( 4pq)m < c» if 4 pq < 1

s p mq m ^ £
m- 1
Km J m=1 (n m )112
= oo if 4 pq > 1.
(using Stirling’s approximation for m\ = ^ 2 n e mnim+V2) 4pq > 1 is impossible

for if 4pq > {p + q)2 then 0 > (p - q)2.
Hence 4pq < 1 if p * q
= 1 if P = q= -
Therefore S }converges faster than the geometric series S (4pq)mif p * 1/2.

m- 0 u m
Hence Random walk is recurrent iff p = -■ and transient iff /? ^ —.
We have shown in Exercise 2.1 that a symmetric Random walk in one dimension
is recurrent. Similarly it can be proved that in 2-dimensions a symmetric Random
walk is recurrent. But Polya proved that in k > 3 dimensions a symmetric Random
walk is transient.
Exercise 2.2 Show that a symmetric Random walk in three dimensions is
transient.
Solution p0(02n+l>= 0, n =0, 1. 2, . . .

p (2n) y _______________( 2 / i) ! _______________
and (1/6)2",
00 o <i+j<n i\i \ j \ j \ - i - j)\ (n-
since the probabilities of a transition of one unit to the east, west, north, south,
up or down all equal 1/6.
(Consider all paths i units to the east, i units to the west,y units to the south,
j units to the north and k = n - i - j units to each of up and down).
p(2n) _ 1 f 2") 2 n\
r 00 ~ (1/3) In (2.4)
10 <i+j<n _i\j\ ( n- i
u
f 2 n\ n\
<C (1/3)" where Cn = max (2.5)
" 2 2" 0 <j<n i \ j\ (n - i - j )!
since I -t-77----------77(1/3)"= 1
0 <i+j<n ll J\ (tl - I - J)\
We shall show that for large n the value of Cn is attained for i = j ~ n/ 3. Let
■ • n■ i •
io, io maximize /------:----sub ect to 0 < / + / < « .
i!j! ( n - i - j ) !
Then
n\ n\
j o '■Oo - 1)! ( n - j 0 - ‘o + 1)! " j o ! ! (« -7o - <o)!’
n! n!
j o ! Oo + 1)! (n - Jo - io ~1)! " i o ! h ! (« - Jo - <o)!
_____________________ n ! _____________________ n ! _____________
(jo ~ 1)! io(«o

! - Jo r 'o + 0 ! ~ jo'io Jo -
P
/l! /i!
and
(Jo + !)! *o! (« - Jo - io ~ 1)! jo'-*o'-(n ~ Jo ~ io)'-
which lead to n - j 0 - 1 < 2/0 < n - j 0 + 1 and n - i0 - 1 < 2j0 < n - /0 + 1-
Hence for large n, /0 ~ rc/3 and y0 ~ n/3.
Putting i - j - n/3 in (2.5), we obtain from (2.4)
p(2n) , n\ ( 2n\
r00
(n/3)\(n/3)'.(n/3)\22n3n \ n J
3V3
i,l n„ 3/2
2 n 3/2
(using Stirling’s formula for factorials).
Thus, Z ^ P0(02n) < oo. Hence, ‘0’ is a transient state. Since the M.C. is
irreducible, the Random walk is always transient.
Example. 2.3 (Success Chains or Runs)
Consider a M.C. with possible states Zs0, E2, • • • with transition probability
matrix
r Po <7o 0
Pi 0 <7i 0
Pi 0 0 <72 0
V •• • • V
Let k represent the ‘age’ of the system. When the system reaches k the aging
process continues with probability qh but with probability p k it rejuvenates and
starts afresh with age zero. The successive passages through the state E0 again
represent a recurren event and the probability that a recurrence time equal k is
again given by the product q^q\q2 • • • cIk-\Pk-\- This is actually a sequence of
trials in which at the Azth trial the system is in state k if the last failure occurred
at the trial number n - k (i.e. index k equals the length of uninterrupted block of
successes ending at the nth trial). By convention we take 0 < pt < 1 which implies
that the chain is irreducible. The nature of the transition matrix P shows that a
Tirst return at the nth trial can occur only through the sequence E0 —>Ex E2
• • • —» En_j —> £0, and so for n > 1,
f w = <70<M2 • • • <7«-2/V b/oO ) = 1 and /d o ’ = P-

The recurrence probability
/oo = n=\
2 /do0 = n=l
2 (q0qi . . . qn- 2 Pn-\)
= 2 (<7o<?i • • • <7n-2 ~ <7o • • • <7*-i)

n=i
(since p„_, = 1 - <?„_,)
=1 - lim (<7o<7i • • • <7«) = 1 - lim Um
where Un = q$q\. . . q,v n>0 and t/_i = 1 by con

Now //firi
0(0ri) = £/„_2 - Un_x. Hence, the M.C. is transient iff/00 < 1
00
i.e. iff lim Un * 0.

n—>°°
We claim that it is positive recurrent iff Z Un <°^,

n=0
null recurrent iff Z JJn = and lim Un = 0.

n=0 n->°°
The proof of this follows from the following arguments.

In case of recurrent M.C. the mean recurrence time becomes
oo oo f oo \ oo oo
z n /< n) = z z i / < ; ) = z z
n=\ J °° n = 1 *=1 00 * = 1 n = K 00
V J
= Z f/;_2(since lim Un = 0)
fc=l A
2—>°°
= t/_, + 2 t/* = 1 + 2 Uk = 1 + 2
1 *=0 * A:=0 £=0 i=0
<7, n
Hence, M.C. is positive recurrent iff Z Uk <°
fc=0
Lemma 2.3 If 0 < < 1 then lim Un = 0 iff Z p t = 00-

n—
>oo
Proof <7, = I - Pi <
n
n - I P,
Hence Un = n q, < e -=°
1=0
Thus, Z /?, = oo u n —> 0.

/=0
Conversely, if —>0 but Z p{ < <», then

jT — = n ( l - Pi) > l - p k - pk+l - . . . - p n > 0 for large n if

U k- \ /=*
£ p; < °o for some k > 1.

z=0 ' '
Hence lim ^ n > 0 for some k > 1.

n—
^°° C k_\
Therefore lim Un = 0 is contradicted and hence
n—>°°
u 0 => £ /?, = <x>.

n z=(T '
Thus, the M.C. is transient iff S p,< °o

/=(T '
and recurrent iff £ p, = °°.

z=(T *
2.4 Limit Theorems for Markov Chains

Definition 2.10 Let d(i) be the greatest common divisor of those n > 1 for
which p[n) > 0. Then d(i) is called the period of the state i. If d(i) = 1, then the
state i is called aperiodic.
Note i <-» 7, then d(i) = d(/).
There exists nx and n2 such that p\*jx) > 0 and p ^ 2) > 0.
Now p (T l+n2) > p . j l ) p ^ 2) > 0 and hence d(i) is a divisor of n{ + n2.
If p > 0, then p (unx+n+n2)> p \jx)p ^ p {?2) > 0 (by Chapman Kolmogorov
equation).
Hence, d(i) is a divisor of n] + n + n2. So d(i) must be a divisor of n if
< ) >° -
Thus d(0 is a divisor of {« > 1 : > 0}. Since d(j) is the largest of such
divisors, d(i) < d(j). Hence, by symmetry d(j) < d(i).
Hence d(i) = d(j). Therefore having a period d is a class property.
Note If pa > 0, then d{i) = 1 and this implies that a sufficient condition for an
irreducible M.C. to be aperiodic is that p n > 0 for some i e S. Hence a queueing
chain is aperiodic.
Theorem 2.7 Limit Theorem (for diagonal elements)
Let j be any state in a M.C. As n —> «>.
(i) if j is transient, then p ^ —> 0
(ii) if j is null recurrent, then p f f 0

(iii) if j is positive (recurrent) and
(a) aperiodic, then p f f —> ——--= —— (mean recurrence time of j )

(b) periodic with period d(j) then 7 —> ------ .

Write d ( j )/ j L i j = n r
Corollary 2.3 Let a M.C. be irreducible and aperiodic. Then j is transient iff
j is null (recurrent) iff nZ= \ JJ

= ©o and JJ
—» 0,
j is positive (recurrent) iff p ^ 7lj > 0 .
Proof of Theorem 2.7 (i) If j is transient, by Theorem 2.5.
Z p ^ < °° => p \ f —> 0 as /i —> ©°.

n JJ JJ
As a part of the proof of Theorem 2.7 let us state the following famous lemma
without proof.
Lemma 2.4 (Erdos, Pollard and Fe//er in Discrete Renewal Theorem) Let {/„} be
a sequence such that /o = 0, f n > 0, Z /„ = 1 and the greatest common divisor
n
(G.C.D.) of those n for which f n > 0 is d(> 1). A second sequence {Vn} is defined
as follows:
v0 = l,v n= > 1).

m-1
dl p if p = Z nf„ < <*>
Then lim v(nd) =
0 if JLL = ©©
By Theorem 2.4, <»*»
and this can be written as yn = Z f m un_m (the discrete renewal equation). Thus
m- 1
the lemma gives lim = 0 (since in null recurrent case p = ©©).

a?—
»°° JJ
(iii) By definition of positive recurrence, Pj = Z n f j ^ < 00.
J n JJ
(a) So by the lemma p ')]1— ——— if the M.C. is aperiodic i.e. <7=1.
(b) p([ f uy) -> — ^ if j has period J(y').

Pj
Proof of Corollary 2.3 follows from Theorem 2.5 and Theorem 2.7.
(i) j is transient iff Z p ^ < 00 (from Theorem 2.5)

n- 1 JJ
(ii) j is recurrent iff Zq p f f = °© (from Theorem 2.5)

^77 —» 0 => —
JX ■ = 0 => ÛjJ = oo i.e. / is null recurrent iff r JJ —» 0
p jj} 7tj > 0 => jUj < oo i.e. positive recurrent iff —>/r;- > 0.
n
Proof i is recurrent => X p[n>= ©o .
n=l
n ,
(i) Now i is positive iff p t = X = lim —----- - , . < »
r /i=l Ml- (1 - s)Pu(s)
i.e. if J L - 1 lim (1 - .i(> 0 (by Theorem 2.5)

s)P
Pi n S— >1—
1
i.e. if lim — X p \k) > 0 (by Cesaro-Tauber Theorem).
Moo n. k=1 w 7
Conversely by Cesaro-Abel Theorem,
if /M
lim
oo —
fj >0
then lim (1 - s)Pl:(.s) > 0

n—>1-
i.e. Pi < oo and hence i is positive recurrent,
(ii) Similarly i is null if
Pi = lim —;-------- \ ■■, x =oo
t'Ml- (1 - s ) P (5)
ie. — (1 -s)Pu
1
=> lim — X p-k) = 0(by Cesaro-Tauber Theorem)
moo n k=oû J '
1
Conversely, lim — X =0
n-)oo ft £=0 "
=> lim (1 - s)Pu(s) = 0(by Cesaro-Abel Theorem)
1
■V — > -
=> Pi = oo => i is null recurrent.

Exercise 2.4. Homogeneous Random Walk (Continued from Exercise 2.1) is null
recurrent if p = q.
This M.C. is transient if p * q (proved in Exercise 2.1 before) and recurrent
if e - i ’ l '
Proof Now p ^ n)b —J f L ---> 0 as n — >

■>jnn 2
Also Pw>n+l) =0 for all 1.

Hence p$ -> 0 as n-> oo
and («) — OO .
? P0Q} - ? ^00
(Here the chain is not aperiodic, the period is 2).

By Theorem 2.7. the M.C. is null recurrent.
Therefore, M.C. can never by positive recurrent.
Theorem 2.8 Limit Theorem (for off diagonal elements).
Let j be a fixed state in a M.C. and i be an arbitrary state. Then as n —>°o
(i) If j is transient, then pjf'*—>0 as n —> °°
(ii) If j is null recurrent, then —» 0
(iii) If j is positive recurrent and the M.C. is aperiodic,
then fa
Corollary 2.4 Let a M.C. be irreducible and aperiodic then lim p j j ] exists for
all i and /, but is independent of /.
Proof If j is transient or null, the result is obvious from Theorem 2.8. If j is
positive, then this can be proved if f f = 1.
Lemma 2.5 If j is recurrent and j —> /, then i is recurrent and
L
Proof Assume j * /, for otherwise there is nothing to prove.
Since f f > 0, there exists n0 such that p ^ o) > 0
and p^ff = 0 for 0 < m < n0 (2.6)
Since p ^ o) > 0, we can find states /j, . . . , ino x such that p Jn . . -PjinQ ] > 0
and none of the states iu . . . , /no i equal j or /, for if one of them did equal j
or /, it would be possible to go from j to i with positive probability in fewer then
n0 steps in contradiction to (2.6).
Suppose f f < 1. Then a M.C. starting from i has positive probability 1 - f f of
never hitting) and that implies it has positive probability . . . p in (1 - f f )
of visiting the states . . . , /„01, i successively in the first n0 steps and never
returning to j after n0 steps. But if this happens then the M.C. never returns to j
at any time n > 1 and that contradicts the fact that j is recurrent. So f f = 1. Since
f f = 1, there exists n x such that p \f^ > 0. Now p \ff+n+n° ^ > p \fl)p ^ p ^ o) and
hence 2 p[n) > 2 p ^ n+nQ) > p-"l}pyfo) 2 (since j is recurrent).

n=1 n=\ ■
’ J n=\
Hence, i is recurrent. Since i is recurrent and i ~^j(fij= 1) from the first part
of proof it follows that f f =1.
Proof of Theorem 2.8 By Theorem 2.4,
(m) (n-m) I f f(m) P (n-m) (2.7)
P ii ^=
m=1i Jl-I ii 2 , / / m>p )T m)
i=n'+1
where 0 < n < n(n > 1).
For £ > 0, take n and n so large that
( 2 . 8)
m=n'+1
For (i) and (ii) when j is transient or null recurrent take n so large that p^n
77
m)
< £ for all 0 < m < n < n (by Theorem 2.7). Now from (2.7) and (2.8), we have
0 °o *7 m=l 7 m= 1 7
< £ + £ = 2£, for all £ > 0.

Therefore pjj0 —» 0 as n —> <*>.
For (Hi) (j is positive recurrent and the M.C. is aperiodic) taken n —>°o and //
fixed. Then
0 < lim pW - lim 2 f!jn)p%-m) < (by (2.9))

n—>oo *•/ /i—>°° m= l 7 77
= lim p<n) - 2 1/y m)- f - (by Theorem 2.7 (iii))

m—>°° 7 m =l 7 jJLj
bm p<?> - - f 2 / / m> c £.
n— 7 p , m-1 7
Now take then ri —> then
0 < lim ,(*) 1

" ------ for e > 0. « v = j / r >)
Therefore p-'0 - >f u l P j .
2.5 Stationary Distribution

Definition 2.10 A probability distribution is [vj] (i.e. Vj > 0, 2 Vj = 1) is called
a stationary distribution for a Markov chain with transition matrix (p^) if
Vj = 2 ViPij for all 7 = 1 ,2 , . . .
=i k
X( Z v kPki)PlJ
= 2 vk Z PklP,j (by Fubini’s Theorem)
= ? « '* * # (by Chapman-Kolmogorov)
(by induction) (2.10)

Suppose a stationary distribution n - (n{, n2, . . .) exists. Also suppose

lim p\*•/p = 7t;7 > 0 for all i > 1
n—>oo
(2.11)
Then n is called the steady state distribution of the M.C. with transition matrix
If the initial distribution {ay0)}(a*0)= P(X0 = is stationary, we have the

marginal distribution of X„given by a\n) (i.e. a\n) - P(Xn = /)) = Z
(using (2.10)).
Thus, the unconditional (or marginal) distribution of Xn is independent of n
and we may therefore say that the system (or the process) is in statistical
equilibrium. Suppose conversely, that the distribution of Xn is independent of n.
Then the initial distribution a0 = (a\°\ a f \ . . .) i.e. a (0) = P(X0 =j ) = =j)
= Z <3-0)pij = a ^ = . . . and consequently, a0 is a stationary distribution. Therefore
the distribution of Xn is independent of n iff the initial distribution is a stationary
distribution. Suppose (2.11) holds. Since a{jn) = Z a f ]p p } /r; , we see that
the limiting distribution of Xn is given by n.
In other words, we can say that if (2.11) holds and a stationary distribution n
as n —> oo. Denote — = n ,.
Hj J
Theorem 2.9 (Limit theorem on stationary distribution)
Let a M.C. be irreducible, aperiodic and positive (i.e. the M.C. is ergodic).
Then (i) lim p \p = n ;
(ii) 7l, > 0, Z n j = 1
7 ./f-S 7
(iii) n .i = %s n kPkj
Moreover, (ii) and (iii) determine [7ih i e S) completely.
(Note that in this case the steady state probabilities or the long run distribution
is identical with the stationary distribution of the chain).
Proof (i) follows from Theorem 2.8 and the lemma thereafter.
(ii) nj = — > 0 (since j is positive recurrent /Uj < °o). Suppose is a
subset of state space S with exactly M states.
Now
Let n —> oo, then Z 71, < 1 .

,/gSm 7
Then taking limit M —» oo, 7ij< 1. (2.12)
2 P ^ P k j |^ = P u +l)(Chapman-Kolmogorov)
Let
n oo then Z n kpk j < 7i j .
keSM * -7 7
Then letting
M —» oo, we get Z 7ikp kj < n j . (2.13)
jeS
]tsn Jp? = Is ,!
= h { h n >pkk) pki- £ s n
By induction, Z J t j p ^ < nx for all n > 1, i e S. (2.14)

jzs
Now 1 (Since|,/eS?^ ”) = D
Z n k= Z/cg5ZjeS^kPkV
7
£eS
L ir
C * t c C .'c C * A, /
= Z Z n kp[ni) (by Fubini’s Theorem) (2.15)

yeS JteS K J
Suppose Z Ki Pi ? < 7T,. Then, Z Z < Z ;r, and

11 teS 7 y s S te S ;eS J
Z 7ik < Z Tt; which is a contradiction.

fceS /e.V '
Thus Z = tt, for 1. (2.16)
In particular, for n > 1, Z n iPn = 71i-This Proves (hi)-

MoreoverTby Lebesgue Dominated Covergence Theorem and part (i), letting
n ->ooin (2.16), Z^ 7T;-7T, = .
Now 7ii > 0, that gives Z Uj = 1.
To show that the solution given by (ii) and (iii) is unique, suppose that
{xh i,e 5} is another such solution satisfying xz>0, Z /r, = 1 and x t = Z x , p /7
jeS ■ jeS ’ ’
for all i e S.
xi = Z Z-XtPkj Pj<
jeS X j P >‘ j£S keS
= Z xk (by Fubini’s Theorem)

keS K jzS P k i P j
= z r n (2) —
~
keS X kPki
= Z x kp k{l ) for all n = 1, 2, . . .
By the Lebesgue Dominated Convergence Theorem, letting n —> «>,

x i - 2 x kn t = 7Tf- Z ^ = n i for all i e
lhus, the solution {7tn i e 5*} is unique.
Note that condition (iii) of Theorem 2.9 can be written in a matrix equation
7i{P —/) = 0 where 7t= Tb, . . .). This matrix P-I is such that its off-diagonal
elements are non-negative and the diagonal elements are strictly negative. In
case of finite chain with k-states, from the irreducibility condition it follows that
the matrix P- 1 is of rank In this case n - (tzu /z^, . . . n k) can be easily
obtained from the homogeneous equations n(P - 1) = 0 and the normalizing
k
condition X n, = 1.
/=i
We state the following theorem without proof.
Theorem 2.10 (Limit theorem for periodic chain)
Let the M.C. be irreducible, positive recurrent having a stationary distribution. If
the chain is periodic with period d, then for each pair /, j of states in S there is
an integer r, 0 < r < d such that p \^ = 0 unless md + r - n for some non-negative
integer m
(md+r) _
7li d__
mlim
—»oo p)j•' /V
Note
(i) if i = j, r = 0, Theorem 2.10 is proved before (see Theorem 2.7).
(ii) if r = 0, d = 1, Theorem 2.10 is proved before (see Theorem 2.8).
It is extremely useful to calculate higher order transition probabilities. We
shall illustrate this with a simple example. In general, Pn is difficult to compute
except for 2-state M.C.
Example 2.4.
0 1
0fI- p p \
Consider the M.C. with P =
H <7 I - Q)
Denote the initial distribution by /r0 = (n Q(0), n Q( 1)), row vector nn - (P(Xn =
0), P(Xn = 1)) represents the marginal distribution of Xn.
We compute p ^ ) = Po(Xn = 0), the conditional probability of Xn = 0 given
that X0 = 0. Let tt0(0) = P(X0 = 0).
For a stationary chain
Poi = P(Xn+1 = 1 I = 0) = p , p l0 =) = q,
Poo = I - p , Pn = I - q,P\o = *r0(l) = P(-X^ = 1) = 1 - ô(0).
Po(Xn+l =0) = P0(Xn = 0)p00 + P0((2.17)
P0(Xn = 0)(1 - p ) + P0(Xn =l ) q = (l
(2.17) is a first order difference equation. The solution is
Poo'” = po(Xn= 1) = + d -p -9 )'(l- — ) (Poo(0) = 1),
Similarly, p ^ = P0(X„= I) = - (I - p - q)" - E - ,

p \l} = — ----- (1 - p - q ) n- 2 — ,
^ 10 p +q y P +q
and = — — + (1 - p - q ) n— q—,
11 p +q
1 ~q p~ ’ P - p~
SX
1
1
Thus, Pn =
p + q _q p_ p +q -q q _
1 q p \0 «> if \
p +q q p p < 1, q < 1
This is the steady state distribution and also a stationary distribution. Hence
the marginal distribution of Xn is given by
q
P(Xn=0) = - ^ + (l
p +q p +q
P( Xn= = -p-q)n P —> p

p +q p +q
Moreover 0 < p + q < 2 i.e. | I - p - q \ < 1.
So we have a unique stationary distribution n determined by,
7I0 = 7" -, 7i\= —7— and , if 0<

p +q p +q lJ J t'
n does not depend on initial distribution.
Alternatively we may find the stationary distribution from Theorem 2.9. This
M.C. is obviously irreducible and aperiodic (0 <p, q < 1). Stationary distribution
is the solution of Xj = Z x p iy Therefore
1-p
( x u x 2) = (*i,-*2)
q
= CÔ - p ) + X2q, x {p + x2(\ - q))
or x j = Xj(l - p) + x2q and x2 - x xp + x2{\ - q)
or x2q - x\p - 0 and x xp - x2q = 0. Hence, we get
X\ + x2 1 (since x\ + x 2 - 1)
x 2!p - X \ ! q -
p +q p +q
Therefore P q
x2= = p +q
p +q
Let us make a few commets about existence of steady state distribution of M.C.
1. Even if a M.C. is irreducible the steady state distribution may not always
exist (since the irreducible chain may be periodic).
2. If the M.C. has more than one closed sets then the steady state probabilities
(i.e. the fixed probability vector n) do not exist.
3. A M.C. is called regular if there exists a positive integer m such that
every element of Pm except those relating to transient states is strictly
positive.
"0.6 0 0.4"
Exercise 2.5. P = 0.3 0.5 0.2 Here state 2 is transient.
v0.7 0 0.3y
f
7 0 4A
11 11
and pn —> 7 0 ± as n —» oo
11 11
7 0 4_
111 llj
and fixed probability vector n = satisfies the equation nP = n.
Proposition 2.1 Let P be a regular stochastic matrix. Then

(i) P has a unique fixed probability vector /rare all positive (those elements
corresponding to non-transient states) and nP = n.
(ii) Pn —> T whose rows are each fixed points.
(iii) Also nPn —» /rfor any probability vector n.
(Proof follows from Theorem 2.8 and Theorem 2.9).
2.6 Special Chains and Foster Type Theorems
2.6.1 Special chains

If the Markov Chain is infinite, the number of equations given by n(P - /) = 0
will be infinite involving an infinite number of unknowns. In some particular
cases we can solve these equations. The following examples will illustrate this
point.
Example 2.5 Birth and-Death Chain (Non-Homogeneous Random Walk)
Consider a birth and death chain on {0, 1, 2, . . . , d) or a set of non-negative
integers i.e. where d = <*>. Assume that the chain is irreducible i.e. pj > 0 and
qj > 0 in case 0 < j < d (i.e. when d is finite) pj > 0 for 0 < j < °° and qj > 0 for
0 < j < °° if d is infinite. Consider the transition matrix
(f r0
r Po 0
Q\ 0 P\ 0.
0 <72 r2 Pi 0.. .
0 0 <73 r3 Pi 0
^ • J
when d < °° we assume that rx = 0 for i > 0 and p0 = 1.

Particular Case: First consider that d is still infinite and rx = 0 for i > 0,
p0 = 1. The stationary distribution is given by
' 0 1 0 0 ••A
q\ 0 p\ 0
X = (x0, x u x 2, • • •) = (x0, x l, x 2, • • •) (2.18)
0 <?2 0 P2
ox X - XP. Let x0 * 0. Then

x0 = x\q\,
x \ = xo + x2q2,
X3 = X2P2 + x*q4>
x4 — • • •
Define yi = - L, y o = l > i = U 2 , 3,
Then yi - 1 1 - q\ p\
y 1= Uq\>y\= i + ^<72 or y2 =
q2 q\Q2 q\q2
P\P2 P\P2 - ■Pn- \
y3= >yn = >0 for all n = 1,2,
q\ci2q?> <7 l <72 • • •
(by assumption that all p, g’s are > 0).
By Theorems 2.9 and 2.11, the non-homogeneous random walk is positive
recurrent if 0 < £ a , < i.e. £ y, < ©o i.e. iff
o ' \ Jl
$ P \ P 2 • • -/V-l
£ ---------------- < oo.
n=1 Q\Q2 • - - <]n
Note that £ y.- = £ a ; / a 0 = 1/a0, since £ y, = 1.

o o o
Therefore xn = y„A0 = gives the stationary distribution provided
£ yj < oo • a0 still has to be determined.

oo oo ,
Now 1= x0+ £ =x0+ £ => *o = -----------
/=1 f=l V
1 + £ v,
/=i '
and so x0 > 0, iff £ y, < 00 i.e. iff £ y, < 00 (since y0 = 1).
/=1 i- 1
In fact if £ yz= oo, the solution to (2.18) is either identically zero or has infinite
sum ( £ x- = 00) and hence has no stationary distribution.
In general case also the stationary distribution exists iff X y, < 00. The system
°1 equations £ = 7lj becomes 7T0r0 + nxqx - 7^, Kj-\Pj-\ + +n^q^ =
ni lor j > 1. Since py + q} + rj = 1, these equations reduce to qxnx - p0jIq = 0
(since q0 = 0),
Qj+\ ftj+\ Pjftj —Qjnj Pj-\flj-b j — ^•

Pj
By induction. g,+j7L+1 - pjTij =o, \ - 1 and hence n J+\ = ----- n 0,j > 1.
Qj+1
PoP\ • • • Pj-1 .. .
Consequently, Uj = ------------- ■
— n 0,y > 1.
1 if y = 0
Set y.i = P0Pl-..Pj-> i f . > L
Then we have Uj = y} 7^ for j > 0.

Thus, the M.C. has unique stationary distribution given by
yj
j > 0.
2 yj
j=o J
Similarly, when d < ©o, then the unique stationary distribution is given by
n / = ~p-—- 0 < j < d .

s yi
7=0 ^ 7
As a particular case of the last example consider the Ehrenfest chain. The
transition matrix is given by
i/d if 7 = i ~ 1
P ij = 1 - i/d if j = i + 1 and S = {0, 1,2, . . . , J}
0 otherwise
such chain arises in exchange of heat or gas molecules between two isolated
bodies. Consider two boxes labelled 1 and 2 and d balls labelled 1, 2, 3, . . . d.
An integer is chosen at random. Then ball labelled by the chosen integer is
removed from a box and put in the other box. Trials are repeated. Let Xn be the
number of balls in box 1 after n trials.
Then Xn is a M.C. on S = {0, 1, 2, 3}, d = 3.
0 1 0 0>
1/3 0 2/3 0
0 2/3 0 1/3
0 0 1 0/
This i$ an irreducible birth and death chain with
Hence, the unique stationary distribution is given by

13 3 1
7T =
8’ 8’ 8’ 8
No steady state probability exists because p \^ = 0 for all odd values of n. It has
periodic behaviour
r 41 0 4
4
0 3
pn ^ 4 when n is even and
7 0 4
» 7
0 4 0 4
3
pn ^ 4 when n is odd and n is large.
0 4 0 4
4 0 4
Modified Ehrenfest chain

Here in addition we select at random one of the two boxes and put the removed
balls into this box.
Pxo = Probability that the ball selected is from box 1 and the box selected is
box ? - I I - I
3 ’ 2 6'
Similarly, p n = j -j = 13 = 0 =1 P lO - P 12 - P 13 - 3.
r 1/2 1/2 0 0 '

1/6 1/2 1/3 0
0 1/3 1/2 1/6
Io 0 1/2 1/2,
Again from Theorems 2.9 and 2.11 the unique stationary distribution is given by
13 3 1
4’ 8’ 8’ 8
Here p u = ~ > 0. The M.C. is aperiodic.
Note If some r, > 0, then pu = rx> 0 and the chain is aperiodic. In particular
Modified Ehrenfest chain is aperiodic. Suppose rt = 0 for all /, i.e. we have a
non-homogeneous random walk. Then in one transition the state of the chain
changes either from an odd numbered state to an even numbered or from an even
numbered state to an odd numbered. In particular, the chain can return to its
initial state only after an even number of transitions. Hence, the period of the
chain is 2 or multiple of 2. Since = p0qx > 0, the chain is periodic with
period 2 and so the Ehrenfest chain is periodic with period 2.
Example 2.6 Consider the Markov chain with transition matrix
' 0 1 0
p = 1/2 0 1/2
v0 1 0
This chain is irreducible. We have
"1/2 0 1/2
P2 = 0 1 0
v1/2 0 1/2
so that p ^ n) > 0, p|J2n+1) = 0 for each i.

The states are periodic with period 2. It follows that X p (nn) diverges so that
n
the state i is recurrent. (Note that a periodic state must necessarily be recurrent,
though a recurrent state may or may not be periodic).
Now /,(,1) = 0 ,/® = 1 so that/,, = f/iV 0 = 1

i.e. State 1 is recurrent and by solidarity theorem the other two states 0 and 2 are
also recurrent.
Now jui = X n f ^ = l x 0 + 2+ l = 2 . State 1 is positive recurrent. Thus the
states of the chain are periodic (each with period 2) and positive recurrent. By
Theorem 2.7, pfl ) dl/u= 2/ 2 =1 for all /.
2.6.2 Foster type theorems

The following theorems, associated with Foster, give criteria for transient and
recurrent chains in terms of solution of certain equations. Assume that the M.C.
is irreducible.
Theorem 2.11 (Foster, 1953) Let the Markov chain be irreducible. Assume that
there exists xk, k e S such that x k = X^xtpik and 0 < X Ijc* I < ©o. Then the
keS keS 1 1
Markov Chain is positive recurrent (this is a sort of converse of Theorem 2.9).
Proof Since yk = -= -j---- > 0, 2 y k = 1.

L \Xk keS
keS 1 1
Without loss of generality {xh k e 5} is a stationary distribution of a M.C.

Then
xk = JC iPjkn)for all ft = 1, 2, . . . (2.19)
Suppose that there is no positive state.

Since the M.C. is irreducible, then all the states are either transient or null. In
that case p (tp —>0 as n —» °° for all /, k e S. By Lebesgue Dominated Convergence
Theorem, taking n —> in (2.19)
xk = Z (x,). 0 = 0 for all /: E S (2.20)

j' g S
But 0 < ,Z x* < °° is a contradiction to (2.20).

Hence, there is at least one positive recurrent state. Since M.C. is irreducible,
by Solidarity Theorem the M.C. must be positive recurrent.
Conclusion An ireducible aperiodic M.C. has a stationary distribution iff all
states are positive recurrent.
Theorem 2.11(a) If the M.C. is positive recurrent the system of equations
a, = Z XjPn has a solution such that 0 < Z x, < «>.

7=0 >r Jl j =0 j
(Proof may be found in Karlin and Taylor’s book.)
Theorem 2.12 The M.C. is transient iff x t = ZqPijXj has a solution for i * 0,
which is bounded and non-constant i.e. all x,’s are not equal.
Theorem 2.13 The M.C. is positive recurrent if x t > XQPijx j has a solution
such that-Xj -> ©o as i -> oo (see Chung’s book on Markov Chains with Stationary
Transition Probabilities).
2.7 Theorems Regarding Finite Markov Chain

Theorem 2(a). In a M.C. with a finite number of states, there is no null state
and not all states can be transient.
Proof Suppose the chain has N <<*>states. If all states are transient, then letting
N
n -> oo in the relation Z )p\'j) - 1 we get 0 = 1 (since by Theorem 2.8,
Jim p \fp - 0 for each j), which is absured and hence not all states in a finite
M.C. are transient. Consider the subchain C\ formed by a closed set of null
recurrent states. Then Z p \p = a (say) > 0. Letting j i —><», 0 = a > 0 which
is also absurd. So there cannot be any null recurrent state in a finite M.C.
Theorem 2(b). An irreducible M.C. having a finite number of states is positive
recurrent.
Proof By previous theorem, there is no null recurrent state and not all states are
transient. Suppose there is one transient state. Then all states are transient by
Solidarity Theorem. Hence, all states are positive recurrent.
Exercise 2.6 If a finite M.C. is irreducible, aperiodic and has doubly stochastic
transition matrix, then show that lim p f(.n) = 1Ik, where k is the number of states
n —>oo
in the chain.
Solution If j is a positive recurrent state in an aperiodic irreducible chain then
p [7n) —> 7tj> 0 (by Theorem 2.9).
for all j and n > 1,
Pn • • •Pu
= 1
P2\ P22 • • ■
•Pik
= \ f lim ptf> = 1
_ j i=l n —>°° lJ
KPk\ • • -Pkk
1 1 . . ,. 1
; 7tj- 1 ==> 71j - 1

' k'
Exercise 2.7. Find the unique fixed probability vector of the regular stochastic
matrix
0 1 0
P = 1/6 1/2 1/3
0 2/3 1/3
What matrix does P n approach?

Solution
1 1 1'
6 2 3
P2 = 1 23 5 > 0, so P is regular.
12 36 18
1 5 1
9 9 3
Fixed vector n = (x, y, z) satisfies the equations

1
6y=* y = 6x
1 , 2 or 6x + 3y = 4z = 6y
or 2 y + 3 Z= y
1 1 y + z = 3z
3y+ r = z
y = 6* Hence on zero solution is feasible.

6* + 4z = 3y ► Setting* = 1, we get
or
y = 6 and z = 3.
Since 1 + 6 + 3 = 10, the probability vector or stationary distribution is given
by;r=(A’^’^)-0bviously
i 6 3
10 10 10
i 6 3
10 10 10
i 6 3
10 10 10
Exercise 2.8 Suppose a machine can be in one of three states:

(1) broken beyond repair, (2) in need of adjustment or
(2) working properly, with the following transition probabilities,
1 0 0"
1 1 1
4 2 4
I 8 9
18 18 18
Under these conditions : (1) what is the probability that the machine will be in
state i (i = 1, 2, 3) after n time period? (2) What is the probability that the
machine is in state i when n becomes infinitely large (i.e. the steady state probability
or long run behaviour of the machine)? (It is given that the machine is initially
in need of adjustment).
2.8 Methods of Evaluation of the n-Step Transition

Probability
(a) Method of Spectral Decomposition
Let P be a NXN matrix with latent roots Ab . . . , XN all distinct and simple. Then
(P - A,/) Uj = 0 for the column latent vector Ut and
Vj\P - A, /) = 0 for the row latent vector Vr
= U t V ' are called latent or spectral matrix associated with Az, i = 1, . . ., N .
The following properties of A,’s are well known:
(i) A - s are idempotent, i.e. A f = A/,

(ii) they are orthogonal, i.e. A, A; = 0 (i * j),
N
(iii) they give spectral decomposition P = 2 A, A,. It follows from (i) to (iii).
( N \ k N N
pk 2 A, A,- = 2 A.*A; = 2 M U M ' . ( 2 . 21 )
m ' 1 i=1 ' i=l 1
Also we know that P* = (/D* £/ 1 (by Diagonalisation Theorem) where

U = (t/b f/2, . . . , I/N)
‘A, 0. . . 0
and D= 0 ^2
0 Aw
Since the latent vectors are determined uniquely only upto a multiplicative
constant, we have chosen them such that U' Vt = 1. From (2.21) one can get any
power of P knowing A,’s and A,-’s.
Example 2.7 We shall illustrate the last method with the help of Exercise 2.8 of
Section 2.7.
1 0 0"
1 1 1
In our problem, P = 4 2 4 with characteristic equation
1 8 9
18 18 18
1- A 0 0
1 1
IA = 0 or (1 - A ) A2 - A + 0.
4 2 4 36
=
1 8 9
18 18 18 ‘
So the eigenvalues are A, = 1, A2 = A3 =

Corresponding to A} = 1, taking a x = 1 in the matrix equation
1- A 0 0
I 1 ~0i ~
4 5a. 4 = 0 we get the characteristic vector U\ =
_L JL 9 ,
_c\ _
18 18 T8Al
Corresponding to A2 = -g-, taking a2 = 0, the characteristic vector is U2 =
Corresponding to A3 = —, taking a$ = 0, the characteristic vector is U2 =

Hence, the matrix whose columns are characteristic vectors is given by
1
o
o
U= 3 3 and r 1= —
24
-4 4_
So
0
'1 0 0] p* 0 0
Pk = UDkU~l= 1 3 3 6 4 -3
1 -4 4j 0 4 3
0
0 0
1 - rk
3s_k n 3sk
or 8 2 8
■ rk sk 2 sk rk
2 8 8 2
k
'5 V
where rk =+ 1 ’sH ,6
Now rk —> 0 and sk —> 0 as k ►oo.
1 0 0
Therefore Pk —» 1 0 0
1 0 0
This shows that irrespective of the initial distribution (state), the machine will
be in state 1 (irrepairably broken). So state 1 is an absorbing or a trapping state.
Exercise 2.9. Derman (1963) designed a model for making the Maintenance or
Replacement decisions.
At the beginning of time period State of equipment at the end of time period
Operating properly Need of Adjustment
Operating Properly 0.90 0.10
Need of Adjustment 0.01 0.99
0.90 0.10'
Here P =
0.10 0.99 *
Suppose the equipment starts out working properly in period zero. Calculate
the unconditional probabilities of the state of the equipments and the conditional
probabilities after time period n.
Solution The initial distribution a(0) = (1, 0),
f. 90
am = a (f>)P= (1,0) = (0.90,0.10) and
v-01 .99
^0.8110 0.1890A
a <2) = a <I)P = (a (0>P 2) = ( 1,0) = (0.8110,0.1890)
_0.0189 0.9811
In general a(n> = a<0)P". We shall calculate Pn by another method.
(b) Method of Caley-Hamilton

Caley-Hamilton Theorem Every square matrix satisfies its own characteristic
equation.
The characteristic equation of P is given by | P - A/1 = 0. In the last example
0.9- A 0.10
= 0 => A2 - 1.89 A + 0.89 = 0.
0.01 0.99
By Caley-Hamilton Theorem,
P2 - 1.89 P + 0.89 / = 0 =* P2 = 1.89 P - 0.89 /
=> P2 = 1.89 P2 - 0.89 P = 1.89(1.89 P - 0.89 I) - 0.89 P
= 2.6821 -1.6821 /.
P
Similarly, any power of P can be calculated in this manner.
(c) Method of Generating Functions
Since 1 is always a latent root of a stochastic matrix P. (/ - tP) is non-singular
for 0 < t < 1.
Hence (/ - AP)~l exists. But since (tP)n —> 0,
/ + tP + P'P2 + . . . = ( / - tP)~x. Therefore, p \yn) is the coefficient of t n in the
( ij)th element of (I -tP)~l, which is the generating function of {p^n)}- Suppose
S= {0, 1, 2, 3} and
0 0 0)
<7 p 0 0
P= ,p + q= l,
0 q p 0
0 q p.
Then | / - tP \ = (1 - t) (1 - tP)2 and
(1 - O '1 0 0 0
tq 1 0 0
(1 - 0 ( 1 - t p ) 1 - tp
(i-tpy' = (tq)2 tq 1
0
(l - 0(1 - tp)2 (l -
t
p)2 1
(f<7)2 (*<?)2 _ _ _1_ _
(1 - r)(l - tp)2 (1 - tp)2 (1 - t p ) 2 1 - tp
Expanding the ( i j ) th element of (/ - tP) 1 in powers of t, we get as the

coefficient of f".
(d) Method of First Entrance
If / ^ m) be the probability that a M.C starting with state j will reach &for the first
time at the mth step, we obtain from Theorem 2.4 the generating functions
Pjk(s) = Fjlc(s)Pkk( s ) U * k ) }
and PjJ(s) = FjJ( s ) P j j ( s ) + l j
Hence we can determine Pjk(s) and Pkk(s ) if we know Fjk(s) and Fjj(s).

Exercise 2.1 Find the closed sets in the following Markov chain with transition matrix:
1 2 3 4 5
1 "0.2 0.3 0.5 0.0 0.0
2 0.7 0.3 0.0 0.0 0.0
3 0.0 1.0 0.0 00 0.0
4 0.0 0.0 0.0 0.4 0.6
5 0.0 0.0 0.0 1.0 0.0
1 2 3 4
1 "0.0 0.0 1.0 0.0
2 1.0 0.0 0.0 0.0
3 0.3 0.7 0.0 0.0
4 0.6 0.2 0.2 0.0
Exercise 2.2 A service agency assigns its jobs to a particular worker in the following
way. The maximum number of jobs assigned to him, in addition to one he is working on
at any time is N(> 1). If he can not finish the assigned jobs on a given day, he starts with
the remaining ones the following day. However, if any time of the day he finishes all the
jobs assigned to him, he returns to his own work and becomes unavailable for any more
agency jobs for that day. Let pj be the probability that j(> 0) new jobs arrive during a
service period. Let Xn be the number of jobs assigned to him at the end of the nth service.
Under what conditions is {X,,} a Markov chain? Determine its transition probability
martix and classify its states.
Exercise 2.3 (Bartky's sampling inspection scheme). In a sampling inspection procedure
successive sampling of size N is taken. If in the initial sample the number of defective is
zero, the lot is accepted. If the number of defective exceeds a predetermined number a,
the lot is rejected. From the second sample onward one defective per sample is allowed.
Thus after n such samples, the lot will be accepted if the total number of defectives is
^ n and rejected if the number of defectives is > a + n. Let Xk = number of defectives out
of N - One at the &th sample. Let Sn = Total excess number of defectives in the lot. Find
the distribution of Xk. Show that {£„} is a Markov chain. Find the transition matrix for
{£„} in terms of distribution of Xk. Also classify the states of {5,,}.
Exercise 2.4 A Simple Waiting Model (Queueing). In a simple queueing model a server
serves one customer (if any) at time instant 0, 1 ,2 ,.... Let be the number of customers
arrive in the time interval (n, n + 1) and we assume {£,„ n > 0} is a sequence of i.i.d.
nonnegative integer valued r.v.’s with P(%o = k) = pk, Zq pk = 1 and there is a waiting
room for at most m customers (including the customer being served). Let Xn be the
number of customers present at time n, including the one being served. Show that Xn is
a Markov chain with states 0, 1, . . ., m. Find its transition matrix in terms of {pk }q .
Exercise 2.5 (Storage Model). Consider a dam which can hold at most m units of water.
Assume that during day n a quantity T]n of units of water flow into the dam, n > 0, where
{r)n, n > 0} is a sequence of i.i.d. nonnegative integer valued r.vs. with P(rj0 = k) = pk,
E pk - 1. Any overflow is lost with no possibility of recovery. At the end of any day
k =0
when the dam is not dry, one unit of water is released. Show that (X '}, the content of
the dam at the beginning of day n and {X''}, the content of the dam at the end of day n
are Markov chains on appropriate state spaces. Find the transition matrix of { X'', n > 0}
in terms of (pk )q.
(Note: {X', n > 0) is a M.C. identical to that with the preceding Queueing example
with states space (0, 1, . . . , m) and hence it is omitted for any further discussion.)
Exercise 2.6 A Markov chain remains Markov if the time is reversed, i.e.
PQ^n ~ hi I Xn+\ — in+1, • • •, Xn+k — in+k) —P(Xn — itj | X,I+1 — in+\)
Exercise 2.7 Show that the eigenvalues of a stochastic matrix are in absolute value at
most equal to 1.
Exercise 2.8 (A model for the spread of a contagious disease in a small community). Let
m > 0 be the number of individuals getting infected at time m (i.e. the number of
infectious individuals at time m + 1) and Xn, n > 0, the number of susceptibles still
uninfected at time n. Let us assume that the chance of contact between two individuals be
0 < p = 1 - q < 1 and the contacts to be independent. Show that {X„} is a Markov chain
with states space {0, . . . . , r] when the initial number of uninfected susceptibles is r. Find
its transition matrix. If r be the duration of epidemic defined as smallest integer r such
that £t = 0, express the number of cases in the epidemic in terms of XT(XT_! = XT).
Exercise 2.9 Show that in a Markov chain, if the present is specified, then the past is
independent of the future in the following sense. If k < m < n then
P(X„ = i„, x* = ik
Ix m = i
Exercise 2.10 A Markov chain is of order m < k if

P ( X „ ttm= x „U m , X„ M = x „M
| X nj = x „ j , 1
~ = x „ l t m, , X nttl = x „ ktl |
for integers nx < n2 < . . . < nk+m.
Show that if {X„, n > 1} is a Markov chain, then it is a Markov chain of order m < k.
Exercise 2.11 If P = (p/; )NxN stochastic matrix, then show that there exists a stochastic
n
matrix Q = (qu )NxN such that lim Z p <
i? ) = qih i , j = 1, 2, . . . , N. Moreover,
a— m=1 ' J
QP =
Exercise 2.12 Prove that in a finite Markov chain a class C of states is recurrent iff it
is closed.
Exercise 2.13 (a) If a Markov chain starts in state i then show that the probability of
returning to i at least r times equals f t .
(b) If a Markov Chain starts in state i then show that the probability that state j occurs
at least r times equals fij[fjjY'X-
(c) Calculate the probabilities of returning to i infinitely often when f u < 1 and when
fa = 1•
Exercise 2.14 Let u(i) be the number of visits to the state i and P,[A] is the conditional
probability of A starting from the state i. Then show that
(a) P,[v(j) = n] =fU[fijr lU-fjln> 1
where fj is the conditional probability of ever visiting j starting from /.
f ki
(b) Pk[vU) °<°] = 1 and n(k, i)=
1 Jii
where p{k, /) is the expected number of visit to i starting from k when i is transient.
(c) Hence show that if i is a transient state, then the M.C. visits i only finite number
of times and the expected number of visits is also finite.
Exercise 2.15 In a Markov chain whatever be the states i and j, prove the Doeblin’s
formula
m
(1)
In particular show that for

1
j = i fa = 1 - lim m ( 2)
Exercise 2.16 Let {Yn, n > 1} be a sequence of i.i.d. r.v.s having discrete uniform
distribution on {0, 1, 2, . . . k}. Let Xn = max {Tj, . . . , T„}, n > 1. Show that {X,,} is a
Markov chain (M.C.) and obtain its one-step transition probability matrix. Discuss about
the states of this M.C.
Exercise 2.17 Consider the polya urn scheme with a white and b black balls in a box.
Each time one ball is drawn and is replaced together with an additional ball of the same
colour. The procedure is repeated several times. Let Sn equals the total number of white
balls in the box at the end of the rcth step. Show that S0, ..., Sn, ... is a nonhomogeneous
Markov chain with transition probabilities given by
i _ a+b+n- i
n Pi,j+1 a + b + n nPl1 a + b + n ’ „p,j = o i f / * i , ; + i .
Exercise 2.18 If Xn = max j K,. Y,. . . . Yn] where K, denotes the number on the face
turning up in the ith toss of a die with faces 1,2,..., 6. Show that {X,,} is a Markov chain.
Obtain its transition matrix.
Exercise 2.19 Inputs at the jth period, Xr into a dam of finite capacity k follows a
geometric distribution P(Xj = k) = qpk(k = 6, 1, . . .).
There is unit release unless the dam is empty. Show that the contents of the dam {Yn)
at any time n form a Markov chain. Write down its transition probability matrix.
Exercise 2.20 Let {AT„} be a M.C. on {1, 2, . . . M}. The conditional distribution of X„+1
given Xn = j is the discrete uniform distribution on [M - j + 1, . . . , M). Obtain the
stationary distribution.
Exercise 2.21 Consider a Markov chain having state space {0, 1,2} and the following
transition probability matrix P.
0.0 1.0 0.0
0.5 0.0 0.5
0.0 1.0 0.0
Identify the recurrent states of the chain.
Exercise 2.22 Consider a communication system which transmit the two digits 0 and 1
through several stages. Let {Xn, n > 1} be the digit leaving the nth stage of the system and
X0, the digit entering the first stage leaving the 0th stage. At each stage there is a constant
probability q that the digit which enters will be transmitted unchanged, and probability p
otherwise, p + q = 1.
Find the rc-step transition matrix of the two-state Markov chain and the steady state
probabilities if there is any. Also find the probability that the digit entering the first stage
is 0 given the digit leaving the rath stage is 0 when the initial distribution is given by
P(X0 = 0) = a = \ - p ( X 0 =l).
Exercise 2.23 Let pxy denotes the probability that the Markov chain starting at state x
will be in state y at some positive time; also N(y) denotes the number of times n > 1 that
the chain is in state y. Show that P[N(y) > 2 \ X 0 = x] = pxypyr X0 being the initial state.
Extend your arguments to establish that in general P(N(y) > m \ X0= x) = pxy pyy~\ ra > 1.
Hence sho,Wthat if y is a transient state, then for every initial state x, E[N(y) | X0 = x] < «>.
Now argue to deduce that a Markov chain having a finite state space must have atleast
one recurrent state.
Exercise 2.24 Deduce that (with usual notation)
(n+m)
f JJm)p(n)<p..
rjj r JJ + (1 - p T )
Exercise 2.25 On a genetic inbreading experiment with single pair of Allelomorphos A
and a, the progeny of a mating Aa x Aa results in 50% heteroztgotes Aa where the
proportion of hetrozygotes being reduced by one-half at each generation. Find the transition
matrix of various genotypes and the proportion of hetrozygotes at rth generation and in
the long run.
Exercise 2.26 Consider a dam for water storage with Capacity k units, k being a positive
integer. Let there be independently identically distributed integer-valued daily inputs to
the dam with probability generating function g, Zy. Any overflow is lost. If the dam
is not empty, m > 1 units is released at the end of each day. The dam contains Zn units at
the end of nth day. Show that Z„ is a Markov chain and write down the transition matrix.
When is the chain irreducible? Find also the transition matrix when the input has a
geometric distribution.
Exercise 2.27 Modelling of soil erosion effect and modelling of a rock crunching process
(.Kolmogorov 1941).
Let {Yn}T be the sequence of r.v.s representing annual yields of a given crop in a crop-
producing area A when it is not affected by erosion. Let Rn be the percentage reduction
of annual yield due to soil erosion in the nth year and Un - R J 100. Then show that when
A is subject to soil erosion the resulting annual yield in the nth year is given by
n n
Xn = Yn n Z,, Z, = 1 - Ui. Incase {Z„}[° arei.i.d. and independent of {Yn}~, Ln = n Z,
i=l i=1
is called “loss-rate function”. Show that {L„}T is a Markov chain.
Exercise 2.28 N black balls and N white balls are placed in two urns so that each urn
contains N balls. At each step one ball is selected at random from each urn and the two
balls interchange. The state of the system is the number of white balls in the urns.
Determine the transition matrix of the Markov chain. Find its steady state probabilities
and show that these do not depend on the initial distribution of black and white balls in
the two urns.
Exercise 2.29 Consider two urns A and B containing a total of N balls. Assume that at
time t there are exactly k balls in A.
(a)_A time t + 1 urn A is chosen with probability k/N or B is chosen with probability
N It
—— —. Then a ball is selected from A with probability p or from B with probability q and
placed in the previously chosen urn. Determine the transition matrix for this Markov
chain.
(b) Now assume that at time t + 1 a ball and an urn are chosen with probability
depending on the contents of the urn, i.e. both a ball and the urn A are chosen with
aj _ im
probability k/N or both a ball and urn B are chosen with probability ———. Determine
the transition matrix of the Markov chain with states represented by the contents of A.
(c) Determine the equivalence class in (a) and (b).
Exercise 2.30 Every stochastic n x n matrix corresponds to a Markov chain for which
it is the one step transition matrix. Similarly every n x n two-step transition matrix of a
Markov chain is a stochastic matrix but the converse is not true. Give a counter example.
Exercise 2.31 If P is the one-step transition matrix of a finite aperiodic irreducible
Markov chain, prove that for some positive integer n all terms of Pn are positive.
Exercise 2.32 If j is a transient state prove that for all i e S, 2 p\^ < °°.
Exercise 2.33 If an infinite irreducible Markov chain has a doubly stochastic transition
matrix, prove that the chain can not be positive recurrent.
Exercise 2.34 Suppose the quantities {nip i j • e 5} are defined as the “Cesaro means”
nn' = lim III imX-1 1P\p.
n m=l ‘j
(a) Show that nxj always exists for all i,j e S.

(b) Show that jzs
2 n ulJ < 1 for all i e S.
(c) Show that n,, = X n ikpkj= X Pikjtkj = X nikn kj.
(d) If j is a positive recurrent state, show that
fij
=fij7tj for all i e S.
n=1 JJ
Exercise 2.35 Show that if a finite irreducible Markov chain has a doubly stochastic
transition matrix, then all the stationary probabilities are equal.
Exercise 2.36 Let /V, be the number of returns to state i, starting at /. If / is transient, then
prove that P(N, = r) = (1 -fu)(fuY. Also find E(Nj).
Exercise 2.37 A positive recurrent irreducible Markov chain is given.
Let /i/; = £ f yn). Show that
Pij ~ Pij + 2 pik [1 + /i*/] •

Exercise 2.38 Let j and &be arbitrary states in a (arbitrary) Markov chain, prove the
following statements:
(a) supp';’ </,*<

n n~l
(b ) j k iff /*> 0.
(c) j <r*k iff/,/*, > 0.
<d> Z / j V - f j * +1
fkk
(e) If/** < 1 then jZ p^0 = } •
Exercise 2.39 Consider the unconditional (marginal) probabilities in a Markov chain

p ”' = P(X„ =j),n>1, p‘0) = P(X0 =j) = ctj and I a , = 1.
(a) Show that p,n) = Z p\Va/ ar|d that
Z p /0 = 1 for all n > 0.
(b) If the Markov chain is irreducible and aperiodic then show that , = n—
-
lim
>°° J
(c) Assume that the Markov chain has a stationary distribution {/r7}. Evaluate p(.n) if
ak = nk, Explain.
Exercise 2.40 Consider a Markov chain on S = {0, 1, 2, . . .} with transition matrix
p jk = e~k ^ j pnq]~nXk~n!{k - n)\, where p + q = 1, p, q > 0.
Show that the Markov chain is irreducible, aperiodic and positive recurrent. Also
prove that Kj = e~Xlq{Xlq)ilj\ is the stationary distribution.
Exercise 2.41 Assume state {0} of a Markov chain is positive recurrent. Take {0} to be
the initial state of the Markov chain. Let {Wn, n > 1} denote successive recurrence times
which are independent and identically distributed r.vs. with finite mean and with a generating
function G(t) = YPiW^ = /:), 111< 1. Define Yn as the time of the last visit to state {0}
before time n.
1 -G(t)
Show that £ r Z\oxJP{Y,,=j) =
n=0 (1 - / ) ( 1 - G ( x t ) )
Exercise 2.42 Consider a discrete time Markov chain with states 0, 1, . . . , N whose
transition matrix has elements
Hi if i= / - 1 i, j =0, 1,2__ N
A, if j= / + 1
1 - Hi - A, if;
0 otherwise
Suppose that ^ = Ao = = XN = 0, and all other ^,’s and A,’s are positive and that
initial state of the process is k. Determine the absorption probabilities at 0 and N.
Exercise 2.43 Consider an irreducible positive recurrent Markov chain with initial state
X0 = i. Let Nn(i) be the number of visits to state i in the first n trials. Prove that
E(Nn(i)) = 1
n—>°° n Hi ’
where fa is the mean recurrence time to state /, i.e. Hi = 2
Exercise 2.44 Consider an irreducible Markov chain. For some state; call it 0. Prove that
if fjo > a > 0 for every i * 0 then the chain is recurrent, where f i0 is the probability of
reaching state 0 from state i.
Exercise 2.45
(a) Let a finite state Markov chain be given. Prove that j e S is transient iff there exists
a state k e S such that j —>k but k + j.
(b) Give a counter example to (a) if we drop the “finiteness” of the Markov chain.
Exercise 2.46 Let a Markov chain be given with only absorbing states (E) and transient
states (Ec = S - E).
(a) Show that transition matrix P can be put in the form
E E(
E (I
EC\ R
(b) If i j e Ec, let Ny be the mean number of times the chain is in state j, starting at
i. Prove that N = ^ Q (k) where Q = Q(0) is defined in (a).
(c) Define By = P {The Markov chain is absorbed at j \ X0 = /} for j e E, i e Ec. If
E * (p(null set), show that B - NR (with B = (By)).
(1
(d) Derive from this nlim
-* o o
Pn = D
Exercise 2.47 Prove that if the one-step transition matrix P of a discrete time Markov
chain is symmetric, so is the n-step transition matrix P(n) for all n > 1. Moreover this
Markov chain cannot be positive recurrent (except for finite chain).
Exercise 2.48 Let M be an irreducible, aperiodic positive recurrent discrete time Markov
chain. Assume that the initial distribution coincides with stationary distribution. For all /,
j e S and integer m > 1, define (R{m\ = Rjtm) = P(Xn = i \Xm=j).
(a) Show that this chain is also positive recurrent, and determine its stationary distribution.
Exercise 2.49 (Markov-Bernoulli chain, Wang (1981))
Consider a sequence of dependent trials such that each trial has only two outcomes
denoted by ‘1’ and ‘O’ and dependency is connected by a simple Markov chain model
having transition probabilities
0 1
Of \ - ( \ - c ) p 0 -C)p \
, 0 < p < 1, 0 < c < 1
1 [(1 -c)(l - p) (1 - c ) p + cj
with initial distribution px - P(XQ= 1) = p = 1 - P(X0 = 0).
Find the marginal probabilities pn = P(Xn = 1). If Sn = Sx + . . . + Xn, show that
E(Sn) = npvar(S„)
, =np(1 -p ) + 2p(l -p ) - c2 / / , ~ j ■If as n “ and
p —>0 such that np —>A, then show that E(Sn) —» A and var (S„) —>A + -p— •
Exercise 2.50 Let P = (/fy) be a transition matrix of an irreducible Markov chain and
suppose that P is idempotent Markov matrix, i.e. P2= P. Prove that p^ = pjj for all i,j, e 5
and that the chain is aperiodic.
Exercise 2.51 (Ergodic theorem in discrete time). Let Xn be an irreducible M.C. Let
n- 1
pi be the mean recurrence time of state i. Let Vt{n) = 2 I\Xk=i\ numt>er of visits
to i upto n - 1 and /b e any bounded function on S. Show that:
(a) ri~lVi(n) ■—> p~x as n —»
1 " '1
(b) if Pi < °o for all /, then —= ^ f ( x k) —> X f(i)/pi a s ^ ~
Exercise 2.52 (Estimating transition matrix). Let {X*, 0 < /:< rc) be a M.C. with initial
distribution a, = P(X0 = /). The log-likelihood function L(P) = log (ccXQpXQX{. . . pXn_lXn)•
Show that:
(a) L(P) = log a*0+ 2 /Vy log ptj where is the number of transitions from i to j,
(b) Viewed as a function of pijt L(P) is maximal when Pij = Pij = N^/2
(c) If X is irreducible and ergodic then ptj —>pi} a.s. as n
3
Random Walks
3.0 Introduction
Let [Xn, n = 0, 1, 2, 3, . . .} be a sequence of independent discrete random
variables taking integral values only and Sn = X x + X2 . . . + Xn(n = 0, 1, 2, . . .).
Then the sequence {S„} is a M.C. whose transition probabilities are given by,
{m)Pij = P(Sm+1 = j | = i) = P(Xm+l = j - /), i j = . . . , -2, -1, 0, 1, 2, . . .
(non-homogeneous random walk).
The chain represents a Random walk of a particle along a straight line, the
magnitude of ‘jump’ at time n being given by the random variable Xn. If X0 is
denotes the initial position of a particle then its position after n jumps (at time n)
is given by Sn. When Xn's are also indentically distributed, = Pj_t where pj
= P{Xn = j). We have then a homogeneous Random walk (RW). Such Random
walks occur in fluctuation theory (sums of discrete or continuous random variables).
In classical RW, P{Xn = +1) = /?, p(Xn = -1) = q = 1 - p.
3.1 Different Types of Random Walks

(a) Unrestricted Random Walk
In this the elements of transition matrix is given by pt J+1 = p, pt i_x = q, for all
integer /(. . . , -1, 0, 1 , 2 , . . .).
If 0 < p < 1, the chain is irreducible. Then we have
n+j-i n-j+i
Pp = P ( s n i 2 if n is even
v( n -j+
= 0 if n is odd.
and n n/2
Poo = 2 (pq)
v J
The period of the chain is 2.

It is transient if p & ■- and null recurrent if p =
(b) Random Walk with an Absorbing Barrier

In this walk the elements of transition matrix are given by pi i+\ = p, pL
(p + q = 1), p00 = 1 for all i > 1.
‘O’ is an absorbing state and the remaining states are all transient. 0,
-3, . . . are condensed into a single absorbing state ‘O’.
Let /.(0Al) = Probability of visiting ‘0’ from /, first time in n steps
f n ) p(n-i)/2 (j (n+i)/2
K( n - l)/2.
Probability of visiting ‘O’ from i ever,
fio = Z //0"} satisfies difference equations
fio = Pfi+i,o + <lfi-i,o for i > l,/io = p/20 + q-

Hence solving we get
1 if p < q
fiO ~
(q/pY if p > q.
(c) Random Walk with Two Absorbing Barries

Here the elements of transition matrix is given by
Pi,1+1 = P> Pi, m = <7 for 1 < / < « - 1, poo = 1, pflrt = 1.
‘0’ and V are absorbing and remaining states are transient.
Examples of Random Walks with Absorbing Barriers

Gambler’s Ultimate Ruin Problem
The fortune of a gambler forms a M.C. with transition matrix
P if / = i + 1
Pu <17 if j = i - 1 and i = 2, 3, . . . , s
0, otherwise
1 if 7 = 1
and i = 1 and 5.
0 if j * 1
More explicitly the transition matrix is given by
" 1 0 0 0 ■
q 0 p 0 0
0 <7 0 p 0
_0 0 0 0 1
Random Walks 53
In terms of gambling this can be described as follows:

If two gamblers play a series of games in which the probability of a particular
player winning is p for each game (q = 1 - p is the probability of losing a game).
If the player loses he gives one unit of money to his opponent and if he wins he
receives one unit from his opponent. If this particular player starts with jc units
of money and his opponent with 5 - jc units, what is the probability of the player
losing all his money? The absorbing barriers are ‘0’ and ‘5’. When barrier ‘O’ is
reached the gambler is ruined.
Solution Let p(x) be the probability of the particular player losing all his money
i f he now has jc units. Then we have the difference equation
p(x) = p • p(x + 1) + ^ - - 1) if 1 < i < 5 - 1

P( 1) = P ‘ p(2) = q, p(s - 1) = q • p(s - 2)
Boundary conditions are: p(0) = 1, p(s) = 0
Auxiliary equation is px2 - x + q = 0 or (x - (x - q/p) = 0
Solutions are jc = 1 and q/p.
General solution is p(x) = A + B(q/p)x
From the boundary conditions 1 = /?(0) = A + B
0 = p{s) = A + B(q/p)s
8 = ------ !------ =
1 - (q/py 1 - ( q/ pY
(qlpY ~ [qlpT
if P * <7
Hence P(x) (q/pY - 1 (3.1)
l - f i f p =q
The last expression follows from the fact that if — = > 1, then
P
lim p( jc) = 1 - - (by L’ Hospital’s Rule).
Now suppose the gambler is playing against a person with infinite capital, i.e.
s -» 00 (a random walk with one absorbing barrier) then from (3.1) we get
1 if q > p
lim p(x) =
S — >°o
(q/ p) x if q <p
Let us investigate the effect of changing stakes.
If the amount of money held by two players are doubled, then
( g / p ) 2v- ( g / p ) 2* (<7/p Y + (<?/

P2 W
( q/ p) 2s- l 1
depends only on the ratio (q/p).
Let p(s) be the Gambler’s ultimate winning probability.
( pi a)s —( picj)s x
Then p(s) = ----------------------- (replacing p in (3.1) by q and x by s - x)
(p/qY- 1
Evidently, p(0) + p(s) = 1.
Let Dx be the expected number of steps (time) required for absorption to one
of the two absorbing states (i.e. the expected duration of the game). Then Dx
satisfies the difference equation
Dx = pDx+x + qDx_i -f 1 if 0 < x < s (3.2)
with boundary conditions D0 = 0, Ds = 0.
This follows from the fact that if the first trial results in success the game
continues as if the initial position has been x + 1. Conditional expectation of the
duration assuming success at the first trial is Dx+X+ 1. More formally
E(DX | first trial is a success) = Dx+X + 1.
Then Dx = (Dv+1 + 1) p + (Dx_x + 1)q - pDx+l + qDx_x + p + q with boundary
conditions: D0 = 0 = Ds.
A formal solution is
Dr = (p * q) with Ax = p Ax+1 + q Ax -\
q -p
(where Av is the difference of two solutions of (3.2)).
To get the general solution we proceed as follows.
Boundary conditions give A + B = 0 and A + B(q/p)s = -
q -p
Solving A and B we get, Dx = + A + B( q/ p) x
q -p
x s 1 ~ (qtpY
q-p q —P \ —(q I p) s
x(s - x) if p = q = j
which is the expected duration of the game in the gambler’s ruin problem,
(d) Random Walk with a Reflecting Barrier
P,.M= P’ Pi,,-i = q for 1 > 1, = q-
Here we imagine a barrier placed at -1/2 such that every time the particle
moves to the left from 0, it is reflected at the barrier and returns to ‘O’. The chain
is irreducible if 0 2), y, = py2-
Therefore by iteration we obtain
y/+1 - y, = y\(qip)1, / ^ i
and yf- - vj = y x{(q/p) + (q!p)2 + • • . + ( q lp T X}
Random Walks 55
Hence y, = ------ ■y,, / > 1, so that y\ is bounded if p > q. Thus by

1 - (q / p )
Theorem 2.13 (Foster-type theorem) The states are all transient if p > q and
recurrent if p < q, then the stationary distribution is given by
71i ~ £ niPiJ ~ p7Tj-1 + ^v'+i u > l)
d -<?)
71q — ^1 ~ ^0
<7
Proceeding successively Uj = (p/g)7 7r0(y > 0),
where 7i0 {1 + (p/q) + (p/q)2 + . . . } = 1. (3.3)
If p = g, the series diverges and consequently 7r0 = 0 and ^ = 0 (y > 0) so that
stationary distribution does not exist. Thus, if p = q, the states are null recurrent.
If p < q, by (3.3) 7T0 = 1 - p/q > 0, and is the stationary distribution (the states
are positive recurrent).
(e) Random Walk with two Reflecting Barriers
If in addition to the barrier at -1/2, there is a second reflecting barrier at a + —,
so that paa = p , pa = q, {pu /+1 = p , pu. { = q, i = 1, . . . , a - 1, p w = q,
p0] = p ). We have a finite chain on {0, 1, 2, . . . , a}. In this case stationary
distribution exists and is in fact the geometric distribution of the previous case
truncated at a, i.e.
= i - pig (1 - p/q) (p /q )J
y = 1, . . .
1 - (p /q )a+l 1- ( plq
and this is irreducible (0 < p < 1) and positive recurrent.
3.1.1 Waiting time for a gain

Let {X, } be sequence of i.i.d r.v.s with common distribution
P{Xk = 1) = p , p(Xk = -1) = q, p + q = 1 and Sn = X1 + . . . + X„, S0 = 0.
In gambling terminology Sn, n > 1 is the Peter’s accumulated gain at the end
of the nlh trial if Peter and Paul play for unit stakes. Now consider the event
A „ = { S ,< 0 , S2 < 0, . . . , S n_ !< 0 , Sn = l} .
Thus, the nih trial is the first to render Peter’s accumulated gain positive. The
event An is called first visit to +1 or the index n is the passage time through 1 in
random walk terminology.
Let (pn = P(A„). Define (fo = 0, 0i = p.
If the event holds for n > 1, then S\ = -1 and there exists a smallest integer v
< n such that Sv = 0. The outcome of the first n trials may be described as
follows: (1) At the first trial Peter looses an unit amount. (2) It takes v - 1 further
trials for Peter to reestablish the initial situation. (3) It takes exactly n - v further
trials for Peter to attain a positive net gain. These events depend on non-overlapping
blocks of trials and are therefore mutually independent, (2) and (3) have probabilities
0i-i and 0w_y.
Now the event An occurs iff the events (1) to (3) occcur for some v < n.
Summing over all possible 1 < v < n - 1, we get
n- 1
0n = <7(01 0n-2 + • • • + 0n-2 01 ) = <7 0^! 0 n_v

Multiplying both sides by 5n and summing n = 2, 3, . . . , we get
2 L 0, Sn = q {s S 0„_i s" 1 }
n —2 \n-2
= 0» j " |
(since 0O= 0) (3.4)
Also 2 0„j" = <£(.?) - 00 - 015 = &(s) - ps

n=2
(3.5)
where d>0) is the G.F. of {(f)n}.

Equation (3.3) follows from the fact that
01 <t>n-2 + • • • + 0n-2 01 = (* “ 1) * term of {0„}* {0n}•
and 2 q{(px (j)n_2 + • • • + 0*-2 0i) ^

n-2
= <7* n=2
2 {0, <P„-2 + ■■■+ 01 } Sn 1
Hence from (3.3) and (3.4), <P(s) - ps = qs<&2(s).

1 - -^1-4 pqs 2
Therefore, 0 (s) = is the unique, bounded solution near
2 qs
5 = 0.
( i\
<P(s) = ^ - - ( - l ) X k s02k
( - \ y (4pq)K
2qs k=\
\ kJ
i 00
H ) H (4 m )^£ ^<.2*-!
2
=J- E
2<y *=i
z' n *-i
( - 1)
Therefore 0 2 * -l “ (4pq)k and 02* = 0, k > 1.
2q
v*/
l-jp-gj \p lq if
Now 2 L0„ = <P(1) = -
n =0 2q 1 if p > q
P (Sn remains negative forever)
\q - p
= 1 -2 if p < q
n=\ i q
o \f p > q
Random Walks 57
Therefore, P(Sn will sooner or later become positive) = 1 if p > q and E (number
ifp = q = 1\2
of trials preceding the first positive sum Sn) = & ' (I) = }
—L~
p-q if rp > Hq
3.1.2 Returns to equilibrium

Let Ak be the event of equalization of the accumulated number of successes and
failures occurs at the kth trial if Sk = 0. Let uk = P(Sk = 0). The number of trials
is necessarily even and the probability of a return to the origin at the 2nth trial
is given by
f _ \ \
U2„ = f 2nl Pnq n = ( - D n 2

(4 pq Y
vn, . n J
The G.F. of {U2n} is U(s) = Z U2n s 2n

n= 0
(
' 2 (4 pqs2)" = (1 - 4 2\~J
= n2- 0 (-i)»
The first return to (origin) equilibrium,

B2n = [Sk * 0, for k = 1, . . . , 2/i - 1, S2n = 0].
Let P(Bln) = f2n.
Consider two sub-events with Xx = 1, Xx = -1 and denote their probabilities
by /+„ and f£n, i.e.
f 2+n = P(Bn n (*i = D) and = P(Bn n (X, = -1)).

Now f 2n = q (because first 2 n - 2 partial sums X2 + X3 + . . . + Xn < 0,
but the next one is positive)
As before let = P [52 < 0, S2 < 0, . . . , Sn = 1]
Then the G.F. of { f 2n} is
F~(s) = Z / 2- s 2n = sq S <h.n-\ s 2n -\
n=\ n- 1
i
2^2
1 - ( 1 - 4 pqs1)
= qs <P(s) = qs
2 qs
By symmetry, F*(s) = F~(s) and hence
Z f 2 nS2n = F(s) = F +(s) = F - 1 - (1 - 4
= 1- in general j
Hence probability that a return to equilibrium occurs sooner or later will be

F( D = n=1 % fl n
= 1 - (1 - 4 p q ) 1= 1- - |
2q if p > q
2p if p < q
1 if p = q= 4-
Hence, if p - q= — a return to equilibrium is certain.

Differentiating F(s) = 1 - (1 - s2)1/2, we get
^waiting time for the first return to origin) = F '(1) = ©o if p = q = —
1 - F(s)
if q=
\-s 2
As both U and F are power series in s2 this relation differs only notationally from
1 -P(s)
Q(s) =
1 -5
where p(s) = Z pj s J ,Q(s)= I () q; s J and = pj+x +pj+2 + . . . .
Hence, by uniqueness theorem for G.F.
^ 2 n = fln+ 2 + /2 n + 4 + • • • •
that is when p = the probability that S2n = 0 equals the probability that the
2n sums S\, S2, . . . S2w are different from zero.
3.2 Sequential Analysis

An important problem arising in Wald’s sequential analysis is concerned with
the random variable N = N(a, b), where N = min {n | Sn < - b or Sn > a] is the
first exist time from the interval (-/?, a).
We ignore the trivial case P(Xt = 0) = 1.
Let Xt are i.i.d. r.v.s and Sn = X x + . . . + Xn.
Theorem 3.1 (C. Stein 1947)
N is a proper random variable with finite moments of all order, i.e.
(i) P{N o<o) = l and (ii) E(N)k < <*> for all * = 1, 2, . . .
Proof (i) We shall show, more specifically that there exists A > 0 and
0 < 8 < 1 independent of n and P[N > n] < A 8 n (3.5)
Let C = a + b and r be a positive integer.
Let Si = X, + . . . + Xr , S2 = Xr+1 + Xr+2 + . . . + X2r,. . . ,
Sfc - X(k-\)r+\ + • • •+ %kr

Random Walks 59
We have, P[N > kr] < P[ \ S? \ < C, \ S 2 \ <C ,. . . , | S*_, | <

where p = P[ \ Sk \ < C] is independent of k and 0 0.
If p = 1, then E(S*k )2 = rEX2 + r(r - 1){EXt )2 (since Xf-’s are i.i.d.)
Since E ( X f ) > 0, E(Sl )2 > C2 by choosing r large enough. But p = 1 =>
)2 < C2, which is a contradiction. Therefore p * 1 and P(yV < ©o) = l.
(ii) For t > 0 and positive integer k, nk < etn for large n,
£ az* P[/V= n] < £ etn P[N> n] <A £ ( Se ‘ )n < oo if 8 e l < 1.

n=m n=m n=m
Hence E ( N k) = £ n k
n- 1
m -1 oo
= E n l P[jV = n] + S P[N = n]
n= 1 ti-m
Definition 3.1 N is called a stopping rule if N is a non-negative integer-valued

random variable and the event [N > n] depends on X2, . . . Xn_x only, i.e.
[N = n] is measurable with respect to M X\, . . . , Xn_{) (Xb . . . , need not
be i.i.d. r.v.s).
3.3 Wald’s Equation and Wald’s Identity

Theorem 3.2 ( Wald'sequation) Let {X,} be a sequence of i.i.d. r.v.s with
E(N)<o
o. if E|X, | < oo then E(SN) = (£X,) EN.
If moreover, cr = var (X,) < oo, then E(SN - Np)2 = d2 E(N), where p = £(Xj).
Proof E(Sn )= S E(SN \N = n) P[N = n]

n= 1
=£ I. P[N = n ] E ( X l \ N
n= 1 i=l
=£ £ P[N= n] £(X,
i = 1 n=i
(interchanging the order of summation)
|£ £ £(X, I N = n) P(N =n)\< £ £ X,I \ N = n ) P { N = n)

1= 1 n = i / = 1 « = /'
= E | Xt | E(/V) < oo (Fubini condition is satisfied)

Therefore
E{Sn) = £ P[N > i] E(Xt \ N> i) (since N > i depends on X {, . . . , XM only)
= l p [ N > i ] E ( X i ) = E( Xi) E( N) .
Let Nn = min (A, n). Now let Nn N monotonically, it follows from the
Monotone convergence theorem that
ENn —> E(N) a s/i-^ o o (3.7)
Since {(£„ - njd)2 - n a 2, n > 1} is a martingale (prove it).
We can apply optional sampling theorem to obtain (see Appendix iv)
E(SNn - np)2=
Now let m > n. Since martingales have orthogonal increments we have, by
(3.7) and (3.8),
E(SNm ( SN
n-pNn))2 = - E(SNn
= cr ( E N -m ENn) —» 0 as >
that is SNfj - fuiNn converges in L2 as n —» ©o.
However, since we already know that SNn- juNn —> SN - jiN as n —> it
follows that
E (S n„~ pNn)2 —> E(Sn - fiN)2 as n —> <»,
which together with (3.7) and (3.8), completes the proof.
3.3.1 Wald’s fundamental identity

Let X], X2, ■. . are i.i.d. r.v.s with Sn = Xx + X2 + . . . + Xn and A is a stopping
rule.
Let Fn(x) = P[Sn < x], 'Fi(x) = F(x) = P[Xx < x] and m.g.f. of Xx is given by
0(0) = J eGxdF(x) < oo if 0(a) < «>, where a = Re(0).
We also assume that

0(a) < oo for all a, -/3 < a < a < °o, a, [5 > 0.
Under these conditions, P[ex < 1 - 5] > 0 and P[ex > l + <5]>0, <5>0.
0(0) has a minimum at 0 = 0O * 0, where 0O is the root of the equation
m =1-
Wald’s Sequential Analysis presented the so-called Wald’s identify
E(e0SN / [0(0)]*) = 1 for 0(0) < oo and | 0(0) | > 1.
Actually we shall give the proof of a more general theorem in Random walk
due to Miller and Kemperman (1961).
Define Fn(x) = P[Sn < jc; N > n], N = min {n | Sn £ (-b, a), 0 < a, b < oo} and
oo /*<7
the series F(z, 0) = 2 z" e9xdFn(x).
n=0 J-b
Then
E(e9s" z N)= 1 + [z0(0) - 1] F(z, 0) for (3.9)
all 6»
which is known as Miller and Kemperman's Identity.
Random Walks 61
If 0(g) = [/z we get Wald’s Identify.

fO if * < 0
Proof Let F0(x) =
[1 if x > 0
and Fn(x) = P[Sn < x\ N > n], n > 1
= P[- b < Sk < a, k = 1, . . . n - 1, Sn < x]
Then dFn(x) =f n(x) dx = P[-b < S\y S2, . . . , Sn-\ < a, x < Sn < x + dx]
= P[N = ft, x < Sn < x + dx] (if x < -b or x > a)
is the joint probability that the time N for absorption is n and that the position
reached when absorption occurs between jc and x + dx. Hence if we take Laplace
transform with respect to n and with respect to x over absorbing states we have
E(eds» z N) = Z z " f ee*dFn(x)+ f eedxdFn(x)

n=1 J_oo Ja
£ -1 etixdFn{x)
= Z ,z " J e0 x dFn(x- 1
n=1
where F(z, 6)= 1 z n \eexdFn(x).

n-0 1
This series converges for all Qfor which -/?< cr < a and | z \ < [0 (cr)] , since
e0xdFn(x) <U|" J e°x dFn( x ) = \ z \ n

Now, Fn ( jc) = F(x -y)dFn_i ( jc) and 5„ = +Xn
J e0xdFn (cc) = J e0xdF(x) | J*

-b
= (Did) |
\-:b
Therefore E(e0s" z N)= £ z n<D(9)

n=] J-b
f -F(z, 9) + 1
"=I J-b
= z0 ( 0 ) £ z"-' f e°>dFn_x
= 1 + [ Z 0 ( 0 ) - 1] F(z, 0) (3.10)
To get Wald’s identity, we let z —> [0(0)] 1 in equation (3.9)
E(eesN[<p(6)TN) =1 + lim - 1] 6)
z->[0(e)rl
= 1.
The Case of Single Barrier

If either a or b —> then we have a single barrier. If a = °o, b < ©o, N = A\b) =
min {n | Sn < - b). In this case the previous Theorem 3.1 (Stein) does not apply
and we have P[N<°°] < 1 but Wald’s Identity still holds as the Identity (3.9), can
be regarded as analytic continuation of the other beyond the common range of
validity.
3.4 Fluctuation Theory

In this section X1? X2, . . . , Xn, . . . are i.i.d. r.v.s.
Theorem 3.3 If E \ Xt \ < then
P[N(b) < oo] = 1 if E X, < 0
< 1 if E Xi > 0
For Proof see Chung and Fuchs (1951) and Chung and Ornstein (1962), Memoirs
of American Math. Society.
Definition 3.2 If S is uncountable, and Sn = Xx + . . . + Xn are Markov, X/s
being independent, then * is called a possible value of the state space S of the
Markov chain if there exits an n such that
P[ | Sn - x | < 5] > 0 for all S > 0. A state x is called recurrent if
P[ | Sn - X | < S i.o.] = 1 i.e. Sn £(x - 5, x + S) i.o. with probability one.
We shall conclude this section by stating two very important and famous
theorems whose proofs are beyond the scope of this book.
Theorem 3.4 (Chung and Fuchs)
Either every state is recurrent or no state is recurrent, (ref. Spitzer-Random Walk
(1962)).
Theorem 3.5 (Chung and Ornstein)
If E | Xi | < oo, then recurrent values exist iff E{Xt) = 0.

Exercise 3.1 In a simple random walk with two absorbing barriers at 0 and a let the
position Xn at the nth step be given by Xn = Xn_{ + Zn where Z„’s are i.i.d. r.vs. taking
values 1 and -1 with corresponding probabilities p and q = 1 - p. Let nk{ri).bz the
probability of absorption at 0 of the random walk in n-steps starting from position k.
Show that the generating function Gk (s) = £ n k (n)sn , | 5 | > 1 is given by
n=0
(<?/?)*
(s) - A" (s)
where AjO) and A2(s) are given by
Random Walks 63
l + ( l - 4 pqs2)m , , N l - ( l - 4
Ai (s)
-------- 2 sJ ' {S) = 2 --------
Also show that
7ik(n) = 2n p{n k)l2 q{n+k)l2 J cos" 1 (nx) sin(^jc) sin(knx)dx.

Jo
What will be the value of nk(n) in case of simple absorbing barrier at 0 when playing
against an infinitely rich opponent?
Exercise 3.2 In a random walk with two absorbing barriers at -n and a, let the position
Xn at the nth step be given by Xn = Xn_{ + Z„, where Z„’s are i.i.d. r.v.s taking values 1,.
- 1, 0 with corresponding probabilities p, q, 1 - p - q.
If = n - b < x„ X2........ x„_, < a,X„=
Show that the generating function of j f -"1) is given by
F , = [A ,(5 ) ]^ -[A 2(.s)P/j

* [A1(i)]B+6- [ A 2(j)],'+*
where A!( 5 ) and A2(j) are the roots of the equation
psX2 - A[1 - 5(1 - pq)] + qs = 0.
If the random walk starts from the origin, what will be the expression of the generating
function.
Exercise 3.3 Show that in a simple Random walk with two reflecting barriers at 0 and
a, the stationary distribution Uj is given by
1 -ptq O'= 0 . 1 , . . . . a),
7 lj = (p/qV
i-(p /r‘
where n,= nlim
—>oop]"1;
J and J has the usual interpretation that it represents the probability
that the particle occupies the state j at time n having started in the state /; what will be the
value of 7tj when the Random walk is homogeneous (i.e. p = q)2
Exercise 3.4 (Wald's Lemma) Let X be a r.v. such that (a) E(X) exists and * 0, (b) the
m.g.f. (f)(6) = E(eex) < °° for all real 0and (c) there exists a 8> 0 such that P(ex < 1 - 8)
> 0 and P(ex > 1 + 8) > 0. Show that the equation (j)(6) = 1 has a unique root $o * 0
(i.e. (p(6) has a unique minimum at 6= 60).
Exercise 3.5 (Continuation of Exercise 3.4). LetZb . . . , Xn, ... are i.i.d. r.v.s satisfying
the conditions of Exercise 3.4. Define SN = X{ + . . . + XN.
Let E0 = E(ee°s» | SN < - b), Ex = | SN > a)
E — i
Prove that P(SN < -b) = —A—77-, where N - min {n: Sn £(-b, a), 0 < a, b < °°}.
t\ - t 0
Exercise 3.6 (Continuation of Exercises 3.4 and 3.5)

0(0) exists for all real values of 6 at any values 6 in the domain | 0(0) | > 1. Give an
alternative proof (by differentiation and putting 0 = 0) of
E(N) = E(SN)/E(Xn) if EXn * 0
= E(S2N)/E(X2) if EXn =0.
Exercise 3.7 Let Xu X2, . . . be i.i.d. rvs taking values in the integres Z = {. . -2, -1,
n
0, 1,2,...} and having finite mean. Show that the M.C. {£„, n > 1} given by Sn = £ Xf
is transient if E(X{) * 0.
Exercise 3.8 Consider a symmetric random walk Sn with S0 = 0. Let T = min {n> 1 :
Sn = 0} be the time of first return of the walk to its starting point. Show that
P ( r = 2" > = 2 ^ 7 ( 2,,")r ! '

and deduce that E(Ta) < iff a < V2.
[Hint. Use Stirling’s formula nl - nn+lf2e~nV2tt-]
4
Renewal Theory
4.1 Introduction
Let Xn,n = 1, 2,. . be the nonnegative i.i.d r.v.s with Sn = Xx + .. . + Xn, n > 1,
Sq = 0. F is the common d.f. of X and assume P(Xn = 0) < 1. Define N(t) =
sup {n | Sn < t}. The process {N(t), t > 0} is called the Renewal Process.
To fix our ideas X, can be taken to represent the life time of the machines
being replaced. The first machine is installed at time t = 0 and is replaced
instantaneously at time t = X x. The replaced machine is again replaced at time
t = X x + X2, and so on. If we write Sn = X x + . . . + Xn, the partial sum Sn can be
interpreted to be the time at which the nth replacement is made. N(t) is the
largest value of n for which Sn < t. In other words N(t) is the number of renewals
that would have occurred at time t. The Renewal Theory, in a sense, is a special
case of a Random Walk with absorbing barrier. We are sampling the Xt until Sn
shoots the barrier at time t and N{t) + 1 is the sample size when we stop. Hence
the Renewal Theory is also linked with Sequential Analysis in statistics.
{N(t), t e (0, <»)} is called the Renewal Counting Process. We can also write
N(t) = max{« | Sn < r}.
We want to find P[N(t) = n] given F. To compute this we proceed as follows:
F(t - u)dF(u)
o
= F* F{t)= F® (0, • • •
P[Sn <t] = F (n)( 0 = f0 Fin-l)(t - u)1
Define
Now P[N(t) = n] = P[Sl < t , S 2 < t , . . . , S n <t, Sn+l > t]

= P[Sn < t, Sn+1 > t] (by nonnegativeness of Xj)
- P[Sn ^ t, Sn + Xn+] > t]

= P[t - Xn+l <Sn <t]
P[t - Xn+\ < Sn < 11u < Xn+] (/) - F (n)(t - m)JJF( m) (by (4.1))
= f F in) (t) dF(u) -f F(n) (r -

Jo Jo
= F (n)(0 - I* F <n) - u)dF(u)

Jo
Deflnition 4.1 H(t) = E(N(t))is called the Renewal Funct
4.2 Renewal Equation

Theorem 4.1
(a) P[N(t) = n]= F (n\t)- F {n+X\
(b) H(t) =£ F (">(0

n=l
(c) //(/) = F(/) + I //(/ - «)dF(w) , the so-called integral equation of Renewal
Jo
Theory (Renewal equation).
(d) {N(t)y te [0, °o)} is completely determined by H(t).
Proof (b) H(t) = S nP[N(t) = n]

n- 0
= P[N(t) = 1] + 2P[N(t)
= F(1\t)- Fil)(t) + 2Fi2\t) - +...
=Fil\t) + F 2,(t) + Fi3\ + .
= Z F (n)(r) provided the series is convergent.

(convergence of the series will be proved in Exercise 4.5)
(c) H{t) = £ FXn){t) = F n) (t) + £ F (n)(r)

n- 1 n-2
Renewal Theory 67
= F(t) + £ F (,2+1)(0 = F (0 + £ f F (n)(r - ii)dF(M)

«=1 "=1Jn
= F(f) + X F (n) (f - u)dF(u) (by Fubini Theorem)

n- 1
Jo
= f(0 + f H(t - u)dF{u)

Jo
(d) H(t) = F(t) + //(r - u)dF(u) = F(t) + H* F(u) where * is the

o
convolution operator.
Taking Laplace transform on both sides
&(s) = e~stdH(t) = ^ ( s ) + &(s) jr ( s )

Jo
or ^(s) = where ^ ( s ) = f e~stdF(t)

1 + ar(s) J0
(5)
and £F(s) = --------- - -■ (Re(i’) > 0). This shows that H(t) and F(x) can be
1- ^(s)
determined uniquely one from the other, since Laplace transform determines a
non-decreasing (specially a d.f.) function uniquely. Hence N(t) is completely
determined by H(t).
Now N(t) = max {n I Sn < t} and EN(t) = £ F{n) (t) if EN(t) <
n—1
The next theorem will prove that all moments of N(t) is finite.
Theorem 4.2 N(t) is a proper random variable with finite moments of all orders
i.e. P(N(t) <
00) = l and for E rf (t) < °°fo
Proof We assume P(Xk = 0) < 1. Since Xk are i.i.d. and Xk > 0 a.s., there exists
a > 0 such that P[Xk > a] > 0. Let us take without loss of generality a = 1. Then
P[Xk > 1] = p > 0 .
Define a modified Renewal process with
_ JO ifX <1
*
Let s n = X, + . . . + Xn and M(t ) = max {n \ Sn < t ) .

Then Xn is a Binomial Process. Since Sn < Sn, N(t) < N(t)-
A[{t) is a proper random variable, since N(t) is. Also
E(ee° Nit)) < E ( e 6oN(t))<oo for some 0 < 00 < l o g y - ^ .
Since nk < ed°n for large n and all k = 1, 2, 3, . . . , ENk(t) < °° for all k - 1,
For Binomial Process renewal occurs at t = 0, 1, 2, 3, . . .

Let Nj be the number of renewals in (j - 1,7], 7 = 1, 2, 3, . . . .

Then /V(/) = + N2 . . . + NT, where T = [/].
Now P[Nj - n] = pqn~l (n > 1) and
EN, = Var N. = qlp2(q = 1

P
E(eeN’ )= £ = £ (<7^ ) " - ' = (1 -

n- 1 «=1
fR e (0 )< lo g i j .
Since N x . . . , NT are i.i.d, 7V(7 ) is a 7-fold convolution of and hence
Pe , the m.g.f. of a negative binomial distribution whose

E(e0Nin) =
1 - qee
T+k- 1
distribution is given by p Tq k , k = 0, 1, 2, . . .
*
T
H(t) = E N (t)= =-
7=1 J P
and Var (iV(r)) = £ Var(V.) = £f-.

7=1
We have derived before the integral equation and showed that the H (t) = 2 F {n)(t)
n=1
is a solution of (4.1). Now we have to show that the solution is unique.
Let V(t) be a solution to (4.1), which is bounded for all finite t. Then,
V(t) = F(t)+ f
Jo
V(t -u)dF(u) for all t
= F il)(t) + f
Jo
dF(u)[F(t u) + V(t - u-
/) + f dF (u ) f
Jo Ju
= F w (t) + F a \t) + f {t-0)d

V f JoJo
(interchanging the order of integration and making the change of limits of
integration)
u< 9 < t 0 <u<G
Q<u<t 0 < G< t
Renewal Theory 69
= F ° \ t ) + F (2\t) + f0
and proceeding in similar manner, we find that
Now since the series X F (n)(0 converges,

n= 1
F (n)(0 —> 0 as n—» °° and therefore.
I* V
(t- G)dF(n)( G<) supV(u)F(n) 0 as -> °o an
Jo
V(t) = X F{n)(t) = H(t).
n—
1
This proves the uniqueness of H(t) = X F ("}(r) as a solution of (4.1). The
n= 1
Renewal (Integral) equation can be generalized as follows:
V(f) = G (f)+ f V(t - x)dF(x) , t > 0

Jo
(4.3)
where G and F are known and V is unknown. Equation (4.3) is called the Renewal
Type equation. An unique solution of V(t) exists in terms of G and F as can be
seen from the following.
Theorem 4.3 If V(t) = G(t) + f V(t - x)dF(x), t > 0

o
= Gif) + (V* F){t) (4.4)
(4.5)
o
Proof Taking Laplace-Stieltjes Transform of (4.4) we get
V*(s) = G*(s) + V*(s) F*(s)
or
= Y - F * t y where v*(i) =
and
= G*(s) + G*(s) H*(s),

since H*(s) = £ [F * (j)r

n=\
= F*(s)I [F*(j)]"”'
n=1
= F*W/[1 -F*(s)] if F*(s) < 1.
Inverting the Laplace transform, we get
V(/) = G (f)+ f G(t - x) dH( x) .

Jo
Example Let Xb X2, . . . be i.i.d. r.v.s with
P(Xk <t) = F(t) = 1 - e"At, A > 0, t > 0.
Then F(n) (?) = P[S„ < ? ] = ! - e - x' f 2

k=o k\
(can be proved by induction on n. Prove first for n = 2).
e - A,(A?)"
Hence F (n\t) - F<n+'\t) = )=
n!
i.e. the Renewal process generated by exponential random variables is a Poisson
process, or in other words, Poisson process is a Renewal process with Exponential
Interarrival times.
In this case, H(t) = E(N(t)) = Xt - / ■ for all t > 0, i.e.
E(X)
H(t)
= A, where A 1 = E(X).
4.3 Renewal Theorems

1. Elementary Renewal Theorem (Feller 1941)
H(t ) l/ju if 0 0 and X, is a continuous random
/z
variable.
lim P [renewal at nd] — where Xt is a lattice type discrete r.v. and d is the
f— fJL
period of the lattice.
Definition 4.2 A random variable X is said to have lattice distribution if

P[X = c + nd] > 0 where c and d (> 0) are real constants and n = ±1, ±2, . . . .
3. Key Renewal Theorem (W.L. Smith, 1953)
Renewal Theory 71
If Q(t) > 0 and non increasing and fJo Q(t)dt < °° then
lim r Q(t - x)dH(x) - i f Q(t)dt whenever X is not arithmatic.

Jo f* J o
A weaker version of the Elementary Renewal Theorem is the following due
to J.L. Doob (1948) and is known as Doob’s Renewal Theorem
v N(t) 1
lim ------= — a.s.
Proof Assume that X2, . . . and i.i.d. with 0 < ju < °o.
and for all 0 < £ < ju, (/u - e)n < Sn < (ju + e)n for all large n.
nt < SN(tn)+} <(ju+ £) (N(tn) + 1). (4.6)
Now N(t) = max{/:: Sk < t] => SN{t) < t, SN(t)+\ > 1. Replace t by nt to get
nt > SN(tn) > (ju - e) N(tn) for all large n. (4.7)
From (4.6) and (4.7)
t N(tn) + 1
- —-— + — for t > 0 and large n. Hence
jl + £ n ju - £ n &
N(tn)
—> — a.s. as h and t > 0,
P
N(tn)
i.e. -j as t —> oo. Putting nt = r* we get the result.
tn
Proof of Elementary Renewal Theorem

Assume 0 < ju < ©o.
Hence, there exists A > 0 such that P(Xn > A) > 0 for all n = 1, 2, . . .
f A if jc„ > A
Define = (truncated r.v.)
[0 otherwise
Then X'n are i.i.d. r.v.s and X'n < Xn a.s. for all n = 1, 2, . . . .
Define S'„ = t X'k and yV'(f) = max {£: 5^ < r}.
k=\
Then S'n < Sn and N'(t) > N(t).
X'
Let P[Xn > A] = p > 0. Then is a Bernoulli random variable with probability
of success p. If t is an integer then N'(t + 1) - N'(t) is the number of times, the
partial sums take on the value t + 1.
Also N' ( t ) = %
j=o
.( N (t) N \t)Y
<E = 0(1) as t—» oo, because
E(N'(t)) = Tip, where T = [f] and

Var (N'(t)) = Tq/p2.
Therefore £(yv'(r))2 = l i + i £ = i i + i £ = l l l l i and
AT(/)
— + —— = 0(1) as t —> oo, 0 < < 1. Thisimplies
P" P “'
N (0 < c (constant) for all large r.
Therefore |fW(0
—-—, t > 11 is (uniformly integrable) u.i. by the following result.
Exercise If E X% < c for all large n, show that {X„} is u.i.
Proof f
J \ X n \>k
f
\ X n\<
]-dP X l d P Z T E ( X ^ ) < f
K J KK
i
0 as k —» if n > n0(4.8)
But max f | Xn | dP < icf k -> °° (4.9)

\<n<no >*
Hence, sup X„ U P 0 as A: -> °°
i.e. [Xn] is u.i.
By Doob’s Renewal theorem —^ —>— a.s. and , t > 1 m s u.i. Hence

t p
N(t) L\ l H(t) 1
—— ---- » — =* —— -> — as t oo.
t JLl t JLl
Proof of Blackweir s Theorem which is a special case of Key Renewal Theorem.
fl//i if 0 < r x > t - h

Then Q(t - x) =
0, otherwise
Jt, , H(t) - H(t - h) 1

Urn f
Jo
Q(t - x)dH(x) = lim
ii: dh(x) = lim ----------:--------- 1 = —
h p
since f Q(t)dt = f 1I h d t - 1
Jo Jo
Renewal Theory 73
r //(r+ h) - H{t) 1
i.e. lim --------- ;-----------= — => Blackwell s Theorem.
h ju
Proof of Key Renewal Theorem
Let I'
Jo
Q ( t - x ) d H ( x ) = f + / 2,
f
fit/ 2
where f = I Q(t - x)dH(x) and /2 = I Q( t - x ) d H( x ) .
Jo JJ //t/2
2
Since Q(t) is non-increasing and non-negative, we have
fit/2
0 < /, < Q(t/2) dH(x) = Q(t!2) H( t f 2) (since //(0) = 0)
Jo
= 1 j G 0 / 2 ) J| (4.10)
{ t2
! J'
By Elementary Renewal Theorem —— ------ > 1/jU as / —> «>.

[Similarly for the continuous case using intergrals, the result follows.]
f ° ° oo *nh+h
pnh-
Now let Q Q(t)dt = Z Q(t)dt. (4.11)
Jo Jnh
Since Q(nh + h) < Q(t) < Q(nh) for (4.11) gives.
h Z Q(nh) = h Z Q(nh+ h< /i Z ) < °°

n=1 m=0 n~0
and 0<Q- h Z 2(«/i) < hQ{0) < £ if we choose 0 < h < sJQ(0) (4.12)
M=1
If [r/2/i] = /V0 then
h= f Q(t-x)dH(x) = f Q(s)[-dH(t - 5)], putting x = t - s

Jt/2 Jo
N 0-l finh+h
=z n=0
Q(s)[-dH(t - s)]
Jnh
/Vo-l
Hence Z Q(nh + /z[f/(r - nh) - H(t - nh - /i)] (since //(/) = Z
n=0 n
F M (t) is
non-decreasing)
N0-l
</2< nZ
-0
Q(nh (4.13)
Now' we can choose t so large that (by Blackwell’s Theorem)
H(t - nh) - H{t - nh - h) l

(4.14)
1, 77 < £
and h 1 Q(nh) < e (4.15)

rt = /Vo + l
(Note that t large => N0 is large).

Using (4.12) to (4.15), we have
~ - s j (Q - 2e) < 12 < |-jj + (Q - £) for arbitrarily small £ > 0.
Hence, I2 —> ^ as t —> «>.
Therefore f Q(t - x)dH(x) — f Q(t)dt as t -> » ,

Jo ^ Jo
4.4 Central Limit Theorem for Renewal Theory

4.4.1 Renewal density and integral equation
If F(t) is absolutely continuous with F \t) = f i t ), then H'(t) = /i(0 exists and is
given by
h(t) = S f {n)(t) where f {n\t) = [Fin\t)]'

n- 1
(since the terms of H(t) are non-negative and bounded we can carry out term-by-
term differentiation and obtain the result). h(t) is called the Renewal density.
If we “Confuse” expectation with probability then h{t)dt can be interpreted as
the probability that a renewal occurs during (r, t + dt). Now if a renewal occurs
during (r, t + dt), then the dying individual is either the initial one itself or the one
which entered the population at some prior time, say t - t(0 < t< t). This leads to
hit) =f i t ) + f hit - r ) f i r ) d z
Jo
which again is a Renewal type integral equation. Such an argument is called the
Renewal argument. We have seen _> 1 a s i f n > 0 (Doob’s Renewal
t fi''
Theorem) and
r Hit) r EiNit)) 1
lim ------ = lim -------------= — as t—> «>
t t ii
(from Elementary Renewal Theorem).
£(/V(r)) = 0 ^ j
We can prove that as t —> ©o. 2
V(/V(r)) = C 2 _ where a 2 = VarX,.
I*
(see Ex. 4.12)
N(t) - t/fi i rx i 2
Central limit Theorem 4.4 lim P - j L r f , 1 “ dU-
*J(ta2//u3) V2 n
Renewal Theory 75
Proof P[N(t) < n\ = P[Sn > f], since

P[N(t) <n] = P[N(t) = n - 1] + P[N(t) = n - 2] + . . . + P[W(r) = 0]
= F (n- l\t) - F (n\t) + F (n~2)(r) - + F m (t) -
=F (0\ t ) - F(
n)(/)
= 1 - F (n\t)
= 1- P[Sn <t ]= P[Sn >
By CLT of i.i.d. random variables,
Sn - nlju
lim P
/ —>o°
e - L 2du
Let n —> oo and r —> oo in such a way that

t - nju
= - x (x fixed) (4.16)
Solving (4.16) for n as a function of t and jc, we get
n = t/fx ± jc(<720 1/2 + 0(0 (4.17)
because we note that t > n/x for x < 0 and t < nfi for x > 0 in the two solutions
of the above equation.
From CLT of i.i.d r.v.s,
t - nix
lim P[Sn > t] = lim p\ H— -x
«fno
-~u2
du
n - t/jx _
Now equation (4.17) shows
V(«T2/ / / 3) " ' • Theref° re’
lim P <X L2
^(rcr2/ ^ 3)
4.5 Delayed and Equilibrium Renewal Processes

There are two types of generalizations of ordinary Renewal Process. When the
first interarrival time Xx has a distribution function G different from the common
d.f.F of X2j X3, . . . , we call such a Renewal process a Modified or Delayed
Renewal Process. A Modified Renewal Process in which
J o F
is called the Equilibrium (Stationary) Renewal Process.
Let HD(t) = 2 G* F (" 1}(0 be the Renewal function of delayed Renewal

n=\
process. Earlier we have seen for ordinary Renewal process
H(t) 1
------------- >— as » oo, wherex dF(x).
t— u
t n
Similarly, —>i- as t —> «> (by Strong Law).
Hence, the question is whether there is any Delayed Renewal Process for
which HD(t) = tip? Also we know that for Poisson process, where the underlying
variables are exponential, H(t) = tip.
Let Fe (x) = 1 ~ ^ (° (4.18)
The following proposition is an answer in affirmative to our last question.

Proposition 4.1 There exists a modified Renewal process where the initial
distribution G = Fe exists such that He{t) = tip,.
Proof If He(t = tip, then Laplace transform
" • ''''T •“ (* )* ■ 5 ? (4' 19)
A|S0 ( 4 '2 0 )
where f e*(s) is the Laplace Transform off e and/* (5) is the Laplace Transform
of/w here F'(x) =f(x).
Equation (4.20) follows from the following argument: Integrating by parts
Laplace transform of d.f. F(jc) is (F(jc))* =f*(s)Js. By multiplicative property of
Laplace transform
I {F<n-'>(0}* = Us I [/*(5)]"-1 = if/* (* )< 1

n=l n=1 VA- J \S)J
and F is absolutely continuous with density/) . Now He (t ) = I F / /r(«-D ( j) .

/2=1
Taking Laplace Transform on both sides
wt* ( o =
Equating <4.19) and (4.20)
=»/«*(«) = [! -f*(s)V(/ is). (4.21)

Inverting the Laplace Transform, we get
1 - Fit) i.e. Fe (X) = dt.

feU) =
F F J
Renewal Theory 77
As a particular case, a Poisson process is an Equilibrium Renewal process.

Stationary or Equilibrium Renewal Process has a simple physical interpretation.
Suppose that the Renewal process started at time t = - «>, i.e. at remote past but the
observation of the process begins at t = 0. Then Xj has distribution function Fe
given by equation (4.18) and He(t) = tlfJL. (It can be shown that H(t) > - 1 for
t > 0. Refer Statistical Theory of Reliability and Life Testing by Barlow and
Proschan. Also AMS, 1964 Paper on Replacement Policy by Barlow and Proschan).
Let Ne(t) be the number of Renewal in an Equilibrium Renewal process.
Result 4.1 We have for all t > 0 and n = 0, 1, 2, . . .
P[Ne(t) > n] > P[N(t) > n] (4.22)
i,T rn asH * <423>

Proof Assume (4.23) holds.
Since — f (1 - F{x))dx = — f {1-F(x)}d.x - f {1-F(x)}d.x

Fit F [J0 Jo
= 1 - Fe(t)
note that for a non-negative r.v., the mean ju = (1 - F(x))dx.
Hence, by (4.23), 1 - Fe(t) < 1 - F(t) for t > 0

or Fe(t) > F(t) for t > 0.
If Sn and S'n denote the nth renewal epochs of N(t) and Ne(t), respectively,
then for n > 1
P[Ne( t ) >n] = P[S'n <t ] = f F (n~l)(t - x)dFe(x)

Jo
> f F {n~l) (t - x)dF(x) = P[Sn <t] = P[N(t ) > n] for n > 1.

Jo
(It also holds for n = 0).
Further, P[Ne(t) > 0] = P[N(t) > 0] = 1 for all t > 0. Thus (4.22) holds if (4.23)
holds.
To prove the converse, assume that
P[Ne(t) > n] = P[N(t) > n] holds for n = 0, 1, . . .
Since equality holds for n = 0, we must have
P[Ne(t) = 0] < P[N(t) = 0]
(since [N(t) > 0] = [N(t) = 0] u [N(t) > 1]) or 1 - Fe(f) < 1 - F(f)
^ P[N(t) = 0] = F (0)(r) - F (1)(r) = 1 - F(t).
Corollary 4.1 H(t) < tin iff F/t) >
Proof H(t) = E(N(t)) = 2 P[N(t) 2 P[Ne(t) > n]

n n
= £(AUO) = ^ i f f
Definition 4.3 A unit (or equivalently its life-time distribution F) is said to be

New Better (Worse) than used NBU (NWU) if
1 - F ( x + y)< (>) [1 - F(x)][ 1 - F(y)]
i.e. F- - —<(>) F(
F(x)
In other words, a unit is NBU (NWU) if the conditional survival probability
of a unit at age x is less (greater) than the corresponding survival probability of
a new unit.
Definition 4.4 A unit is said to be New Better (Worse) than used in Expectation
NBUE (NWUE) iff
1 ~ F(x)
dx < (>) p.
1 - F(t)
In other words, a unit is NBUE (NWUE) if the conditional expectation of the
residual life of a unit at age t is less (greater) than the expectation of the life-time
of a new unit.
The above result may be restated as
P[Ne(t) >n]> P[N(t) > n], t > 0, n = 0, 1, 2, . . .
as well as H(t) < tip iff distribution F involving the Renewal process is NBUE.
Note If F belongs to NWUE class then H(t) > j j - 1 for t > 0.
4.6 Residual and Excess Life-Times

To a given t > 0, there corresponds uniquely a Renewal process N(t) such that
SN(t) < t < S)v(,)+1, i.e. t falls in the interval of length Xô+i-
(i) The residual life-time of the individual alive at age t is given by the time
Y(t) from t to the next Renewal epoch, i.e. Yt = SN(,)+i - 1. It is also called Forward
Recurrence time at t, or Excess life time at t.
(ii) The Spent life-time of the individual alive at age t is given by the time
upto t since the last renewal epoch, i.e. Z(t) = t - SN{t). It is also called Backward
Recurrence time at r, or current life time.
(iii) The length of the life-time containing t is given by
Y(t) + Z(t) - SN^ +\ - SN(t) - XN(t)+\.
Theorem 4.5 P[Y(t) < X] = F(t + x) - f [\ - F(t + x - y)\dH(y)

Jo
(4.24)
If in addition, F is not lattice, then
Renewal Theory 79
lim P < JC] = - f [1 - F( y ) ] d y if/i< ° ° (4.25)

l->°o M Jo
= 0 if // = °o
Proof It is clear that Y(t) > x iff no renewals occur in [?, t + x].
Let P(t)= P[Y(t) > jc]. By conditioning on X\. we get
P{t)
■r P [ Y ( t ) > x \ X l = Y]dF(y).
Now following three situations may arise:

(i) y > t + jc, in which case it is certain that no renewal occurs in [/, t + x] so
that P[Y(t) > x | X] = y] = P[No renewal in [t, t + x]] = 1.
(ii) r < y < / + jcin which case one renewal (the one corresponding to X } = v)
occurs in [r, t + x] so that
P[Y(t) > x \ X l =y] = P [No renewal in [*, t + jc] | Xx = y] = 0.
(iii) 0 < y < t, in which case the first renewal occurs in [y, y + dy) and then the
process restarts itself and
P[Y(t) > x| X, = y] = -P
(t y)>
[Y
considering the above three cases, we get
Pit) ■ f P(t - y)dF(y) +

Jo f
= 1 - F(t + jc)
•'O
+
f
P ( t - y ) d F ( y ),
which is a Renewal type equation and the solution is given by
P(t) = 1 - F{t + jc) + f [1 - F ( t + x - y ) ] d H ( y )

Jo
which is equivalent to (4.24).
Therefore
•'o f
P ( t ) = l - F ( t + x)+ \ [1 - F(t + x - y ) ] d H ( y ) (4.26)
Now suppose that F is not lattice. Then applying the key Renewal theorem
with Q(t) = 1 - F(t + jc), we get as t —>
f [1 - F ( t + x - y ) ] d H { y ) - > ± f[1 - F ( t + x)]dt

Jo r1 J o
Hence from (4.26) and (4.27), we have

-c [1 - F(y)]dy (4.27)
lim P[Y(t) > jc] = t—

t—)oo
lim
>oo
P(t) [1 - F(y)]dy
or lim P[Y(t)
t—>oo
< X ]= 1 - i f [1 [1
- F ( y ) ] d y = r±
PJr P'JO
Theorem 4.6 P[Z(t) < x] =

F(t) r
Jo
1 if x > t
[1 - F(t - y)]dH(y) if x < t
lim P[Z(t) < x] = —

fijx
f
[1 - F(y)]dy if F is not lattice.
Proof The distribution of Z(t) can be obtained immediately by noting that

Z(f) > x iff no renewals occur in [t - x, t\ i.e. iff Y(t - x)> x.
By previous theorem,
[1 - F( t - y ) ] d H( y ) if x < t
P[Z(t )< x] =
1 if x > t
(since Z{t) = t - SN(t) < x Sm > t - x < 0 if x > t, so Z(t) < x is a sure event)
and if F is not lattice then,
lim P[Z(t) < x] = f [1 -F(y)]dy/fi.

^ Jo
Note It is to be noted that the result
\im P[Z(t) < x] = f

Jo
[1 -F(y)]dy/JLL
also holds for the modified Renewal process regardless of initial distribution G.
4.7 Renewal Reward Process

Consider an ordinary renewal process generated by the sequence [Xrt, n > 1} of
interarrival times. Suppose that each time a renewal occurs, a reward is received
or a cost is incurred (e.g. for replacement of a failed component). Denote the
reward or cost associated with nth renewal by Yn, n > 1. Now Yn will usually
depend on Xn, the duration of the interarrival time between (n - l)th and nth
renewal. In such a case we assume that the sequence of pairs [Xn, Yn, n > 1} are
i.i.d. The sequence of pairs of i.i.d. random variables {Xn, Yn} is said to generate
a Renewal Reward process. Now Y(t) = I gives the total reward carried (or
n- 1
cost incurred) by time t and {Y(t), t > 0} is called the renewal reward process
generated by {Xn, Yn}.
Theorem 4.7 Suppose E(X) < «>, E | Y | < °°, then
(a) Y(t)it -> E(Y)/E(X) as t -> oo a.s.
(b) E(Y(t)/t) -> E(Y)/E(X) a s t-^ o o .
Renewal Theory 81
Proof
N( t )
no_ nom o _ n-iYh jv(o

W / W(/) ' / N(?) ' t
By Kolmogorov’s SILN for i.i.d. r.v.s Sn/n —» E(X) a.s. as n —>
Let A/(7) = inf {n: > t} = N(r) + 1.
Since (t)/ or = (NO) + 1)//

N 1/£(X) > 0 a.s. as -> °°
A/(r) or jV(0 -> oo A.y / oo if 0 < £(X) < oo.
n
Let S* = l>Yi, A = {ft): S*„((0)ln—» £(L)}, fl = (ft): A/(f, f
i=l
Then C c A£/£. But P(A) = 0 and P(B) = 0 and hence P(C) = 0.

N(0
This is same as S*M(t)IM(t) -—> E(T) as t —>°o which implies 2 Yn/N(t)
-> as t —» oo and also Af(Q 1 as / —> oo a.s.

£ (* )
u Y(t) E(Y)
Hence — -> ; ;■a s M ° o .
t E(X)
[Note that N(t) is not a stopping time since [N(t) = n] is not independent of
Xn+\y Xn+2, • • • but N(t) + 1 is a stopping time since [N(t)+] =.n] is independent
of Xn+\> Xn+2, . . .]•
(b) We first note that N(t) + 1 is a stopping time for Yx, Y2, . . . since the
independence of N(t) + 1 and {X„+1 • • •} implies the independence of
N(t) + 1 and {Yn+l, Yn+2, . . .}. By Wald’s equation,
(N (t) > ~N(t)+\
______ i
--------- 1
E(Y(t)) = E 2, K„ = E
1
l " =1 /
=« + 1 )^ 0 0 - £ ( W i )
= ( / / ( / ) + 1) E(Y) ~ E(YN:t)+l).
Hence
EY(t) = j/(Q + 1 _ £ (* W i)
t t 1 ' /
Thus, it is enough to prove E(YN(t)+llt) —> 0 as t ©o.
Let 5 (0 = E O W i ) = f ^ ( ^ ( , ) +. I X, = x)dF(x)
Jo
Now i i f
[£(^1 | Xj = x) if x > t.
,
where h(t) = J E(YX | X\ = x)dF(x).

Also note that, since
E \ Y } |= f
Jo
E[ | Y\ | |X, =x] dF( x ) <oo,
/i(r) —> 0 as t —» and h(t) < E | Yx \ < °° for all t > 0.

The solution of the Renewal type of equation (4.28) yields
g(t) = h(t) + h(t - x)dH(x) where H(x) = Z F{n)(x).

f'
Jo
n- 1
Now | h(t) | < e if t > T.
Hence
t
wi+r
? Jo
h(t - x)
dH(x) + r
J l- T
i
1
h(i- dH(x)
<L + e m i ^ +EW m z J i ^ ( ,• //( 0 ) =o)
H(t)
sin ce ■—^ — as t —^ °°
t P
—> e !\x as t -> oo (ju > 0)
H(t-T)
—> ~ as t —> 00
t
Since £ > 0 is arbitrarily small, —>0 as t ©o-

Denote EYIEX = and ET(/) = c(0, the average expected reward (or cost) in
[0, t]. Then from Theorem 4.7 (b) we get for large r, c{t) = t%.
A more refined asymptotic form of c(t) has been given by Christer (1978)
c(t) = %t + 77, where £ and 77 are functions of parameters and decision variables
of renewal process.
4.8 Replacement Policies Connected with Renewal Theory

4.8.1 Age and Block Replacement Policies
The usual replacement policy implies replacement of a component as and when
it fails, by a similar new one. The most important replacement policies general
use are age and block replacement policies.
Definition 4.5 In age replacement policy a unit is replaced upon failure or at
age T, which ever comes first.
Definition 4.6 In a block replacement policy the unit in operation is replaced
upon failure and at times T, 2T, 37, . . . .
The age replacement policy is administratively more difficult to implement,
as it requires the age of the unit to be recorded. Whereas block replacement
policy, though simpler to administer as the age of the unit need not be recorded,
leads to more frequent replacements of relatively new items. The word “block”
Renewal Theory 83
is used to imply that in practice a block or set of components will be replaced at

times T, 2T, 37,. . . regardless of which have failed. Suppose that the successive
life times of the units are r.v.s with a common d.f. F(x) having mean f,l
T is called the replacement interval under these policies.
Let N(t) = Number of renewals in [0, t] for ordinary renewal process,
NA(t, T ) = Number of renewals in [0, t] under age replacement policy,
ND(t, T) = Number of renewals in [0, /] under block replacement policy,
Denote by
H(t) = EN(t), HA(t, T) = ENA(t, 7), HD(t, T) = ENB(t, T) the corresponding
renewal functions.
N A(t,T) HA(t, T) F(T)
lim
Theorem 4.8 (a) Ho = lim
/ —> o o
(a.s.)
O
(1 - F{x)dx
fJo
N D(t, T) N D(t, T ) H(t)

(b) lim lim (a.s.).
/—»°o t t —> o o t T
Proof (a) {NA(t, T)} is a renewal process with interarrival times {Lt } and FA{t) =
P[Yl < t]. Since t lies between one of the intervals nT < t < (n + 1)7, n = 0, 1 ,2 ,...,
1 - EA(t) = P[Yi >*] = [ ! - F(T)Y [1 - F(t - nT)].
Hence fiA = ETl = P[Yt > t]dt

I0
(n+\)T
= z [1 - F(T)]n[1
n=0 JnT
= S [ 1 - F (T,\' I
n=0
J
( l- F ( ,)H (putting x = t - nT)
0
f u - F(x)]dxlF(T), since
■ Jo
1
, F(T) < 1.
Then applying corresponding limit theorems for ordinary renewal process we

get the desired result.
(b) Note that after successive block replacements at 7, 27, 3 7 ,... the process
starts anew, and if N Br (7) is the number of failures in the interval [(r-1 ) 7, r7],
r = 1, 2, 3, . . . following block replacements, then N Br (7) = N(T).
Suppose that for some specified integral value of n, nT < t < (n + 1)7, i.e.
t = nT+ x, 0 < r < 7 then NB(t, 7) = Z N B (7) + N (r).
r=1 r
Z N Dr(T)
N D(t, T) N( t )
We then get, lim lim ------ ------- + . By Strong’s
nT lim nT + r
law the last expression is equal to
hoj N ( r )
E , since = 0.
T lim nT + r

Exercise 4.1 Assume that customers arrive at a store according to a renewal process and
that, independently of the process and of each other, every customer makes a purchase
with the same probability p. Show that those customers who make a purchase also form
the “renewal points” of a renewal process. Let the time scale be expanded by a factor of
\/p. Find the interarrival distribution of the new process in terms of the original interarrival
distribution.
Exercise 4.2 Let the Laplace transforms of F(x) (the interarrival distribution) and
Wt(x) = P(Y(t) < x), the d.f. of Residual life time Y(t) in a renewal process be LF(s)
and Lw(s) respectively. Show that
Lw(s)e~st = [1 - Lp(s)] I e~sydH(y) , where H(y) is the renewal function.
Exercise 4.3 (a) If the random lifetime of an item has d.f. F(x), what is the mean remaining
life of an item of age xl
(b) Find the mean total life time of an item when the d.f. F(x) = 1 - e ~ x > 0, A> 0.
Hence show that the mean total life is approximately 2 times the mean life when the
renewal process has been in operation for a long time.
Exercise 4.4 Let [Xk, k > 1} be a renewal process, Xk has distribution F{t), N(t) is the
renewal counting function, and H(t) = E(N(t)) is the renewal function.
(a) lim H(t) = oo
t—>00
1. Show that
(b) for every t >0 ,0 < H(t) <^ 1
1 - F(t)
2. Let H2(t) = E(N\t))
(a) H2(r) = nS=1(2 v' n- 1) F tf

Show that
(b) H2(t) = H(t) + 2 f H(t - u)dH(u)
3. Assume that Xk has finite first moment ju > 0.
Define F*(f) = ~ J [1 - F{x)]dx, t > 0.
Show that (a) F*(t) is a probability distribution of a nonnegative r.v. X*

(b) If the Laplace-Stieltjes transform of F(t) is/(s), then the Laplace-Stieltjes transform
of F * ( t )is
1 z J l iL .
H
(c) Use (b) to show that X* has finite first moment iff X has finite second moment p2.
4. Assume that Xk has non-lattice distribution. Define Q(t) = 1 - F*(t) where F*(t) is
defined in Problem 3.
(a) Show that

f Q(t - u)dH(u) = / /( /) - tfp + F*(t).
Jo
Renewal Theory 85
(b) Apply Smith’s key renewal theorem to the left hand side of this equality to prove
that if Xk has finite second moment ^2 then
lim Pi ~2jU2
t—>°° 2(x1
(c) Interpret this result graphically.
Exercise 4.5 Show that ENk(t) < is finite for all k > 1 and that
E[Nk (t)]= £ [ (n+\ )k - nk ] F(n+]) (t)

n=0
Exercise 4.6 Derive the alternative formula for Residual life time T(/) in a renewal
process
p t+ X
P(Y(t) <x) = [ l - F ( t +x - y)]dH(y\t > 0, jc > 0
by arguing that Y(t) < x means that there is at least one renewal point in the interval
(r, t + x]. If the largest such point is Sn, then
P(Y(t) n= l
<x)= £ Pit 5„+1).
Exercise 4.7 Prove that Residual life time Y(t) and spent life time Z(t) have joint
distribution given by
P(Y(t) > jc, Z(t) < z) = f [1 - F(t + x - y)]dH(y) for t > 0 and 0 < z. < t.
j t-z
Exercise 4.8 (Uniform Renewal Process) Let Vh V2, . be a sequence of i.i.d. r.v.s with
EV, = 0, EVi2 = (T2.
Assume also that {Tn} and { } are independent and Fx is the distribution function of
7j. Define a stochastic process {X(/)> / ^ 0} as follows.
If 0 < / < Tx let X(t) = Vlf if + . . . + Tn_x < t < Tx + . . . + Tn let X(t) = Vn+X
(n = 1,2,...). Show that {X(0} is stationary with covariance function K{t) = cr2(l - Fx{t)).
Also assume that F{ = — I (1 - F(x)dx and m is the expectation of F, the common d.f.
of r 2, r 3, ______ m°
Exercise 4.9 Suppose that EV{ < for some r > 1 and the renewal process is arithmatic.
Let Y(t) = SN(t+l) - t be the residual lifetime. Show that
(a) E(Y(t))r ^ ^ — EX| as t —^ °°
rfi 1
2 2
(b) E(Y(t)) —> ° ■ + ^ as M «, where EXx - ju and var (X{) = <72 <
Exercise 4.10 We say that the random walk (or renewal process) is arithmetic if the
common distribution function F has its support on {0, ± d, ± 2d, . . .} (or {0, d, 2d, . . .})
for some d > 0. The largest d with this property is called the span. A random walk or
renewal process which is arithmetic with span d is also called d-arithmetic.
n
If the renewal process is d-arithmetic, then show that un = /„ + Z un_kf k where
k =0
kd)( k> 0) and/* = P(X{ = kd).

Exercise 4.11 Let 0 < fx = EXx < Then show that for renewal counting process N(t),
E(N(t)/i)r -» — as -t > °°
Exercise 4.12 Suppose that 0 < \i = EXx and a2 = Var(Xx) < «>. Show that
(a) E(N(t)) = t/fx + 0( V7) as / -> «
(b) Var (N(t)) = — - + as -> «>.
Exercise 4.13 Let Yt = SN(t)+l - t be the excess life at time t in a renewal process and
Z, = / - SN(t) be the spent life.
Then show that
limP(Y, > z.)= n 1 J* {1 - F(y)]dy, 0, where n = E(X{)
and lim P(Zt > y, Y, > x) = \x~x fI {1 - F(z)}d(z).
Also show that the total life St = Yt + Z, has limiting distribution
Exercise 4.14 (Replacement policy) If f(x) is a p.d.f. associated with a lifetime d.f.
F(x), the hazard rate is r(x) =f(x)/[l-F(x)]. Suppose T, is the cost of the /th replacement
under age replacement policy where a planned replacement at age T costs Cx dollars,
while a failure replaced at time x < T costs C2 dollars. Show that the long run mean cost
per unit time is given by
0(7) = Ct[l - F(T)] + C2F(T) / [1 - F(x)]dx and also show that the
replacement age T* that minimizes 0(7) must satisfy
r C
r(r* )J [1 - F(x)]dx - F(T*) =Ci _‘c ^ .
Exercise 4.15 Let Nx(t) and (N2(t) be two independent renewal processes with the same
irfteroccurrence distribution F with mean i. Let N(t) = Nx(t) + N2(t). If N(t) is also a
renewal process, then show that A)(/), N2(t) and A(/) are all Poisson.
5
Branching Process
5.1 Introduction and History

Branching process (popularly known as Galton-Watson process) dates back to
1874 when a mathematical model was formulated by Galton and Watson for the
problem of “Extinction of families”. The model did not attract for a long time
but during the last 40 years much attention has been devoted to it.
An important class of Markov processes with countably many states is the
class of branching processes. Let us suppose that we are observing a physical
system consisting of a finite number of particles either of the same type or of
several different types. With the passage of time each particle can disappear or
turn into a group of new particles, independently of other particles. Phenomena
described by such a scheme are frequently encountered in natural science and
technology, sociology and demography, for example, showers of cosmic rays,
growth.of large organic cells, the development of biological populations and the
spread of epidemics. All of these processes are characterized by the same property,
namely, their development has a branching form. A precise definition of processes
of this type within the frame-work of the theory of Markov process brings us to
the concept of a branching process. In fact, the mathematical model of such kind
of emperical processes lead to the idea of a branching process.
The theory of branching random processes is a rapidly developing area of
general stochastic processes. A great number of papers and the books of Harris
(1963), Jagers (1975), Athreya and Ney (1972), Assmussen and Hering (1983)
and other books are some of the results of this growth and they show the
development of the theory. The interest in this theory is connected with its
applications to a wide spectrum of practical problems. They include the description
of various biological populations, the investigation of the transformation processes
of particles in nuclear reactors, the investigation of cascade processes, of chemical
processes, problems of queueing theory, the theory of graphs and other problems.
Suppose we start with an initial set of objects (or individuals) which form the
0th generation of these objects called ancestors. The offsprings reproduced or
the objects generated by the objects of the 0th generation are the “direct
descendents” of the ancestors and are said to form the first generation; the
objects generated by those of the first generation form the second generation,
and so no.
5 .2 Properties of Generating Functions

Mathematically, we have a sequence of i.i.d. random variables {4} with
oo Xn
P[£ = k ] = p k >0, Zpk=1 and Xn+1= Z £ , n = 0, 1,2
*=0 /=!
Here Xn is the number of elements in the nth generation and {Xn, w= 0, 1, 2,. . .}
is a discrete parameter integer valued Markov chain with transition probability pik
= P[X„+\=k\Xn = i],
We also assume /’[Xq = 1] = 1.
OO
0(z) = Z pkz k be the probability generating function of Cr 0 o(z) = z
Let
and 0](z) = 0(z).
Define 0„(z) = Z />[X„ = k]zk

, n = 0, 1 ,
k=1
The nth iterate of 0(z) is 0 [ 0 [ . . . [0(z)] . . .]
n-times
Theorem 5.1
(i) 0 n+1(z) = (0„(0(z)) (5.1)
and (ii) 0 B+I(z) = 0(0„(z)) (5.2)
Proof
(i) 0„+1(z) = k=
Z0 m B+i = *]z*
= Z f Z P[Xn+l = k\Xn = l}P[Xn = Z]lz*

k=0 yl=0 J
= Z />[X„ = /] Z P[& + £2 + ... + £, = *]z*

1=0 k=0
(interchanging summations by Fubini Theorem)
= Z P[Xn = l]Z P[% = k] zk( 4’s ar

1=0 \ k =0 I
= l P [ X n = l][0(z)]1
= 0„(0(z)).
Also 0 o(z) = P[X0 = l]z = z for n = 0
0,(z)= Z P[Xo = /]0(z) = 0(z)
1=0
(ii) By (5.1), 0„+1(z) = 0„(0(z)) = 0„_,[0(0(z))]

= 0„_ ,(0 2(z))
= . . . = 0[0„(z)], by induction on n.
Branching Process 89
Note In fact for any k = 0, 1, 2, . . . , n

0 n+l(z ) = 0 „ _ * ( 0 * +1(z )).
Let E(î) - EXx - m.

Corollary 5.1
E(Xn) = mn (5.3)
mn - 1
Var (Xn) a 2mn~1 if m * 1 (5.4)
m- 1
ncr if m = 1,
where o2= Var(£) = Var(X,).

Proof From 0 n+1(z) = 0„(0(z)), we get
0 ; +1(l) = £ (X n+l) = 0 '( 0 ( l ) ) 0 '( l ) (5-5)
oo
But 0 (1 ) = 2 = 1 and 0 '(1 ) = m = E £ = E( Xi ) .
k=0
Hence E( Xn+i) =0'n

( l)m = E( X
= mn+'0E
(X) = w"+1, since />(X0 = 1)
Note If we assume / ’(Xq = n0) =1, then EX„ = n0m".
We also get from 0 „ +,(z) = 0(0„(z)),
0 « +1 (1) = 0 ' ( 0 n (l))0 n (1)
0^,(1) = 0"(0«(l))[0n(l)]2 + 0 /(0„(l))0;,(l)

= 0"(l)(/n")2 + «0?( 1) (v 0„(1) = 1, 0'(1) =
= m 0 " ( l ) + Km2n,where 1).
Thus, we have Q"+i(l) = m 0 " ( l) + Ar/n2'1 (5.6)
m 0 " ( l) = m20"_,( 1) + Xw(2n-1)
m n- l0 " ( l ) = 1) + Km2n-(,,-])
Adding both sides of the above equations, we get
0 " +i (1 ) = mnK+ Kmn+] + ... + Km2n
- 1 +
wj ”+* —1
= mnK --------— ifm * 1.
m- 1
Now Var (X„+1) = 0 " +1 (1) + 0'n+l (1) - ( 0 ' +, (l))2

»n+\ _ 1
= m nK ------ -— + mn+l - /n2(n+1) if An * 1
m- 1
r [A:w',+1 - K + m 2-m n+3 + mn+2]

m- 1
am” if An * 1.
( Y 0 "(1) = K = Var(X) - m + m2 = o2 - m + m2)
i(1 ) = 0J,'(1) + K
0"+ (from (5.6), putting = 1)
If m = 1, 0+
n\' 1( ) = (n + 1)K.
Also Var (X„) = 0 „ (1) + 0^(1) _ (0n (I)2)

= nK + mn - m2n = n£ if An = 1
Hence Var(X„) = ncr2 if An = 1.
Note If An > 1, the population increases (i.e. explodes). This case is called
supercritical case.
If m < 1, the population dies out and the case is called subcritical.
If m = 1, the case is called critical.
Also note that
if An < 1, Var(Xn) —» 0 as n —> and if An > 1, Var(X„) —» °o as n —» ©o.
Alternative Proof of (5.3) and (5.4)
xn
Now the number of members in (n + 1)* generation can be written as Xn+x = S ^ ,
i=i
where 4 ’s are i.i.d r.vs. By Wald’s Identity £(XW+1) = E(£x)(EXn) = mEXn and the
rest is the same as before.
Exercise Show that (i) Var(X) = £(Var(X |K)) + Var(£(X|T)) and
This is a particular case of Wald’s Identity for second moment and follows
from (i)
Now Var(Xn+1) = E(Xn) Var(^) + ( E ^ f Var(X„)
= m V + nr Var(X„) (5.7)
This is a first order difference equation.
A particular solution of (5.7) is
Var (X„) = -Cr2- -V ifm * 1.

m- m
A general solution of the difference equation (5.7) is given by Var(XJ = A(m2)n

and a complete solution is given by
Var (Xn)=A( m 2)n+ • - - - i l l for n >1 and Var(X,) =2

m - nr
gives A = crlm(m - 1).
Therefore
m n~x ( m n - 1)
Var(X„) = ----------- ----- - a 2 for n > 1if m * 1
m- 1
and Var(XJ = no2 if m = 1 (from (5.7)).
5.3 Probability of Extinction

Let P[Xn = 0] = qn. Assume p0 > 0.
We are interested to know when the population dies out, i.e. to know lim qn .
n —»°o
Lemma 5.1 qn is non-decreasing, lim qn = n exists and 0 < n< 1.

n—>°°
Moreover n is the smallest positive root of the equation 0(z) = z (0 < z < 1).
Proof q0 = 0, since P[X0 = 1] = 1 (by assumption).
0 „ (z ) = 2 P[X„ - k]zkand 0„(O) = 0] = q„.

k=0
Since 0 n+1(O) = 0 ( 0 n(O)), = 0 ( ^ ) , n = 0, 1, . . . (5.8)
Claim q\ - 0(qo) > 0
= 0(0) = p0 > 0 (by assumption)
Now q0 = 0 and hence > ^0-
Suppose qn_x < c/w. Then since 0(z) is increasing in z(0 < z < 1), we have
0(q n. l)< <p(qn) (5.9)
or qn < qn+x (by (5.8) and (5.9)). Hence qn is nondecreasing in n.
qn+l = 0(qn) < 0(1) = 1 for all n = 1, 2, 3, . . .
Therefore lim qn = n (say) exists.

n— >°°
Now qx > 0 and hence qn > 0 for all n and {qn} is nondecreasing and n > 0.
We have ^fn+1 = 0{qn). Taking limit n —> ©o on both sides, we get
7l= lim qn+x = lim 0 { q n) (v 0 ( z ) is confinuous)
n—>°° n —>°°
= 0(7T).
Now 0(z) = z=> 7t is positive and hence 7Tis the positive root of 0(z) = z-
Assume that S0 is another positive root of 0 ( z ) = z. Then q0 = 0 < s0. Assume
qn < an^ which implies
qn+\ < s0 (v <7n+1 = 0 ( ^ ) < 0(so) = s0)
Now 0 „ (0) = P[Xn=0]=qn <1.
By induction on n, qn < Jq for all n = 0, 1, 2, . . . . Hence, lim qn = n< s0 .

n—>°°
Therefore n is the smallest positive root of 0(z) = z.
Lemma 5.2 0(z) = z has a real root in 0 < z < 1 iff m > 1.
0(z) = z has only z = 1 as root in 0 < z < 1 iff m < 1.
Before giving the proof of lemma 5.2 let us give an interesting example
illustrating lemma 5.2.
Example 5.1 P[% = 0]= ju > 0, P[£ = 2] = A > 0, P[£ = 1]= 1 - ju - A > 0.
Then 0(z) = /u+ (1 - n - A)z = A

To find the solution of 0(z) = z.
iu + ( l - / i - A ) z + Az2 = z
=> fl - (H + A)z + Az2 = 0
=> (z - 1)(Az - /i) = 0 =» z = 1 or /i/A.
Then m = 0'(1) = 1 - / i - A + 2A = 1 - /t + A .
Therefore /n > 1 => 1- ju + A > 1 or - + A > 0 or A > //.
Therefore, if m > 1, i.e. A > //, the population dies out with probability
7i- pdX < 1 and if m < 1, i.e. A < ju i.e. pJk > 1, then n = \ i.e. the population is
sure to die out (by Lemma 5.2).
Proof of Lemma 5.2 Now G.F. 0 (z ) = 2 p^z*.

k=0
Then / ( z ) = ——^ + Pi + + P3^2 + . . . (0 < z < 1).
Since this is a power series which is uniformly convergent
/ '( z ) = - + P2 + 2 p 3z + .. .
"7^
and ru ) + 2 + .. . > 0(0 < z < 1).
This =>/'(z) T z and/'(()+) =

Case 1 m > 1
/ '( l ) = ~Po + p2 + 2p3 + 3p4 + . . .
0 = Po + P\ + P2 + • • • -1
/ '( I ) =P\ + 2p2 + 3p3 + 4p4 + . . . -1
- m - 1 > 0.
Thus, there exists a 0 < /? < 1 such that/'(/3) = 0 and which =>f'(z) < 0 for
0 < z < /?. Hence/(z) >linzin0<z</3 and /(z) T in z for /? < z < 1.
0 ( 1)
Now /(1 ) = 1
/(0+) =
Therefore there exists 0 < a < 1 such th at/(a ) = 1. i.e. 0 (a ) = a.
m Case 2 m < 1
Then j8 = 1. Hence
i
f'(z) < 0 in z for 0 < z < 1 orf(z) 4.
/(0 +) = o o /(l)= 1.
t Hence, there does not exist 0 < a < 1
such th at/(a ) = 1 and hence z = 1 is
0 the only root of 0(z) = z.
Theorem 5.2 If m > 1, then with probability n, the population become extinct
and with probability 1- n, the population explodes. If m < 1, then with probability I ,
the population becomes extinct. [Regardless of the actual value of the mean
EXx = m > 1, the probability that the nth generation will consist of any positive
finite number of individuals tends to zero as n —»©o, with probability of extinction
tending to n. In these circumstances, we say that Xn —>0 as n —> with probability
1 - n\.
Proof qn = P[Xn = 0],
0 n(z) = <Jn +2, P(Xn = |z| < 1.
Then lim 0 n ( z ) =n + 2 P[X„ = k ] z (k5.10)

n->o° k=1
(by Lebesgue dominated convergence theorem)
Now P(Xn — = E(I*x =^|(w)) = I dP

n J \ x n=k\
Claim lim 0 n (z) = n.

n —>°°
Case (1) Suppose 0 < z < n.Then by Lemma 5.1. 0(n) = n a

0„{n) = 0 ( 0 n_l(n)] = nby
( Theorem 5.1).
Now n = lim qn = lim 0 „ (0) < lim 0 n (z) (v 0 „ (z) T in •
n —>°o °° n— v n v /7
< lim 0 n (n) = n.

n—*°°
Therefore, if 0 < z < n,ilm 0 „ (z) = n.
n—
>°°
Case (2) n< z < 1. Suppose 0 n(z) —> y * 71 as n —> oo.
Therefore, lim 0 n+\(z) = lim 0 ( 0 n (z)) = 0(lim 0 n (z)) and hence y= 0(y).
ft—>°° n—>°°
If 7t < y < 1, then /(y ) < 1
i.e. 0 ( y ) < Y ^ - ' - ^ p - = f ( Y ) <
is a contradiction to 0 (y) = y.
If 0 < y < 7T, 0 (y ) > y and again a
contradiction arises.
Therefore lim 0 n(z) = tt, i.e. 0 n(z)
f t—» °o
converges to a constant function, viz. zrand that implies = P(Xn = 0) —> n as

/M o o and all other coefficients of z* in (5.10), P(Xn = k) —> 0 as n —» ©° for
£ = 1 , 2 , . . . and the rest of the proof follows from Lemmas 5.1 and 5.2.
Let T be the time till extinction (first passage time to state 0).
Then [T = n] = [Xn = 0, Xn. * 0].
Hence P\T = n] = qn - qn_x = P\Xn = 0] - /^ X ^ = 0]
(v = 0 => X, = 0, i.e. [X„_j = 0] c [X„ = 0])
Hence E(T) = S - <7/2_i]

n= 1
= <7i - <7o + 2^2 - 2<7i + 3r/3 - 3^2 + • • • (5.11)

= - q0 - q} - g2 - <73 - • • < 0 which is absurd.
(This is because the series I / i ( ( j i „ - </„_i) is convergent but not absolutely

n~ 1
convergent so that a rearrangement of the series, as done in (5.11), is not possible.)
Now lim qn = 7i>0=>qn = d > 0 . Therefore, Z qn is divergent.
ft—>°° n
Example 5.2 0(z) = p0 + 0 < Po<E and + Pi = 1-
Then 0 2 (z) = Po + P\ (Po + Piz) = Po + PoP\ + P\Z-
Thus 0 „ (z) - p0 + PoPi+ PoPi +. . . +P
1 - Pi"
- d - /» r ) + p " z
Hence qn - P (X n = 0) = 1 - p[7—> 1 as n —> <».
Also 0(z) = z = Po + Piz => z(l ~ P\) = Po => <: = 1

i.e. the population is sure to die out. Here
P(T = n)=qn- qn_x =1 - pj' - 1 + p

i.e. the population is alive with geometric distribution.
E(T) = £ npl Vo 2 1
m= 1 n=l
__ 1 Po 1
°(1-Pi)2 p% Po
Example 5.3 It is frequently assumed that the problem of a mutant gene having
k distinct discordants {k = 0, 1, 2,. ..) is governed by a Poisson distribution with
mean A = 1.
Then 0(z) = ez~x and n - 1, because
0(z)= £ = e -A (l-:)
*=o a:!
0(z) = z has the explicit solution for A < 1 though for A > 1, the extinction is
proved with probability 0 < n< 1 (iff A > 1). If A = 2, then z = e2(z~l) has a root
0.2 < 1 and 7T= 0, 2.
The graph of 0(z) in 0 < z < 1 between (0, e~k) and (1, 1) 0(z) is convex and
that the curve y = 0(z) always lies above y = z when A = 1. There is no other root
of z = 0(z) except 1 in [0, 1].
Hence the probability of extinction is 1.
Theorem 5.3 E(Xn+r\Xn) = mrXn. (5.12)
Proof We first prove the case of r - 1.
E ( X n+x\Xn) = E [ 1 £j\xh] = (Xn)E(Zj) = mXn (5.13)

7 =1
(v are i.i.d. with E(^) = m).

Assume (5.12) is true for r. We shall prove that it holds for r + 1.
Now E{Xn+r+,\Xn) = E(E[Xn+r+t » \Xn+n Xn+r_\... x„]\xn)
= E(E[Xn+r+]\Xn+r]\Xn) ( v is Markov)
= E(Xn+lm\Xn) = mE(Xn+r\Xn) ( v X„ is stationary, apply (5.13))
= mr+'Xn.
Therefore, by induction on r we have E(Xn+r\Xn) = mrXn for r - 0, 1, 2, . . . .
Now consider the new process
Wn = — ,n = 0, 1 ,2 ,...
mn
Claim W„ is a Martingale, i.e.
E ( W n+r\W ,n Wn_u W0) = Wn a.s. for r = 0, 1, 2, . . .
Pr° ° f E(W„+r\Wn) = - L - \X„)

mn
=— — Wn a.s.
m
and by Markov property E(Wn+r\Wn, . . . , Wh W0) = E(Wn+r\Wn) a.s. (for Martingale

or fair game or expectation constant process, see author’s book “An introduction
to measure theoretic probability,” Prentice Hall, New Delhi).
Now EW0 = EWr = 1 and E(Wn) = 1 for all n = 0, 1, 2, . . .
EXl m n- ' ( m n - 1) 2
Also = m 2n + ------ -----;---- - a 2
m In m- 1
= 1+ (1 - m~n) —» 1 + if m ^ 1
m(m - 1) m(m - 1)
and V(Wn) = ; -(l - m '") ^ , g2 n if m * 1.

m(m - 1) m(m - 1)
If m > 1, the limiting variance of Wn < a 2.
5.4 Distribution of Total Number of Progeny

Let Xn denote the size of the nth generation n = 0, 1 , 2 , . . . and = 1.
n
Then the random variable Yn = Z X * = l + X i + . . . + X„
k =0
denotes the total number of progeny, i.e. the total number of descendents upto
and including the nth generation and also including the ancestors.
Theorem 5.4 The G.F. Rn(s) of Yn satisfies the recursion relation
Rn(s) = s0(R n_l(s)), (5.14)
0 being the G.F. of the offspring distribution.
Proof Let Zn = Xx + X2 + . . . + Xn and Gn(s) be its G.F.
Then Rn(s) = sGn(s) (Yn = 1 + Zn and G.F. of 1 is s).
oo
We have Gn(s)= S P(Zn = k)sk .
k =0
P(Zn = k) = X P [total number of descendents in the succeeding (n - 1)

/=o
generations following the first generation is k-i given X x - i] P[X{ = /].
If the process starts with one ancestor then the probability of having r descendents
in succeeding m generations is the coefficient of sr in Gm(s) and if it starts with
1 ancestors then the probability of having r descendents in the succeeding m
generations will be the coefficient of / in [G ^s)]1.
Thus P(Zn = k) = X coefficient of s k~l in {G„_! (5))'/?,

!=0
OO
= X pi (coefficient of s k in {jGn_i (s)}')
= coefficient of s k in X p { {sGn.\ (j)}'

1=0
= coefficient of sk in 0(sGn_\(s)).
Therefore Gn(s)= I P(Zn = k)sk =

k=0
and Rn(s) = s0(sGn^(s)) = s0(Rn_l(s)).
Theorem 5.5 lim Rn (5) = H(s) = X p ks k , where p* > 0 (5.15)
and //(^) satisfies the relation H(s) = s0(H(s))y 0 < s < 1.

Further //(s ) is the unique root of the equation t = 50(0 such that H(s) < y,
where y is the smallest positive root of x = 0(x) and H( 1) = X pk = y.
k- 0
Proof For 0 < s < 1, R\{s) = s0{s) < s ( v 0 (j) < 1).
R2(s) = ^0(/?j(j)) < s0(s) = Rfs). Assuming Rm(s) < Rm_\(s), we get
Rm+1(^) = s0(Rm(s)) < Rm(s) if 0 < j < 1.

Hence by induction on n, Rn(s) < Rn-\(s) for all n = 1, 2, . . . .
Thus for 0 < 5 < 1, {Rn(s)} is a monotone decreasing sequence of functions,
and bounded below, and lim Rn ( 5 ) = H ( s ) , say, exists. From Continuity Theorem
of G.F.s, it follows that H(s), being the limit of a sequence of G.F.s is the G.F.
of a sequence of non-negative numbers pk such that H{ 1) = £ p k < 1. Taking

k =0
limit n —><*>in equation (5.14) of Theorem 5.4 we get H(s) = s0(H(s)). Since 0
is continuous, 0 < s < 1 for some fixed ^(0 < s < 1), H(s) is a root of the equation
t = s 0(0- For fixed s < 1, y = 50(0 is a convex function of f and the graph of
v = 50(0 intersect the line y = t in at most two points. Let 6 be the smallest
positive root of the equation x = 0(x). Then clearly 6< 1. The function t - 50(0
is negative for t = 0 and positive for values of t bewteen 6 and 1. Thus t = 50(0
has exactly one root between 6 and 1. The unique root of t = 50(0 equals H(s)
and H(s) < 6.
Clearly H( 1) is a root of x = 0(x) and since 6 is the smallest root of this
equation, //(l) = 9.
Note H(l) = 1 or < 1 depending on whether 1 + Xx + . . . + Xn + . . . is finite
or not with probability 1 and H( 1) = 1 whenever m < 1 (i.e. the population is sure
to die out).
Some well known branching models

1. Lotka Model (1939): Here
pk = bck \ k = 1 ,2 ,.. . , 0 1
Then m = ---- -—y and P.G.F. 0 ( z ) =5 pkz k

(1 - c )2 *-o
= 1- bz
1 - cz
The quadratic equation z = 0(z) has roots 1 and -— = £o(say).

c 11 ~ c)
If m - 1, z0 = 1, and the extinction probability is 1.
If m > 1, zo < 1, and the extinction probability is n = z0 < L
The 1920 US census figures give b = 0.2126, c = 0.5893, (m — 1.25 > 1), n - z0
= 0.819.
2. Suppose 0(z) is of the form 0 ( z ) = a + pz , aS - /?y = 0 (functional linear

7 + Sz
transformation or bilinear form).
In that case 0 „ (z) = (Lotka’s model is a particular bilinear form).

+ Snz
5.5 Continuous Parameter Branching Process

(Markov Branching Process)
Let X(t) denote the size of the population at time t so that X{t) takes only non
negative integral values. We consider only homogeneous branching process, that
is, processes for which ptf t b t2) = Pij(t2- 1{). Let us consider at present, processes
with a single type of object. Since in branching processes every object evolves
independently of the others we may assume that at the initial instant only one
object exists. With the passage of time t it either disappears or turns into k
individuals that are all of the same type and constitute the first generation.
Every object of the first generation “lives” independently of the other, and the
same probability laws obtain for it as for the initial object “the ancestor”. At
some instant of time the object either disappears or turns into second generation
objects and so forth. The entire process is described by a single integer valued
stochastic process X(t) that is equal to the number of objects existing at the
instant t. By assumption X(0) = 1 a.s.
The set of possible states of the process is the sequence of natural numbers 0,
1, 2, 3, . . . . Here 0 be an absorbing state. If X(t) = 0 then X(t) = 0 for all t > t0.
If X(t) = 0 a.s. for some r, then the process is said to be degenerate process. In
other case X(t) vanishes after a finite interval of time with probability < 1. This
probability is called the probability of degeneration of the process. It is possible
that X(t) increases without bound in time. In case of nuclear reaction this situation
can be interpreted as explosive. Thus we can ask the questions: What is the
probability of degeneration and what is the lim X( t)l
/ — >oo
We state the formal definition of a continuous parameter Markov Branching

process as follows.
Definition 5.1 A Stochastic process (X(/), t > 0} with values on the set of non-
negative integers is called a continuous parameter Markov branching process if
(i) it is a stationary Markov chain
(ii) its transition probabilities pi}{t) = P(X(t) = y|X(0) = i) satisfy the relation
P i j ( t ) = p\lk}( t ) , the i fold convolution of the probabilities.
Let pijit) denote the conditional probability that the system consists of j
objects at the instant t + T if there were i objects at the instant r. To solve the
problems that arises in the theory of branching processes, we shall find it convenient
to use the method of generating functions. We, therefore, introduce the generating
function G/(z, t) of the distributions for j = 0, 1, 2, . . .
G,(z ,t) = t Qz kplk(t),\z\< l , / = 1 ,2 ,...'

= E ( Z kU)) (5.16)
G(z, t) = Gj (z, t) and G(z, 0) = z
Since the individuals act independently, the probabilities pik(t) for the fixed i
correspond to the distribution of i independent r.v.s all having the same distribution
determined by Gi(z, t).
Thus, we have
G,(z, 0 = [G,(z,/)]' (5.17)
The formula (5.17) characterizes and distinguishes branching process from other
continuous time Markov chains. In view of the time homogenity, the Chapman-
Kolmogorov equation takes the form
Pik (t + ?) = ^ Pii (0 Pi k (r )- We shall now obtain a continuous time analogue
7=0
of the relation (5.1) satisfied by a Galton-Watson process.
Lemma 5.3 G,-(z, t + t) = [G(z, t + t)]1= [G(G(z, t), T)Vi i - 0, |z| < 1 and in
particular,
G ( z ,f + r ) = G(G(z, 0, t ). (5.18)
Proof Using (5.16) and Chapman-Kolmogorov equation, we get
[G(z, f + t ] 1 = S pu (t + r ) z 7, i > 0
r=0
= Iz^ £ P ik ( t ) p k j ( T )
7=0 k=Q J
= 2 p ,* (o f 2 p jty (r)z y)
*=o V/=° 7
(interchanging the order of summation by Fubini’s Theorem)
£ P tt(0{C (z. r)}* = [G(G(z, t), o r

k =o
and in particular G(z, f + r) = G(G(z, r), /).
We shall now proceed to obtain two differential equations (Backward and
Forward) satisfied by Gj(z, r); these can be obtained from the system of
Kolmogorov Backward and Forward differential equation for discrete state
continuous parameter Markov chain (Chapter 6). According to general theory
(Theorem 6.2), the derivatives
= A,-, / * 1 and lim 1 - P n ( 0

Pij(t)
lim
/->o t 7J t->o t
exist for a regular Markov process. Let us also suppose that the process is
conservative so that
A\ —A0 + Z A; < °° (5.19)

7=2
Now (5.18) implies pi}{t) = Af + o(t),j = 0, 2, 3, . . .
and Pn(0 = 1 - + o(t) (5.20)

With the given initial conditions, the differential equations can be solved in
terms of the infinitesemal probabilities A*’s.
Let u(z) = Z Aj zj be the generating function of the sequence {Ak}.
7=0
Theorem 5.6 Let X(t) be a Markov branching process with infinitesimal transition
probabilities defined by (5.19) and (5.20). Then the generating function of the
probabilities obeys the equations
^ = «(G) (5.21)
and = (5-22)
where G = G(z, 0 is the generating function of {/?u(/)}-

Proof By (5.19), (5.20) and the same argument used to derive Backward
Kolmogorov equation in Chapter 6 and time homogenity of branching process
X(t), we obtain
dp\j{t)
= ~^\P\ j( 0 + 2 Xkp kj(t) (5.23)
dt
k*1
Multiplying both sides of equation (5.23) by zf and summing over j from 0 to
we obtain
d Gi (z , t)
= -A ,G ,( z ,0 + I A*G*(z.f)(|z|^l). (5.24)
dt k=0
k*\
or on the basis of (5.16),
= - A,G(z, 0 + 2 A *G * (z , 0 , ( 5 .2 5 )
Ot k=0
k*\
where we set G(z, t) = Gj(z, /). Finally we obtain the following non-linear
equation
3G (z,/)
= m(G(z, t)) with initial condition G(z, 0) = z. (5.26)
Or
Suppose that the conditions under which the second system of Kolmogorov
forward differential equations may be applied are satisfied. From the definition
of the branching process it follows :
Pkk(t) = (1 - k xt f + o(t) = 1 - + o(t),
k-i(tP)= *(1 - Xxt)k~xX0t + o(t) = kX0t + o(t),
Pk,k-j(t)= o{t),j > 2
Pk.k+jiO = k(l - Axt)Aj+xt + o(t) = kk J+\t + o(t), j > 1.
From which it follows (in the notation of Section 6.1) that
*7jj ~ = A>. Q j j - k = 0» k ^2,
Qj\j+k = j^k+1’ j — 1•
Thus Kolmogorov’s forward equation (Section 6.2) takes the form

dpij(t)
- + ^ P k j i O k A j-k+l (5.27)
dt
k*0
Multiplying (5.27) by z1 and summing with respect to from 0 to °° we obtain
dG(z,t) d
= u ( z ) ----—— (|z| < 1) (5.28)
dt dz
with initial condition (5.26).
Solution of equations (5.22) and (5.21)

G{z, t) can be obtained by solving either the backward equation (5.21) or the
forward equation (5.22) with the help of initial condition (5.26).
Noting that --- I f ( x ) d x = f ( z ) and using G(z, 0) = z, equation (5.21) is

az J 0
equivalent to
dx t (5.29)
s u(x)
provided u(xf * 0 in G(z, 0) = z < x < G(z, t). Thus G(z, t) can be obtained
theoretically from (5.21). Equation (5.29) can be rewritten as
6(G) Hz) = J‘ (5.30)
Similarly, solution of (5.22) and (5.26) takes the form
G(z, 0 = V (5.31)
dz
where y/(t)is the inverse of the function t - <j>(z) = fJo u(z)‘
This solution coincides with (5.30).
Example 5.4 (Linear Birth and Death Process)

In this special Markov branching process if we put A2 = q, X$ = p, X\ = p + q and
Xk - 0 we get a generating function u(z) = p - (p + q) z = qz2- In this case a
particle disappears with probability pt + o(t) after an interval of time f, divides
into two particles with probability qt + o(t) at the end of the same interval, or
keep its original state with probability 1- (p + q)t + o{t) for that interval. The
probability of dividing into more than two particles is equal to o(t). The process
in question coincides with a linear birth and death process. We have
= d)(. \ = f' dz _f___ dz 1 In - — ~ ( p * q)

Jo “ (z) J ■z‘ q - p z-p
For the inverse function z - i\p(t) we obtain the expression
1_
= VO= and (3 = p/q (5.32)
1
Then from equation (5.31), we get
z - p - p ( z - l)er" ° - p)
G U ,0 = V'[1 + 0U)] =
z - (z - l)e 9' (l P)
Expanding G(z, t) in a series of powers of we get
(1 - P)2[\ - eKq
G(z,t) = 1 2 (5.33)
1 _/?-V<9-p) n=\ IP-
which leads us to the formulae
1 - e,iq~p) (5.34)
P\o(t) =
1 - p - le,(l’-,,)
„ , [1 - g '(< 7 -p )l" -l
P\n (0 =(1 - P) [ 0 _ e H q - p ) ] n +\
,n>
------ 1
---------- (5.35)
Special case: If p = g, then
Jo p (l - z ) 2
1
P 1-z
z , = v(t ) =1-1 +1pt' (5.36)
Using equations (5.36) in (5.31), we get

n-1
G(z, t) = Pt + I (pO (5.37)
1 + p r « = i(l+ p r) fl+1
Consequently, powers of z give
(5.38)
(1 + pf)”+l
The following asymptotic relationships as t —>°o are immediate consequences
of formulae (5.34) to (5.38): If q < p, then p 10(f) —> 1. P\n(t) —> 0 for n > 1 as
t —» oo. If q < p, the branching process degenerates with probability 1, i.e. all
objects eventually disappear. If q > p, the probability of degeneration of the
process is equal to j8 = plq < 1. However, if the objects do not disappear the
number of them increases without bound in time, because
i N
P(X(t) > N\X(t) > 0) = 1 ------ — Z 1 for every N.
Pio(0*=i
Let us look at the question of asymptotic behavior as t —><*>of a branching
process in the general case. In what follows we shall need the moments of X(t).
5.5.1 Moments of the process X(t)

Since we are using generating functions it will be convenient to introduce factorial
moments. We define
Mk(t) = E[X(t)(X(t) - 1) . . . (X(t) - k + 1)]. It is not difficult to set up linear
differential equations that are satisfied by Mk(tYs.
Differentiating equation (5.32), we get
d<h(z, t)
m'(G)0!(z, t) and 0i(z, 0) = 1, (5.39)
dt
where <h(z, t) = ^ -G (z , t) = E [ X ( t ) z X(l)-']
and u \ z ) - -Aj + Z k X kz k 1 (5.40)

k~ 2
It follows from (5.39) that (z, t) = exp (\z\< 1).
As z — ¥1, m'(z)—» + E kAk = mi <

°° (assume) and G(z, t) »1, increasing
k-2
monotonically and hence uniformly with respect to t.
Therefore Hm ft (z, t) = lim £ [X (r)z x^>-'] = EX(t) = £ k p k

ztl *=1
Thus Afj (r) = lim 0] (z, 0 = lim exp fJo u \G )dt = e m' (5.41)
This result is immediately generalized to factorial moments of higher order.

Lemma 5.4 Suppose that
mr = Z n(n - 1). . . (n - r + 1) r = 1, 2,. . . k , . . . (5.42)

/j=i
Then the factorial moments Mr(t), for r = 1, 2, . . . £ of the process X(t) are
tinite and they satisfy the first order linear differential equation with the constant
coefficients.
Proof Let us differentiate equation (5.22) successively with respect to z (for
I z I < 1). Setting
& (z, t) = ■k^ y ' t) = (t) - 1 ) ... ( X ( t ) - k +

[X
E
dzk
we obtain
d<h(z,t)
= u ' ( G ) h ( z , t ) + u " (G )tf (z, t),
di
d<t>k(zt) (5.43)
= u'(G)<j>k(z,t) + ii/k(z,t).
dt
where \pk(z, t) is a polynomial in ^(z, t), ty2(z, t),... y<t>k-\(z, t) and the coefficients
depend on m"(G), .. ., u{k)(G). The initial conditions are given by 0*(z, 0) = 0 for
k > 2. The solution of equation (5.43) is of the form
<t>k(Zy t) = exp J u \ G ) d t J \f/k (z,s)exp -J u\G)dO ds.
Using the same consideration as in the case of k = 1, we obtain by induction
Mk (t) = lim (pk (z, t) = 6 m>' f V * (1, s)ds.

Jo
Here Mk(t) obviously satisfies equation (5.43), in which we have set z = 1. In
particular
( tUl- (e mi‘ if *0
m,
(5.44)
( m2t if m{ = 0
and variance of X(t) is given by Var(X(/)) = - 1J em{t (em{t- \ ) , (ml * 0).
Let us proceed to the discussion on probability of extinction. Since the events

X(t) = 0 constitute an increasing sequence of events, the probability of extinction
is given by
q = P(lim X(t) = 0) = lim P(X(t) = 0) = lim p\o(t).

/—
»°° /—
>00 >DO
The following theorem illustrates under what conditions the branching process
become extinct.
Theorem 5.7 The probability of degeneration of a branching process coincides
with the smallest non-negative root of the equation u(x) = 0. If u ( 1) = mx =
-X\ + Z kXk < then for u( 1) < 0 the probability of degeneration is q - 1. If

k-2
u( 1) > 0, then q < 1.
Proof Since /?10(1) = G{0, f), it follows from (5.22) that
= u(p]0(t)),pl0(t) = 'J o(0) = 0(5.45)
we have u(l) = 0 and u(0) > 0.

If A0 = 0, we get a trivial solution p\o(t) = 0 for equation (5.45) and the
theorem is trivial (note u(0) = A0). Suppose that u(0) = A0 > 0. We note that x0
is the smallest positive root of the equation u(x) = 0 in 0 < jc < l.ln other words,
u(x) = 0 for x = jc0, while u(x) > 0 in 0 < x < x0. Now p 10(0 is non-decreasing
function of t and so from (5.45) we find w(/?10(0) > 0 for t > 0, i.e. u(p\0(t)) is
non-negative for all t > 0. Hence p 10(0 < *o f°r all t > 0.
Now p 10(0 cannot be equal to jc0 for all t > 0, because if Pio(ô) were equal to
x0 for t0 > 0, then by virtue of uniqueness of the solution of equation (5.45) we
would have p\o(t) = jc0, which is impossible. Since the limit q = lim p 10 (t) < 1
t —»oo
exists, it follows from equation (5.45) that lim p{o(t) = u(q) also exists by
continuity of u and this implies u(q) = 0. Otherwise the quantity
P\o (0 = f P\o (t)dt + pio (t0) oo as t >oo.

Jto
Thus q = Xq. If x0 < 1, u(x) is an increasing function at the point x0 = 1 and
u \ l) > 0 provided it exists. If x0 = 1> then u \ 1) < 0. In case of linear birth and
death process
u(x) = A(jt - 1)(jc - pJX). Thus q = 1 iff A < p iff u (1) < 0.

Exercise 5.1 L et/(j) = 1 - p( 1 - s)&, where p and are constants and 0 < p < 1,
0 < (3 < 1. Prove that f(s) is a probability generating function and its iterates are
f n(s) = \ - p l+P+ +Pn~l ( l - s ) P n for n - 1, 2,. ..
Exercise 5.2 Show that E = m/( 1 - m) when m = E (Xj) < 1 in a

branching process.
Exercise 5.3 Let f(s) = as2 + bs + c, where a, b, c are positive a n d /(l) = 1.
Assume that the probability of extinction is d(0 < d < 1), prove that d = c!a.
Exercise 5.4 Let c\n = P{Xn = 0) = 0„(O), where Qn is the nth iterate of the
probability generating function 0(5) = £ pksk. Prove that qn = 0w_i(po)» n> \ .
k- 0
Exercise 5.5 Let Xn denote the number of offsprings in the nth generation of
a discrete time branching process. The function (pn(s) is the nth iterate of the
probability generating function 0(5), m = EXx, a 2 = Var(Xj) and P(XQ= 1) = 1.
1. Assume n > I. Show that the bivariate generating function of Xt and Xn is
£ £ P i x , = k, X„=j] ukx ’ = ft
k=07=0
2. Prove that E(XnXt) - mn~’E(Xf ) if n > I.
3. Find Rnh the correlation coefficient for Xn and Xh and discuss the behavior
of lim Rni for fixed values of /.
n—
>°°
Exercise 5.6 Show that 0(z), the generating function for each of the cases (i)
Poisson, (ii) binomial (iii) negative binomial the offspring distribution satisfies
a differential equation of the form (c + dz) 0'(z) = a + bQiz), where a, b, c and
d are constants. Conversely, if such a differential equation holds for 0(z), show
that the following recursive formula applies:
if P(Xn = k + 1) > 0, then, for k > 0 and n > 1
(c + dqn_i)P(X„ —k + i) = aP(Xn. , = k - l ) + j
P(Xn =j)P(Xn.I = * + 1 -7), where qn = P(X„ = 0).
Exercise 5.7 Let m > 1. Assume that X0 = 1 and 0 < EXf < °o. Show that the
sequence Xn/EXn is an a.s. convergent martingale.
Exercise 5.8 At time 0 a blood culture starts with one red cell. At the end of
one minute, the red cell dies and is replaced by one of the combinations with
probabilities as indicated: 2 red cells with probability 1/4, 1 red cell and 1 white
cell with probability 2/3, 2 white cells with probability 1/12. Each red cell lives
for one minute and gives birth to off springs in the same way as the parent cell.
Each white cell lives for one minute and dies without reproducing. Assume that
individual cells behave independently. What is the probability that the entire
culture dies out eventually?
Exercise 5.9 Let En be the event that extinction occurs at sometime after n and
Yn = Xn/(na2), a 2 > 0. Show that if m = 1 and 0"(z) < 00 then
P(Yn < y | En) -> 1 - €“2^, as n - ) 00
Exercise 5.10 (Branching immigration) Each generation of a branching process
(single ancestor) is augmented by a random number of immigrations who are
indistinguishable from the other member of the population. Suppose that number
of immigrants are independent of each other and of the past history of the
process, each such number having p.g.f. H{z). Show that 0w+1(z) = 0n((p(z)) H(z)-
6
Continuous Time Discrete State
Markov Processes
6.0 Axiomatic Definition and Transition Function

Here T - [/: 0 < t < °°], S = denumerable (countable) set. Recall that in discrete
parameter Markov chain transition probabilities satisfy the following conditions:
( 1) 0 < Py < 1 (2) S p y = 1
(3) p l n+l) = ^ p ^ ]p k] (4) p (°} = (Kronecker’s delta).
Now define p t f = P [X (0 i]
= P[X(t + s) = j \ X ( s ) = i)
(by stationarity) as the transition function of a continuous parameter Markov
process. Let (X(r), t e R+] be a Markov process with countable state space S and
tanstition probability function p ^ t ), / yj E 5, t e R+.
We assume the following axioms about transition function Pij(t)\
1. 0 < py{t) < 1
2. ^>s Pij(t) = 1 for all / = 1, 2, . . . and t > 0.
3. Chapman-Kolmogorov equation (or semigroup property)
Pu (t + h) = Zsp ik( t)pk(j
4. p,j(0) = S,j
5. lim Pij (t ) = <5<y (makes Markov process standard or regular)
Note that by axioms 1 to 5 {/?jj(0} are right continuous functions at t = 0 and

in terms of Transition (Markov) matrix, P(t) = (p//(0)» the above axioms can be
rewritten as
(1) 1 >P( t)>0
where 1 and 0 are matrices with all elements 1 and 0, respectively
(4) P(0) = I
(5) lim P(t) = I.
do
Suppose now P(s) for 0 < s < his known. For each t we can find an integer
n such that nh <t < (n +1 )h. T h e n
P(t) = P(nh + t - nh) =P{nh) P(t - nh) (From (3))
= P(h + h + . . . + h) P(t - nh)
n times
= P(h) P(h) . . . P(h) P(t - nh) = Pn (h) P(t - nh).

We want to define pij(t) at least for t = 0. It will actually be shown that
owing to the rich additional structure reflected in (3), continuity implies that P(t)
is in fact differentiable in /, i.e. p'j (t ) = dptj (t)/dt exists for all pair (i,j) e S
and all t > 0. At t - 0, of course “derivative” refers to right hand derivative. Let
us first prove continuity of p^t) for all r > 0 and i, j G S.
Lamma 6.1 P(t) is continuous for t > 0 (6.1)
Proof From (3) P(t + s) = P(t) P(s), t, s > 0.
If 5 = h > 0, then we have
lim P(t + h) = P(t) lim P(h) = P(t)I = P(t)
/i— >0+ /i—
>o+
i.e. P(t) is right continuous.
If t > 0 and 0 < h < t
P(t) = P(t - h) P(h) (6.2)
But || P(h) - / || < € if | h | < S [by axiom (5)].
Hence for | h | < 5, P~l(h) exists and tends to / as h —>0.
Therefore P(t) = P{t) lim P~x(h) = lim (P(t)P~l (h))

/i—
>o+ /?->o+
= lim (P(t - h)P(h)P~l (h)) [by (6.2)]

h-+Q+
= lim P(t - h).
h-+o+
Thus, P(t) is left continuous and hence P(t) is continuous.
6.1 Differentiability of Transition Function

From Chapman-Kolmogorov’s equation
Continuous Time Discrete State Markov Processes 109
P(t + s) = P(t). Pis)

or P it + s)- Pit) = Pit) Pis) - Pit)
or P it + s)- Pit) = P(t)[P(s) - /]
lira^ = lim W M
A'-lO ^ A-iO $
=> P '( 0 = P (0 lim — - if lim ■— — - exists

a I o a4 o
Lemma 6.2 Pu(t) > 0 for all r > 0 and i e S.

Proof pu(0) = 1 > 0.
If t io, Pait) —> 1 continuously by (5) and (6.1).
For some
t0 > 0, pu (t) > 0 for 0 < t < t0 (a)
(by continuity of Pl7(f)).
Choose n such that
H n)r'° (b)
Pait) = Zp iji tln )P j
^ Pu Pu
By induction (or repetitions) on n > Pu I - I I > 0 by (a) and (b).
Theorem 6.1 (a) lim -— = qu exists but may be °°

tio t
= ~Pu( 0).
Pi -—
(b) rlim — j M = qij < oo if
.r .
i^ .
rio t
(c) For every state i of a finite state Markov process, qit is finite and is equal
to I q,j.
(d) For all i e 5, Z q < qu.
Proof (a) We shall first see that for every i e 5,
1 - Pu( t )
-Pu (0) = Jim
t
By semigroup property pu(t + s) > pu(t) pu (s).
Taking log on both sides and writing - log pu{t) = 0(/)
(this is well defined since pu(t) > 0 for t > 0), we get
0(t + s)< 0(t) + 0(s) (6.3)
Since 0 < pn(t) < 1, 0(f) > 0.
iLet♦ qu = sup ------
0 ( 0 . Then
tu 0n <
^ qu <
^ <*>.
t> o t
If qlt < oo, there exists f0 > 0 such that 0(fo)/fo > qi{ - £.
By (6.3)
0(fo) < 0(nt) + 0( 8) < 0((n - l)f) + 0(f) + 0 (5 ) < <n0(t) + 0 (5 )
if f0 = nt + 5, 0 < 5 < f, n = 1, 2, . . .
Hence 0 ( * o ) ^ /i0 (O + 0 ( 3 )
{o 10
0 (5 )
=^0(t)/t +
to to
m 0 (f) 0 (5 )
Therefore qu - e< lim (6.4)
rro h ' t + t0
nf
But f —> 0 =>------> 1 and 0 (5) —» 0 (by axiom (5) and pn(5) —> 1 as 5 —>0).
to
nt_ 0 ( f ) 0 (5 )
Therefore lim 0 (f)/f = lim (6.5)
t —>0 / —>0 L
f0 f fo
Hence by definition of qih

r
0(0 .
lim — < qti ( 6 .6 )
r-»0 f
By (6.4), (6.5) and (6.6), we have
qu - £ < lim 0 ( f) /f < lim = qi(

o r->o f
Since £ > 0 is arbitrary, we have
— 0 (0 ,. 0 (0
r->0
If qu = oo, we can replace qu - £ by an arbitrary large constant M and then
obtain M < lim ~ t h u s «> = lim (0 (f)/f).

r->0 ^ rTo
T • U case we U
In either have 1lim
- -------
0 ( 0 ^< qu.
t->o t ^ 11
Now
f->0 f /->0 0 (f) f ™
(since 0(f) —> 0 as f —> 0, by L-Hospital’s rule we get the last equation).
(b) Let c E (1/2, 1) be fixed. Then by lemma (6.2), there exists an 5 > 0 sufficiently
small so that pH(s) > c and pt]{s) > c. Let 0 < s < nt < <5. Consider a stationary
discrete parameter M.C. {Xn }q with state space S such that
P(Xn+I = j \ X n = i)
Then for all n>
1, we have
Point) = P(X„ = j \ X 0 =i) > P(Xl = j, X„=j \ Xo i)
+ I P(X, * j , . Xr_, * j , Xr = Xr+i

r-1
n- 1
CPij (0 l + I P( XX x r_! ± j , x r = i \ x 0 =i )
Since each term on the right corresponds to a possible way of going from i to
j in H-steps (relative to (py-(f)) ) and these paths are mutually exculsive but not
necessarily exhaustive.
On the other hand,
P(X{ *j , . . . ,X r_! * j , X r = i \ X 0 =i)
= P(X, *j , Xr_2* j, Xri) - *j,
*j , Xr_, = y, X r= iIX0
= P{Xx *y, . . . , Xr_3 * j, X, = I X0 = i) - *y, . . . , Xr_3
^ y\ Xf_2 = y, Xr = i|Xq = i) —PCX] ^ y , ..
X. = / 1X0 = 0.
and so on. Continuing in similar way we obtain
P(Xl * j , . . . , X r_l * j , X r = i \ X 0 =l )
P( Xr = i \ X 0 = i ) - l P( Xx* j ,...,X * _ , *y,X*
k<r
= i|X* = y ) > c - ( l - c) X />(X, * y ,...X * _ , * j X0

k<r
= / | > 2c - 1.
Consequently,
Pij(nt) > (2c - 1)npij(t). Let h < 8, t < 8, and n = [h/t],
where [_*] is the interger part of jc; then, Pij(t)lt < Pij([h/t]t)/{[h/t]t0(2c - 1)}.
Letting, t —> 0+, we obtain
lim sup (Pij(t)/t ) < Py(/i) {/ic(2c - 1)} < ©o for all r > 0.
Therefore,
lbn sup (p i; (f)/r) < Hm in f(/^ (/i)/{/i(2c - l)c},
which means that the limit qtj = lim (Pij(t)lt) < ©o exists.
N
(c) We shall prove <7,,<°° for a finite state process. Now 1 = /?,,(/i) + Z Pu(h ),
7 =1 7
j*i
N
where N is the number of states, or 1 - (h) = Z p u (h).
7*‘ 7
Dividing by h and letting h —>0 and by part (a)
N
<lU = y?i ^ < "•

j*i
(d) Since Z p u ( h ) = 1 - Pu(h), Z < 1 - pu(h) for every finite number

j*i 7 j*i 7
/V of states.
N
Dividing both sides by h and leting h —» 0 leads to Z < qi{.
Since N is arbitrary and all terms are positive, the result follows. The matrix
Q = (qij) is called the generator of the Markov process.
Example 6.1 A birth and death precess has state space S = {0, 1, 2, . . .} and
its generater Q (for definition see Section 6.3) has elements
A, if / = / + 1
qu = + Mi and qij = 0 if j * i - 1 or i + 1, i = 0, 1, 2,. . .
Pi if j = i - 1
A,- and are called birth and death rates, respectively.

Definition 6.1 A state i e S is called instantaneous if qu = °o and is called stable
if 0 < qu < oo. A state i E S is called absorbing if = 0.
Exercise 6.1 Show that qu = 0 iff pu(t) = 1.
We shall see that Kolmogorov backward equation P\t) = AP(t) gives a solution
p(t) = eAt (Section 6.3) from which it follows that P{X(r) = i , 0<z<t \ X( Q) = i} =
e~qiit and is equal to 1 for alD > 0 if qu = 0 and this shows that pu(t) = 1. We shall
also derive this as distribution of waiting time of the initial state i.
Definition 6.2 A continuous time standard Markov process is called conservative
(nonexplosive) if Z qu = qu < for all i G S.
j*i 7
Note that a birth and death process is conservative, since Z qu = A, + p t =
j*i
qu < oo.
Being a pure birth process a Poisson process is also conservative.
6.2 Kolmogorov Differential Difference Equation

Theorem 6.2 In a conservative Markov process, the transition probability
functions Pij(t) are differentiable for all i, j, e S and t > 0 and they satisfy
Kolmogorov’s differential equations :
(i) Backward Equation
Pii V) = 2. qikpkj (?) - (?) for all i j 0.
(ii) Forword Equation
Pa V) = q k j P ki( ? ) - qjjP ij ( for all e 0
Proof (i) pij(t + s ) - Pij (?) = Zs pik(s)pkj( - (?)
= Pik(s)Pkj (0 + (a ) Pij (?) - Pa (?)
= 2. pik(s)pkj (?) - [1 - (t
„ . V Pik ( s ) P k j ( 0 ~
Claim hm Z =Z (?) (6.7)
. . Z P ik(s) , v
hm inf Z — -— />*, (?) > Z < 7 ,^ (?) ( 6 . 8)
viO k*i S J k±i J
for every positive integer N (by Theorem 6.1 (b))
and hence 2* qtkPki \t) < hm inf Z (6.9)

k* i J 5-fo1« ^ =*
N
Take N > k,then ^ p ik{s)pk](?) = ^ p lk(s)pkj (l) +
(?)
- ^ 0 Pik(s)Pkj(t)+
N f N >
= k2= 0 Pik i s) P k j (?) + 1 - A:=l
Z Pik (s)
V
/V
= £ 0 Pik(s)Pkj(t) + 1 - / > „ (•*) - z
2 : P«* ( i ) -
Hence Z kjJ)< £
Pjk^lP Pu^Pkj + 1 ~ P »(s) _
5 ~~ k * i S S k*i S
TV, f r V ik(s)jt ) £
P / N £
Therefore hm sup Z ------- -------- < Z <7,*p*; (r) + qn - Z qtk
v io J k±i
(by Theorems 6.1(a), and 6.1(b)).

Now letting N 00 (since the M.P. is conservative),
A“ sup* ? .~ —
1 P kjM i& qikP kjif)
( 6 . 10)
(the last two terms cancel).

Combining (6.9) and (6.10), the backward equation follows from
.. Z Pik(s) ., , , , .. ■ t Z P i j ( S') , ,
lim sup Z. — -— < Z jQ)^ lim inf
ikP
.vio 5 J k*i J S
I oo OO
since lim - 2 Pik(s)pki(t) exists and is equal to 2 qikPkiiO-

.slO *5 *** **/'
(ii) p„ +0 - ij s() = Zs p,*
P
= I Pik(s)[Pkj(t)-
and hence the result (ii) can be proved similarly as (i) after dividing by t and
letting t -> 0.
Since the scope and validity of the forward equation is rather involved we are
not discussing any more details of the proof.
6.3 Infinitesimal Generators and Examples

Definition 6.3 The infinitesimal generator or matrix or transition rates of a
Markov processes is the matrix A = {ai}) defined by
( hj* } \
<
aU ~ r •
\-<hi if* =JJ
The important problem is, however, to construct transition probabilities p^t)
having prescribed infinitesimal matrix A satisfying atJ > 0 for all /, j e S, i = j
and
-an = 2 atj (6.11)
Now Kolmogorov’s backward and forward equations can be written in the

matrix form as
P\t) = AP(t) and P\t) = P(t) A respectively (6.12)
In case that state space S is finite it is known from the theory of ordinary
differential equations that, subject to the initial condition P(0) = /, the unique
solution of (6.12) is given by P(t) = etA, t > 0, where the matrix etA is defined by
= 2 (tA)n/n\ = I+ 2 t nA nln\ (6.13)

n =0 n=1
Suppose that A is an (N + 1) x (/V + 1) matrix with eigenvalues Aq, A], Xn,

possibly with repetitions, and corresponding eigen vectors U(0\ f / (1), . . . , f /(N)
which are linearly independent. Let
U = (U(0\ U(l\ . . . , U{N)). Then by orthogonality
A U = a 0U {0\ A ,f/(1), . . . , XNU(N))
A 2U = A(A0£/(0), A1f /(1),. .. ?in U{N) ) = (A02f /(0>, . . ., )
A nU = (A0" f/(0), A{*t/(1), . . ., Xn
NU{N)).
Hence etAU = I t n/ n\ An U= I, t " / n U 5 U (0),. . . , A nNU(N))

n= 0
- ( e ‘*o ( j ( 0 ) ' e tM y { \ ) ^ e t&N £/(W) )
= U A (I),
0
where a (0 =
Therefore e,A = U a (t) U~'.

Hence the method of spectral decomposition gives the unique solution of
N
(6.12)as P(t) = S exp(A70 ^ (;)y (;) ), where and are the correspon-
7=0 J
ding right and left eigen vectors of A.
In case the state space S is countably infinite, results analogous to those for
the finite case can be obtained under the following restrictive condition:
a = sup | atJ | < oo (6.14)
ij
In view of condition (6.11), condition (6.14) is equivalent to
sup | an | < oo (6.15)
i
The solution P(t) is still given by (6.13), i.e. for each j e S,
Pij (t) = S ij+ tdjj+t 2/ 21 aj f t nln\
for all i e S, where a'"’ are the elements of An, and A = (a,;j-
Proof of (6.16) By (6.14), (6.11) and by induction on n, we have
| a-p | < a (2a )n~x, for i,j e S', n > 1 (6.17)
Thus, the series on right side of (6.16) converges to an analytic function /i,y(r),
say, absolutely for all t. Hence term-by-term differentiation of this series for
lift) one can easily verifies Kolmogorov’s backward equation with initial conditions
p,;/(0) = 0 for i * j and pn(0) = 1.
To verify the Chapman-Kolmogorov equations, note that
P(t + s)= e0+s)A = I + ( t + s)A +--- t — 2- + ...+

2! n\
I+tA +—j A 2+ . . . +t2/n\An +
I +SA + “9?T A 2 + . . .+±~rAm+

ml
= etA<?A = P(t)P(s)
which follows from the identity
m+n=v ml n\ ym+n=v mini J v\
Also, the functions Pt (t) = Z Pa(t) satisfy

jeS J
p‘Xt) = h p 'i iit) = h £ s a‘kPkJ(t)

= g s aikPk (t),t> 0
with initial conditions P, (0) = I <5(,- = 1 for i e S. Since Pt (t) = 1 for, all t > 0,
j eS J
all i e S clearly satisfy these equations, we have Pt (t) = 1 for all t by uniqueness
of such solutions. Solution (6.17) have been shown to satisfy all conditions for
being transition functions except for non-negativity which will follow from the
following more general result.
Theorem 6.3 Given any infinitesimal transition matrix
A = (ciij) satisfying - a /; = Z <zt/, atj > 0 for i * j
there exists a smallest nonnegative solution P(t) of the backward equations
Pi' (t ) = ’L a i kPkJ(t),i,jeS(6.18)
with initial conditions ptj{0) = 8ip i, j € S.

This solution satisfies £ pa (t) < 1 for all z, e S, t > 0.
jeSr J
Moreover, when the equality holds in the last inequality for all i 6 S and t > 0,
the solution is unique.
Proof Write A, = - au for all i e S.
Multiplying both sides of backward equations p[j(t) = Z aikpkJ{t) by the
“integrating factor” eîSwe get
e X‘sp'ik (s) = eAiI Zs (s)
°r l s ('eX ip‘k= ^ i eXiSPik(s) + eXiS 2 a,jp]k (s) = 2 (s)
On integrating between 0 and t we have, remembering that pik(0) = Slk,
e Xi'PikU) = S ik + 2 f ,
J*' Jo
or pik(t) = 8,ke~Xi' +Z j*e~x' (,~s)ai(jp for 0,

j *1 Jo
Reversing the steps shows that (6.18) together with (6.19) follow from (6.20).
Thus (6.18) and (6.19) are equivalent to the system of integral equations (6.20).
To solve (6.20) start with the first approximation

Pik (0) = for i, k e S and t > 0
and compute successive approximations, recursively, by
p\nk \ t ) = 5,ke - x>' + 2 f (6.21)

3*l J0
Since a tj > 0, i ± 7, it is clear that p$ (t) > pf°}(t ).
Then from (6.21) by induction it follows that pj/1+1)(0 ^ P& \t) for all n > 0.
Thus p ik (t) = lim p \ ^ (t) exists.
6.4 Birth and Death Processes

Definition 1 A birth and death process is a conservative M.P. with infinitesimal
generator
-A q A0 0 0 0 \
Mi —(Aj + P i) Ai 0 0
0 M2 -(M2 + a 2 ) A2 0
v : : : : J
where A, and P p i - 0, 1, 2, ... ,7 = 1, 2, 3 ... are the birth and death rates,
respectively.
Definition 2 (Karlin) A birth and death process is a continuous time Markov
process on the state space S = {0, 1, 2, . . .} with stationary transition functions
P i j ( t ) = P(X(t + 5)) = 7 | X (5) = i) and such that
(i)PlJ(0) = S0
(ii)Pu+\(h) = Xjn + o(h) for i > 0 and h i 0
(iii)Pu-\(h) = pth -v o(h) for i > 0 and h i 0
(iv) Pn(h) = 1 - (At- + Pi) h + o(h) for i > 0 and hiO and
P i j ( h ) =o(h) for7 * 1 + 1, i - 1, i as h i 0 and all i > 1.
(v) A/ > 0 for i > 0 and p t > 0 for / > 1 and p$ = 0 are the birth and death rates,
respectively. A, and p( are also called parameters of birth and death processes.
Note These two definitions can be proved to be identical.
The backward Kolmogorov equation will be given by
Pij ( 0 = MiPi-ij(0 - (Ay + Pi )pu (t) + A,/?/+1 (0 , i > 1
and p',y(/) = - A0p„y ( 0 + A0p i;-(O with initial conditions p,y(0) = <5^.
The forward Kolmogorov equation for birth and death processes are
Pi j ( 0 = Ay-l Pi ,7-1 ( 0 - (Ay + Pj )Pij (t) + /iy+iPj-,y+i (0 ,7 ^ 1
and Pi'«(0 = -A 0p iV>(0 + P1P/1 (0 with initial conditions pt/ 0 ) = <5,,.

Proof For birth and death processes, Chapman-Kolmogorov equation gives
Pij 0 + h) = pik ( h) pkj(t) = Pi, ,-i

*
+ Pa (h)Pij (t) + Pi,j+\ (h)P

\j ?) +
i+
*
where £ is over all k * i - 1, i, i + 1. Using axioms (ii), (iii) and (iv) we obtain
k
* *
^ Pi k ( h) pkl (t) 0.
Hence ptf t + h) = h p ^ / t ) + (1 - (A, + ^,)/i) + A,/i + o(h).
Transposing the term py(f) to the left hand side and dividing the equation by
h, we obtain (after letting /i —> 0 the backward Kolmogorov’s equation)
Pij(t)= -iP
\j(t) (A, + pdpijit) + Xi pMJ (t
6.5 The Yule Process

This a pure birth process (that arises in physics and biology) with parameters
A, = (%N + i) and jit = 0, ft > 0. Kolmogorov’s backward equation becomes
(putting i =n)
PN+n (t ) = + n ) ^ +„ (/) + 0( W + n - 1) pN+n-i (0, /I > 1 (6.22)
and p'N(t) = ~/ 3NpN(t)
where pN+n(t) = pN,N+n(t) = P(X(t) = N + fl/x(0) = AO-
If /V = 1, (6.22) becomes
Pn+i(0 -P(n + l)p„+i(0 + (3npn(t) (6.23)
with initial conditions p x (0) = 1, pn(0) = 0, n = 2, 3, . . . .
By induction, pf(r) = - f i p\ ( t ) implies p\(t) = and initial conditions
give the solution of (2) as
pn(t) = e~pt(l - e ^ ') ”"1, n >0 .
To get a general solution of (6.22) (i.e. when N > 1) we shall first express the
Kolmogorov’s differential equation for the Yule process as a partial differential
equation in terms of the G.F. of the process.
Let G(t, z) = £ PN+n (0 z N+n »| z | < 1, n > 1. Then it is uniformly COnver-
Ô
gent and
^ = 2 , p'N+n(t)zN+n=I., [-P(N+n)pN+n(t)+P(N + n-
Ot n=0 n=0
= -p %A N + n) pN+n(t)zN+n+ p l ( N + n - l ) PN+n_i(t)zN+n
n-0 n=0
—~pz l ( N + n ) p N+n« ) z N+n-'+Pz2 2 ( N + m ) p N+m(t) zN+m-\ni =n-l)

n- 0 m- 0
,SG a 2 SG
8z
Therefore, = -/3z(l - z)-~— or + j3z(l - z)-~—= 0 with boundary

ot oz ot
conditions (1) PN+„(0) = SNiN+n.
(2) I ' P N+n(0)zN+n =G (0,z). (6.24)
Now u ( j c , y , z ) ^ + v(x,y, z ) ^ = 0 ( x , y , z)
represents a Recatti type of partial differential equation (p.d.e.).

To solve Recatti type p.d.e. we find the so called ch. equations (susidiary
equations) given by
dx _ dy _ dz
u ( x , y , z) ~ v ( x , y 9z ) ~ 0 ( x , y 9z)'
The fundamental property of Recatti equations is that the general solution is
homographic function of constant of integration. Hence
^ = /i + cf.
In fact their solution comprises two one-parameter families of surfaces which
can be denoted by
P(x, y, z) = cj and Q(x, y, z) = c2-
Here in our Yule process
u(x, y ,z) = 1 and
z = G,x = t,y = z, 0 ( x 9y 9z) = 0;
u(x,y,z) = /3z(l - z).
Thereofore
dt _ dz dG dG dz
G(f, z) = c \ , (} dt =
T “ p z(l - z) " “F dt T =o z( 1 ~ z)
or ^ = j3z(l - z) => Pt = c2 -
Elemenating one of the constants Ci and c2 we get the general solution given
by G(f, z) = Z■ wbere y is an unknown function to be determined
from initial conditions.

The initial condition gives G(0, z) = z N = y
Put z = or u = —5—. Then
M- 1 Z- 1
N
y/(u) . Substituting u - g & we get
G(t, z) = (6.25)
1 - z ( l
For N = i, coefficient of z n in (6.25) gives pn(t) = e~^'{\ -

Theorem 6.4 Let Txbe the time that the conservative M.P. stays in the state i.
Then F(t) = P|T, < f] = 1 - e~Qiit. Note Tt is called holding time at state i.
Proof F(t) = P[Ti >t] = P[X(s) = i , 0 < s < t \ X(0) = i]
(by separability of stochastic process, since a conservative continuous parameter

M.P. is separable; see Doob (or appendix I).
To see why the last equality holds, define An = jx j^ /j = /,/: = 0, 1 ,... nj.
i 00
Then An Tin n and An = lim A = n An = {X(^) = i for all s in [0, /] which is
n—>°° n=\
a rational multiple of t] = {Tt > r)^ c A.
Since the sample paths are step functions, if a sample path is not in {Tx > t]
then there occurs a jump to state y, different from /, at some r0(0 < t0 <t). The
case t0 = t may be excluded, since it is not in A. Because each sample path is
right-continuous step function, there is a time point tx > t0 such that X(s) = j for
tQ<s< ti. Since there is some 5 of the form s = tk/n < t in every nondegenerate
interval, it follows that X(^) = j for some s of the form (k/n)t < t\ this implies that
this sample path is not in An and hence, not in A. Therefore, {Tx > i ) d A .
Hence {Tx> t] = A.
f n- t ^ (t\
F(t) =limX(t) = i x ------ = ; , . . . , X - = i, X(0) = i
n—
>°° V « J Vn J
x P X(0) = i
n- 1
= lim P X(t) = i /,. . . | X(0) = i
n—>°° n
(by Markov property)
= lim (by induction)

n—
-in
= lim ---- qH + o
n—>°° n
lim-— P"Sl.l -q.. (by Theorem 6.1 )=>/?„ (t)=l-tqii+o(t) as t >1 0

/loo t
=> F(t) = e ' tqii => F(t) = 1 - e~tqn .

Note If qu = 0, then F (rt > / 1X (0) = i) = e° = 1 for all t > 0.
Hence P(Ti = oo | x (0) = i) = 1, i.e. the state i is absorbing.
6.6 Some Population Models
6.6.1 Non-homogeneous birth and death process

Here we make the following assumptions regarding birth and death processes.
A .l: During small interval of time (t,t + A f), the probability that a birth occurs
is Xn(t)At + o(At). The probability of no birth in (t, t + At) is 1 - Xn(t) At + o(t)
and probability of more than one birth occurring is o(At). Births occurring in
(f, t + At) are independent of time since the last occurrence.
A.2: The probability that a death will occur in small interval of time (t, t + At)
is pn(t)At + o(At). The probability of no death is 1 - jJLn(t) At + o(At) and the
probability of more than one death is o(At).
A.3: n = 0 is an absorbing state of the process.
A.4: For the same population size, the birth and death occur independently of
each other.
Let X(t) be the population size at time f(0 < t < <»).
Define pnN(t) = P[X(t) = n | X(0) = M]. Then the Chapman-Kolmogorov’s
equation reduces to
P n flt + A t ) = [1 - (A„(/) + p „ ( t ) ) A t ] p n+ [A„_](0 A t + o (A t)}
[1 - + o( + [pn+l(t)A
[1 - + o(At + )] ) (« > !)
Neglecting the terms of o(At)
pnN(t + At) = [1 - (A„(r) + pn(t))At]p+ A„_i N(t)
+ Hn+i(t)Atp„+lN(t) + (n > 1).
Since ‘0’ is an absorbing state, p^{t)= 0 and we get
p0N(t + At) = (1 - Ao(t)At)pON(t) +
Taking limits as At —» 0, we get
S P nN (t)
St :
(n > 1) (6.26)
SN
Pt()
O
^ ---- = -Ao(t)POiV(0 + (6.27)
w'th initial condition 0) the Kroncker’s delta.

Let G(z, t) = l p nN(t)zn, | z | < 1 be the generating function of {pnN(t)}.

Multiplying (6.26) by zn and summing over n we get
(<5/<S0| 2\QpnN{t)zn I = \ Zn [p„N

n=0 + Hn (t))
+ I ' Zn p „ - w ( t ) A n - ! ( t ) +
n=0 n=0
SG(z, t)/St = - Z z"A„ ( 0 p nyv(0 - 2 (t)p„N (0

n=l n=0
oo oo
+ £ I, z nAn(t)pnN(t) + z~'
n=0 /i =0
= 2 z nPnN {(t) (z - 1)A„

n=0
with initial condition G{z, 0) = z N if P(X(0) = N) = 1.

In the simple linear birth and death process (Generalized B-D processes : see
Kendall-1948) An(t) = nX(t) and jun(t) = n/u(t) and (6.28) becomes
SG(z, 0/5/ = 5G(z, 0/& {(z - 1)UA(0 - MO]} (6.29)
The auxiliary equation of (6.29) is
dzldt+(z-l)[zM t)-V(t)] = 0 (6.30)
which can be obtained from the characteristic equation
dt/l = dzJ{{ 1 - z)[zM0 - MOD = dGI0.
Making a change of variable 1/w = 1 - z, (6.30) can be written as
duldt - [A(0 - p(t)]u + A(0 = 0
which now involves a linear function of u. The integrating factor for the above
equation is eA{t\ where A(/) = - I [A(t) - ji{T)\dt.
rx->| , J0
Then we have
du
[A(0 - p(t)]ueA(t) + A (/)eA(0 = 0
dt
or d/dt[ueA^] + A(0 eA(/) = 0.
Integrating this and replacing u by 1/1-z, we get
e A('V(l - z) + f X(r)eA(t)dr = a constant C2.

Jo
Another equation we get from the characteristic equation is G(z, t) = C\.
The general solution can be written as
G (z,t)= f
1- z f
Jo
A(r)eA(t)dr
If P(X(0) = AO = I, then Giz, 0) = / ( l / l - | z < 1.

Writing 0 =1/1 - z or z = 1/1 - 0,/(0) (1- 1/ O f if|1 - 1 /0 1< 1.
The required G.F. is
Giz, t)= [1 - l/{eA(,)/(l - z)+ f A(T)e*(,)dT}

Jo
1N
r' ")
( e A (,) + J |* A (r)e A(T)< /r- 1) +
H Air)e*(T)dr
'o J
+ J|* Air ) e ^ T)dI* Air)eA(T)dr

Jo
r r l
Letting a(t) = 1 - + eh w X{T)dr and /3(f) = 1 - eA(T)[ - a it)], the
Jo
G.F. can be written as
ait) +[1 - « ( f ) - m u
Giz, t) = (6.31)
1 - /j(0z
Let us first calculate pnit) = P[Xit) = n | X(0) = 1] in the simpler case when
An(t) = nA(t) and p„it)= njuit) and population start with one memb
Then the Kolmogorov’s equation becomes
dPnif)
=in + l)p it) p n+1it)+in-l)Ait)pn-iit)-niMt)+Pit))Pnit), « ^ 1
dt
and -p (t) pl it). Equation (6.29) becomes

dt
dGiz,t) dGiz,t)
\ i z - DlzAit) - pit)]}
dt dz
with initial conditions Giz, 0) = The auxiliary equation is
— = -Pit) + iMt) + H(t))z - Ait)z2 .
The fundamental property of this Recatti equation is that the general solution
is a homographic function of the constant of integration, so that z = /i + cf i

h + CU
r r
and equivalently c2 = c = where / b / 2, / 3 and/4 are all functions of
h - z/4
time t. Thus, the general solution is of the form G ( z , t ) = y / j y ---- j j - j^ and from
the boundary condition y { z and PutinS “ =

f i (0) - z /4 (0)
/ , ( 0 ) + / 2(0)«
w e get z =
h (0) + / t (0) u
Hence
/ i ( 0 ) + / 2(0) z h -/■
G(z,t) = / 2 - Z /4 g\
zh -f\ gs(t) + zgi(t)
/ 3( 0 ) + /4(0)
h
where g2,g2 and g4are functions of t.
On expansion, we get
Po(t) =a(t) and pn(t)= {1- Po((t)}1 - (« ^ D
where a (t) and fi(t) are functions of t. Thus for generalized birth death process
the population size at any time t is distributed in a geometric distribution with a
modified zero term.
In the generalized birth and death process starting with N individuals the
probability of extinction is given by
1N
1
PON (0 “
eA<'> + J A ( r ) e A(T)d r
Now
eA(t) + f
Jo
A ( r) e A(r)d r = eAU) + f
Jo
p(z)e*{T)dz - f
Jo
e A(T)[u(z) - A(z)]dz
= eA(0 + f
Jo
p( z) e * {T)d z - [eA(T)Y0 = 1 + f
Jo
/ u (z) e^T)dz.
f
Jo
jn(z)eMT)dz
Hence, Pon(0 =
1+ p ( z ) e MT)dz
Jo
We observe that chance of extinction tends to one as t —» °o iff
lim f
'-*°° Jo
p ( z ) e A{T)dz = °o.
The first two moments of X (t) are given by

m(t) = Nem
a 2 (t) = Ne~2Mn f [A(r) + p(z)\e*(T)dz.

Jo
In Arley process (1943) the birth rate A(f) = A is a constant, death rate is linear
p{t) = jut and A(0 is given by
A( 0 - fJo [p(r) - A(t)]dr = 1/2 -At.

Also the mean and variance functions are given by
m ( t ) = exp[Ar(-/xf2)/2], cr2(0 = m ( t ) [1 - m { t ) + 2Am(0]2 f

Jo
dx/m{x).
Since lim I jit exp [lit2/2 - Xr\dr = «>, the extinction is certain in Arley
'->°° Jo
process. There are four important special forms for birth and death rates. If
Xn = a, a constant we have immigration at rate a. If Xn = nX, we have simple
linear birth process with birth rate X per individual. If jin = /? and = 0, we
have emigration at rate /}. If ^ = n/i we have linear death process with rate fi
per individual.
6.6.2 Linear birth and death process

If Xn(t) = nX and iin(t) = n/t in the generalized birth and death process, proceeding
as before, we get from equation (6.31) the P.G.F.
-1N
a(t) + (1 - a(t) - p(t))z
G(z, t) = where
1- P
(6.32)
jx( 1 - e {X- ^ ‘ )
a(t) = and P(t) = (X/ii)a(t)
The general solution pnN{t) is the coefficient of zn in the expansion of G(z, t).
Now {a(t) +[l-a(t)-P(t)]zY [(a(t))N- j [ l - a ( t ) - p(t)Y z J]
(6.33)
f-A A 'N +i- r
Since | fi(t)z | < 1 and = (-D '
' J V ' >
AA N + i-V
[1 - p(t)z]-N= I (-l y m t u y = z [P(t)Yz‘ (6.34)
1=0
^ *i / )=0
Using equations (6.33) and (6.34), we get

min(V,Ai)^ N Y A
P„N ( t ) = I
7=0
W A n ~J
1)
and p(),v (?) = [a(?)]‘v, the probability of extinction of the process. Now from
(6.26) the asymptotic behaviour of the probability of extinction PovM depend on
the parameters p and A. Hence
1 if p >A
lim pon V) = (6.35)
( p /A ) N <A
From (6.32) p0N(t) = [ a ( r ) f = G(0, t). Hence for | z 1
1 if ju>A
lim G(0, t) =
/—»°o
(6.36)
(ju/A)N if (jl < A
Equation (6.35) shows that it is certain that the population dies out when the
death rate exceeds the birth rate, but when the birth rate is greater than the death
rate, the probability of eventual extinction is equal to (plA)N. From (6.36), since
the limiting P.G.F. is constant, the probability that the population will increase
without bound is 1 - (jn/A)N. The relative magnitude of A and ju also influences
the asymptotic value of the expectation and variance of X(t).
The moments can be calculated for linear birth and death processes by taking
derivative of G (z, t) at z = 1. We get
m(t) = EX(t) = n \ — ~ = ,
1
N[ 1 - a(Q ][g(f) + Pit)]

a 2(X(t))
[1 ~ P ( t ) 2]
0 if H > X
Hence lim m(t) = N if X = n
/ — >oo
OO if A > n
if /i > A
and lim
/—>oo
a 2{X{t=
if ju < A
When A = jl we have an interesting case in which though the probability of
extinction tends to one, yet the expected population size tends to N. Most population
will eventually become extinct while a few others will attain huge sizes, so that
the average size is N.
6.6.3 Effect of imigration

The Kendall Process
If with a birth and death process we take into account immigration, we have to
assume that during an interval t, there is a probability y&t + o(Ar) of a new
individual being added to the population from outside the country, we have a
birth and death process for which An = nk + y and fin = n/Li(n> 0), the forward
equations being
P«(0 = + AO + Y)Pn(t) + [A(n - 1) + y ]p n-i(0 + ji{n + l)pn+i(0 0)
with initial conditionsp0(t) = 0 ,pn(0) = 0 forn > 1, i.e. X(0) = 0 a.s. LikeFeller-
Arley process the P.G.F. G(z, t) satisfies the partial differential equation
~ ( z - l ) ( A z - / i ) |^ = y (z -l)G
The auxiliary equation is

dt dz dG
1 ~(z - l)(Az - p) y (z -l)G
dz
Now = - ( z - l)(Az - ju) and hence, we get
(u - X)t = log —— — + constant or * ± - J ± e (tl-X)‘ =C2 (6.37)

M z -/i
From
dz dG
we get G(z, r)(Az - / x ) ^ = C, (6.38)
(Az - /x)(z - 1) y(z - 1)G
or G(z, = C,(Az -
Az - /x ,(M-A)f
From (6.37) and (6.38) G = 0 (Az - p) y/A for some unknown
V
z “- A1 /
function 0 to be determined from initial conditions.
Initial conditions give G (z, 0) = 1.
Az - p
Hence (Az - n) -ya = 1.
z- 1
„ - Az - /LI . 0- p
Put 6 = ------ — i.e. z = ■, then we have
z- 1 e -A
y/A
0(0) = e ^ a -jo ^ e -A )-^
Now replacing 0 bv —— we get the general solution

z- 1
-,y/A n -y/A
^ _ A z_ _ £ e(p-x)t
G(z, = AiZiLtM -A)/ ( A - ; i ) r/A ( A z - p ) y/A.
z- 1 z- 1
Simplifying the above expression we get the generating function G(z, 0 as

y/A
G(z, 0 = P (6.39)
1 - qz
(A - /x)/(Ae(A- ^ -
where P= •(6.40)
(1 + At) * if A = /x
(applying L’ Hospital’s rule)
and q = I - p .
From (6.39) we see that X(t) has a negative binomial distribution. As
(6.40) shows that p -» 0, <7 —> 1 if A > p, while p —> 1 - A/p, g —> A/p if A < p.
Therefore, if A > ju, no limiting distribution exists for X(t) as t —» <*>, i.e.
X(t) —> oo with probability one. If X < ju we find from (6.39) that
lim G(z, t) - (6.41)
which gives in particular /lim/?0(O

—>oo
= (1 - A/jU)y/A < 1 (constant term in the
expansion (6.41) which shows that there is a positive probability that the population
avoids ultimate extinction).
The mean number of individuals is given by
^ |_ i =( e U - ^ ' - i ) + e < ^ > ' f0 r A * «

oz “ A - jU
= yt + 1 for A = // (apply UHospital’s rule)
7
Therefore, lim m(t) = ------ r- if A * u.
t->oo // - A
These formulae relate to the stable distribution of population size which
immigration can just maintain against the excess of jU over A. It is clear from
(6.39) that the distribution will still be negative binomial in form for every finite
value of t when fj. = 0, the process being one of immigration and reproduction
only. On the other hand, when A = 0 so that the process is one of immigration and
mortality (or emmigration) only the distribution assume a Poisson form.

Exercise 6.1 Let {X(0, />0} be a stationary Markov process with state space 5 = [-1, 1),
an initial distribution n( 1) = P(X(0) = 1) = 1/2 and a transition function ptj(t) satisfying
Pu (t) = p_x_,(/) = a(t) where a(t) is a continuous function on [0, °°) such that a(0) = 1.
Find Pn(X(t) =1) = P(X(t) =1 |
t >0.
Exercise 6.2 A stochastic process X(t) (t >0) may take the value 0 and 1with probabilities
p0(t) and Pi(/), respectively. Also
P[X(t + h) = 1 | X (/) = 0] = ah + o(h)
and P[X(t + h) - 0 | X(t) = 0] = /3h + o (h) as h —» 0, a and
being positive constants. Show that
aPo(t)
+p =—[ j+ ( pa+ojS( 0J ) - ~ r ) c - <a+/,)'
a + fi = -a A+ P^ + - ^ ■ (a+/’,'
if po(0) = 1.
Exercise 6.3 (Continuation of Exercise 6.2) Let ptj(ij = 0, 1) be the expected length
of time in (0, t) that the Markov process [X (/), t >0} with two states {0,1} spends in state
j, having started at X(0) = /(/= !, 2) initially. Then show that PijiiJ = 0, 1) are given by
a [1 _ e-a]
Poo(0 = pt/(d+ P) +
(a + P)2
at a
M o i(0
(a+P) (a+p)2
Pt P
Pio(0 (a+p) (a + P)2
at P [1
Pu(t) a+P)
( H (a+P)2
Exercise 6.4 (Contagion or Accident Proneness) Suppose that the probability of an
accident occuring in tto / + Atis
previous to time t, for v. Ay> 0. Show that this is a pure birth process on the set of integers
S = {0 , 1, 2, . . .) which satisfies the forward equations
Pon (0 = -(u + nX)p0n (t) + (v +(n-l)A)p0i„_l (
Poo(t) = -vpoo(t)
and the transition probabilities of ) are given by
- j - v/X^
P j t U ) = e ~ (v*j x v («rA' ( 2)
V k-j ,
In particular show that
v(v + A)... (v + nX) (j _ )w+1
PVn+\ ( 0 “ (3 )
(n+ l)!An+1
Exercise 6.5 (Branching Birth and Death Process) Consider a collection of particles
which act independently in giving rise to succeeding generation of particles. Suppose that
each particles, from the time it appears, waits a random length of time having an exponential
distribution with parameter 6 and the splits into two identical particles with probability p
and disappears with probability 1 - p. Let X(t)y 0 < / < «> be the number of particles
present at time /. This branching process is a birth and death process. Find the birth and
death rates.
Exercise 6.6 In exercise 6.5 suppose that new particles immigrate into the system at
random times from a Poisson process with parameter A and then give rise to succeeding
generations as described in the previous exercise.
Find the birth and death rates of this birth and death process.
Exercise 6.7 (Application of Birth and Death Process in a Telephone Traffic Problem)
Consider a telephone exchange where the number of available lines is large that for all
practicle purposes it can be considered infinite. Assume that Y(t), the number of lines in
use at time t follows a birth and death process with P(Y(t +h) = i+\ \ Y(t) = /) = Xh + o(h),
t > 0, h > 0 and i > 1.
Also P(Y(t + h) = i~ 1 | Y(t) = 0 = Pih + o(h) as h -> 0
(In the first case we have assumed that a call will occur in (r, t + h) is independent of
the number of busy lines at time t): Asume also that p, = ip. Find the probability generating
function (P.G.F.) of Y(t), P(Y(t) = k) and E(Y(t)).
Exercise 6.8 Consider the Feller-Arley process (linear growth process) which is a
hirth and death process with birth and death rates A„ = nX and pn = np respectively for all
n ^ 1 and initial condition P(X(0) = n) = 6nl.
(a) Show that the generating function G(z, t) of the size distribution at time t satisfies
the partial differential equation
dG 2xdG
- f o- = [ n - ( k + n ) z + k z 1} - ^ .
(b) Show that the size distribution is given by

p„(t) = P{X(t) = n I X(0) = 1} = (1 - $) (1 - T tfrr1, n > 1
where £ = £(/) and rf = r\(t) are functions of t, A and p.
(c) What is pn{t) if A = pi
(d) Discuss the limits pn(t) as / —»«>, for A > pyA = p and A < p.
(e) Compute the mean population size and its limit for the three cases p < A, p = Aand
p > A. Also do the same for variance.
(f) Classify the states of this process.
Exercise 6.9 (Feller) Let (X(f), t > 0) be a pure birth process with distinct birth rates
A, and Xq > 0. Then show that: (a) the transition function p/y(f) satisfies the recursion
relation pii+l (/) = A, I exp(-(/ - s)A/+1 )e~ski ds
Jo
A, [exp (-/A; ) - exp(-/A/+1)]/(A/+1 —Af-) if Af- * Af+,
A, exp(-rAy) if A, = A,+1
(b) Show that the forward equations have the unique solution given by
Pij (0 = 1 B('k' exP(-,/lt ) ifJ * i

0 if j < i
A/A/+1 . .. Xj _j
where B(k,i,j) =
(A/ - Xk) . . . (A*_j - Ajt )(A^+1 - A*) . . . (Ay- - A^)
(c) Show that in the linear growth process (A* = kX)
Pi j ( t ) = exp (-/A/) (1 -e-*A)H

1
Exercise 6.10 (Feller and Lundberg) Let {X(r), t > 0} be a pure birth process with
distinct birth rates A, and Xq > 0. Then show that only finite number of transitions occur
with probability one, i.e.
2
j=k
Pi i (t ) = 1 iffj - k^ 1/Aj = for all finite k. (1)
Also prove that if (1) holds, the wth jump time

Wfl —> oo a.s. as n —><».
Exercise 6.11 Let (AX/), / > 0} be a pure birth process with birth rates [Ak }q . (a) show
o° .
that X(t) —>oo a.s. as t —>«> iff £ —- < oo. Hence show that a Yule process or a linear
*=° Xk
growth process (i.e. Xk = kX) does not move out to infinity.
Exercise 6.12 (a) Show that for a stationary distribution to exist in a birth and death
00 A A A
process it is necessary and sufficient that the series £ —— (pk > 0, k > 1)
*=1 P\Pl - Pic
converges (A*’s and p*’s have their usual meanings).
(b) (The servicing of lathes) Suppose that m lathes are serviced by a crew of a repairmen.
When a lathe fails to function properly it is repaired immediately unless all of the repairmen
are working on lathes that have already failed in which case the lathe must await repair.
The following assumptions are made about functioning of a lathe.
(1) The probability of getting out of order during an interval of time (/, t + At) is
indpependent of t and is equal to A(A/) = XAt + o(At), independently of the “history” of
its operation. Similarly, if a lathe is being repaired, the probability of its being putback
into operation during an interval of time (t, t + At) is equal to p(A/) = pAt + o(A/) and
is independent of the nature of its work and its length of service upto the instant /. Let Ek
denote the event that at a given instant the number of lathes being repaired or awaiting
repair is equal to k subsequent removal of a single lathe from service denotes transition
to the state e m , and completion of the repair of one of the lathes indicates transition into
the state Ek_x. Show that this is a birth and death process with finitely many states E0,
. . . , Em. Find also its stationary distribution.
Exercise 6.13 Let M be a discrete time a periodic Markov chain with state space
S = {0, 1, 2,...}, transition matrix (/?,-,) and initial distribution {a„ i e 5}. Let Abe a fixed
positive real number and s any real number of [0, «>). Define for all i, j e S
~pii { s ) = l Qe - ^ a s ) " l n \ p \
1. Show that (pn (5)) is a transition matrix of a continuous time Markov chain
with state space S, and initial distribution [ah i s 5}. [Note : Only check the conditions
for transition functions of a continuous parameter Markov process. Do not prove actual
existence of M.]
2. Evaluate the infinitesimal generator A of M in terms of the original matrix P of M.
Is M conservative?
3. Prove the following statements:
(a) M is irreducible iff M is irreducible.
(b) State j e S is positive recurrent iff j s S.
(c) State j e S is transient iff j s 5 is transient.
Exercise 6.14 Continuation of Exercise 6.13.4. Assume that M is irreducible and positive
recurrent with {Kpj s 5} as stationary distribution and there exists a constant p, 0 0
\ p \ ? - n , \ 0
| pij(s)-7ij | < exp [-A (l -p )5 ].
Is there any anologous result for M transient ? Null recurrent ?

Remark In the proof of the next exercise, use the following theorem of Chung: If the
infinitesimal generator A = (a^) satisfies Sup | a,, | < K < then A determines the
transition matrix P(t) uniquely.
Show that a Markov chain with Sup | | < K <°° is positive iff there exists a
i
solution {X.,i
i * e 5} tfor /=0
2 jt.-a.-.-
' ’J = 0 for which 0 < E/=oIx.1 'I1< «>.
♦♦ Exercise 6.15 Let P(t) be the transition matrix of a continuous time Marikov chain
with infinitesimal generator A given by
-a p Y
Y -a P , where p + y - a > 0.
P Y -a)
By the remark to Exercise 6.14, we can determine P (t) as follows:

1. Prove that the eigenvalues of A are 0, -p + /cr, - p - io where
p= — and l/2(0-y)V 3.
2. We put for all t > 0
P(t) = R + S exp(-(p + ia)t) + T exp (-(p - io)t)
where R, S and T are 3 x 3 matrices with (possibly) complex entries.
Use the following facts to compute /?, S and T.
(a) A is symmetric; (b) P{t) is real for all t > 0;
(c) P(0) = /: (d) P'(0) = A\ (e) Refer to Exercise 6.14. Show that all entries of the
matrix R are 1/3.
3. Using the above results, indicate how it is possible for an infinite number of
different continuous time Markov Chains, to have a common skeleton chain.
4. Use the above results to prove: If the one-step transition matrix of a descrete time
markov chain is of the form
Q
P where p + q + r = 1, p * 1, then there exist infinitely many
V? r P)
solutions to Elfving’s problem.
♦♦Exercise 6.16 Consider a generalised birth and death process with k n(t) = wA/, A> 0
and p n(t) = njxt, p> 0 for all n > 0. Also pn(0) = Sin-
(a) Solve this birth and death process by exhibiting an explicit expression for pn(t) =
P(X(t) = n | X(0) = 1).
(b) Compute the mean population size, and evaluate its limit as t —><».
(c) If T denotes the time till absorption, then show that P(T <t)= p0(t). Compute E(T)
if A = 0.
Exercise 6.17 Show that Poisson processes are the only renewal processes which are
Markov Chains.
7
Poisson Process
7.1 Definitions and Their Equivalence

Definition 7.1 Let Nt be a non-negative integer-valued stochastic process, where
t e T = [0, °o], satisfying the conditions:
(1) P[N0 = 0] = 1
(2) {jV„ t eT) has independent increments, i.e., for
to < h < • . • < tn in T Ntn - Ntn_x, Ntn_x - Ntn_2 ~ N t0
are independent random variables.
(3) {N„ t e T } has stationary increments, i.e. for t > 0, Nt - Ns has same
distribution as N/+h - Ns+h for all h such that t + h and s + h are in T.
(4) For t > 5, P[N, - N, = k] =A > 0, 0, 1, 2, . . .
A is called the intensity of the Poisson process.

Let {X,, t e L}, be a stationary independent increment process with P[X0 = 0] = 1.
Then 0 , „ +(2 (u)= E { e iuX^)-"
= E(eiu{Xri+t2~Xtl)eiuXr')
= E(eiu(Xtx+l2 ~x‘0 £ ( e irlXn )
(because of independence of increments)
= [em(X'2“*o)] £ ( e iM
*'i ) (because of stationarity)
= 0 *,2 ( « ) - 0 x,1(iO
Let / ( t ) = 0 ( m). Then/(r) satisfies the following functional equation (known
as Cauchy equation)
/U i + t2) = /( r 0 ’/ ( f 2) for all tx and t2.

If/is continuous, the solution is /( l) = [/(l)]' for all real t. Hence we get the
following proposition.
Proposition 7.1 If {X„ 16 T } be a stationary independent increment stochastic
process with P[X0 = 0] = land 0 Xf(«) = / ( 0 is continuous in t for every w, then

0X,(K) = [0X,( u)]f, where ( 0 X[(u) is called the characteristic functional of Xt.
Definition 7.1a A characteristic function 0(u) (random variable X with
characteristic function 0(u)) is called infinitely divisible if [0(w)]' is a characteristic
function for all t > 0. Thus, the Poisson random variable is infinitely divisible.
Similarly, the Poisson process has infinitely divisible characteristic functional,
since 0 X[(u) = e- ^ x-elu^ is infinitely divisible.
Poisson process can be characterized as pure birth process. Hence we have
the following alternative definition of a Poisson process.
Definiton 7.2 It is a continuous time Markov chain (Birth process) with
Xn - X for all n and j.in = 0, i.e. dpn(t)ldt = -Apn(t) + hpn-\(t) is its Kolmogorov’s
differential equation, where pn(t) = P[Nt = n \N0 = 0].
The G.F. G(t,z) = Z pn(t)zn ,\z\ < 1 of Poisson process is given by
#a i = i
n=0
[--W 0 + 4 v i(0 ]z "
= -AG + XzG = A(z - 1)
=>4JZ = A (Z - l ) d t ^ G ( t , z ) = c.eXiz-'}l.
Cr
The boundary condition 1 = G(0, z) gives c = 0. Hence
G(t,z) = e - Xt 2 n\
e~h (Ar)"
Therefore P[Nt = n] = pn (t) =
n\
Axiomatic Definition 7.3 It is a continuous parameter stochastic process
satisfying the conditions:
(1) P[N0 = 0] = 1
(2) {Nt, f eT] has independent increments
(3) {Nt, t e f ) has stationary increments
(4) P[Nt > 0] > 0 for all / > 0
P[Nh > 2] _
(5) lim
hi o 1]
or
(4') P[Nh =1] = Ah + o(h), h1 0, A > 0
(50 P[Nh 2>] = o(A) ash 0i
Conditions (4) and (5) or (4') and (5') are known as regularity or oderly properties
of Poisson process.
Definition 7.4 Poisson process is a Renewal process with interarrival times as
exponential distributions (see Renewal theory).
Let Nt denote the number of occurrences upto time t.
Poisson Process 135
Theorem 7.1 Under the axioms (postulates) (1) to (5) Nt has Poisson distribution
with mean function Af, i.e. pn(t) is given by the Poisson law
e x\A t)n/n\ n= 0, 1, 2,. . .
Proof Let pn(t) = P(Nt = rt|Af0 = 0) = P(Nt = n) (by axiom 1). Actually
P(Nt+h - Nt = y\Nt = jc) is independent of x. Axioms (4') and (5') => p0(t) =
1 - Xt + o(t). Consider pn(t + h) for n > 0.
The n events by epoch t + h can happen in the following mutually exclusive
ways A x, 4 2, 4 3, . . . , An.
For n > 1
Ai = {n events occur by epoch t and no event occurs between t and t + h)
we have P{AX) = P[Nt = n\P[Nt+h - N t = 0|A, = n]
= Pn(t)PoW = PnW 1 o(
A2 = {(w - 1) events occur by epoch t and 1 event occurs between t and t + h]
P(A2) = P[N,= n - 1]P[AU - 1|AT, = 1]
= Pn-\U)P\(h) = pn-\(t){kh) +
4 3 = {(n - 2 ) events occur by time t and 2 events occur between t and t + h]
P(Al) = Pn~2 (t){o(h)} = o(h)
and so on for A4, A5, . . . , A n i.e. P(Ak) = o(h) for 4 < k < n.
Therefore pn(t + h) = pn(t)( 1 - A/i) + pn_\(t)A,h + o(h) for n > 1
nr Pn(t + h) - Pn(t) o(/l)
or ^---------- - - Ap n + Ap„-\ (t) + — —
Taking
lim h-> Oyieldsp^(t) = -Ap+ 1 ... (7.1)
as the Kolmogorov’s differential equation for Poisson process.
For n= 0, p0(t + h) = p0(t)p0(h) = p0(t)( 1 - Ah) + o{h).
Po(t + h) o(h)
Therefore -Ap{t) +
h
Talcing limit h —> 0. we get
p'0{t) = - A p 0{t) (7.2)
Initial conditions: Ng = 0, i.e. p0(0) = 1, p„(0) = 0 for 0 (7.3)

Equations (7.1), (7.2) and (7.3) give by generating function method (done after
definition 7.2)
(Aty
Pn(t) = , n > 1. (7.4)
n\
Alternatively by induction on n, for n= 0 (7.2) gives
= 1, c = 1.
Therefore p0(t) = e~Xt (7.5)

Consider equation (7.1) for n = 1 and substituting the value of p0 and solving
the equation by using (7.3), we get
Pi(t) = Xte~Xt.
Assume (7.4) holds for n = 1. In a similar manner it can be shown that it holds
in general for n + 1 if it holds for n.
7.2 Poisson Process and Renewal Theory

Theorem 7.2 If {Nh t e T} is a Poisson process with intensity A > 0. Then
(a) {Ni} is a Renewal process with exponential interarrival times 7) i.e.
P(Tt < t) = F(t) = 1 - e * .
(b) Wn Z 7} j , the waiting times are gamma (Erlang), i.e.
P(Wn < t) = F(n) (t) = 4 =

V n Jo
f (Ax)"-1
(Waiting time Wn = Time from origin to the nth subsequent occurrence)
(c) H(t)^ = Z F (n) ( o j , the renewal function in this case is given by At.
Proof
(a) See Theorem 7.5. (b) This is trivial.
(c) Note that P(Nt < n - 1) = P(Wn > 0 = 1 - P(Wn < t)
n -\
= X e * (At)k/ k \(integrating by parts)
k=0
Therefore, P(N, = n) = e \ f o )n/n\
Hence H(t) = £ F (n)(r) = E(N,) = h .

n- 1
Recall the definitions from Renewal theory that the Excess life time or Residual
life-time or Forward recurrence time is defined by Y(t) = - t. TheBack
recurrence time is Z(t) = t - WNi. (Note that we have used the notation Wn
instead of Sn used in the Renewal Theory Chapter 4).
Recall the Theorems 4.5 and 4.6 from Renewal theory.
Theorem 7.3 (i)

ft+X
P[Y(t) <*] = J [1 - + jc - for0,
= FU + x ) ~ [1 - F(t + x - u)]dH(u) if 0
Jo
Poisson Process 137
(ii) P[Z{t)<x]- f [1 - F(t - u)]dH(u)

Jt-x
1
F(t) - f
Jo
[1 - F{t - u)]dH(u) if x <t
1 if x > 1
Proof
Let Si < S2 < . • . < Sn be positive renewal points with Sn be the largest such
points. Then
[T(/) < x] = [At least one renewal point fall in the interval (f, t + x]].
Hence P(Y(t) < x) = P( u {t < Sn< t + x < S„+1})

n> 1
= £ P ( t < S n < t + x < S n+l).

n=\
By conditioning on Sn = u and taking expectation, we get
P{t < Sn < t + x < Sn+1) = J P[Sn+i ~Sn > t + x - u]dF(n) (m),
where F {n) is the n-fold convolution of F.
Therefore
P[Y(t) P[5„+1 - S n > t + x - u]dF(u)y

-=r
[ l - F ( t + x - u ) ] d H ( u ) , where H(u) = E F (n)(u).
■r n =\
The rest of the proof is given in Theorems 4.5 and 4.6.

Corollary 7.1 If {Nt, t e T} is a Poisson process, then
-Xx
1- e if x < t
P[Z (t) <x ] =
1 if x > t
r ‘- x
Proof P[Z(t)<x]= 1 - Ae-H'-^ds = 1 - e-**
Jo
Note If x > t [Z(t) < x] is a sure event.
Corollary 7.2 P(Y(t) < x) = 1 - if jc > 0.
Thus the excess life processes have the same exponential distribution as every
life, a manifestation of the memoryless property of the exponential distribution.
The spent life has a truncated exponential distribution.
Note (1) This shows that the Poisson process is a stationary or equilibrium
process.
(2) Y{t) and Z(t) are independent iff {N(t), t eT] is a Poisson process
(see Exercise 7.9).
7.3 Properties of Poisson Process
Elementary Properties of Poisson Process

(1) Sum of two independent Poisson processes is a Poisson process.
(2) The difference of two independent Poisson processes N\ (t) and N2(t) Nt =
N](t) - N2(t) has probability distribution given by
n/2
Ai
P[N, = n] = e"a 'A2)' I\n\ (2fÂ|A2) (7.6)
hi
n = 0, ± 1, ± 2, . . .
“ (x/2)2r+n
where ln{x) = 2
r=o r !|n + + 1
is the modified Bessel function of order n(n > - 1).
Proof The p.g.f. of ,Ns

i £ ( / ' ) = E (s N' ll)- N2(n)
N20)
E ( s N' (n)E\ - = exp {kxt{s - 1)} exp \ X2t\ -■ - 1
= exp {-(Aj + A2)r} exp ts + X2tls\ (7.7)

P[Nt = n] is the coefficient of sn in the series expansion of (7.7) in positive and
negative powers of s
or directly, P[Nt =n] = £ P[N](t) = n + r]P[N2(t) = r]

r=0
~ g - A>'( A 1Q w+r e -A2'(A 2/) r
r=o (n + r)\ H
^-(Ai+A2)/ M ”/2 y (?VA,A2) 2'"

A2 JT=o r!(rc + r)!
Note (1) /_„(0 = /„(/) = / H (/).

(2) £W, = (A, - A2)t, E(Nt)2 = (Aj + A2)t + (Ai - A^V and Var (Nr) = (A! + A2)f.
Example 7.1 If passengers arrive (singly) at a taxi-stand in accordance with
a Poisson process with parameter Aj, and taxi arrive in accordance with
an independent Poisson process with parameter A2 then Nt = N^t) - N2(t) gives
the excess of passengers over taxis in an interval of length t. The distribution
of Nt P[Nt = ri], n = 0, ± 1, ± 2, . . . , is given by (7.6). Then Mean of Nt is
(Aj - A2)f > 0 according as
Aj > = < A2 and Var (Nt) = (At + A2)t.
Poisson Process 139
The Poisson type of occurrences are called purely random events and the
poisson process is called a purely (or completely) random process the occurrence
is equally likely to happen anywhere in [0, T] given that only one occurrence has
taken place in that interval. We state this by the following theorem.
Theorem 7.4 Given that only one occurrence of a Poisson process Nt has
occurred by epoch T, the distribution of the time interval T, in [0, T) in which it
occurred is uniform in [0, T], i.e.
P[t< T< t + dt\NT = 1] = y , 0 <
Proof Since r is exponential, P[t < X

< +
P[Nt =l] = e-xrU T ) and P[Nr = l |r ==e~MT-,)
which is the probability of no occurrence in the intrval of length ( ).
P[t '<
r< =1
P[t < T< t + dt\NT = 1] =
P[NT =l]
= P[t < T < t + dt]P[NT = 1 | r = t]IP[NT = 1]
(Ae-A>dr)(g- ^ r -'> _ dt
( e~XT))T
{X T
Note It may be said that a Poisson process distributes points at random over
the infinite interval [0, <*>] in the same way as the uniform distribution distributes
points at random over a finite interval [a, b].
Interarrival time
With a Poisson process {Nt), the number of occurrences of an event E by epoch
t, there is associated a random variable—the interarrival time X between two
successive occurrences of E.
Theorem 7.5 The interarrival time between two successive occurrences of a
Poisson process {N,} having parameter A has a negative exponential distribution
with mean 1/A.
Proof Let X be the r.v. representing the interval between two successive occurrences
of {jV,} with P[X < jc] = F(x) be its distribution function. Let E{ and Ei+X be the
successive events and Et occurred at time tr Then
P\X > x] = P[Ei+1 did not occur in (th t{ = jc) given that E, occurred at instant fj
= P[Ei+{ did not occur in (rt tt + jc)| Nti = i]
= P [no occurrence takes place in an interval (th tt + jc) of length jc | N tj = /].
= P [ N t i +x - N ti = 0 \ N t . = i]
= P[NX = 0] (since Nt is stationary and has independent increments)

= Po(x) = e x > 0.
F ( jc) = P[X < jc] = 1 - P[X > jc] = 1 - e-**9x > 0.
Actually the property of the Poisson process stated in Theorem 7.5 is a

characterizing property which will be discussed in next theorem.
7.4 Characterization of Poisson Process

Theorem 7.6 If Tt denotes the interval between Et and £ t+1, i = 1, 2, 3,. . . , in
a renewal process then Tx, T2, . . . , are i.i.d. r.v.s with
(a) P[Ti < x] = 1 - e"'1*, x > 0 iff the renewal process is a Poisson process.
(b) If Tn = Ti + T/+i + . . . + Tn_i, then Tn —> °° a.s.
Proof (J.L. Doob) Consider only two interarrival times Tx and T2. By previous
Theorem 7.5
P[TX< * , ] = ! - e~M' = [N) ,- N(0) > 0], *, > 0.
P
Also
n -\
P[Tx < t u T2 < t 2]< I P *1 El ) _ m = o, i v f i i i ,, ] - El
j =o
n- 1
+ 2 P N\ | - N(0)
j =o
= 0,N\ — fi) - 4 — I ^ 2
n- 1 ^ M ^2 f
Now the second sum < Z e~Xjtl,n 1 - e n - e n A —
7=0 n
V
. 0 („-2 ) =0ri
i - e~**1n \n
n-I / 2, \
First sum < S e(-*jt\in-h\in) \ —1_ (i (by L’Hospital’s rule and
j= o V. nJ
convergence of G.P. series) —»(1 - e~**' )(1 - e~ ) as —>
Hence P[7i < t u T2 < t2] < (1 - )(1 - <T^2)

= P[TX< t,]P[T2 < t2].
To prove the reverse inequality.
Poisson Process 141
n- 1
= s *(-- fy titn ) 9- * * \ I n A t2
-i)
>=o 1- e
(I - e **1)(I - e **2) as /i —> <».
Therefore P[TX< t x, T 2 < t2] ^ P[TX< tx]P[T2 < t2].

Hence by induction Tx, 72, 73, . . . are independent.
Note Implicitly the idea of Separability of stochastic process (see Appendix I)
is used here.
(b) Define Tn = 7y + 7}+1 + . . . + Tn_x, j = 1, . . . , n - 1
The characteristic function of 7, is

Jo
f
t e ie,Ae-x,dt= | 1 - - y | = 0 ( 0 ) .
Since Tp 7}+1, . . . are independent random variables the ch. function of rn is
J~n
2
0 „ (0 ) = ^1 - y j i _ *|0i„ ( 0 ) | = 0 as n->
j and hence
,n>
°o for all Q > 0. rn Now, being sums of non-negative independent r.v.s, rn —» °o
in distribution as n —» «>. By Levy’s theorem rn being sums of independent r.v.s,
Tn — > oo a.s. as n —> °°.
Note . . . 7/ = inf [* | A(r) = i + 1] if P[N(0) = i] = 1, i.e. 7} is the time of occurrence

of the first jump after the ith step.
Alternative proof of (b)

P(rn+1 - Tn > a) > exp (-Aa), a > 0.
It follows by Fatou’s lemma that
P( lim sup ( r n+i - Tn) > a) > lim sup P ( r n+1- r n > a) > exp (-Aa).
n —>oo n —>oo
This implies 7 ( lim = <») > exp (-Aa) for every a > 0.
In particular it follows that 7(lim Tn = °o) = 1.
Now we give an alternative proof of Theorem 7.6(a) based on renewal
argumant.
Proof of (a) Let 0 < Sx < S2 < S3 . . . are the renewal points (arrival points) of
the Poisson process. We prove the theorem by induction. For 71? the theorem has
already been proved in Theorem 7.5. Now calculate the joint distribution of Tj
and 7Y After conditioning on Tx = x and taking expectation, we get
P(TX< u, T2 > v) = f
Jo
P{S2 > v + ;c|7i = x)he~*Jcdx
(since Tx has exponential distribution).

Now given that Tx = 5j = x , {52 > u + jc} = {No renewals in (x, v + jc]} =
{N(v) = 0} and hence
P(S2 > v + jc 17, = x) = P(A(u) = 0) =
Therefore P(T, <

Jo
u,T2 >v) = f =- «-*“
Hence the theorem is proved for and T2.
Next, assume that the theorem has been proved for Tx, T2, . . . T*. Then the
joint distribution of 7}, 1 < y < k + 1, calculated by first conditioning on 7} = Xj,
1 < j < k and then taking expectation
P(Tj < uj,1 <
j<
k,Tk+l > xk+1)
[U\ *uk k+\ / k \
-XkJ ... J P(Sk+i > 2 Xi\Ti = x 1 < / < k) exp J ^ - A E jc,-^ d x x ... d x k
Now [Tt = x t , 1 2 xj | 7} = x t , 1 
=Pu Z Xi 1 ~ N = P{W(**+i) = 0}
i \ 1 )1 J J
(By stationarity and independence of increment of Poisson process N(x)) = e **k+l.
Evaluating the multiple integral, we get
P(Tj < uj, 1 <j < k, Tk+i > xk+l)= - .. (1 - e ~ ^ )
Therefore Tx, T2, . . . , Tk+l are i.i.d. exponential r.v.s.
7.5 Generalization of Poisson Process

Using the notation N(t) instead of Nt it can be easily shown that a Poisson
process {N(t), t > 0) satisfies the following axioms.
axiom 0: P[N(0) = 0] = 1
axiom 1: {N(t), t > 0} has independent increments
axiom 2: P[N(t) > 0] > 0 for any t > 0
axiom 3: for any t > 0
P[N(t + h ) - N ( t ) > 2 ]
n P[N(t + h) - N(t) = 1]
axiom 4: N(t) has stationary increments.

Following generalization of the Poisson process are very well known and
useful:
1. Non-homogeneous Poisson Process or Time Dependent Poisson Process

Let {N(t), t > 0} be an integer valued stochastic process satisfying axioms 0, 1,
2, 3, and
1 - P [ N ( t + h ) - N ( t ) = 0\
axiom 4 ' lim v(t).
h^>0 h
Poisson Process 143
is a nonrandom function of time. In this case Kolmogorov difference differential

equations become p'n(t)= - v ( t ) p n(t) + v ( t ) p n„\(t),n > 1.
Define if/(z, t) as the generating function of P(N(t) = n) and p'0 (t ) = v(t) p0 (t )
as the initial equation.
Since (N(t), t > 0} has independent increments,
£ [ z W(/+/i)] = g [ z N ( t + h ) - N ( t ) ] g ^ z N ( t ) y
OO
Now i/a[z, f] = 2 z nP[N(t) = n]

n -0
Therefore, y/(z, t + h) = y/(z, t)E[zm +h y m ].

Hence
z Af(f+A)-N(0}_i] = I{ P [N (f +N(t) = 0] - 1}
+z T P{N(t + h ) ~ N(t) = 1} + -£■ 5 = n]

n n n- 2
Therefore
Hm j-{ E[ z N(t+h)~N(t)] -1} = v(t)[z - 1] by axioms 3 and 4'.

Hence
= £ % £ > ,£ [,» < .« .> -« .> ,_ u .
A—
.0 h A—
>0 h
Thus for | z | < 1, f > 0
Jj- vr(z,r) = vr(z,r)u(r)[z - 1].
Under initial condition y/^z, 0) = 1 and therefore yf(z, 0 = exp [m(0 [z - 1]],
where m (0 = I v(t')dt' is the mean function of N(t), because m(t) = EN{t) =

Jo
d\f//dz | z = 1.
II. Generalised Poisson Process

Let {N(0, t > 0} be an integer valued stochastic process satisfying axioms 0, 1,
2, 4 and axiom 3': there exists sequence {pk} k = 1, 2, . . . , / > 0 such that
lim P[N(t + h) - N(t) = k\N(t + h) - N(t)> 1} =p*.
h—>0
Again desired yf(z, 0 has to be determined.

Since {N(t), t > 0} has independent increments,
E[zN^t+h^] = E[zN(‘t*h^~lsr^ ] E [ z N^] .
Due to stationary increments y/(z, t + h) = y/(z, t) y(z, h)
Therefore j^{y/(z,t + h) - y/(z,t)} = y/(z,t)j^{y/(z,h) - 1}
Taking limit as h —> 0
= y/(z,t) Y m j {-
But L{ y ,( z, h ) - l) = -{F[A f(/t) = 0 ] - 1}
+ -£
n n =1
Then we can show (as in the case of ordinary Poisson process) that there
exists a constant u > 0 such that
lim j- {1 - P[N(h) = 0}] = lim | P[N(h) > 1]

/i—>0 n h—>o n
Applying axiom 3' and Lebesgue dominated convergence theorem
lim £ z "[/>[A^(/i) (h)>1] = £ z npn

n\N
=
h—>0 n- 1 n=l
Hence, y/^z, t) = y/(z, or y/(z, 0 = c

n£=0 znp n -Po
Note yf(z) can be the characteristic function of a non-negative integer valued

r.v. with probability distribution {pk}.
III. Compound Poisson Process

A stochastic process (X(f), t > 0} is said to be a compound Poisson process if for
N(t)
t > 0 it can be represented by X(t) = L Yn = sum of random number of
n=l
independent and identically distributed r.v.s, where {N(t), t > 0} is a Poisson
process,
{Yn, n = 1 ,2 ,...} are independently and identically distributed r.vs. and {N(t),
t > 0}, {Yn] are independent of each other.
Let yfxv)(u) be the characteristic function of X(t).
Since {N(t), t > 0} has independent increments and {Yn} is a sequence of
independent and identically distributed r.v.s, it is obvious that X(t) had independent
increments.
For n = 0, 1, 2, . . . and for any t > s > 0
E[e iu{X(,)-X(s)]\N(t) - N(s) = n]= {(pY(u)}n
where (pY(u) is the ch.f. of Y.
Since given n events have occurred in (5, f], X(t) - X(s) is the sum of n
independent r.v.s distributed as Y9s.
Then
W m - x t ju ) = £ £ [ e i « |X ( , ) - X ( . v ) ) |yV ( 0 _ n] . _ = n]
AI= 0
Poisson Process 145
= J o {^K(«)}"«-p('-5)^ n^ - = e - vl,-s)exp[v[t - s](pY(u)]
Therefore V x ( t ) W = evt{*r(u)~l]
where (pY(u) is the common characteristic function of {Yn}.
Thus we have shown that {X(0, t> 0} has independent stationary increments
with the above characteristic function for any t > 0.
Note If Y is integer-valued then the compound Poisson process is a generalised
Poisson process.
Compound Poisson process arises in bulk queues where customers arrive or
are served in groups. An important application arises in “collective risk theory”.
Suppose that claims against a company occur in accordance with a Poisson
process with mean Xt and the individual claims X-s are i.i.d. with distribution
{P*}. Then X(t) represents the total claims at time t. If A represents initial reserve
and c the rate of increase of the reserves in the absence of claims, then the total
reserve at time t is A + ct - X{t) and negative reserve implies “ruin”. The
moments of X(t) are given by Wald’s identity as
E(X(t)) = E m ) ) E ( X x) = Xt{EXx\
Var (X(t)) = E(N(t) ) Var(X,) + E 2(X,) Var = At EX,2 (EX, = 0)
and Cov(X(r), X(s)) = min t) AEX2

(since Cov (N(t), N(s)) = A min (s, t)).
IV. Poisson Increment Process

{ Y„ €t T}
is called the Poisson increment process.
Theorem 7.7
(i) EY\ = A/
A (/-|r - s|) if/>|r-s|
(ii) Cov( f s) =
0 if / < - s|
(iii) ( Y„ t eT) is covariance stationary.
Proof (i) EY, = E(Nl+i) - E(N,) = A)t + /) - Ar = A/
(ii) Cov (Yh Yx) = Cov (Nl+l - N„ Ns+I - Ns)
= Cov (Nl+l - Ns+l + Ns+l - N„ Ns+l - N t + Nt -
+ Cov (Nl+l - Ns+h Ns+l - N,)
+ Cov (Ns+l- N t N „ , - N J
- Var (Ns+t - N,) (stationarity and independence of increment)
= Var (Ns+i_, - N0)
= Var (VJ+M) = A(i + I - t) =Ml
(iii) The proof is obvious and hence the proof is omitted.
7.6 Order Statistics and Poisson Process

There is a close connection between Poisson process and so-called order statistics.
Let Kb Y2, . • •, Yn be i.i.d. random variables with common d.f. F(t). Assume,
for simplicity that F{t) is strictly increasing.
r«) = /th smallest among Yu Y2, . . . , Yn is called /th order statistics. Consider
Xf = F(Tt), then X/’s are uniform on [0, 1] i.i.d. random variables.
Let Xi , X2 ,. . •, X„ be the order statistics of Xb X2, . . . Xn. Let us state a
well-known lemma from statistics for our later use.
Lemma 7.1 The joint probability density function of the order statistics
(X j\ . .. ,X*) is given by
fft!, if 0 < Jtj < . . . < * „ < 1
0, otherwise.
Theorem 7.8 Let {Nt>t e T} be a Poisson process with intensity A. Let t be a

fixed real number. If Nt = ft, then the arrival times rx, r2, . . . , rn are distributed
as the ft order statistics of the n independent random variables uniformly distributed
over [0, t].
Proof We note that the interarrival times are independent and exponentially
distributed with parameter A. Hence
P[t\ r<, < fj + hu . . . , tn < Tn < tn +hn\N, = n]
= ~P[ N = ai] — ^1 ^ h^"hi,...tn ^ T n < t n +
e~MU
Xnh\h2. . . hne Mn\ n\

= hn-
Xnt ne - h
Let Z(1) < Z(2) << Z{n) denote the order statistics in a sample of size n from the
exponential distribution
P[Z < t] = 1 0<r<oo (7.8)
The joint p.d.f. of Z(n)’s is
ft!exp | - I Zr if 0 < Z] < ^Zn <°°
which may be written as

n
ft! exp - Z (« - r + l)(z f - . Zo = 0.
Let Yr = (n - r +l)(Z(r) - Z(r_1)), r = 1, 2 ,. . . , n(

7.9)
Note that range of each Yr is (0, oo) and Yr are independent random variables.
Now Z(r) = £ (Z0) - Z(W)) = £ n f j + x (7-10)

Poisson Process 147
a linear function of independent exponential variates and hence Z(l), Z(2), . . . , Z(n)
is a Markov chain (cf. Renyi, 1953).
As a general case, let X(1) < X(2) < . . . < be the order statistics from a
strictly increasing distribution function F(t).
The transformation Uk = F(Xk) converts into £/(r), r = 1, 2, . . . n, the order
statistics from uniform /?[0, 1] distribution. Let Z* = -log £/*.
F[Z* < x] = P[-log £/*<*] = P[t/* > *-*]
= P[F(Xk) > = P[Xk > F~\e~x)]
= 1 - F(F~\e-x)) 1 - e~x
i.e. Zk s are exponential random variables.
Hence, Z(r) = log f/(n_r+d are order statistics from exponential distribution.
Now X(„_r+1) = F - 1(t/(n. r+1)) = F - 1(«"*<'>)
Yi , >2 + Yr
= F - 1 [exp - . . . + (7.11)
n n- 1 n - r+r
Thus, X<„-r) = F -' [exp (-(K|/n + . . . + Yr+i/n - r))J
= F~' exp j log Fn-r+1)) JWi_ (7.12)

n- r
Since T/s are independent, hence by (7.11) X(„_r+1) and Tr+1 are independent.
Therefore, by (7.12), X(n), . . . , X(1) form a Markov chain and hence
W o . X(2), . . . , X(n)} is also a Markov chain.
From this the following theorem follows.
Theorem 7.9 For random sample of n from a continuous distribution function
the conditional distribution of X(i) given X(r) = xr(s . r) is just the distribution of
(s - r)th order statistics in a sample of (n - r) drawn from the parent distribution
truncated on the left at x = xr
Proof
Hr) e ~Z(n-r+1) —e(Z(n-r)-Z(n-r+1))

Mr+1) Hn-r)
= e^Yn~r+]),r, Yn+i = 1. are independent random variables.
Hence, *co “ = e Yn~r+' is a sequence of i.i.d. uniform [0, 1] random

l(rtl)
variables (Malmequist 1950).
Lemma 7.2 The density of Z(k) is given by
fk(t) = ke~nt [e‘ -l]* -1

w
Proof For k = 1,f\(t) = ne nt.

Then prove by induction on k.
Proposition 7.2 Let {Nt, t E T} be a Poisson process with intensity A. If Nt = n,
r
then the density of Wr (waiting times of Poisson process = X 7}; 7} = T, - T/.j)
i=i
for r < n is given by
nl
( n - r ) l ( r - 1)!
Proof The density of fcth order statistic X(k) from uniform [0, t] distribution is
given by
n\
( n - k ) l ( k - 1)!
Now from Theorem 7.8 given Nt = n arrival times are distributed as order
statistics from uniform [0, t] distribution.
Therefore, Wr is distributed as r th order statistics from uniform [0, t\ distribution.
Theorem 7.10 Let {Nt, t e T} be a Poisson process with intensity A. Let s eT,
s < t E T, k< n.
n-k
Then P[NS k\N t = n] =
P[NS = k,N, = n]
Proof P[NS = k\N, = n] =
PIN,=n].
P[ Nt - N s = n - k , N s = k]
PIN, = n]
P [ N , - N s = n - k ] P [ N s =k] . . . .
-----------------------^ -------------- (due to independent increment)
e-A<,-,)[A(/-s)]"-* e~h (Xs)k

-k!)
(n it!
e~M(At)n/n'.
(t - s) n-k ok
yh tn

Exercise 7.1 Assume a device fails when a cumulative effect of k shocks occur. If the
shocks happen according to Poisson process with parameter A, find the density function
for the life T of the device.
Exercise 7.2 Let { X( t ) y t > 0} be a Poisson process with intensity parameter A. Suppose
Poisson Process 149
that each arrival is “registered” with probability p, independent of other arrivals. Let
{Y(t), / > 0} be the process of registered arrivals. Prove that Y(t) is a Poisson process
with parameter Xp.
Exercise 7.3 Suppose g(t) is the conditional failure rate of an article at time /, given that
it has not failed upto time r, i.e. P (failure in time (t, t + h)/no failure upto time r} = g(t)h +
o(h) as h I 0. Assume that g(f) is positive and continuous on (0, °o). Find an expression
for F{t) = P (failure at some time r, r< t) in terms of g(t).
Exercise 7.4 Let X(t) and Y(t) be two independent Poisson processes with intensity
parameters Aand p respectively. Let W and W* be the times of two successive occurrences
of Poisson events of the processes X{t), i.e. W < W*, X(t) = X(W) for W < t < W* and
X(W*) = X(W) + 1. Define N = Y(W*) - Y(W). Find the probability distribution of N.
XU )
Exercise 7.5 Let Y(t) = Z %n be a compound Poisson process. Then show that
n= 1
(a) the process Y(t) has independent increments, (b) The characteristic functional 0y(/) of
Yi() is given by 0y(o(u) = exp [Af(0|(w) - 1)] where 0|(w) is the common characteristic
function of {£„} and Ais the parameter of the poisson events, (c) If E(%2) < °° then show
that E(Y(t)) = XtE(£) and Var (Y(t)) = XtE£2.
Exercise 7.6 Consider a Poisson process with rate A(> 0). Let Txbe the time of occurrance
of the first event and let N(TX) denote the number of events in the next Tx units of time.
Show that EiN(T{)Ti) = 2IX and find the variance of N(Ti)Tx.
Exercise 7.7 Let X(t), - oo </<«>, be a Poisson process with rate A. Let Y(t) = X(t + 1)
- X(t), - oo < t < oo. Find the mean and covariance function of Y(t) process and show that
it is a second order stationary process.
Exercise 7.8 Let X(t) be a Poisson process with parameter Aand Tabe the first time X(t)
reaches a > 0. Calculate E(Ta), Var (Ta) and also m.g.f. of Ta. Can you guess what type of
random variable will be Tal
Exercise 7.9 Prove that if the sum of two independent renewal processes is a Poisson
process, then both renewal processes must be Poisson processes.
Exercise 7.10 A telephone exchange has m channels. Calls arrive in the pattern of a
Poisson process with parameter A; they are accepted if there is an empty channel, otherwise
they are lost. The duration of each call is a r.v. where distribution function is exponential
with parameter p. The life times of separate calls are independent r.v.s. Find the stationary
probabilities of the number of busy channels.
Exercise 7.11 Show that if the interarrival distribution function F(x) of a renewal
process Y(t) is continuous and if the expected residual life time E(Y(t)) is a finite constant
for t > 0, then show that the renewal process is a Poisson process.
Exercise 7.12 In a factory, accidents happen according to a Poisson process with intensity
A. The times that the victims stay in the hospital are independent random variables with
common distribution function F(t). Let X(t) be the number of persons in the hospital at
time t € (0, oo).
(a) Show that P(X(t) = k) = e-XaU){Xa(t)]klk\, = 0, 1, 2___
where a(t) = f
Jo
[\-F(x)]dx
(b) Derive the distribution of tlim X(t) if F(t) has a finite first moment.
—>°o
(c) Compute P(X(t) = k) if all the victims stay in the hospital for a constant length of
time L.
8
Continuous Time and Continuous
State Markov Process
8.1 Diffusion Processes

Let us consider a continuous time and continuous state space stochastic process
{X(f), te T } with T = [0, <*>), 5 = (-oo, +oo).
More specifically let {X(0, t e T ] be a Markov process on a probability space
(Q, Jf, P) assuming values in a set S. Denote by ifa a -algebra of subsets of S.
The measure space (S, if) will be called the “state space” of the stochastic
process X(t). The process X(t) is said to be a homogeneous Markov process if its
transition probability has the property P(X(t + s) e£|X(s) = x) = P(x, t, B),
B 6 i f One may imagine a continuous parameter Markov process (X(r), t e T ]
that is not a process with independent increments. Suppose that given X(s) = x,
for small times t, the displacement X(s + t)-X(s) = X(s + t)-x has mean and
variance approximately t/u(x) and t<J2(x), respectively. Here /.i(x) and a 2(x) are
functions of the state of x, and not constants as in the case of Brownian motion
W(t). The distinction between {W{t) } and {X(0} is analogous to that between simple
random walk and a birth-death chain. More precisely, suppose
E( X( s + 0 - X (j)|X (5) = x) = tji(x) + o(t)

E((X(s + t ) - X( s ) 2 |X(s) = x) = t o 2 (x) + o(t) ( 8 . 1)
E(|X(5 + r) —X( s ) |3 1X(s) = jc) = o{t)
hold, as t i 0 for every x e i f

Note that (8.1) holds for Brownian motion.
A more general formulation of the existence of infinitesimal mean and variance
parameters, which does not require the existence of finite moments will now be
discussed. Using Kolmogorov’s consistency theorem (see A. Basu) it can be seen
that a Markov Process can be defined by the following simpler alternative form.
Let tx < t2 < h < . . . < tn < t e T, (8.2)

and X, Xx, X2, . . . , Xn are values from state space S. Then (X(r), t e T] is said
to be a Markov Process if it satisfies
Continuous Time and Continuous State Markov Process 151
P[X(t) < x | X(tn) = X(tn. j) = Jcn_! . . . , X{tx) = x x]

= P[X(0 < x | X(tn) = xn] (Markov Property)
for all t\, t2, • • • , tn satisfying (8.2) for all n> 1. We have seen in a continuous
time discrete state space Markov Process the transition function Pij(t) = P[X(t +
s) = j | X(s) = /] satisfies the following properties:
(i) p ^t ) > 0, i j e S
(ii) Z (/) = 1 for all t e T , j e S
jes J
(iii) pij (t + s) = Z p ik (t) pkj (5) (Chapman-Kolmogorov equation)

kes
(iv) P(0) = /, i.e. Pij(0) = Sjj, the Kronecker’s delta, and I is the Identity Matrix.
We shall develop an analogous theory in the continuous state case. To avoid
many measure theoretic difficulties let us define transition distribution function
as
F (y, s\ x, t) - P[X( t ) < x | X (s) = y], s < t, s, t E T.
To avoid many pathological difficulties we shall assume that the transition
distribution function of a continuous time Markov process satisfies the following
axioms:
(i) F ( y , s \ x , t ) > 0
(ii) F(y, s ’, - 0 0 , t) = 0, F{y, s\ + 00, t) = 1.
(iii) Chapman-Kolmogorov equations: For every t, s and t satisfying s < r < t ,
we have
F(y, s;x, t) f F (z , T\x, t)dFz (y, s\ z, r).

J—
00
[0 if x < y]
(iv) F(y, t\ x, t) = \ [ = S (y , x), Dirac delta function.
[1 if x > y\
(v) For all 8 > 0 the limits
(a) lim I (y - x)dFy (x, t\y, t + At) = 0,

A t J[\x~y\>6)
(b) lim I (y - x)dFy ( x , t ’, y , t + At) = a ( x , t )

At->0 At J {jjc— } (driftcoefficient)
(c) ]im 4 - (y - x ) 2 dFy ( x , t \ y , t + At) = b ( x , t ) > 0

J{|jc-y|< < 5} (diffusion coefficient)
exist, the convergence being uniform in x.

Condition (v (a)) implies that small changes of state occur continuously during
small intervals of time, i.e.
P[| X(t) - X(s) | > <5 | X(s ) = x ] = o(t ~ s ) ( s < t).
Definition 8.1 A Markov process that satisfies axioms (i) to (v) is called a
Diffusion process.
Special Case (When density function exists)
If / ( y , s; x, t) = F(y, s \ x, t) exists then

/(y , 5 ; x, f) is called the Transition density function. Transition density of a M.P.
satisfies.
(iii) /(y ,j;x ,f ) = J f(z,T\x,t)f(y,s;z,r)dz
(Chapman-Kolmogorov equation)
(iv) /(y, t\ x, t) = <5(x - y), Dirac delta function,
where <5(x) = 0 for x ^ 0 but f 8 {x)dx = 1

J—00
(<5(x) is also called Generalized Distribution in Mathematical Physics).
If F(y, 5 ; x, f) = F(y, 0; x, t - s) depends on (t-s) only, then the Diffusion
process is called Homogeneous. For homogeneous process we write F(y \ x, t) =
P(X(t) < x | X (0) = y).
Example 8.1 A Diffusion process can be considered as the continuous limit of
a simple (homogeneous) random walk.
Informal Proof
Consider the n-step transition probabilities in a simple random walk P(Sm+n = j
| Sm= i) = P(X{ + ... + Xn- j - i \S 0 = 0) = P(Sn= j - i), whereXf-’s arei.i.d. Bernoulli
random variables taking values 1 and -1 with P(X{ = 1) = p and q = 1 - p.
0 if n + j - i is odd n-j+i
Then p( n + j- i) / 2 q 2
Pr = if n + j - i is even
(n + j -
= v ( j -i, n) (-<» < i j < °°),

( \ n+ X n~ X
where n) = ( (/| j p* q '
Here x is the displacement of the particle (performing random walk) in n~steps.

Consider a (Brownian) particle performing a random walk such that X is the
displacement in time t in rc-steps.
Now we assume that each step has length Ax and time between two consecutive
steps is At. Then n = i.e. during the interval (0, the interval of length t is
divided into n equal sub-intervals of length At. Note that E(X) - n(p - q) and
V(X) = 4npq.
n
If the total displacement during (0, t) be X(t) = I Z , where Z; are i.i.d.
Bernoulli r.v.s. taking values ±Ax, then
E[Total displacement] = n(p - q)Ax = - ~ ( p ~ <?)(Ax)

and
Var [Total displacement] = n • 4pq(Ax)2 = - —(4 pq)(Ax)2
In order to get a meaningful result we have to assume that
(Ax)2/At —>a limit 2D if At —» 0 and Ax —> 0
and p - q = / ? - ( ! - p) = 2p - 1~ — Ax: (a multiple of Ax).
where C and D are constants. (8.3)
C (Ac)2 a
E [Total displacement] ~ t
D D
= 2 Ct ~ a(x, 0 = m(0, the mean function
Var [displacement] ~ 4t ^ - C 2 At j — ■
—» 2Dr - 6(*, 0 = o2^), the variance function.

Thus the Binomial distribution v(x, n) of displacement tends to f ( x , r) dx, where
1 (x-m(t))2 (by CLT)

/< x - ' ) = v r i ^ e’‘p 2 o 2 (t)
that is, the displacement during (0, f) X(£) - X(0) and the increments in the
displacement during (0, t) X(t) - X(0) and the increments in the displacement
during 0 < s < t that is, N((t - s) 2C, 2D(t - 5)). p \ p satisfies recurrence relation
p \ ^ X) = p • p j + qp^jh which can be written as v (j, n + 1) = pv (j - 1, n) +
qv(j + 1, n) and in our new units
v(x, t + At) = pv(x - Ax, t) + + Ax, t) (8.4)
By Taylor’s expansion and substituting p, q from (8.3) in (8.4) and proceeding
to the limit as Ax —> 0,
v(x, t + At) = v(x, t) +At + o(At) (8.5)

ox
± Ax, t) = v(x, t) ± A3
^ + (Ax)2 ■+ ( 8. 6)
dx 2 dx 2
By (8.3), (8.4) and (8.5) and letting 0, we get
^ i £ )= d |! £_2C ^,
dt ddx
the forward Kolmogorov partial differential equation for Diffusion process as

the continuous limit of the difference equations in two variables of homogeneous
random walk.
8.1.1 Alternative method to show Brownian motion (most important

example of a diffusion process) as an approximation of simple
random walk
Suppose Brownian particle starts at the origin and jumps every At seconds,
moving to the right a small distance Ax with probability 1/2 or to the left a
distance Ax with probability 1/2. If Xn(t) is the position of the particle at time t
= nAt (n being large and At being small time interval), then Xn(t) is the sum of
independent r.v.s. Yx, Y2, . . . ,Y„, where P{Yt = ± Ax) = 1/2 for all i > 1. Now Var
(Xn(t)) = n{Ax) 2 and hence
y. + + Y r- i_
Xn (t) = ------'f=------ — ^fnAx = Zn 'JnAx,
VnAx
where E(Zn) - 0, Var (Zn) = 1.
Now n - t/At and *fnAx - ^ A x / J ~ A t . If we assume that (Ax)1!At —» o2 > 0
as At -» 0, the CLT for i.i.d. r.v.s implies Xn (t) —1d-> N(0y cr2r), that is
Xn (t) W(r). Now we show that all the finite dimensional distribution of
1X^(0} converges to that of (W(t), t > 0}.
If 0 < * ! < . . . < then
Xn (t j ) - x n (tH ) W(tj) - W(tH ) for each fixed j > 1.
Since the random walk and the Brownian motion (W(r)} have independent
increments, the joint ch. f.s of Xn(tx, . . . , Xn(tk) is
exp \i S u jXn (t A
7= 1
k
= E exp z Uj(Xn(t 2 ) ~ X n( ti ))
7=2
E[exp {iuk(Xn(tk) - Xn(tk. x))}]

—> 0(wj, . . . , uk), the joint ch. f. of W(tx)f . . . , W(^).
Hence by continuity theorem for ch.f. the random
vectors (X„ ( f , Xn(tk)) ...,
In principle, Kolmogorov’s consistency theorem proves the result. But complete
proof of convergence of the entire process to Brownian motion needs techniques
(Invariance principle and tightness-see Billingsley (1968)) beyond the scope of
this book.
8.2 Kolmogorov Backward and Forward Diffusion

Equations
There are various methods for determining transition probability function or
transition distribution of a Markov process, ranging from purely analytical to
purely probabilistic. The method presented here was developed by Kolmogorov
in 1931.
Theorem 8.1 Let {X(t), t eT] be a diffusion process. Assuming that
3 32
— F(y, s;x, t) and —^ F(y, s; jc, r) exist and are jointly continuous in all the
oy dyz
variables, we get
F(y, s;x, t) = - a ( y , s) F(y,
the Generalized Heat equation or the Backward Kolomogorov’s Differential

equation.
Assume that /( y , s\x, = y »s ' x ' 0 exists and that f ( x , t) =/(y, s\ jc, t).
Also assume that ~-[a(jc, t ) f ( x , /)] and -— -[£(*, t ) f ( x , t )] exist and

df dx 3jc2
are continuous in all the variables. Then we get
[a(jc, * )/(* . 0] + 5 0 / (*, 01.
the Kolmogorov’s forward differential equation or the Fokker-Planck equation.

Proof of Backward Equation
F(y, s - As, x, t) - F(y, s,x, t) = J F(z,
J —As;
Let I= J* [F(z, s\x, t) - F(y, s\x, t)]dFz (
By assumptions for | z - y \< 8a

nd for some 0,
9F (y,s; x,t)
F(z, s,x, t) = F(y,s-,x, t) + (z - y)
dy
. x1 , x2 d 2 F ( y , s - , x , t ) n2n
+ 2 ( * - y> -------— ------- + o((z - )•
dy 2
Then
/
-JJ \z- y \> S
[F(z,s\x,t)~ F(y,s;x,t)]dFz(y ,s-As;z,s) + I
J|.iz-si<5
Kz-
°y
+ 2 (z - y)2| ^ £ F
(y,s-,X
,t) o((z ~ yf)]dFz(y, - As; z, s)
~ /i + h-
where \I{| < 2| I dFx (y, s - As;z , s)\, and

J\z-y\> 8
h = f (z - y)dF. (y, j - 4s; z, s)
+ —d 2 F/dy 2 f ( z - y) 2 dFz (y,s - As\z,s) + o ( ( z ~ y ) 2).

Z J\z-y\<8
Dividing by As and letting As —>0 we get the left hand derivative

dF
lim (F (y, s - A s \ x , t )- F(y ,s\x , t))/As = - - j -
A s —>0 OS
as a ( y , s ) ^ F ( y , s \ x , t ) + - b ( y Js ) —- £ F ( y , s \ x , t )
(first term l\ tends to 0 due to u(a)).

Similarly, the right hand derivative exists and is given by the last expression.
The proof of Kolmogorov’s forward equation is difficult and is given at the end
of Ornstein and Unlenbeck process in section 8.5.
8.3 Wiener Process (Brownian Motion)

Definition 8.2
Let {X(0, t 6 T) be a continuous space and continuous time stochastic process
satisfying the following conditions:
(i) P{X(0) = 0} = 1
(ii) (X(r), t e T] has independent increments
(iii) {X(/), t eT] has stationary increments
1
(iv) P [ X ( t ) - X ( s ) < x ] = exp[-M2/2D (f - s)]du
(2nD(t- s ) ) 112
Then (X(f), t e T] is called Wiener Process (or Wiener-Levy process or Brownian
Motion).
The erratic movement of small particles suspended in fluids, caused by impacts
of the molecules of the fluids, is called Brownian motion (after Robert Brown,
1827, an English botanist). Statistical treatment of this was given by N. Wiener
(1923). Before that Einstein and Smoluchowski (1905) also used it. We shall
outline the theory as given by Uhlenbeck and Ornstein (1930).
Theorem 8.2 Wiener process as defined above is a Diffusion process.
Proof The transition distribution is
F(y, 5; x, t) = P[X(t) < x | X(s) = y]
= P[X(t) - X(s) < x - y | X(s) - X(0) = y] (from (i))

= P[X(t) = X(j) < x - y] (by independent increments)
1 e\p[-w a /2 D(f - s)]du

(2 n D ( t - s ) ) ' n wr — oo
(x -y)/D ( t- s )
= —i — f exp(-w2 ! 2 )du (8.7)
■J2n J_,
Obviously (i) F(y, s; x, 0 > 0; (ii) F(y, x; = 0 and F(y, s\ 1.
f0 if x < y
(iii) F(y,s;x=
[0 if* > y
From condition (iv) of Wiener process, the transition density exists and equals
1 -(x-y)1!\2D(t-s)\
f(y,s\x,t) =
(2 J tD { t- s )) m
and satisfies Chapman-Kolmogorov equation
(iv) f(y ,S\X,t)= j* f(z, r ,x,t)f(y,s;z T
Check that by putting, for example, y = s = 0 we get

1 -jc z /2 Dt
/(0 , 0; jc,O =
(2 n D t ) ul
and using the simple additive property of normal distribution, viz. that sum of
two independent normal distributions with 0 means and variances , cr|
respectively is again normal with mean zero and variance cr2 + cr|
(v(a)) Jim 4 - I dFv(x, > 0
= lim -j-f L—
t|Jy-x|>
A SJlFDiAT)
-M2/2
= lim I du y
AI-+0 At V2 nJ\ujDAi\>8 V d ZF
= nm ± J i r e 14 12 du (by symmetry)
At->0 At v 71 JSi
S/fDAt
= lim . ulDALe-uh 2 du s <u ^ < 2l ™

0 At 5ljDAt O VDd7
= 0 (by absolute continuity of integral)
(vb) lim
fcl
. (y-*)
^jlnDAt
1 ~ - ( y - x ) 2 l(2 D A t)
dy
= 0 (by symmetry)
= a (x , 0-
(v(c)) lim At ( y - x )2 ■. . .1 e -(v-j02 /(2D40 ^

A t- > 0
I |y -x |< < 5 - y J ^ T l DAt
lim 1___ 1 _ f u2
£ "2/2 du(puty - x = u-yfDAt)
At-ô At
^ 4 l n J|„|<s| / d I7
D r I 1 uwe2du (by Monotone Convergence Theorem)

-J2 n J —oo
= D = b(xt)
Note (a) If the Diffusion process is homogeneous, then t) and t) are
independent of t.
(b) If the Diffusion process is additive (i.e. given X(r0) = x() the increment
X(t) - X(t0) depends on t and t0 and not on a0) then a{x, t) and t) are
independent of x. For example Wiener process is additive.
8.4 Simple Properties of Wiener Process and

First Passage Time Distribution
Some Simple Properties
(1) A Wiener process with drift coefficient fU= a(x, t) and diffusion coefficient
c r = b(x, t) = D satisfies Kolmogorov’s forward partial differential equation
V +—1 (T2
2 ---—
d2/
dt = - » 2 U dx 2
( 2 ) It is not covariance stationary, since
Cov (X(t). Xu)) = a 2 min (r, s) (compare with Poisson process)
(3) Inversion
K(r) = tX(llt), 0 < 1 < 1 with T(0) = 0 a.s. is again a Wiener process.
Since P(Y(t) < y]= P[tX(\!t) < y]
=P w i l l < ill
aJUt oJUt
= & ( y ! o 4 t ),where <t> is the c.d.f. of 1).
Hence, E(Y(t)) = 0 and Var(K(/)) = t2a 2lt = a 2t. In fact X(1 and X(Uu) - X(1
are independent for 1 It< 1 lu.The independe
by showing that they are uncorrelated.
E(uX(l/u)[tX(l/t) - uX(Uu)}
= l/)EX
(utX (1It)- u2X2 (Mu)) = - u\Uu) = 0
(since E(X(s)X(t)) = min U. /)). Hence for 0 <
E(exp [id{tX(l/t) - mX(1/m)}])
= E exp {i0((r - u) X(1 It)- u[X(l/u) - X(l/r)])}
= exp j - 0 2 / 2 ( r - u ) 2 1It - J ~ 1/0
= exp { - 6 ll 2 (t - u)}.
Hence Y(t) - Y(u) has Gaussian distribution.
(4) (a) X(0, t < 0 is a Wiener Process (follows by symmetry).
(b) (Origin change and strong Markov property). {X(t + r) - X(t), t > 0, T> 0
may be a stopping time} is a Wiener process
(c) j~ r * a (constant) > 0 1 is a Wiener process (scale change).
(d) \X(T) - X( T- t), 0 < t < T (fixed)} is a Wiener process.

Proof of (2) We shall show Cov (X(t), X(t')) = ci2 min (/, tf).
I f 0 < / ' < r , Cov (X(0, X(t'))
= Cov (X(t'), X(t') + X(t) - X(t'))
= Cov (*(*'), X(t')) + Cov (X(t'f X(t) - X(t'))
= Var (X (/')) + 0 (due to independent increm ents)
= crV.
Similarly Cov (X(t'), X(t)) = a 2t if 0 < t < t'
Therefore Cov (X(t), X(t')) = a 2 min (t, t').
Min(r, t')
Hence Corr (X (f), X (f')) = P(C t ’) =
Vm 7
8.4.1 First passage time distribution for Wiener process
The possible realizations of a stochastic process are called sample path or
Trajectories. For a Wiener process the sample paths are almost surely continuous
functions (see Appendix 1). Let us state the following well known result in
Probability Theory which will be useful for our next result.
D. Andre’s (1887) Reflection Principle There is a 1:1 correspondance between
all paths from A(au a2) to B(b]y b2) which touch or cross the x-axis and all paths
from A'(ab - a2) to B(bx, b2).
Let Ta = inf {r: X(0 > a) be a random variable and in fact a stoping time
(continuous valued). Then [Ta, a e R+] is a stochastic process.
Lemma 8.1 Let (X(r), 0 < t < T) be a Wiener Process with X(0) = 0 a.s. and
P = 0. Then for any a > 0,
P[M(T) = max X(t) > a] = 2 P[X(T) > a].

o < t< T
(This is due to Bachlier (1900)).

Proof Consider the collection of sample paths {X(0, 0 < t < T] such that
X(T) >a,a> 0. Now X(0) = 0 and X(t) is continuous, there exists a random time
ra t which X(t) first attains the value a, denoted by Ta. The hitting time (or first
passage time) Ta is a random variable.
For t > Ta, we reflect X(t) about the line x = a to define
\X(t) \ i t < T a
Xa(t) =
[a - [X(t) - a ] = 2 a - X(t) if t > Ta
i.e. the new Stochastic process Xa{t) gives for t > Ta the reflection of X(t) about
the line x - a.
By symmetry, X(t) and X0(t) have same probability distribution.
[M(T)> a] = [M(T) > a, X(T) >a]U[M (T ) > a, X(T) <a]U[M(T)> a, X(T) = a]
Note that Xa(T) < a, since X(T) > a. Because the probability law of the path
for t > Ta given X(Ta) = a, is symmetrical with respect to the values x > a and
x < a and is independent of the history prior to time Ta. From reflection principle
to every sample path X(t) for which X(T) > a, there exists two sample points such
that M (T ) > a and vice versa i.e. to every sample path X(t) for which M{T) > a
there corresponds two sample paths X(T) > a and X(T) < a with equal probability.
P[M(T) >a] = P[M(T) > a, X(T) > a] + P[M(T) > a, X(T) < a]
(Since P[M(T ) > a, X(T) = a] < P[X(T) = a] = 0, the other two are mapped onto
one another by reflection about the line x = a after the time Ta)
= 2P[M(T) > a, X(T) > a] = 2P[X(T) > a].
Note Tm a> 0 is a.s. right continuous but may not be continuous. P[M{t) > a] =
2P[X(t) > a] with ju = 0, X(0) = 0 and a = 1.
f
V27Z? "J„a
e x p (-* 2 1 2 t)dx
=~i= f exp
x/2 n J ai41 p u tV7
= 2 1 - 4> a
vV7y
Hence < /] = F[A /(0 > a] = 2

i - * ' t
2^ 9
=J L f s 3/2 exp d5,5 > 0 s = fl f
V2tf Jo 2 ,y *2 V ;
( „2 \
h , (0 = K (0 = ^ exp ,t > 0
2 f,
is the density function for first passage time of the standard Brownian motion.
E{Ta) = °°, since E(Ta) - | P(Ta >x)dx

Jo
M OO MOO
= {1 - (2/7T),/2 I y-i\!2)Z2 fa
JO Ja/Jx
f(i/Jx ( \ 1/2 /•oo fAT2

e - (\/2 ) z2 I dx
- <2",MI AJ. <ra, * - 0 f) Jo Jo
= a 2(2lx)'n f" - y * > 0 : (2 /< r)l,!

Jo z
In fact Laplace transform of Ta is 0(5) = exp ( -a ^ 2 s ) and hence no moment
of Ta exists.
But if /a * 0 and <7 > 0, i.e. if X(t) is an Wiener process with drift p and
diffusion coefficient cr then the first passage time r.v. T* has density
t~m exp (a z £ ) l , t > 0.

fr :(0 = o jln 2 a 2t
Then, Laplace transform will be exp ^ 2 + 25<t 2 } .

.<7 J
Hence E(T*) = a / n and V ar(T*) = a a 2/^.
If ]U> 0, then / * (0 is called Inverse Gaussian density. For properties of this
distribution, see Johnson and Kotz (1970) (Statistical Distribution, Vol. I) and for
statistical applications, see Folks and Chhikara (1978) in J.A.S.A.
8.5 Ornstein and Uhlenbeck Process and Gaussian

Processes
8.5.1 Ornstein and Uhlenbeck process (1930)
We have seen that for a Wiener process X(t), the displacement Ax in a small
interval of time At is small, being of the order 0 ( ^ At), the velocity of which is
0 (■\J~At/ At) = 0(1/-yf~At) —^ °° as A t —^ 0.
Thus the Wiener process does not provide a satisfactory model for motion of
small values of t although for moderate and large values of t it does so. An
alternative model which holds for small t was proposed by Ornstein and Uhlenbeck
in 1930.
The theory of Brownian motion developed by Einstein and Smoluchowski
were not based on Newtonian mechanics. Langevian’s approach, on the other
hand, relies heavily on Newton’s Second law of motion. In what follows we
shall give a brief account of Langevian’s model. Let V(t) denote the random
velocity of a large solute molecule immersed in a liquid at rest. For simplicity,
let V(r) denote the vertical component of the velocity. The solute molecule is
subject to two forces: gravity and friction (resistence). It turns out that the
gravitational force is negligible compared to the frictional force exerted on the
solute molecule by the liquid. The frictional force is directed opposite to the
direction of motion and is proportional to the velocity in magnitude. In the
absence of statistical fluctuations, one would to the direcion of motion and is
proportional to the velocity in magnitude. In the absence of statistical fluctuations,
one would therefore have mdv (t) dt = -fidv(t), where m is the mass of the solute
molecule and /3the constant of proportionality known as the coefficient of friction.
However, this frictional force ~/3V may be thought of as the mean of a large
number of random molecular collisions. Assuming that the central limit theorem
applies to the superposition of displacement due to a large number of such
collisions, the change in momentum mdv (t) over an interval (/, t + At) is
approximately Gaussian, provided dt is such that the mean number of collisions
during (f, t + dt) is large.
In the Ornstein and Uhlenbeck process instead of the displacement X(t),
V(t) = X'(t) is assumed to be a Brownian motion.
Let u0 = V(0) and {V(t), t e 7} be a stochastic process representing velocity
of unit mass molecule. The equation of motion of Brownian particle can be
written as
--j- = -pV{t) + f ( t ) (Langevian equation)
Since the derivative may not exist we replace it by AV(t) = -(3V(t)At,

where f(t)dt is replaced by AF(t). Here -fiV(t) represent a symmetric part due
to resistance of the medium and dF(t) represents the random component.
Assume that F and V are independent and F(t) is a Wiener Process with ju = 0,
Var (F(t)) = a h = 2 Dt.
E(AF(t)) = E[F(t F(t)] = 0

V(AF(t)) = 2DAt, >0
E(AF(t)AF(s)) = O i f r < r + z l r <s <. s + As.
Then
a(v,t)= \ i mo ± E [ A V { t ) \ V ( t ) = v ]
= lim -^7 { - P E [ V O ) A t \ V(t) E[AF(t)\V(t

At-^>0 /At
= lim RE[(vAt)]+ E(AF(t))] (Since is independent of V(/))

ZW —>0 At
= - pv
b(v,t)= \\mn ±-E [(AV(t))2\ V(t) = v]
(At ) 2 . V ar(4 F (0 ) „ „
= lim —r— + lim ------- = 2D.
At-Ô At A t-* 0 At
Since these limits exist, the stochastic process V(t) is a diffusion process and
its transition density function g(v0, v, t) satisfies the forward Kolmogorov equation.
More specifically let g(v, t)dv = P[v < V{t) < v + dv | V(0) = v0].
Hence, the Kolmogorov’s forward equation is
dh
(vg) + D ( 8 . 8)
dv dv 2
with initial conditions:
g(v, 0) = d(v - u0), Dirac delta function.
vg(v,t) —> 0 if v —» ±oo and ^ —> Q if v —> ±°°.

ov
To solve the differential equation for Ornstein-Uhlenbeck process ope of the
methods used is called separation of variable, i.e. we assume g(v, t) = V{v)T(t),
say and use the so-called Weber’s function. But we shall use the method of
Laplace Transform (ch.f.). Let 0(u, t) be the ch. function of g, i.e.
J ei9v g(u, t)dv = 0 (u , 0-
From equatin (8.8) we get
ei6u j ^ ( v g ) d v J ei6v
Integrating by parts and using the boundary conditions we get

5 0 ( 0 , t) _ _ £>020^ a jjnear p.d.e of Lagrange type.
ot 06
The characteristic equation is given by
dt dO d0
Hence, Pdt = =$ fit = log 0/c{ => = 6 /cx.

Therefore cx = 0e~ (8.9)
Also ^ = - - | 0 J 0 = * l o g 0 = --— + logc.
Therefore 0 ( 9 , t) = c 2 e~De~'2 P. ( 8 . 10)
Also c2 = W(ci), say.

Eliminating one of the constants c x and c2., we get.
e D0 2 l2 P0 ( e t ) = w ( Qe -fSt) ( 8 . 11)
Initial condition gives g(v, 0) = S(v - v0).
ei9v g ( v , 0 )dv elf>v S(v - v0)dv
e‘ev°
Therefore 0 (0 ,0 ) = ei9v° = exp(- ( 8 . 12)
and y/(0) - e(D°2 / 2 P)+i8 vo5( js the Dirac delta function)
Hence 0 (0 , t) =D02/2êxp ^ ( e e - P ' ) 2 +i(de-P')vo (8.13)
From (8.11) and (8.12) (8.13) can be written as

0(0, t) = exp [i6 m{t) - 1/2 0 2cr2(f)],
where m(t) = v0 e P f and a 2 (t) = ~ (1 - e 2^).
Note Y(t) = e-ptX(ae2pt) is the Ornstein-Uhlenbeck process if X(t) is a Brownian

Motion.
Therefore Y(t) = — X(ag(t)), where g(t)
So, Ornstein-Uhlenbeck process is a time transformation of Brownian motion.
Now we want to study what happens if t —> °o.
m(t) —> 0, for P > 0 and a 2 (t) —> = (Tq (say) when f —» <».
Then g 0 (v)dv = lim g(v,t)dv
1 - U v / 0 0)2
= - ---- e
7— 1 dv, i.e. the distribution of velocity
V27T (To
is normal with mean zero and variance Dip. In Maxwell Boltzman distribution
(kinetic theory of gases), <Tq = where ATis the Boltzman constant, T the
temperature and m the mass of the particle.
They assume the initial distribution is MB (Maxwell-Boltzman), i.e.
P[v 0 < V0 < v 0 + dvQ] = e -|/2(u°/<T)20•
The unconditional density of velocity is given by

g(v, t) = P[v < V(t) <v + dv]
P[v < V(t) <v + dv | V(0) = v0] P[vo < V(t) < v 0 + dv 0 ]dv0
- [
.If I l f
o A <nn J -I ffo I dv o
^/(2?rcr2 (f)) J(27ral)
2 rr^ i| &
<y{t)
f (2^<r2( f) )f (2^ cTo) r dv o
1 -~(l;/CT0)2 I , V,
e 2 dv0 =go(v)dv0
*j2 n o 0
which is known as the Maxwell distribution of velocity of gases (<7q = £>//?).
Hence, if v 0 is Maxwell Boltzman then E(V(t)) = 0, Var(V(r)) = cr^ and
Cov (V(f), V(j)) = a 0V ^ ' - vl.
Thus, V(r) is covariance stationary.
But K(r) = —dx where X is the displacement. Hence
E(X(t) )= v0, Var (X(r)) =-1 +/fr).

P
**Proof of Kolmogorov’s forward differential equations for diffusion process
Let Q(X) be an arbitrary continuous function with Q\x), Q"{x) and Q(x) = 0
outside [a, b]. Then from continuity Q(x), Q'(x) and Q"(x) vanish at jc = a and
x = b.
Consider
/= { Q {y)^f(y,t)dy= ^ ^ Q(y)f(x0 ,r,y,t)dy
(since Q(y)f(y, t) is continuous in y and t and df/dt is continuous)
= lim
At->0 "a f Q(y) f ( x 0 ,T;y,t+
At
Aty t)
dy
= HZ-T t {j d y Q ( y ) ^ f(x
-r"Cl
Q ( x ) f ( x 0 , r ; y , 0 dy j* (by Chapman-Kolmogorov)
= lim J /(x 0T;y,o|J f i x , t;y, t

A t-* o
[For interchanging the order of integration:

g(x, y) = Q(y)f(x, t\ y , t + At) /(x, v, y, t)
is continuous in
(x, y) e (-« , oo) x [a, 6], | 0(y) | = | f

J — oo
#(■*>y)dx\
; GOO if
•J —o
/(x,
is uniformly continuous]
<2(x) being continuous on [a, 6], it is bounded by diffusion condition v(a) and
hence we get
Thus,
I
l.y-*l><5
/( x , t \ y , t + At)dy = o(At), as At —> 0
/ = lim
At-*°>
d x f ( x 0, r ; x , o j j
Expanding Q(y) in the neighbourhood of x, we get
GOO = Qix) +(y-x)Q'(x) + I (y - x)2 G * (jt) + o ( y -
Using (8.14) and diffusion conditions (a), (b) and (c), we get
1 |y-*|£<s
f ( x , t ; y ,t +At)Qiy)dy - Qix)
1|y - A |< 5
f i x , r,y, t + At)[Q(x) + (y - x)Q'(x) + ^ i y - x ) 2 Q"(x)
+ o(y - x)2] dy - Qix)
= oiAt) + \x) f (y —x )/(x , t \ y , t + At)dy

*Iv-.
\y - x \ < d
+ jQ "(x) f
*\y-x\<d
[(y - x ) 2 + o(y - x )2]/(x , t\y, t + At)dy
(since by diffusion condition (v (a))
I
J|;y-Jt|< 8 |y-x|<3
J |v -.
f
f i x , t \ y , t +At)dy/ (=x ,1V-, f, + At)dy = 1 -
Therefore / = I /(jc 0, r;* , r)[Q,(x)a(jc, r) + 1/2 "ix))]

Qr (8.15)
-I
[by diffusion conditions (v(b)) and (v(c))]

Integrating by parts
[
( a f ) Q \ x ) d x = [ a f Q ) ba
•b
r
"a
-(af)dx
Q(x)-^(af)dx
(8.16)
-]> *
(since Q(a) = Q(b)= 0)
and (bf)Q"(x)dx = [bfQ ' t - f Q''(x)4fc(bf)dx

[ *a
= - fQ'(x) J ^ ( bf)dx (v Q'(a) = Q'(b) = 0)
= - j b Q(x) (bf)dx
= - -H-(bf)Q]ba + f Q(x) ^ - ( b t f d x
(bf)dx (8.17)
By (8.15), (8.16) and (8.17), we get
/=J 1/2 £ T (bf)^Q(x)dx
_ r w
Since / Q(x)
.L dt
r
* -o o
^ + ± { a f ) - l / 2 £ - (bf ^ Q i x ) d X = 0.
Since the last expression in {•} is a continuous function and Q(x) is arbitrary,
fv . ±
l r + i < « / ) - i / 2 - W ) = o.
dt'dx J '" ” dx 2
8.5.2 Gaussian process

Definition 8.2 The stochastic process {X(0, t e T) is said to be a “Gaussian
process” if for each {fj c T, n > 1, (X(^), . . . , X(tn)) is multivariate
normal.
The covariance function TO, t) = £(X0) - fi(s))(X(t) - //(/)) possess the
following properties:
(i) r(j, t) = T(t, j);
(ii) For any { ^ ,. . . , tn] c T, n > 1, and any real numbers ux . . .,

, I X
i=i i=i
utujFiti ytj) > 0 i.e. the covariance function is non-negative definite.
This follows from the fact that the covariance matrix of the multivariate
normal random vector is positive definite.
Theorem 8.3 The Gaussian Stochastic process {X(0, t e T) is stationary iff
the covariance function T (m, n) = V (m - n, 0) for all m, n.
Proof If {X(/)J is stationary then E(X(m)X(n)) = E(Xm^JCn_m) = T(m - n, 0).
Conversely, if true, then the characteristic function of X(l), X(2), . . . , X(n) is
exp ^ 1/2 2 2 ujukT (k - y, 0)j.
But this is exactly the characteristic function of X1+m, . . . , Xn+m.

From uniqueness theorem of characteristic function the result follows.
Theorem 8.4 If a Gaussian process {X(0, t e T] have mean function EX(t) =
jji{t) = 0, and TO, t) = EX{s)X(t), then it is a Markov process iff
r(rj, t3) = ■■ —- for any tu t~>yt3 e T with tx < ti < t 3 (8.5.1)

A(h* h )
Proof
Let us first prove that if X = (Xb . . . , Xn) is multivariate normal N(0, 2),
2 = (a,,)"1 then
£(X „|X . =jr„ i= l ^ o nn J

(8.5.2)
n n n-1 n - l n -l
Now 2 2 OijXiXj = 2 Z <7y.x,.r, + xj?<7„„ + 2 x n Z cr,,,*, (8.5.3)
/=1 ; =1 <=1 7=1 ;=1
Then
n -1
exp(-l/2cr„„.^ - Z a inXi)
1= 1
r
*n-i) = n -l
exp(-l/2crnAI^„ —x n ^ OinXi)dxn
V —o i- 1
n-l
exp <- 1 / 2 <7*, ■^n ^ (Gin ^Gnn)Xi
i=1
l
n-l 2 '
f " e x p ■ - 1 / 2 ( 7 „„ ■^n "A" ^ (^Ln ! Gnn )■*■*
J -o o i=l
I 1
n-l
x C exp j -l/2cr„ + Z ( CTin (8.5.4)
/=!
endent of x u . .., jc„_i. Equation (8.5.4) is clearly a
/?—
1
normal p.d.f. with mean - 2 (o in! o nn)Xi.
1=1
Now if X(t) is a Markov process, it follows from (8.5.2) that for any s<t from 7,
E(X(t)\X(s) = x) = - o X2/o22x (with n = 2).
-1
r(.s, 5) - r ( j , t)
Now a, are elements of Z 1=
-r(/,j) r(r,r)
1 T( / , / ) - r ( j , o '
T ( ^ , 5 ) r ( / , r ) - r 2( 5, 0 - r ( j , 0 r ( i , i )
Therefore £ (X (0 |X (s) = x) = x (8.5.5)

r (s ,
Since given the present, in a Markov process the past is independent of the future,
r (f b h) = E(X(t{)X(t3)) = E{E(X(tx)X(t3) | X(f2))}
=£ { « | X(t 2))E(X(t 3 | X(t2))}, tx <t2 < t3
Hence by (8.5.5) we have
( T(t\ , t2) F(t2, t3)
T(tx, t 3) = E X(t2) X(t2)
k F(^2»h) r(* 2 *h )
r r ( t i , t 2 ) r ( t 2 , t 3)^ E{X{t 2 ) ) 2 = E(t 1, t 2 )T(t2, t3) (8.5.1)

( r ( t 2 , t 2) ) 2 r ( r 2» ^ ) 2
for tx <t 2 < t3.
To prove the converse, i.e. a zero mean Gaussian process whose covariance
function satisfies (8.5.1) must have the Markov property, we first prove that a
zero mean Gaussian process is Markov iff, for any
t\ < t2 < . . . < tn, n > 2,
E{X(tn) | *(/,) = jcj......... X(tn_x) = xn. x] = E(X(tn) | X(Vi) = xn. x] (8.5.6)
The necessity is obvious. To prove its sufficiency note that from (8.5.2) and
(8.5.6), it follows that
G\n = a 2n = • • • = Gn-2, n = 0
~\2
G n-\n
so that by (8.5.4)/(x„| x h x2,■■• ,x„-i) = C exp -j - j Xn + —-----Lj-1
-|2
Obviously /( x „ | = C. exp I - - - <7„ An ~ _ An-1
Thereforey(;c„ | jcb . . . , jc„_t) =/(x„|xn_1) i.e. X(0 is a Markov process.

Let us now show that (8.5.1) implies (8.5.6). Equation (8.5.1) implies that for
l < k < n - I.
, tn-[)T(tn__x, tn)
» t n) - (since tk < r„_i < tn)
T(tnz.x, rn-1)
and that implies
r ( ^ _ i »tn) ^ N
X{tn) - *('*) =0
for all it = 1, 2, . . . . n - 1, = X(t„) - X(f„_i) are normally

1 , t„_i)
distributed r.v.s, are independent of X(^), . . . , X(tn_j). Therefore,
s jx
[ uj - U
rr//"~1
'/i-h ,/
V "lJ\ x(/"-i)lX(f1) = ■*■•• • •’*('«->) =
or E{X(tn)| X(f,) = x,.X(f„_,) = }

= n v , ,fj ^ =
1 , tn_\)
This proves (8.5.6) and hence the desired result.
8.5.3 A>Dimensional Brownian motion

Let {Xj(0; t > 0}, . . . , [Xk(t), t > 0}, be k independent Brownian motions, i.e.
for any finite set of points
*11>*12» * • • * * l n i * * 2 l , / 2 2 * * * ’ ' t l n 2
and tk] , t 2k , • • • *h,nk *
the k vectors
Xl (t]) = (X l (tu \ . . . , X l (t]ni) \
X 2 (t2) = (X 2 (t 2 l) , . . . , X 2 i t 2n2) ) 9
x k (tic) - (Xk (tk\)»•••» Xk (tknk )),

are independent. The vector-valued process defined by
x(t) = (*!«,..., x km t > o,
is called /:-dimensional Brownian motion.
The movement of a particle undergoing Brownian motion in the plane and in
space are described by two- and three-dimensional Brownian motions, respectively.
8.5.4 Radial Brownian motion (Bessel process)

Let (X(f); t > 0} be a ^-dimensional Brownian motion process. The stochastic
process defined by
R(t) =[ X,2 (0 + . . . + X t )]1/2,
is called the Radial Brownian motion or the Bessel process with parameter
(1/2) k - 1. It is a Markov process having continuous sample paths in (0, °o). The
transition probability density function from x to y is
pin X, y) =t 1exp {-(x 2 + y 2)l(2t)}{xy)x U2t\kn_\-\(xylt)yk ', t > 0, *, y > 0

where Ij(z) is the modified Bassel function
Ij(z)=S ( z/ 2)2k+Jk
[/ ],1
J k=0
We shall prove the case k = 2.
Use the polar coordinates by defining
R(t) = JX
)2 ( 0 + * ! ( / ) , 0(f) = tan"1
Since Brownian motion is Markov,
P[R(tn+ t )< >-1X(f0) = x , . . . , X ( r „ . 1) = x

~0 ~ n- 1
= + t) <y\X{tn)P[R(t) < y
where 0 < t0 < . .. < t„ , t> 0, and x , . . . , are arbitrary points in the plane,
~0 ~ n- 1
x = (jcj, x2). Then
P[R(t) < y | X(0) = x]
= ff _L exp (u\ - -Xi)2 + (u 2 - X2)2

du\du 2
JJ U} +Uj < y 27
21
(r sin - x \ ) 2 + (r cos - x 2) 2
■ffJTi
0 0
exp d 6 rdr
It
where transformations U\ - r cos 0 and u2 = r sin 0 gives dux du2 = rdrdO. Since
(r sin 0 - x x) 2 + (r cos 6 - jc2)2 = r2 + 2r(jcj sin 0 + jc2 cos 0) + || x ||2
where ||x\\ = x 2 + x \ ,
/>(/?(') <y|X (0) = * ) = JJ 2^ exP j/(r ,x )rfr .
where I(r ,x ) = J exp |y(jci sin# + x 2 cos0)J</0.
Defining an angle y/ by writing

sin y/ = jc^H x ||, cos y/ = jc2/|| x ||, we get
• 2/r
,*) = f
I(r,x)= I
Jo
exp £H 1 cos (y/+ 0) \ d 6
12
exp n m i cos 0 ^ 0 ,
■I 2/
Since cos (y/+ 0) is periodic over the interval [0, 2 n],
I[ ln exp ,(a cos 0) dO = I 2- n, C n —V

------ ( adO
------ c o s=d )2-k -7-7
Jn I v cos a* k6f 2"
d6
Jo Jo *=° *! * = ° k -J o
pin 0 if A: = 1, 3, . . .
But cos k9d9= k'ln
Jo , , if* = 0 ,2, 4,
24 [(A/2)!]2
pin
Therefore | exp (a cos dO = 7r^_£
Jo _2*[(*/2)0:
, 2*
=2n Z = 2 nl 0 (a)
'*=o 22*(£!)
Hence P ( R ( t ) < y \ r 2=
X(0) = x) +J*
l MI^2 exp
2 nh dr
2t
= f / , M r)dr (when 2)
Jo
Therefore
P(P(tn + < t) y| X( r 0)= x 0, . . . , x ( t n) = x) = f |x||,

/,i r
Jo
Let r.-= (xf, + *!,-), / = 0 , n. where x = *2i) and let = tan 1(x2j/xXj).
The conditions X(t0) = x ,. . . ,
- 0 - n
R(t0) = r0, 9 (t0) = 90, , R(tn) = ) = Gn.

Denoting the joint probability density of G(t0) , . . . , 9(t„) by f ( 9 0, . . . , 9n),
we have
P(R(tn + t) < y | R(t0) = r0, . . . , R(tn) - )
pin pin
= • ■• P(P(tn +t)<y I
Jo Jo
G (t„)= 0n}o
f(Q- • • • -Gn ) d G 0,. . . ,d 9 n
pin pin p
f i ( r n , r ) \ f ( 9 0 , . . . , 9 n) d 6 0 . . . d G n
Jo Jo [Jq
=Jof , r)dr
This verifies Markov property of R(t) and its transition density when k = 2.
Similarly, the general case can be proved by induction.

Exercise 8.1 If X(t) is a standard Brownian motion process, show that the conditional
density of X(t) given X(0) = X(l) = 0 a.s. is given by
-oo < < °o, 0 < t 1.

H
(2*/f(l - O r 1'2exp - i . * 2/(/(l -O ) X <
Complements The conditional process of X(t) given X(0) = X(l) = 0 is known as

Brownian Bridge or Tidedown Brownian motion and is denote by B{t). This can also be
written as B(t) = X(t) - tX(\), 0 < t < 1. It is easy to see that 5( 1) = 5(0) = 0 a.s. and
Cov (5(0, 5(5)) = 5(1 - 0 if 0 < 5 < t < 1 and Var (5(0) = f(l - /), 0 < / < 1.
Exercise 8.2 (Interpolation formula for Brownian motion) Show that the conditional
density of X(t) for tx<t <t 2 given X(o) = A and X(tf) = 5 is a normal density with
Mean = A + ——— (t - t x) and Var = (t 2 - t)(t - tx)/(t2 - 0)*

h ~h
Exercise 8.3 Let {X(0, t > 0} be a Brown motion process with X(0) = 0 a.s. Then show
that
(a) p = P[X(t) has at least one zero in an interval (a, b)]
= 2 /n cos~l(a/b)m.
(b) P[T < t] = 2In sin"l(a/0l/2 where T is the largest zero of X(t) with T < t.
Exercise 8.4 Show that a stochastically continuous stationary Gaussian process
{X(0, t > 0} with EX(t) = 0 £(X(0)2 = 1 is Markovian iff the Correlation function
p(t) = c> 0.
Exercise8.5 Let Zand 5be independent r.v.s such that Zhas a density function/^ given by
[0 if z < 0
f A Z ) ~ \ze-w2»2ifz > 0
and 6 is uniformly distributed in the interval [0, 2n\.
Define X(t) = Z cos (2nt + 0).
Show that {X(t), -«» < r < ®o) is a Gaussian process.
Exercise 8.6 Suppose that (X(r), -°° < r < °°) is given by the form
X(t, a)) = A(co) cos [2m + 0(co)]
where A and 6are independent. A is nonnegative and Qis uniformly distributed on [0, 2k)
.
(a) Show that {X(t)> -oo < *< oo} is a stationary process.
(b) Show that E(X(t)) = 0 for all t provided that E(A) < Does
M(t) = 1/2T J X(t)dt converges a.s. to E(X(t)) as T —» «>.
(c) Show that {X(0, -°° < t <°°) is a Gaussian process iff A has a Rayleigh distribution,
that is, A has a density/* given by
f a( a)= S - exp(-l/(2cr2a 2)). 0.
(J
Is {X(t), ~oo < / < 00} a Markov process?
Exercise 8.7 Determine the Covariance function of U(t) = e~* W(e2t), t > 0 where W(r)
is the standard Brownian motion.
Exercise 8.8 Let X(t) =

Jo
f
l^(5)^s.Calculate EX(t) and Var (X(t)). Also find the
conditional distribution of X(t) given that W(t) = w.
♦♦Exercise 8.9 Prove Kolmogorov’s inequality for a separable Brownian motion

P[sup\ W(u) \>C]<t /e2, e > 0 .
0 <u<t
Exercise 8.10 Prove that lim - W(t) = 0 a.s.

/—
>°° t
Exercise 8.11 Let Wx(t) and W2(t) are two independent Brownian motions and Tx be the
first time t at which Wx(t) = x. Find the distribution of Y(x) = W2(TX). Show that the
stochastic process {Tx: x > 0} and {Y(x)\ x > 0} have stationary independent increments.
Complements In general, given any Markov process {X(t), t > 0} and a process {7\jc),
x > 0} having stationary independent increments and increasing sample paths, with T(0)
= 0, it is possible to derive a new process K(jc) = X(T(x)), x > 0.
The process of forming Y from X is called subordination, and the process {T(x)), x > 0}
is called subordinator. Under the above, conditions {T(jc)), x > 0} will be a Markov
process. If, in addition, {X(t), t > 0} has stationary independent increments, then so will
{}%*), jc>0).
Exercise 8.12 (Geometric Brownian motion) Let {X(t), t > 0} be a Wiener process with
drift fj. and diffusion coefficient cr2. Find the E[Y(t)| K(0) = y] and Var [Y(t) | Y(0) = y] of
the Geometric Brownian motion Y(t) = eX(t\ t> 0.
Also calculate the diffusion coefficients
l;_ E[{Y(t + h ) - Y ( t ) ) \ n t ) = y ] , , A^ .
lim----------------- :----- = a(y), 0 < y < <*>
hio h J J
andJ £[{y(r + A)-y(/)}2|K(/)=^] = a(y),

lim------------------;------------------ / X 0A <. y <. oo
hi o h J
Exercise 8.13 Let M(t) = max W(s) and Y(t) = M(t) - W(t)
Find the joint p.d.f. of M(t) and T(t).
Exercise 8.14 Let {W(t)t > 0} be a standard Brownian motion. Let a < 0 0} be a Wiener process with drift fi ± 0 and diffusion
coefficient a2 and suppose W(0) = x. Find the probability that W(t) reaches level b > x
before hitting a < x i.e. P[W(Tab) = b \ W(0) = jc]. If ju < 0 and M = max W(s) - W(0),
show that M has exponential distribution.
Exercise 8.16 Let T be the first time the Brownian motion process crosses the line b(t)
= a + bt{a > 0, b > 0). Find the Laplace transform of T.
Time Series Analysis
9.1 Introduction and Stationary Time Series

A typical time series can be considered to be the result of four distinct components.
(1) Trend
A smooth long-term movement covering a number of years reflecting the general
tendency of the series. Growth curves are good examples of time series exhibiting
trend. Naturally, characteristics of the process depend very much on the time of
observation. One method of eliminating the trend factor from a time series is the
method of moving averages which is well known and we shall not discuss it in
some detail in this book.
(2) Cyclical Component (Seasonal Component)

The result of a periodic movement in definite time periods and can be represented
by a strictly periodic function of time i.e. Xt = Xt+e = Xt+2Q= . . . (the period is
6 ). Fourier analysis techniques may be employed in the investigation of a known
periodic movement like a seasonal variation. Periodogram analysis is a method
based on the harmonic analysis of the Fourier representation. We shall consider
only sine and cosine functions later on in our studies.
(3) Oscillatory Component

It is the Irregular periodic movement of a time series. Economists and many
others go for this type of non seasonal cyclic patterns. The sunspot cycle and
outfall of many rivers are good examples.
(4) Random Component

Random disturbance term is the basis for the stochastic Process representation
of the time series.
Difference Between (2) and (3) ;

In cyclical series, apart from the disturbances due to superimposition of the
random element, the maximum and minimum occur at equal intervals. Moreover,
the amplitude also remains the same.
In an oscillatory series the distance from peak-to-peak and trough-to-trough
varies. The oscillation also does not remain the same around some constant
value. (See the following figures).
We would, however, consider only those oscillatory series which have a constant
amplitude. When the trend and cyclical component are eliminated from a time
series we are left with a series which apart from the disturbance term fluctuates
round some constant value.
t —>°o
The question would arise whether it is possible to represent such a series by

some function of t and in fact three systemmatic procedures have been developed by
which an oscillatory series can be represented appropriately. These procedures
are called the Schemes.
Consider, three schemes which can account for a typical oscillatory series.
(i) Moving Average of a purely random component may generate an oscillatory
series.
(ii) Sum of cyclical components, e.g.
„ . 2nt . 2 nt
Xt - a\ sin -----+ a 2 sin ------
1 0)\ co2
where a>\ * co>i (c being, a constant multiplier).
Thus the sum of a number of cyclical components may generate an oscillatory
series (since the amplitudes change due to superimposition).
(iii) Autoregressive scheme Let us consider the representation of a time
series as
%t = 8 (Xt-u ^/-2» • • •* Xt„k) + £t
where g is some mathematical function and et is the random component. Then
the series Xt is an oscillatory series.
9.1.2 Stationary time series

Denote by (Q, P) the probability space and let T be an index set. Then a real
valued time series (stochastic process) is a real-valued function X(t, co) belonging
to the product space TX Q.
For fixed t e T, X(t, co) is a random variable defined over Q and is denoted
by Xr The time series {Xt, t G T} is therefore a collection of all such random
variables.
The series {X,} is said to be strictly stationary if
L ■. , X J = FxH+h Xln+h(xn , . . ,.x„) for (/,, . . and
(Ji + A, . . ., tn + h) c T and (Xri , . . . , Xtfi) lies in the range of the r.v. X,.
Note
(i) The distribution of any point in the index set remains the same.
Time Series Analysis 177
(ii) From (i) it is clear that the joint distribution of a finite number of points
in the index set do not involve the points themselves, instead it is a function of
the distance between the points,
(iii) t \ 9 . . ., tn are not necessarily consecutive.
(iv) If the second order moment E(X,2) < <», then
E(Xt ) = constant for all rl

>for such a stationary time series.
V(Xt ) = constant for all rj
Recall that {Xh t e 7} is covariance stationary if
E(Xt) = constant
and Cov (Xt, Xt+h) = y(/i) for all t e T.
9.1.3 Stationary vector stochastic process

In many areas of statistical application such as economics, business, engineering,
earth sciences, environmentals sciences, data are collected over time, typically
quarterly, daily, hourly, in second etc. Often such data is available on several
related variables. An example is the daily measurement of dissolved oxygen
concentration and biochemical oxygen demand in a river or ocean coast as
indicators of water quality. Another example may concern the study of the relation
between immigration and emigration from a region of a country as they relate to
economic variables such as capital growth.
There are two basic reasons for analyzing the data jointly as opposed to
singly:
1. To understand the dynamic relations among the series, the effect of change
in our series may show in another series simultaneously or over a period of times
and with varying amplitude effect. Some series may lead others and there may
be feedback connections between them.
2. To provide more accurate forecasts, the joint modelling allows the use of
joint information; intuitively this enables forecast errors to be reduced.
We assume the availability of observations on a set of variables, say Xt =
(Xtt, . . ., Xkt). In order to model the data generation process, we associate the
random vector Xt with each integer t = 0, ±1, ±2, . . . . This section is mainly
concerned with the first two moments of the random vector Xt We call them
stationary (second order) if these moments are finite and constant through time
t. Formally { } is stationary if:
1. E( Xt ) = \x < oo for all t
2. E[( Xt - Ji) ( Xt - nY] = I < oo for all t
3. E[(Xt - Ji)(Xt+h - /i)'] = T(h) for all t and h.
We define strict stationarity in the same way as in the univariate case (see
Chapter 1, Section 1.2).
Note Strictly stationary may not imply covariance stationary. Since the variance
restriction may not be fulfilled.
A covariance stationary series is also known as weakly or second order stationary

series (generally referred as a stationary time series). The following notation
will be used in this chapter.
y(h) = autocovariance function and
Y(h)
P(h) = autocorrelation function.
noy
9.1.4 Some important results for a Weakly stationary time series

involving autocovariance (autocorrelation) function
A function/(x), x e S is said to be positive semidefinite (p.s.d.) if for any set of
real number (a{, . . an)
n n
I I afljfit- tj) >0 for (t, - e S.
f=17=1 ■
’ J'
Result (1) Autocovariance function is p.s.d.
Proof Let [Xt, t £ T } be a time series.
Consider a finite number of points (tj,. . ., tn) e T and real numbers (ab . . ., an)
i.e. observe Xt{,. .., Xtn. Then to show y(h) is p.s.d., it is enough to show that
n n
,2=17S=1 aiOj y(ti ~ tj) ^ 0.
Now 0 < V = Z cov(X, X, ) = H a ^ j y(tf - t j ) .

i 7 1 ' i 7 ^
Hence the result.

Result (2) Autocovariance function is uniformly continuous if it is continuous
at /i = 0, i.e.
| y (5) - y(0) | < r) => | y(h + S) - y(/i) | < e if 8 < <50.
Proof Let y(h) = Cov(Xt+h, Xr). Since this is a weakly stationary time series,
without loss of generality assume E(Xt) = 0.
Then y(h)= E(Xl+h X,) and y(h+ <5) = £(X,+fc+5X,).
{y(h +8 ) Y
-m 2 = [E{X,(Xt+h+s- X l+h) } ] 2
<E(X?)E(Xt+h+s- X l+h) 2
(by Cauchy-Schwartz inequality).

Therefore E(Xt+
hs- X l+h) 2 = + Xt+h 2 Xt+h+s Xt+h)
= E ( X 2+h+s)+ -
= 2[y(0) - y(5)] < 2

Therefore {Y(h + 8 ) - Yih) ) 2 < {y(0)} {2/y(0) - y(<5))}
o E2
< e 2 by taking r? < ^ q)-
Hence | y(h + S )- y(/i) | < ei

f 5 < <50.
Example 9.1 Non-stationary time series
Consider a time series Xt - A + j§21, t e [0,1], where
E(X, ) = a function of r = /\ (r) ]

Then ^ x r x M.e. they are not constant.
Cov(X,,X,+/l) = f 2 (t) J
Any realization of this process is a continuous function in the interval [0, 1].
Such a process can be considered as the result of an experiment to estimate
the linear response due to a variable t measured on the interval [0, 1]. Each
experiment is conducted to obtain a set of the regression coefficients A> /?2.
Since (A , /J2) ~ W2 ((j^ )> 2 ), E(Xt) = A + P2 ^ is not a constant for all t, and
Cov(X„ X,+„) = E((X, - £(X,)) (X,+* - £(X,+„)))
= £ ( ( ( A - A ) + ( f t - f t ) 0 ( ( A - A ) + ( f t - f t ) ( * + * )))
= cru + cr12 r + cr120 + h) + <r22 + h)
is not a constant for all t.
Hence it is not covariance stationary.
Result (3) Autocovariance function is an even function of h, i.e.
y(h) = y (- h).
Proof y(h) - Cov (X„ X/+/f) = Cov(X ^, Xt>) = y (- h) where t - t' - h.
9.1.5 Useful stationary Processes representing different time series

models
1. Purely Random Process A time series {Xty t E T} will be called a purely
random process if the r.v.s X/s are a collection of i.i.d. random variables with
E(Xt) = n, V(X,) = a 2 and Cov(X„ XM) = y(j) = 0.
A purely random process {X„ t E T) is also called white noise (W.N.) when
the process is a discrete time process.
For example, in autoregressive process Zt = OL\Zt_x + . . . + ccpZ t_p + Xt where
X, is a White noise. For a continuous parameter process we have a ‘continuous
white noise’. Purely random processes are used as a constituent of more complicated
stationary process viz. moving average process, Autoregressive process, etc.
2. Moving Average Process MA (q) A time series {X„ t e T) is called a moving
average process of order q if it can be expressed in the form
%t = Po£t + A ef-1 + • • • + Pq£t-q*
As are constant, and where {£,} is a white noise with E(et) = 0 and V(et) = (J2.
This is simply a moving average of a random series generating a stationary

 q
° 2 % Pi Pm t f k < q for all t
y (—k ) if k <0
Hence this process {X,} is covariance stationary.
0 if k > q
1 if* = 0
q-k
Also P(k) = £0 PiPi+k
if k= 1,2,
I Pf
1=0 1
and p(k) = p (- k) if k < 0.

3. Autoregressive Process AR (p) A time series {Xt} is called an autoregressive
process of order p if
Xt = a xXt. x + . . . + ap Xt_p + et s
where et is a white noise with
E(et) = 0, Var (£,) = o2, Cov(£„ £,') = 0, t * t'.
Here the process arises from an oscillatory series giving rise to an autoregressive
scheme.
(i) When p = 1, this is a Markov process
(ii) When p = 2, this is a Yule process.
For p = 1, Xt = aXt_x + £,
= a ( a X t_2 + £,_i) + £,
= a 2X t_2 + otet_i + et
= £, + a £ M + a 2£,_2 + . . . if | a | < 1.
= P 0 et + P\£t- X+ /32£/-2 + • • • (where )3t = a ', i > 0)
is a MA (oo) process.
In general an AR(p) is an MA («>) (a duality property between AR(p) and MA
(oo) exists) if all the roots of the equation z p - a xzp~l - ct2z p~2 - . . . - a p = 0 are
less than one in absolute value. Back to AR(1) we have now
or
E(Xt ) = n i P i = 0 i f ,?o^'2 <
and j=0 ^
(these conditions are
Var(X,) = <72 i p ? ,?0 1/?' 1<
satisfied if - 1 < a < 1).
Y(k) = Cov (X„ Xl+k) = a 2 I PiPi+k < oo if | « | < l

i- 0
(by Cauchy-Schwartz inequality).
Therefore {X,} is a covariance stationary process.
To show AR(p) is an MA (<») process, let us write
X, = a xXt_x + . . . + apXt_p + et
= a xBXt + . . . + apBpXt + et
where B is the Back-shift operator such that Xt_p = BpXt.
Therefore white noise
et = (1 - a xB - . . . - a pB p)Xt
= /(* )* „ say.
Therefore if f( B ) is invertible we have Xt =f~l(B) et
= (1 - a xB - . .. - a pBp
)-' e, = e,+ , + P2e,_2 1
which gives E(Xt) = 0 for all t = 0, ± 1, ± 2, . . .,
oo oo
V(Xt) = a 2 £ Pf for all f = 0, ± 1, ± 2, . . ., and y(k) = cr2 S PiPi+k.

i=0 /=o
This again shows that {X;} generates a second order stationary process.
4. Autoregressive Moving Average ARMA (p , g) Consider the stochastic process
Xt = aiXM + . . . + a pXt_p + P0£t + A ^-i + • • • + Pq £t-<r
This is also second order stationary by means of results (2) and (3) To describe
a autoregressive series or a moving average process with a large order (which is
difficult to handle) we take recourse to the ARMA process.
(5) Random Walk Xt = XM + £t with
E(£t ) = /a,
K(£,) = cr2 and
Cov (£t , £s ) = 0 if t * s
Take X0 = 0. Now
Xi = ex
x ‘ - Si
Then E(Xt) = t/a and V(Xt) = t o 2.
Hence the process is not second order stationary.
However, the first order difference of {X,} is a stationary process, since £t =
Xi - XM, with expectation jll and variance = a 2 (independent of t). For example,
share price on tih day = Share price on (t - l)th day + error.
9.1.6 Stationarity and ergodicity

Consider a stationary process {X,} with E(Xt) = p for all t and Var(X,) = cr2 for
all t.
The expectation and variance of the averages of realization averages are
1 m ( ni ^ i m
E - E — Z X, =E E - AC the ensemble average and
"I /=! [ ,1j '=1 j m j=\ M7)
Var — 2 X,(./) £1
tn j =i m
1 —
If the number of realizations is large enough then — Z X( j\-^> JUas m —> °q,
m J=i
since E = ju and Var 1 Z x (7) —> 0 as m —> °°.

2 x (7)
7= 1 7= 1
If we have a number of realizations then the average of all the realization

averages would converage in ‘quadratic mean’ (q.m.) to ju as the number of
realizations get large (cf. Theorem 9.1).
Question If instead of a large number of realizations we have only a single
realization, then will it be possible to estimate p on the basis of it? The answer
to this question gives rise to Ergodicity.
Ergodicity Property of a Time Series

A stationary time series {Xr} is said to be ergodic if a single realization of the
series can make all possible conclusions about the probability law that generates
the process. Specifically, the sample moments obtained from a single realization
converge to the population moments with probability 1 or in mean square.
Ergodicity of p
Consider a discrete stationary time series {X,} with E(Xt) = p for all t. Then the
_ 1 n
series is mean ergodic if the sequence of sample means Xn = — E X„ n = 1,
2, . . . converges in q.m. to p.
Theorem 9.1 A stationary time series {X,} whose covariance function as
~ = -1 E
y(h) —> 0 as h —> °° is ergodic for p. Hence we can suggest that X„ n X,
n t=\
is an optimal estimate of p.
Proof To show that Xn ^ we are to show that lim E( Xn - p ) 2 = 0-
n —>oo
- n n
Now Var ( Xfl) = Var E x, = — Z I Cov
n /=i
X,-)
n~ i=i i'=\
I n
Z Cov , ,X,-)- V Z Var (A",).
(X
t'=l n ~ i=l
Since Var (X,) = cr2, second term of right hand side —> 0 as n —» ©o.
VI
Now 4 -i 2 Co - £
n “ r=l /'=! AZ“ /=! t t'=\
1 '- 1
Z 7 Z Y(h) since y(/i) = y (- h) and — < 1
n /=i t h=0
i n 1 r’ 1 1 '- 1
Z - 2 y(h)
n ,=i t h=0
+^n /=yv+1
z 7 Z y(A) ,(n > AO.
t h=o
The second term on right hand side can be made
1 /_1
< ■£ !- for r > jV since 7 2 y(A)
2 n * h—Q
Now take n> N, so that the second term < -■. Also for n > N, first term < .
Hence Var(X„) = E(Xn - ji)2 -> 0 =» Xn ju.
Corollary 9.1 A stationary time series with absolutely summable covariance
function is ergodic for mean. Also
lim n Var(X„) = /j=-oo

n—
>oo
Z y(/i).
Proof Z | y(/z) | < oo y(A) —> 0 as h —> <*>. Ergodicity follows from the
h
theorem.
Also we have
n V a r ( X „ ) = in j=\
i /=i
i y ( r - y ) = ni- h=-(n-1
"l ) |)
= I (1 - \h\ln)S y(A)
l/J
(by Kronecker’s lemma lim Z — | y(/z)| = 0.)
«—>°° /i w
An alternative expression of lim zzVar(X„) given by the following corollary
n— >°°
should be studied after section 9.2.4.
**Corollary 9.2 If the spectral density of a stationary time series X, is continuous,
then lim zzVar(X„) = 27r/(0), where /(A) is the spectral density of Xt (to be
/?—>00
defined in section 9.2.4).
Proof Since /(A) is a continuous periodic function on [-n, n] the Cesaro
1 * *”
sum C„(A) = — 2 Z (ak cos kX + bk sin kX) + a 0l2 of the Fourier series of
H r-2 lc=1
f(X ) —>f(X) uniformly in X e [- /r, tt].
Therefore
2 n f(0 ) = lim 2zr a 0/2 + i £ Z a *

n r=2 *=1
n r- 1 1/
= lim 7(0) + - Z I 7(*) note . i r cos kX f(X ) dX = 7(/:)
A r=2 *=1 j v
= lim 7(0) + — Z (n - r)y(r)

n—*°° n r=\
= lim «Var(Xn)
n—>oo
(from the proof of corollary 9.1).
9.2 Different Approaches of Time Series

Analysis of time series is done by two distinct approaches.
Analysis of Time Series
^ (Analysis in time Analysis in frequencyl . g

' [domain domain j *
A : For fixed (0 e Q, the observations used are Xt in

(i) determining the internal structure of a given time series {Xr}.
(ii) to estimate the parameters of the given stochastic model of {Xt} and
is done using the ergodicity property.
In (i) the basic tool used in drawing any inference about the given time series
{X;} is autocovariance (autocorrelation) function.
The analysis here is known as correlogram analysis.
B : For fixed t, observations are functions of co, Xt. co e Q.
The basic tool used in drawing any inference from frequency domain approach
is the spectral density function (spectrum). The analysis here is known as
periodogram analysis.
In order to understand analysis of time series in frequency domain, let us
introduce some of the tools and terminologies pertinent to time series analysis
through system engineering vocabularies. Let Xt denote an electric current at an
instant t andto know the energy dissipated by this current per unit of resistance,
we have the following natural definitions. The quantity C2 X} dt is called the

J t\
energy of the stochastic process {X,} in the course of the interval
[r,, t2]. The integral J X} d t, if it converges, is called the total energy o f the
1 CT ,
ser/es {X,, - oo < t < o°}. The limit lim I X fd t is known as the average
T—>°° " * J _ t-
power of the random process. If the stochastic process is complex valued, we
write | X, |2 in place of X}.
In many situations time series are simulated by the sum of harmonics with
given frequencies and random amplitudes and phases. In other words, one considers
processes of the form
n
Xt = Z ak cos (ukt+ Qk) (9.2.1)
k=\
where uk are given numbers, amplitudes ak s and phases 0k s are random quantities.
The probabilistic structure of this process is completely determined by the joint
distribution of the r.v.s ak and 6k for k = 1, 2,. . ., n. If a second order process X,
is the sum of n harmonic oscillations with amplitudes | ak | and frequencies
uk, then all the frequencies {uk}, for k = 1, 2,. .., n, considered as a set of points
on the real line, is called the spectrum of the process {X,}. In many cases it is
convenient to consider complex-valued processes of oscillatory nature
n
Z, = k=1
I (9.2.2)
with complex amplitudes yk = ak + ibk for k - 1, 2,. . ., n. The process Z, can be

split into real and imaginary parts
Z, = X, + iYt where
n
Xt = £ (ak cos ukt - bk sin ukt)
k=\ (9.2.3)
n
Yt = X (ak sin ukt+ bk cos ukt)
*=l
If we set a* = | % | cos 6k and bk = | % | sin Qh we obtain an expression for the
function X, and Yt in the form
x, = 2 I r* I cos (ukt+ e k) and = Z | y* | sin + dk).

k- 1 k-1
The mean value of the process Zt
sin Tuk
r* Tuk (9.2.4)
where ^ - ^ = 1 at t = Oand y k =\yk \ e'°k and its average power is given by
2T | Tr I Z* I2 dt = J 7^ i z y*f
sin ^ - n , )
= ^ \Y k \2 + 2 YkYr
T(uk - u r) ' K
k=l *,r=l
1 f7 "
Therefore lim — \Z t \2 dt= 2 y? a.s. (9.2.5)
T— k= 1
Thus the average power of the oscillatory stochastic process Z, is equal to the
sum of the average power of the harmonic components of the process.
From (9.2.4) we see that if 0 is a point of the spectrum, then
lim
T~>°° 1
y f
J- T
=
where y0 is the amplitude corresponding to the frequency at t = 0.

Suppose that complex amplitudes yk haveE(yk) = 0, E(yky r) = 0,k ± r . Then
E(Zt) = 0. Note that if 0 is not a point of the spectrum of a stochastic process then
E(Zt) coincides with the mean value over the infinite interval (- °°). On the
other hand, if 0 is a point of the spectrum of the process, then the mean value of
Z over time lim — i rr Zt dt is a random variable.

T—>«> I J _j
Now Z, has covariance function
\
v Pi(<ukt\-Urt2)
R(tu t2) = E(Zt Zt2) = E S S YkYre 2 ^ 2 e iuk ( t ] - t 2)^
1
k = r=\
y it=i
where C\ = £ | y k |2. Thus covariance function /?(rb r2) depends only on the
difference, i.e.
R(ti» ^2) - ^(^1 ~ ^2) (9.2.6)
n
where R(t) = 2 C,2 (9.2.7)
ik=i
Thus, if the amplitudes are uncorrelated with zero mean, then the process Zt is
a stationary process in the wide sense and the covariance function is given by the
formula (9.2.7) called the spectral distribution of the covariance function. It
determines the spectrum of the random process, i.e. the set of frequencies {w*}",
of the harmonic oscillation constituting the process {Z,} and C\ = E |y k |2 of
the process in a unit time. The quantity C\ can be called simply the mean value
of the power of the harmonic components of the second order process with
frequency uk. It is obtained by first averaging the power over time and then
averaging it in the probabilistic sense, that is, by taking expectation. In connection
with these energy representation we introduce the important characteristic of a
second order stochastic process, the spectral function F(u) of the process (9.2.2)
defined by
F(u)= 2 Cl
U k<U
This shows that F(u) is equal to the average power of the harmonic components
of the process Z, (whose frequencies are less than or equal to u) in a unit time.
It completely characterizes both the average power of each harmonic component
of the process Z, and the overall average power of the harmonic components of
the process, whose frequencies lie in a given interval. Thus
c j = F(uk) - F(uk - o). Z C 2k = F(u2) - F(UX).
u\<uk<u2
By means of spectal function, the covariance function of the process Z, can be
written in the form
(9.2.8)
Therefore, spectral function is a non-negative nondecreasing right continuous

function that is piecewise constant with finitely many jumps of magnitude Ck .
This concept of spectral function will be introduced for arbitrary weakly stationary
n
stochastic processes by passing to the limit of the processes Z, = Z y ke lUkt and
k=\
decreasing the complex amplitudes yk whereas the spectrum of the process (i.e.
the set of all frequencies uk) fills the whole real line (- °o, over more densely.
For arbitrary second order process in passing to the limit we obtain a process
that has a continuous spectrum for which analogue of (9.2.8) is valid. This is
given by the celebrated (Wiener) Khinchine's theorem for spectral representation
of covariance function.
9.3 Correlogram Analysis

Consider a covariance stationary time series {X,}. Let pk be the autocorrelation
of lag k. Different covariance stationary time series possesses different forms of
correlograms pk when plotted against k is called the correlogram. By analysing
the shape of the correlogram we can infer about the internal structure of {X,}.
For different standard stochastic models we present the correlograms :
L MA (q)
X, = P 0£, + Pi e ,.1+ . . Pq£,_q
where et is a white noise with
E(et) = 0, Var(£,) = cr2 and
1 for k = 0
P* = for k = 1, 2 ,.. ., q
0 k >q
In particular, /3, = 1 for all /, indicates simple MA process.

q+ 1
k if /: = 0, 1,
Then q+ 1 q
Pk =
if k > q
The autocorrelation oscillates

between the points (0, 1) and (q + 1,0)
and then coincides with k-axis.
If it is a simple MA process then
autocorrelation will decline smoothly.
Otherwise it may have some oscillatory
movement depending on the coefficients
Ph and it will ultimately coincide with £-axis for k = q + 1.
2. First order autoregressive (or Markov) process
Xt = a X t_i + | a | < 1 (this assumption is needed to make the time series
stationary) where et's are white noise with E(Xt) = 0 and Var(X,) = a 2.
Xt = a ( a X t_2 + £,-i) + $
= a nX t. n + a n- ]e t_n+l + . . . + et
n -\
= a nXt_n + Z
/=o
will be stationary and convergent, if | a \ < 1.
Xf ~ Z a l£t_i is a MA («>) process if I a | < 1.

i=0
Now £ (* ,) = 0,V ar(X ,) = tr 2 £ ct2' = g2 , (I a I < 1)

i=o I - a2
and y* = E
^ oo ^ ^ oo ^
=E 2 a JE,_k_j
l ' =° J { ■ J
t t+k
oT
7
M
= £ 2 a l+k (where / = t i, m = t + k - j)
[ = -oo m=-«*>
t
= cr2a* Z a 2(t l) (only those terms which satisfy / = m contribute)
/= - o o
= o 2a 2 a 2'
i=0
= - a 2).
Therefore, pk = a k.
To get an even function defined for all integer k, pk = a}k\ k = 0, ± 1, ± 2 , . . . .

As k -» oo, p* -> 0 if | a | < 1.
3. Correlogram of AR(2) (Yule process)

Consider
X( ~ aX(_\ + bXt_2 + £f>
where et is a white noise with Eet = 0, Var(e,) = a 2.

Multiplying both sides by Xt_h taking expectation and dividing by Var(X,) we
get
pk = aPh-i + bpk-2 (Yule-Walker equation).
The general solution is given by
pk = A \rk + A2r k, | rx | < 1 and | r2 | < 1. (9.3.1)

where A { and A2 has to be found from initial conditions and r2 are the roots
of r2 - ar - b = 0 (characteristic equation of the process).
a ± *s](a2 + 4b)
Case (i) rx and r2 are real, i.e. are real, i.e. if a2 + 4b > 0.
A j and A2 are found as follows :
Since p0 = 1 = A x + A2. From first Yule-Walker equation, we have
Pi = aPo + bp-1 = a + bp_x = a + bpx
Therefore p x =
1 - b’
Also p x = A xr{ + A2r2 (from (9.3.1))

= A\rx + (1 - A x)r2 = A x (r, - r2) + r2
Hence Ay =
1 - b -~ 'r22 i /VM
K ri ~ r2) andA2 = l-A ,
------
For real equal roots of the characteristic equation rx = r2 = r and
,2 n
0 as &—><*>.
« =^ * TTT^ ^
The correlogram is oscillatory and decaying in character.
Case (ii) rx, r2 are complex conjugates if a2 + 4b < 0.
Write r} = p (cos 0 + i sin 6) and r2 = p (cos 6 - i sin 6).
Therefore pk = pk[Ax (cos 0 + i sin 6 )k + A2 (cos 0 - i sin 0)*]
= pk[Ax (cos 6k + i sin Ok) + A2 (cos 6k - i sin 6k)]
- pk [A* cos 6k + 0* sin 6k]
where A* = A x + A2 and 0* = i(Ax - A2).
*
If II
II
II
A = 1, Pi = p [cos 0 + 0 * sin 0]
* = -1, p_! = [cos 0 - 0 * sin 0]
since p x = p_b equating the above two equations
1 - P~2 cos 6 1 —p 2
R* = ------ —cot 0 = cot y/ (say)
1 + p 2 sin 0 1+p
Now Pk = Pk [cos + cot y/ sin 0/:]
/ sin (0& + yO
k s‘n (Ok + iff) ^Q
since <c
p sin y/ v
sin y/
as k —» oo and p* is a damped sine curve.

Now p 2 = rxr2 and also from the theory of quadratic equations
rxr2 ~ - h since a2 + 4 b < 0 = > b < 0
Therefore, p = 4b < 1 if | rx | < 1, | r2 | < 1.
4. Autoregressive model of order p, AR(p)

Xt = atXt_x + . . . + apXt_p + et with white noise et,
E{et) = 0, Var(£,) = a 2.
Multiplying by Xt_k, taking expectation, dividing by a 2 and using the fact
Pk = P-b we get Yule-Walker equation
Pk = a \Pk-\ + alPk-2 + • • • + apPk~p(k > 0).
p
The general solution is pk = Z Atr k, | rt | < 1 for all i = 1, 2, . . . , p where r- s
i~\
are the roots of the equation rp = a xrp~x + a 2r p~2 + . . . + a p.
Here also | rt | < 1, i = 1 ,... ,p are necessary and sufficient conditions for {X,}
to be a stationary process.
In case of real roots pk —> 0 as k -» «>.
9.3.1 Interpreting the sample correlogram

Sample autocovariance is defined as
Ck = 2i (X ,- X ) ( X l+k- X) or — "s (X, - X)(X,+* - X)

/=i n —k t~i
and sample autocorrelation is similarly defined as

n-k / n-k
rk = 1 (Xt - X ) { X t+k- X ) 2 (X , - X )2.
t= \ / t= \
A useful aid in interpreting a set of auto-correlation coefficients is a graph called

a sample correlogram in which rk is plotted against lag k.
(a) A completely random series

In this case rk ~ 0 for large n and for all non-zero values of k. This follows from
sampling distribution of rk - rk ~ N(0, 1In) for large n (Quennoule, 1957).
rk
0
lag k —►
(b) Short term series

Stationary series often exhibit short term correlation characterised by a fairly
large value (also from theory of equation) of r{ followed by 2 or 3 more coefficients
which, while significantly greater than zero, tend to get successively smaller
values of rk for larger lags tends to be approximately 0. A time series which gives
rise to such a correlogram, is one for which an observation above the mean tends
to be followed by one or more further observations above the mean and similarly
for observations below the mean. This is an indication of autoregressive model.
(c) Non-stationary series

If a time series contains a trend, then the values of rk will not come down to zero
except for very large values of lag k. This is because an observation on one side
of the overall mean tends to be followed by a large number of further observations
on the same side of the mean because of trend. Little can be inferred from a
correlogram of this type, as the trend dominates all other features. Simple
autocorrelation should be calculated only for the stationary series and so any
trend should be removed before calculating {rk}.
Upward trend Downward trend

(d) Alternating Series
(e) Seasonal fluctuations

If a time series contains a seasonal fluctuation, the correlogram will also exhibit
an oscillation at the same frequency.
If X, follows a sinusoidal pattern then so does rk, e.g.

Xt = a cos too, frequency CO, 0 < oo < n
then Xn —» 0 and rk ~ cos koo for large n.
( f) ARM A
The correlogram of a MA(q) process is easy to recognise as it “cuts o ff’ at large
q whereas AR(p) process is a mixture of damped exponential and sinusoids and
dies out slowly (or attenuates). The correlogram of a mixed ARMA model also
generally attenuates rather than cuts off.
Example Show that the infinite MA (oo) process {X,} defined by
Xt = £t + C(£t_i + £,_2 + . . .)
where £,’s are white noise with E£t = 0, Var £t = a 2 and c is a constant, is a non-
stationary process. Also show that the series of 1st differences {Yt} defined by
Yt = X, - XM is a first order MA(1) process and also 2nd order stationary. Find
the autocovariance function of { } .
Solution Var (Xt) = cr2[l + c2X 1] is infinite, obviously Xt is not stationary.
Yt =£t + c{£t. x + . . . ) - £ t_ x -c (£,_2 + . . .)
= £t + ( C - 1) £t_x
is MA (1) process with
E(Yt) =0 +(c- 1) 0 = 0, Var(K,) = + (c - l) 2 2 = <r2(l + (c - l ) 2)

do not depend on t and hence {T,} is second order stationary.
Y(k) = E(Y,Yl+k) = (c- 1) <x2 if* = ± 1
= cr2(l + ( c - l)2) if &= 0
= 0 otherwise
1 if Jt = 0
Py (>c) = ( c - 1)/[1 + ( c - l)2 ] if Jt = ±1
0 otherwise
9.3.2 Integrated models (ARIMA)

If the observed time series is non-stationary in the mean then we can difference
the series and this approach is widely used in econometrics. If Xt is replaced by
VdXt in the ARM A (p, q) model we have a model capable of describing certain
types of non-stationary series. Such a model is called an “integrated” model
because the stationary model which is fitted to the differenced data has to be
summed or “integrated” to provide a model for the non-stationary model. Writing
Wt = the general autoregressive integrated moving average process (ARIMA)
is of the form
W, = + ...+ a p W ,_ p + e , + +
and is denoted by ARIMA (p, d, q). Usually d = 1. Also random walk is a
ARIMA (0, 1, 0) process.
ARMA (p, q) model can be generalised as
i <XjXH = £ PiEt. i (9.3.2)

7=0 i=0
with a 0 = j80 = 1 and {£,}’s are uncorrelated (0, a 2) r.v.s. We assume that the
characteristic equation
z p + a \Z p~x + . . . + ap = 0
has all the roots less than one in absolute value.
(i) Then Xt has the MA (°o) representation
Xt = £ Vj£t_j where v0 = Oq = 1, ^ = A ~
7=0
v2 = P z - C h - a2v x, . . .
min( 7, p )
Vj = P j - X -iif /'< 4
jV
(X
/=1
min(7,7>)
Vj = - X a jV H if/' > q
(ii) If £,’s are white noise

Here Zj are the roots of the characteristic equation a( z ) = 0.

p
Proof Let ut = X a , X,_7 be an AR (p).
7=0
oo
The Xt = X w jUt_j is a MA. (<*>). (9.3.3)
7 =1
The case p = 1 has been done before.

In case of general p by partial fraction method we can prove
oo ( <i \ oo
X ,= X w, 2 P i E ,- h = 2 V r E,-r (9.3.4)
;=i l'=o J r=0
Q P p ( oo \
S p ,e ,_ i =2 a.j Xt-j = 2 (Xj X V r Et_ j _ r
j=0 7=0
n
1=0
o
P
w |
X C t j £ t_ j _ r
ii
7=0
and obtain v fs by equating coefficients of £t_j, j = 0, 1, 2, . . . .
(ii) Equation (9.3.2) can be written as
d>P(B)Xt = y/q(B)et, (9.3.5)
where QP(B) and y/q(B) are the polynomials in Backshift operator B. Since
E(etXt_k) = 0, multiplying (9.3.5) by Xt_k(k > 0) and taking expectation on both
sides
<l>p(B)pk = 0 (k > q)
and so, X &iPk-i = 0. Now a 0 = 1 and the last equation is a homogeneous
i=0
p
difference equation of order p . Hence = X (Xr Zr ( k > q ) .
r=\
9.4 Wold’s Decomposition and Wiener-Khintchine and

Bochner’s Theorem for Stationary Time Series
9.4.1 Deterministic and indeterministic processes
Deterministic process A time series {X/} will be called deterministic if there
exists a function g(r) of past and present values of Xt
g(t) = g(XHJ = 0 , 1 , 2 , . . . )
such that £(X/+1 - g(t))2 = 0 (9.4.1)
If a function g(t) is a linear function of Xt_r j > 0, then {X,} will be called
linear deterministic; e.g. consider the series X, = aebt so that X/+1 = ebXt. If b is
not known, it can be perfectly estimated from the past of the series, but the
estimate will be a nonlinear function of the (sequence) series; for instance
b = 1/2 [log X} - log X 2_x\ allowing for the fact that a may be negative. Other
linear deterministic functions are the periodic functions X, = a cos (cot + 0)

provided a is known and there exists an integer k such that Inklco is an integer.
For example, if X( = a cos (2 n t/\2 + 0), then X,+1 = XM1 so that X, is clearly
linearly deterministic. If co has to be estimated from past data, the series becomes
n
nonlinear deterministic. Let X,= X dkt k so that the sequence is a polynomial
*=o
in t. Then AnX, = n\dn and An+1X, = 0, i.e. X, series obeys a homogeneous linear
difference equation and hence is linearly deterministic. To use this, procedure
one has to know the value of n or n > n.
A typical time series consists of a deterministic part and a random part, e.g.
X, = Asin (Qt) + et
i I
Deterministic white noise
Indeterministic Process Consider a discrete time stationary time series {X,},
E(Xt) = 0 and Var (X,) = a 2. Suppose the residual variance obtained by regressing
X, on (Xt_q, Xt_q_\, . . .) is c 2(< a 2). Certainly a 2 is a non-decreasing bounded
sequence, i.e.
\o 2
lim o l = (Jo = \ (considering two extreme situations)
[0
(i) If residual variance a 2 > 0 then it is useless to consider the regression of
X, on {Xt_q, XM_{, ...} . Such a process {X,} is then called indeterministic, since
the process cannot be used for forecasting purposes (MA, AR, ARMA, satisfy
this property). Then linear regression on the remote past is useless for prediction
purpose.
(ii) If residual variance is 0 then it can be used for forecasting purposes and
hence is deterministic.
Theorem 9.2 Wold Decomposition Theorem (1930)
Any discrete stationary time series {X,} can be expressed as the sum of two
uncorrelated processes, one purely deterministic Vt and the other purely
indeterministic process Ut. Further, Ut may be represented as a infinite moving
average MA (<*>), i.e.
Ut = 0
£ t +
+ 2 -2 +
x£ t- \ • • •
& £ t
where e/s are uncorrelated. The proof is given in Appendix 11(B), p.284.
9.4.2 Analysis in frequency domain

Inference based on the conceptual tool known as spectral density function
(spectrum) is called analysis in frequency domain. The spectral density is the
natural tool for considering the frequency properties of a time series.
Theorem 9.3 Wiener-Khintchine’s Theorem Suppose {X,} is a covariance
stationary time series with autocovariance function y(&). Then there is a function
F(A) T in A such that
cos (Xk) dF(A) (9.4.2)

This is called the spectral representation of autocovariance function, where

Ojr
A = frequency of the series yt = cos (A t) and A can be written as A = — where
C7
0 is the period of oscillation. Note that A e (0, n).
Following is the direct physical interpretation of spectral representation (9.4.2):
If k = 0, y(0) = f dF(X) = F(n) = a 2 (from equation (9.4.2))

Jo
As F(A) T in A, max F(A) = F(n) = a 2 = Var (X,).
0 < A<7T
Therefore F(A) is the contribution to the variance of the series which is

accounted for frequency A (0, n). F(A) is an absolutely continuous function
E
for any discrete stationary time series {X,} satisfying £ | y(k) | < ©o. Therefore
dF(X)
/(A ) = spectral density exists in this case and (9.4.2) becomes y(k) =
dX
I cos (Ak)f(X) dX.
Jo
The autocovariance function and the spectral density function are equivalent
in representing underlying time series. In some situations, the autocovariance
function is seen to be useful in determining the underlying structure of the time
series while in some situations the spectral density function /(A) is seen to be
more useful. Fourier Analysis of spectral density/(A) plays an important role.
9.4.3 Fourier Analysis

Suppose that a function f{t) is defined on (-7T, n) and satisfies the Dirichlet
conditions (It is absolutely integrable over the domain (- n, it), has a finite
number of discontinuities and a finite number of maxima & minima). Then/(/)
may be approximated by the Fourier series
+ 2 (ar cos rt + br sin rt),

r=\
where f{ t) d t,a r = ^ ^ cos
and fit) sin (rt) dt r - 1, 2, 3, . . .
k
It can be shown that the finite Fourier series sk = -■ + 2 (ar cos rt + br sin rt)
2 r=\
converges to /(/) as k —> except at points of discontinuities where it converges to
Theorem 9.4 (From Fourier Analysis) If 2 ( | ak \ + | bk \ ) < ©°, then the
associated trigonometric series a0/2 + 2 (ak cos kX + bk sin kX) converges

k- 1
absolutely and uniformly to a continuous function of period 2 n o f which it is the
Fourier seires. Let us recall the definition “ A stochastic process {X;} is said to
be mean-square continuous if lim Var(Xr+/, - Xt) = 0.”
h->0
Theorem 9.5 (Bochner-Khintchine) If the function ph is the correlation function
of a mean-square continuous stationary time series with index set T = (- °°),
then it is representable in the form
(1) Ph =
£ e i(0h dF(co) where F(co) is a d.f.
£ e,mh p(d(d),p is a finite measure on (
If the index set T is discrete, i.e. 7 = {0, ±1, ± 2 ,...} then we have a theorem
due to Herglotz.
Let the correlation function pk of a stationary time series be absolute summable
(i.e. X | pk | < <*>). Then it is representable in the form
k
(2) pk = f
* -7 1
eM dF(A)
= I elXk p{dX) where p is a finite measure on (E = [- n., tt], SB).

* -7 1
In fact more is true.

There exists a continuous function/(A) such that
(i) pk = f /(A) cos k k d K

* -7 1
(ii) /(A) * 0,
(iii) f
J -7 1
/(A) dX = 1
(iv) /(A) is an even function of A.
Proof From Fourier Analysis
1
g(A) = T + X p* cos LA
^ *=i
is a well defined continuous (since X | pk | c °°) function.

k
By positive semi-definite property of correlation functions pk
n n
X X pm_n cos (wA) cos (qX) > 0
rn= 1 <7=1
and X X pm_n sin (mX) sin (qX) > 0.

m = 1 <7=1
Hence Z Z pm_n [cos (mX) cos {qX) + sin (mX) sin (qX)]
m —1 <7= 1
n n
= S S pm. qcos (m - q ) A > 0
m- 1<7=1
m- q = 0 n times
= 1 —> n - 1 times
Note
= n - 1 —> 1 times
Letting m - q = k, we have
*=-(«-1) V n )
Now {pkcos k }Ais absolutely summable ( E |cos (kA) | < oo) and hen
£
Kronecker’s lemma
(" - 1) . . .
lim Z *1— I pk cos (kX) = 0.
n-+oo *__(W_1) n
(n - 1,1 00
Thus, lim Z - — '— I-p* cos (&A) = Z pk cos (kX) = 2g(X) > 0
£=-(<i-l) ^ k=-oo
Now pk are the Fourier coefficients of g shows that
A = [ # U ) cos
J -7 1
We have only to multiply g(X) by a constant to get (iii). Since
g(X) 1,
the appropriate constant is — and we define
/(A ) by /(A ) = i •- + Z p^ôs (kX)

* k-\
Note that Z p* cos kX = — 1 + 2 Z p* cos (£A) if a 2 = 1

I n *=-~ ;r *=i
(iv) The function /(A) is an even function since it is the uniform limit of a
sum of even functions (cosines).
The stationary process {X,} itself has a spectral representation (Cramer 1942),
J ei a dZ(A) = J e ia n{dX) (9.4.3)
where Z(A) (- °o < A < °o) is a stochastic process with finite second moments has
orthogonal increments such that E \ Z(Z\) - Z(A2) |2 < 00 and
E[(Z(fi2) - Zi/uO) (Z(A2) —Z(Aj))] = 0, p x < & < A, < A,;
^ ( - o o ,r ) i f r = / ? 1
also Z(/) =
/i[- 7T, t] if 7 = [- /r, /r].
The representation (9.4.3) is called the spectral representation of the stationary

process {X,} and the function F(A) = E | Z(A) |2 is the spectral distribution
function. The measure ju is uniquely determined by the spectual function via the
equality p[a, b) = F(b) - F(a).
9.5 Spectral Densities of some Standard Processes and

Autocovariance Generating Function
9.5.1 Determination of /(A ) in terms of y(k) and pk

From the spectral representation theorem
1 + 2 2 pk cos (A:A)
k=1
a 2 + 2 2 Yk cos (*A) (Yo = a 2)

*=1
Exercise 9.1 MA(1)
1 if* = 0
P
X' = e, + pe,_u pk = if &= ±1
l+P2
0 otherwise
/(A ) = 2 - 1 + ----- —r cos A for 0 < A < n

J n 1+
1 cr2 [1 + 2/3cos A +
« 1+ p2
2P 2P
and /(0) = 1+ J in ) 1-
n 1 + P2 n 1 + P2
Here is the variance of E,process and not of X,.

1+p2
Special Case (3 = 1 and j n

=0
f /(0 ) = 0
If ft = - 1, then ^ 9
} /W =
The shape of the spectrum depends on the value of (3. When /3 > 0 the power is
concentrated at low frequencies giving what is called a low frequency spectrum,
Exercise 9.2 AR(1) Xt = a XM + |a |< 1

Here we have pk = a 1*1.
/U ) = ^ 1 + 2 Z a k cos (Xk)
k= l
In
1+ Z a k (eiXk + e~iXk)
*=i
2;r
1 + Z (a V A)* + Z (ae-/A)*
*=i *=i
51 ae iX ae - i X
1+
2/r 1 - ae iX 1 - ae -iX
a2 1 + a 2 - a (e lX + e lX) + a e lX - a 2 + ae~iX - a 2
2n 1 + a 2 - a (e iX +e~iX)
0
5 1 1 - or
In 1 + a 2 - 2acos a
a 2(1 - a 2) _______ 1_______

In 1 + a 2 - 2acos
cr2 1 - or2 a2 1+ a f 2 and/(ff) = a 2 1 -

/ ( 0) =
2?r (l _ a )2 2s 1 - a [ 7r J 2n 1 + a
Note here X, has variance cr2(l - a 2) and a 2 is the variance of et.
The shape of the spectrum depends on the value of a. For > 0, the power
is low frequency spectrum and for a < 0 a high freq
If a < 0, the values of Xt will tend to oscillate: a positive value of Xt will be

followed by a negative value and vise versa. The rapid oscillations correspond
to high frequency variation.
Exercise 9.3 Let {Xt} be a white noise process with E(Xt) = 0 for all t and
G2 if 5 = 1
E( XsXt )
0 if s * t
Now covariance function is given by y k = e lkX dF(X).

J -n
In the discrete case, by inverse Fourier transform, the spectral density is
given by
Here Yk = 0 if k * 0
= a 1 if k = 0.
So,/(A) becomes, o 2! 2 n , - n < X < n.
Autocovariance generating functions are very useful for finding spectral densities
of many time series models.
9.5.2 Autocovariance and autocorrelation generating function (AGF)
Define C(z) = 2 ykzk, where z is such that C(z) is convergent and C(z) is called
k=-oo
autocovariance generating function.
Replacing yk by ft Y k ' , we get autocorrelation generating function

Po
OO
C(z)
S(z) = X pkz k =
k=.„- Var (X,)
The spectral density can be expressed in terms of AGF as follows:
m = ^ 2rn.*=_,
2 2n y ke~i=
V ar(X ,)
S(e~a ).
2n
Theorem 9.6 Let Yt be a linear process, i.e.
Yt =Z a i X t_ ; with Z Ia,-1 < °o

7=0 7=0
is a stationary process where X t is a white noise.
If Cy(z), Cx(z) are the AGF’s and f y(X) and/t(A) are spectral density functions
{Ff} and {X;} respectively, then
Cy(z) = h(z)h(z
and f y(X = h(e,A)h(e ' )A(A)
where h(z) = 2 a ,zj -

i=o
Proof Assume E(Xt) = 0 for all t, then E(Yt) = 0 for all t. Denote
Y(k) = E(XtXt+k), R(k) = £(K,K,+*).
\ ( \
then R(k) = E\ L a j XH £
j=o J l /=0
= 2 I a j a , Y( k +j - I)
j=o /=o
Hence Cy(z)= 2 /f(AT)z*

/c=-oo
= 2 z*-{2 2 a
*=-~ j=o /=o
oo oo
= 2 2 2 a > a , z - j+lY ( k
7=0 /=0 k=-°°
( oo ^ / \(
£ a{z l £ ajz j £ y(s) zA L setting j = k + y - 1
1=0
/ V"
[Note: £ | aj | < oo => X a 7- < <*>. Hence changing the order of

j j
summation is valid.]
h(z)h(z~l)Cx(z).
Again, f yU ) = 2 7 Cy(e~iX)= ± h(e-a )h
= K e a )h(e~a ) f M )
Exercise 9.4 MA(^) Xt = £ aj£t^ Var(£,) = o2, where et is a white noise. Then
7=0
A (A ) = f e /l(e 'A)/l(e' U)
<7 > f <1 \
2 a ,e iAi 2 a re-"*
2;r
O
II
O
II
2 n n-k
7T— £ £ cos(&A)
^ 7T k = - q r=0
[/ - r = k or / - k = r].
Since spectral density is a real function of A only cosine terms appear.
Exercise 9.5 AR (P)
Z (X}Xt_i = £,, a 0 = 1
7=0
Assume that roots of the equation z p + cc]z p~] + . . . + a p = 0 all lie within the
p
unit circle. Applying Theorem 9.6 to £t = Z (XjXt_j, spectral density of {£,}
y=o
f />
-ikX
/.(A ) = Z OLji Z A (A),
;=0 k=0
where /*(A) is spectral density of X,.
f P
-1
Hence /,(A ) = Z a ,e ° A Z -ikX
W=0 J V*=0
Particular case p = 2
/*(A) = ^ [(1 + a xeiX+ a 2e2U) (1 +
_ ___________________ 1___________________
l + a \ + a \ + 2 a j(l + a 2) cos X + l a 2 cos 2A
Exercise 9.6 ARM A (p , q)

p <1
Z a rXt_r = Z PfEt-rt olo = 1, {£,} is a white noise.
r=0 r=0
Assume roots of + ajZ ^"1+. .. + a p = 0 all lie within the unit circle. Putting
n
Zt = Z p r£t_r and applying Theorem 9.6 we get
r=0
Z' <1
/ za ) = f i f r ^ Z -j'.vA
Z/T ^ r=o s= 0
where/Z(A) is the spectral density of {Z,}.

<7
Again from Z, = Z a rXt_r we get
r=0
p \ f p \
/z(A ) =
= Z ar Z a re /.(A )
o
II
11=0 J
Equating the two expressions we get
<7 ( <7
£ P ,eia 2 p te -z.vA
—2 I r=0 s= 0
/,<A) = f j > 7 -
( P
Z a re irX Z a se~isA
r=0 v=o
Exercise 9.7 Ornstein-Uhlenbeck Process Unlike other time series process,

this is a continuous parameter process.
It s correlation function is given by
p(f) = ae~b^\ a, b > 0
Falls off rapidly

Figure 1
The spectral density
/ U ) = 2“ | e-"xp,dt
=27F £ C""A
= 2tt e<b~'X>' dt+ \ eAb+'X]f
a f 1 _ J _ N) _
2n y b - i X . b + i A) n (b 2 + A 2)
is a Cauchy type density.
The /(A ) has maximum at zero, remains
/(A) constant when A is small compared to b and then
falls off slowly. In Figure 1, pt in fact falls off
exponentially with |t | and can already be con
sidered virtually zero when the distance from the
origin is only a few multiples of b~l. The b~x
-2-10 12
Figure 2 characterizes the time needed for any correlation
between X(s) and X{s + t) to die out.
9.6 Estimation of Spectrum

For a discrete parameter stationary time series the spectral density is given by
/(A ) = I e ~ i kXYk-
2 n k=
We are interested in suggesting some estimate of the spectral density/(A) on the
basis of n observations from {X,}. Given any finite realization, a covariance
stationary time series based on n observations can be represented as
Xt = ~~ + Z (ak cos (Okt + bk sin cokt) assuming n odd, m - n - 1

2 *=i
where 0)k = — , k= 1,2 . . . , m

K n
a0 = X, ak = — Z Xt cos (Okt ,
n t=\
bk = — Z X , sin k = 1, 2, . . . , m.
w r=i
Any function/(0 defined on n integers can be expressed in a finite Fourier
series as
L(n)
f ( t ) = Z (ak cos cokt + bk sin (Dkt), t - 1,. . . , n
*=0
2nk
where cok = — , ab bk are suitably chosen and L(n) is the largest integer not
exceeding -j.
'/ (
Un)
{ak cos cok 1^ ^ bk sin (Ok \^
/(2)
= Z + (9.6.1)
k=0
Kak cos (Okn bk sin cokn
\h*)j
In each row, corresponding to each/(/) there are thus - * + -- + 1 = n

terms. With these ^-elements and n columns, we can thus form a n x « matrix of
ah bh i.e.
fm '
\ =pxai + ... +pnan,
v /("),
^cos a)0 r r cos cok r ^sin cok r

where a { =
81
* • • • Gik+i -
n
*<■N
*
^cos coQn^ vcos (Okn^ ^sin coknJ
Hence we can always represent the/(0 as a linear combination of the orthogonal

vectors a ]y. . . , a n. Since by the trigonometric results we have
1. Z cos (cokt) = Z sin (cokt) = 0

t=i /=i
0 ifktk'
2. Z COS (CQkt) COS (CQk't) = ni l ifk = k ' * l
/=!
n if k = k' = %
0 if***'
n
3. £ sin (cokt) sin (cok't) n il if* = * ' *
t=1
n
II
II
n
4. £ cos (ay) sin (cok't) = 0 for all (oh cok'.
t~\
we can easily check that the above sine, cosine functions appearing in (9.6.1) are
orthogonal functions. Thus a set of n functions defined on n integers can be
obtained such that they form an orthogonal basis in ^-dimensional space. Thus
for suitable choices of ah bk we can always represent any function/(f) defined
on n integers in the form (9.6.1).
To obtain ak, bk we apply the usual regression technique (least square), i.e.
minimize
n f n) 2
£ \ f ( t ) - £ (ak cos cokt + bk sin cokt) with respect to ak and bk s.
/=1 k=0
9
a* = — £ f ( t ) cos cokt, k = 0, 1, 2 , . . . , L(n)
We get n '=' (9.6.2)
6* = — Z / ( / ) sin A: = 0, 1 , . . . , L(n)
n t=1
If n is even. L (« ) = f
2 n
= - % f ( t ) cos (nt)
bL{n) = \ £ / ( f ) s i n ( w ) = 0
i.e. although contributes something does not. Hence we take n odd.

We have here represented a observations by a function having n parameters.
As in usual regression analysis, we can partition the total sum of square (SS) into
(m -»- 1) components
n m n
£ X 2 - n X 2 = £ £ (ak cos 0)^+ bk sin a>*f)2,
(=1 fc=ir=i
where ak and bk are given by (9.6.2).
n n
Sum of squares = £ (ax cos a y + bx sin a y ) 2 + £ (a2 cos co2t + b2 sin ay)2
/=i f=i
n
+ . . . + £ (am cos comt + bm sin (Omt)2
t=\
- — (a 2 + b2) + . . . + -■ ( a 2 + b^)
ANOVA Table
Source d.f. SS
Due to mean 1 na\!2

frequency at 2 n/2 (af + b?)
... 0)m 2 nil (a2 +
Total n Total SS
The residual sum of squares expressed by the periodic component at frequency

(Ok is given by the SS n!2 (a \ + b\) which is called the “periodogram”.
Let l„(cok) = \ { a \ + b\-)
(n \ 2"
2 Xt cos (okt Z Xt sin cokt
n V=' )
2
2 Xte - it(0k , k = 0, 1, • . . , tn
n t=\
= Normalized SS at frequency (Ok.

The plot of ln{cok) versus cok is usually called the periodogram even though
In((Qk) is a function of frequency rather than period. Anderson (1970) described
the graph of {a\ + bk) versus the period nlk as the periodogram and suggested
the term spectogram to describe the graph of In((Qk) against the frequency cok.
Hannan (1970) defined the periodogram in terms of complex number as
1
2 71n /=l
The periodogram appears to be a natural way of estimating the spectral density
function. But we shall see that for a process with a continuous spectrum, it
provides a poor estimate and need to be modified.
Under normal autocovariance stationary time series N(0, cr^) we have
ak = — Z Xt cos a)kt - N
n /=1
2<t 2
and bk = — 2 X, sin (Okt AM 0 , for k *
n ,=i *
cov (ak, bk ) = 4 cov S X, cos (o^t, S X, sin tin2

'=i '=> )/
A 2 n
= —t— X {cos cokt sin cokt]
n L t=l
2<j 2 n
= —r— X sin 2a)kt = 0 [since X,’s are uncorrelated].
/=i
Therefore, ak, ^ ’s are independent (since being linear combinations of normal
r.v.s a*, bk are normal).
Hence 2 * , * I are independent

f^ ~ J
and /„(*»*.) = ~ { a l + bf) ~ 1,2,...,
Thus Var [In(0)k)]= 4 a 4,
Therefore periodogram ordinates In(cok) are independently and exponentially

distributed with means a 2 - 2nf(cok), where/(.) is the spectral density of {X,}.
Theorem 9.7 For any covariance stationary time series with absolutely summable
autocorrelation function, /*(&>) = ■— In(co) is an asymptotically unbiased but
inconsistent estimate of the spectral density
Proof !n(0)) = - I X ,e -im

t=\
X Xte~ita) X Xr elt'a
t=\ /'=1
X X XtXr e-i(t~r)a)
/=! /'=1
Now EXt = 0. Therefore
E(In(co)) = ± X X E(XtXr ) e - i(t-r)a)

t=\ t=l
= - 2 Z Y .- r e - ^
n t=\ t '= i
n-1
= — X y h e lh(° (n - \h\) [yt. is an even function]
n h=-(n-\)
n- 1
=2 Z yh e~ihu> - 2 I ■ihcô
h=-(n-\) h--{n-\) n
t - f = 0 —» n times,
= 1 —> n - 1 times,
= —> n - 2 times and
= n - 1 —> 1 times]
We have to show that Z 1A1 < £ for n > N.
n- 1 n- 1
Z 1A I Yhe -ihw < 2 -ihco |
lA im e
&=-(«-1) n *=-(#!-1) n
AJ-1
<2 Jy*.
/?=0 «
z
By applying the Kronecker’s lemma (since autocovariances are absolutely
summable), we get
lim Z \A1 y
n—>°° / i = - ( a 2- 1) n
Thus lim E (ln(a>))= 2 2 Yhe~h<0= 4tt/((U) (assuming y0 = <?2= !)•

n—>°° h~-00
Therefore, lim E(I*(co))=f(co).
n—>°°
Hence I* (co) is an asymptotically unbiased estimate of the spectrum/(co).

In case of normal error N(0, cr2), we have shown
V a r ( / ^ ( c u )) = - p - y * 0.
A n1
Bartlett (1966) even showed that
lim Var(/;(fl)» = - p y * 0
47TZ
for stationary process with continuous spectrum. Thus l*n {(D) is not a consistent
estimate of the spectrum f{(D).
9.6.1 Use of the periodogram

It is needed to search for “cycles” or “hidden periodicities” in the data. For
example, let us assume that a time series is represented by
Xt = + A cos (cot) + B sin (cot) + et (9.6.3)
here et are normal independent N{0, a 1) random variables, A and B are fixed, and
2 jik
(D(known) is of the form, (0 = — , where k is an integer.
To test Hq \ A - B = 0 against
H\ : A ^ 0 or B 0 (9.6.4)
We can use
(Total SS-SS due to p) - (Total SS-SS due to A, B - S S due to ju)

Fi 2(m-\) -
Total SS - SS due to p - SS due to A, B
(al + b*)/2
-, d.f. 2(m - 1) = n - 3
2 (a* + bj )/ 2(m - 1)
j*k J J
Ink
If (0 cannot be expressed as — , then the regression associated with (9.6.3)
can be computed and the test be constructed by usual regression technique.
Fisher's test for hidden periodicities

To search for hidden periodicities we have to make the test mentioned in (9.6.4).
When 0) is unknown, we have to search for the largest periodogram ordinate and
ask whether this ordinate can reasonably be considered the largest in a random
sample of size m selected from a distribution function, i.e. a multiple of a x \-
A statistic used to test the hypothesis is
- 2 I„((0 In(L)>
tn t=\
where In(L) is the largest periodogram ordinate in a sample of m periodogram
ordinates each with 2 d.f. Fisher (1929, Proceedings of Royal Society, Series A)
demonstrated that for g > 0,
r i , i / m' I(COj)
T
p — ^m > 8 (1 - g j ) m = rn max
II
m J=i i< j< m

m
J ; I 1(0) j)
where ^m is constructed from the periodogram of a sequence of independent

/V(/r, o2) r.v.s and k the largest integer less than g~l (Davis 1941, Analysis of
Economic Time Series).
9.6.2 Some Consistent Estimation Procedures

We shall discuss some methods each of which will provide a consistent estimate
of the spectrum. Although the periodogram is itself an inconsistent estimate the
procedure considered here are essentially based on periodogram (by using some
sort of smoothing technique).
We assume that the available time series data is free from trend or seasonal
variation since otherwise the results of the spectral analysis are likely to be
dominated by these effects, making any other effects different or impossible to
see. Trend will produce a peak at zero frequency while seasonal variation produces
peaks at the corresponding seasonal frequency and at integer multiples of the
seasonal frequency.
The methods are: (i) transforming the truncated autocovariance function
(ii) hanning, (iii) hamming and (iv) the fast Fourier transform.
Method (i) consists of taking a Fourier transform of truncated sample

autocovariance function using a weighting procedure.
Using trigonometric result
n n
Z cos (Opt = Z sin (0pt = 0
/=i t=\
where (0p = 2npln,
(
= 2[(Z (Xt - X)cos (0pt) 2 + (Z (Xt - X) sin (Opt )2]/n

t t
n n __
=2 Z Z ( Xt - X)(cos (Opt cos (Ops + sin (Opt sin (Ops)!n
t = \ s= \
= - I I (X ,-X )c o sft)„(r-.s)
n t=\ s—l
(n-l) M—
^ _ _
=2 Z C* cos where= and Q = Z ( Xt - X ) ( X t+k- X ) / n
k = -(n -l) t=\
n-1
ln(0)) = J ^ l n(0)p)= 1 Un + 2 Z Ck COS (0 pk (9.6.5)
2n k=\
which is the obvious estimate of

oo
y(0) + 2 I Y(k) cos (ok .

k=\
Equation (9.6.5) is obtained by simply replacing y(k) by its estimate Ck for k =

0, 1, 2, . . . , n - 1 and yk = 0 for /: > n. Hence /* (cu) is the discrete Fourier
transform for the complete sample autocovariance function.
The precision of the Ck decreases as k increases so that it would be intuitively
reasonable to give less weight to the values Ck as k increases. An estimator is
then constructed as
i f A#(«)
/(ft)) = ■{b0C0 + 2 S bkCk cos {(ok) ■.
LU ^ k=\
where {bk} are a set of weights called the Lag Window and M(n) is called the
truncation point. Here we note that the values of Ck for M < k < n are no longer
used, while that of Ck for k < M are weighted by bk.
In order to use the above estimator we shall have to choose a suitable lag
window and a suitable truncation point. Two lag windows generally used in
practice are
(a) Tukey-Window
bk = j ( l + cos n k / M) , k =1,2,
This window is also called the Tukey-Hanning or Blackman window.

(b) Parzen-Window
This has an advantage of not giving negative estimates for /.
9.7 Forecasting
Suppose we have an observed time series X2, . . . , Xn. The forecasting
problem is to estimate XN+i or more generally XN+h (h is a positive integer). The
prediction of XN+h made at time N of the value h steps ahead will be denoted by
X(N, h). This integer /i is called the /ead time. There are many forecasting
procedures which can be classified into three broad categories,
(a) Subjective: Forecasts can be subjective using judgement, intuition,
commercial knowledge and any other relevant information procedures of this
type (e.g. Delphi method) but they will not be discussed here.
(b) Univariate: These are forecasts entirely based on past observations of a
given series, by fitting a model to the data and extrapolating. We shall discuss
only two types.
(c) Multivariate: Such forecasts can be made by taking observations of other
variables into account. For example, sales may depend on stocks. Regression
models are of this type, as are econometric models. The use of a “leading
indicator” also comes into this category. Multivariate models are sometimes
called causal or prediction models. We shall not discuss this but only two of
several forecasting procedures available for time series data.
9.7.1 Exponential smoothing (Holt-1958, Brown 1963)

Given a stationary, non-seasonal time series X {, X2, . . . , Xn, it is natural to take
as an estimate of XN+l the weighted sum of past observations
X(N, 1) = CoXN + CjJfon + . . . (9.7.1)

where {C; } are weights. Geometric weights Cj = a ( l - ct)J, j = 0, 1, . . . where
0 < a < 1 are chosen to give more weights to recent observations and less
weights to the past observations.
Since in reality we have only finite number of observations, let us modify
observations and then (9.7.1) can be written as
X(N, 1) = a X N + (1 - a)[aXN_x + a ( l - a)XN_2 + . . .]
= aXN + (1 - a) X{N - 1, 1) (9.7.2)

If we set X(l, 1) = X\ then equation (9.7.2) can be used recursively to

compute forecasts. Also it reduces the amount of arithmetic involved, since
forecasts can easily be updated using only the latest observation and previous
forecast. The procedure defined by (9.7.2) is called exponential smoothing.
The adjective exponential arises from the fact that the geometric weights lie
on an exponential curve.
Equation (9.7.2) can be written as
X(N, l) = a[XN- X ( N - 1, 1)] + X ( N - 1, 1)
= a e N + X(N - 1, 1), where eN = XN- X( N - 1, 1)

= prediction error at time N.
It can be shown that exponential smoothing is optimal if the underlying
model for the time series is given by
Xt = p + a E £j -f et.
j< t
This infinite MA(«>) is non-stationary, but the first differences Wt = Xt - Xt_

t form a MA(1) process so that Xt is ARIMA (0, 1, 1) process. The value of the
smoothing constant a depends on the properties of the given time series. Values
between 0.1 and 0.3 are often used. The value of a may be estimated from past
data by a similar procedure as used in estimating the parameter of a MA process.
The SS of prediction errors is computed for different values of a and the value
is chosen which minimizes the S.S. i.e. X(l, 1) = Xb 02 = X2 - X(l, 1). 03 =
X3 - X(2, 1), and so on. In general 0N = XN - X( N - 1, 1), N = 4, 5, . . . , N.
. N
Compute E Of. Repeat for different values of a between 0 and 1 in steps of .01.
i=2
Usually the SS surface is quite flat near the minimum and the choice of a is not
critical.
9.7.2 The Box-Jenkins seasonal model (SARIMA)

In practice, many time series contain a seasonal periodic component which
repeats after every s observations. For simplicity we consider monthly observations
where s = 12. Box and Jenkins (1970) have generalized the ARIMA model to
deal with seasonality and define a general multiplicative seasonal model in the
form (using their notation).
<Pp(R)lp(Bn )W' = 6q(B)PQ{Bn )at, (9.7.3)
where B denotes the Backshift operator, <Pp, Ip, 6q, PQ are polynomials of order
P> P, q, Q respectively, and [at} rather than {£,} is the Box-Jenkins notation for
a purely random process, with Eat = 0, Var(a,) = a 2 and Wt = which
removes both trend and seasonality, e.g. d = D = 1,
w, = vv12x, = v12x ,- v 12xM
= ( * ,- * M 2 ) - ( * M - * M 3)
The model (9.7.3) is called Seasonal Integrated Autoregressive Moving Average

(SARIMA (p, d, q) X (P, D, Q)).
Usually d and D do not exceed 1 and B n Wt. = Wt_l2
Suppose p = Q = 1 and P = q - 0 in (9.7.3). Then (9.7.3) becomes
(1 - aB)W t = (l + 6Bn )at
so that Wt = aW t_i + at + 9at^ 2 (9.7.4)
If, for example Wt = V12X, then (9.7.4) becomes
%t = X(-\2 + tf(X,_i - X;_13) + at + 0a f_i2-
9.7.3 Box-Jenkins forecasting procedure

This procedure is based on fitting an autoregressive integrated moving average
(ARIMA) model to a given set of data and then taking conditional expectations.
Following are the main stages in setting up a Box-Jenkins forecasting model:
/. Model identification
Examine data to see which member of the class of ARIMA Process appears to be
the most appropriate.
//. Estimation
Estimate the parameters of the chosen model by L.S. (Least Square) or Maximum
likelihood.
III. Diagnostic checking
Examine the residuals from the fitted model to see if it is adequate.
IV. Consider alternative models if necessary
If the first model appears to be inadequate for some reason, then other ARIMA
models may be tried until a satisfactory model is found.
V. Forecasting
The first step in the Box-Jenkins procedure is to difference the data until it is
stationary. This is achieved by examining the correlograms of various differenced
series until one is found which comes down to zero “fairly quickly” and from
which any seasonal effect has been largely removed. For non-seasonal data, first
order differencing is usually sufficient. For seasonal data of period 12 month the
operator VV12 is often used if the seasonal effect is additive, while the operator
Vj*2 may be used if the seasonal effect is multiplicative (e.g. Xt = Lt • St + %
where Lt is the linear trend, St is the seasonal component). Sometimes the operator
V j2 be itself will be sufficient. Over-differencing should be avoided. For quarterly
data, the operator V4 mey be used, and so on.
The differenced series will be denoted by {Wt, t = 1 , . . . , n - c] where c terms
are “lost” by differencing, e.g. if the operator V12 is used then c = 13.
If the data is non-seasonal, an ARIMA model can now be fitted to Wt. If the
data is seasonal, then the general seasonal ARMA model may be fitted (done
before). “Reasonable” values of p, P, q, Q are selected by examining the correlogram
of the differenced series {Wt). Values of p and q are selected by examining the
first few value of rk. Values of P, Q are selected by examining the values of rk at
k = 12, 24 . . . (where seasonal period is 12), e.g. r 12 is large but r24 is small
suggests one seasonal moving average term. So take p = 0, Q = 1.
After finding reasonable seasonal ARMA model, L.S. estimators of the
parameters are obtained by minimizing residual S.S. in a similar way to that
proposed for the ordinary ARMA model. In case of seasonal model, it is advisable
to estimate initial valuers of at and Wt by back-forecasting (or back casting)
rather than set them equal to zero (see Box-Jenkins section 9.2.4). If the model
contains a seasonal moving average parameter which is close to one, several
cycles of forward and backward iteration may be needed. The adequacy of the
fitted model should be checked by what Box and Jenkins call “diagnostic-checking”.
This consists of examining the residuals from the fitted model to see if there is
any evidence of non-randomness. The correlogram of the residuals is calculated
and we can then see how many coefficients are significantly different from zero
and whether any further terms are indicated in the ARIMA model. If the fitted
model appears to be inadequate, then alternative ARIMA models may be tried
until satisfactory one is found.
When a satisfactory model is found, forecasts may readily be computed.
Given data upto time N these forecasts will involve the observations and fitted
residuals upto and including time N. They rely on the fact that the best forecast
of all future residuals is simple zero (i.e. aN+\, aN+2, • • • = 0), e.g. consider the
seasonal ARIMA model example
X, = Xt_i2 + cc(Xt_i - XM3) + at + 6 a t_l2-

The forecast of X N+1 is obviously given by
X(V, 1) = X„_n + a ( X N - X„_12) + 0aN_n ,
where a , 6 are estimators of a , 9 and aN_u is the estimated residual at time

N-l 1.
The forecast of XN+2 will involve the (yet unknown) value of XN+l and hence
insert the forecast value instead, i.e.
X(V,2) = Xyv-io + &2 (Xn - XN_\2) + 9 a a N-\\ + 0aN_\Q
9.8 Estimation of Parameter in the First Order

Autoregressive Model AR(1)
Let a be a real number and {en, n> 1} be an i.i.d. sequence of r.v.s defined on
a probability space (Q, ^ P) Given X0 an arbitrary initial r.v. independent of
{£„}, define recursively the sequence of r.v.s {X„, n > 0} as follows:
X0, Xj = aX 0 + £ j,. . . , X„+1 = a X n + en+l(n > 0). (9.8.1)
Then [Xn\ n > 0} is a Markov process on the state space S = R1, having the
transition probability of going from x to A (a Borel set of R1) in one step
P(x, A) = P(Xn+i e A | X0, Xb . . . , Xn)
= P(ax + en+i e A | Xn = x)
= P(en+1 e A - ax | Xn = x) = F(A - a X n)
where F is the common d.f. of en and the initial distribution given by the distribution
of XQ. Without loss of generality we take X0 = 0 a.s. Now Xn has the stationary
representation
Xn = Z a j en-i, I a \ < 1. (9.8.2)

j=0
Then the least square estimate of a is given by

n > 1( n >
( n > / f * "y
? <°
a +
M
i x ^ x , =
X
p , 2- , /
v ml t 1 1 ) V "1 ) / . k 1 )
( n } f * "y
or a n - a - E etx t_\ 2 * 2,
k'=1 J / i ,=i j
We shall study the asymptotic optimal properties of a .
Theorem 9.8 If {en, n > 0} is an i.i.d. sequence of r.v.s with P(£x = 0) * 0,

E£i = 0, Ee^ = ( j 2 > 0. Then a —> a a.s. as n —» «>, i.e. a is a strongly consistent
estimator of a.
Without loss of generality we shall take a 2 = 1.
From equation (9.8.1) we get
Xn = a ( a X n_ 2 +£„_!) + £„ = . . .
= a n~l£l + a n~2£ 2 + . . . + a e n_i + en (9.8.3)
To determine the convergence properties, one needs to consider three cases
( l ) | a | < l , ( 2 ) | a | = l and (3) | a | > 1. The corresponding processes are termed
stable, unstable and explosive, respectively. Since EXn = 0 and from (9.8.3)
Var(X„) = E a 2(*“1}, it follows that lim Var(X„) < iff | a | < 1 since
k=l o
(1 - a 2n)/( 1 - a 2) if * 1
Var(X„)
n if a = 1
n 1/2( 1 - a 2) 1/2 i f | a | < 1
Let g(n, a) = n /V 2 if | a | = 1 (9.8.4)

| a| " / ( l - a 2 ) 172 i f | a | > 1
Note that g(n, a) is called the Fisher information in statistics. To pro

theorem let us state a lemma.
n
Lemma 9.3 Let Vn = Z Xr2 r Then g(n, a)-2 V as n —» 00 and
P (V = 0) = 0.
Proof We shall prove the lemma only in the stable case | a | < 1. The proofs of
the other cases are quite involved and hence are omitted. It is enough to prove
that Wn -£> cf2 > 0. By equation (9.8.3) we have
fn-l 2"
1 £ * , 2- , = 1 £ 2 + ( £ 2 + CtE j)2 + . . . +
« i=i ' n
l*=i J
2 £ a 2(*-D + e 2 % a 2(k-l) + _ + e2
1 k= 1 2 *=1
+ J [a(e,^ + £2£3 + • • • + V 2V 1) + • • • + a"~2£,e„_,]

The second expression has mean zero and variance
- - [ a 2(rc - 2) + a A{n - 3) + . . . + a 2("~2)] < — S a 2*

n2 w *=o
= ---------- r---- ) 0 a s n - ) o o .
n(l - a 2)
Hence the second term -£» 0. The first term is equal to
I 1 - a 2(n~l) 1 - a 2^ )
+ £^ . . + 1 - « 2 _2
n 1- a2 1- a 2 1- a 2
i n-1 i
= 1 E £2 - 1 t£2a 2("-') + • • + a 2£n2_,]
n (l - a 2 ) i=i ' n (l - a 2)
The second term has mean zero and variance —»0 as » whereas the first
term by the Kolm ogorov’s strong law for i.i.d. r.v.s tends to
2
£(£?)/( 1 - a 2) = ———r a.s. Therefore, by Slutsky’s Theorem as [(1 - a 2)/
1 - cr
n]Vn -£> g 2 > 0 as n —>
**Proof of Theorem $.8

Since Vn > 0 a.s. for each jc > 0,
P(Vn > jc) = P(Wn > rg-\n, a)) = 1 - Fn(x g -\ n , a)) (say)
where Fn is the d.f. of Wn. By Lemma 9.3 Fn —> F(x), jc e c(F) where F is the d.f.
Hence given £ > 0, there exists t]> 0 such that F( 8 ) < rj. Choose jc) such
that n > n0 => xg~2{n, a) < 8 .

Since Vn T, lim P[Vn > x] = F[lim Vn > jc] > 1 - F{8 ) > 1 - r\.
n —>°°
Since rj > 0 and x > 0, P[lim Vn = ©o] = 1.

n
Hence lim V~l = 0 a.s. and Vn > kQ> 0 a.s. for large n.
n
Let $8n = <7(£i, . . . , £n). Consider Zn = 2 j£t
>t=i
Since E(ek) = 0 and E{\Xk \V[') < [ E ( X 2 kIVk )E(V; ' ) ] y2
< { E V k ' ) 112< °° for k > k0and [Zn, is a martingale
EZ- = E 2 v r * X l +2 Z E(Vk\v;}_xXk_,X
k=\ \<k<k'<n
=Z E { V ^ X 2_ { E { e 2 \ 3
k- 1
+2 Z E(Vk '_xV~}_xX*., X*._, ek E( ek. | JgV-i)

n
=a2 Z ) (sin ce E{e\ \ &k-\) = ££* = a 2 ’
&=i
£(e*l^*-i) = £(e*) = 0))
= cr2 £ 21 X*2*-,‘ (9.8.5)

*= y
« n y y " r* dt r dt
Z x 2 V,-2 = Z -* - < Z
*=1 * * *=2 V.2 £,
= l
*“2 7
^ - \*0 I <9-8 «
where c = V 2 > k0 > 0 (taking n0 = 1 for convenience).

For large /i, Vn > 1 a.s. and from (9.8.5) and (9.8.6), we get sup EZ 2 < ©o.
n
Hence {Z„, £8 n] is L2-bounded martingale and hence uniformly integrable. Thus
by martingale convergence theorem Z„ —>Z a.s. and in L2 (mean-square). So by
1 n~l
Kronecker’s lemma — Z Xk_xek -» 0 a.s. a s n 4 » . Therefore, a n —> a.s. as
yn *=i
Theorem 9.9 (CLTfor L.S. estimate a n)

Let o 2 = 1.
(i) If | a | < 1, then ( a n - a ) g ( n , a ) N(0, 1)
and v ? ( a „ - a ) - ± > N ( 0, 1).

(ii) If moreover £„’s are jV(0, 1) and | a \ > 1, then
(a n - c c ) g ( n y a ) Cauchy
i
and Vn2 ( a n - a ) - ± > N ( 0, 1).
^n
Proof Now a n - a = Z Xt_xet p i, If we assume I a I < 1
/=i i /
n j( n iA
then g(n, a) ( a n - a) =Vn Z £,X,_, / Z X,2, (1 - a 2)J
i=i y
= (/ln / V ^ ) [ - ^ | (1 - a 2 ) 1' 2 (9.8.6)
where An = Z £,XM = Z £,£,_i + 0 f l £ ,£ ,_ 2 + 2£„£i

7=1 7=2 7=3
n
Now by Lemma 9.3, Z X^_xln-^> y(0) = a 2/( 1 - a 2) and hence g(n, a)
7=1
(a„ - a ) has same asymptotic distribution as
n mAnl { ( j \ 1 - a 2)"172).
Now {enX/I_|, n > 1} is a martingale difference sequence, since Een = 0 and
£n and are independent.
Also Var(X,) = — - cr2 and Var {Anl j n ) ------——r- for large n. Let
1- cr I - a1
us state the following Martingale CLT (due to Billingsley 1968).
n
Let Sn = Z Yt be a martingale and Yt’s be stationary with finite (common)
/-i
variance.
Then 5„/(Var(5„))1/2 = SnlV ^(Var(y,))1/2 A /V(0, 1) as
4
X,_i£,. Then Var(X,_i£,)----- ——- . By Slutsky’s theorem (a„ - a) g(n, a)
1 - a2
N(0, 1).
Again V y 2 ( a n- a) =Vn(A„)(V„/n)~y2has the same asymptotic distribution
as Vn A„/ct( 1 - a 2)~U2.This shows that
Vny 2 (a„ - a ) -U 0, 1).

Now £,’s are independent N{0, 1) and ek = Xk - a Xk_x. The transformation
from the et to the Xt is one-to-one with jacobian unity, and hence the joint density
of Aj, . . . , Xn is
/( * ! ,..., x n)= (2 exp £ (** - a x k.x) 2 (9.8.7)

*=i
= (2 n) nl2exp | - j Ax) j,
where Anxn is a symmetric positive definite matrix given by A = (.ay) with

an = 1 + a 2, i = 1, 2, . . . , n - 1
ann = 1
aii+1 = - a, i = 1, 2, . . . , n - 1
atj = 0 otherwise and x = (jq, . . . , x„).
Also a n - a can be written as
a n - a = 2 X,_,£( / Z X 2 , = Z (X,X,_, - crX2, ) / Z X,2

7=1 / 7=1 7=1 7=1
= (X'BX)/(X'CX) (9.8.8)
where X' -(X\, X2,. ■. , Xn) and Bnxn is a symmetric matrix with
diagonal except for the «th element, which is zero, -1 for the first line parallel
to the diagonal and zeros elsewhere, and
1 0 0 O'
0 1 0 0
0 0 1 0
0 0 0 0 0 1 0
0 0 0 0 0 0 0
The joint m.g.f. of the two r.v.s in (9.8.8) is given by

m(u,v) = E(exp{u(X'BX) +n(X'CX)})
-n/2
= (2 n )
If exp \ - | — | (x'Ax) +u(x'Bx) + v(x'Cx) d x x... d x n
= (2n) -n/2 exp<j- ( ^ | (x' Hx) [• d x x. . d x n (9.8.9)

II
where H = A-2iiB-2vC, which is positive definite if w, v are sufficiently small so
that the integral in (9.8.9) exists. Since H is symmetric, there exists an orthogonal
matrix P such that H = P'DP, where D is diagonal with eigen values > 0 of H.
Setting y = P'x and noting that the absolute value of the jacobian is equal to 1,
we obtain from (9.8.9)
m(u, v )= (2 ; r)m/2 J J exp j - D y )| d y , . . . d y n
-m/2
1/2
= 1 n A, = 1H -
(9.8.10)
J=i
Settingp - 1 + a 2 - 2v + 2 au, q = - {a + u), then writting Hn for the H which
depends on n, we obtain by expansion (of Hn = An- 2uBn - 2vCn) the difference
equation
= pHn-i - q2Hn_2, (9.8.11)
with initial values = 1 and H2 = p - q2- Let z\ and z 2 be the roots the
characteristic equation of the difference equation (9.8.11) z 2 - pz + 1 = 0 so that
z u z2 = 1 / 2 p ± 1 /2 ( p 2 - 4 q 2) m (9.8.12)
1-
H„ = 1 - *2
Z\
Hence •*2 (9.8.13)
^1-^2 1 Z2 - Z \
Substituting (9.8.13) in (9.8.10) we get the m.g.f. of the r.v.s for each n.
Now X'BX X'CX

g(n, a ) ( a n - a ) =
g(n, a) g 2 (n, a)
and we can obtain the joint m.g.f. from m(u, v) by setting
0i 0? 1/2
Mn(Ou 0 2) = m = 1Hn
-
(9.8.14)
g(n, a ) ’ g2(„, a )
where Hn is // with w, v , replaced by 01/g(n, a) and 0 2/g\n, a), respectively.

Recalling the values of g(n, a) and expanding the radical in z2 of (9.8.12), we
get the following expressions.
Case 1. |a |< 1
0 2 + 202
z\ = 1 - + 0(n~1/2)
0 2 + 2 a 26 2
z 2 = a 2 + 2a 0^(1 - a 2)/n]m + + 0 (« -i/z)
Case 2. |a |> 1
(0 2 + 202) ( a 2 - 1)
Zj —1 + + 0 (| a I-3")
a In
2a 0 , (a* - 1) _ + 0(| a
| a |" a 2"
S u b s t i t u t i n g t h e s e v a l u e s in ( 9 . 8 . 1 4 ) a n d s i m p l i f y i n g , w e g e t
r aj ma\ {exp (0 2 + if | « ( < 1

lim M JQ i , 0 2) = * 1/n
[(1 - 2 0 2 - 0,2 ) - 1/2 if | a | > 1
R ep la cin g 0 x,02 b y it xa n d it2, we o b tain the
0 { t u t 2) = lim M n(itx, it2) (9.8.15)
j exp ( it2 - t f 12 if | | <1

(9.8.16)
= { (l-2 ih +if | a | > 1
To get the limit d.f. of g(n, a)(a„ - a ) , we see that
f X'g„X
w r, „ < x rc„x„ \
F(x) = lim P
n—*°° g(n, a) ~ x g2(n, a)
X'BnX
= nlim P - x ■* ^ < o
—>°o g (n ,a ) g 2( n , a )
= P(U -xV< 0)P^y< x
X'BnX X'CnX
where (U,V)
g(n, a y , a)
(from Lemma 9.3 and Theorem 9.8).
The ch.f. of (UyV) is given by (9.8.16) and hence by Cramer (1937) Theorem
for ratio of two r.v.s and the joint ch.f., we get the density of U/V as follows.
Case 1. |a |< 1
30
F'(x)=f(x) = — r (v, - xv) dv
2 nlI V —ex 3u
=j_ r . exp {-{ivx + v 2 /2)}dvby

( (9.8.15) and (9.8.16))
2 ni
„ - x z /2
a/2 tt
This was proved without the assumption of normality of et.

Case 2. | a | > 1
By Lemma 9.3 P(V > 0) = 1. Now 0(tb t2) being the limit of ch.f.s is a ch.f.
Again by Cramer’s Theorem the density of U/V is given by
/U)=2^L ^ (t’ - t X)dt
=_L r jdt
2 nii( + t l + 2 itx)m
*“ J _______ t + ix_______
( (1 + x 2)^/l + + 2 J
= J _ lim r _________1 + (ix/T)________

2 nr
->~ I (i + x 2)(l + T 2+ 2iTx)IT2)m
___________ 1 - (ix/T)__________
~(1 + x 2)(- l)([l +T 2 - 2iT]/T 2 ) ' 12
1 2 1 1
1 + jc2 71i + x2
which is the cauchy p.d.f.

Case 3. When £„s are normal and | a | > 1 then V„1/2 ( a n - a) N( 0, 1).
The proof is rather involved and hence is omitted.

Exercise 9.1 Prove that for the ARMA (1, 1) process Xt = aXt_x + e, + st is a
UW(0, <t2), the autocovariances of Xt are given by
1 + 2afi+ P 2 (1 + a/3)(a+ 0)
/( 0) = cr2, y(l)
1- a 2 1- a 2
and the autocorrelation p(k) = ap(k - 1), k > 2. Hence show that autocorrelations of X,
obey p{k)= a*~‘p( 1), k>2 and p(l) = + "•
1+ 2
Exercise 9.2 If X, and T, are two independent Moving average processes MA(qx) and
MA(q2) respectively. Then show that Z, = X, + T, is a MA(q), where <7< max (ql%q2).
Exercise 9.3 If Xt and Yt are two independent ARMA (px, qx) and ARMA (p2, q2)
processes respectively. Then show that Z, = X, + Yt is also a ARMA (p, q) process, where
p < px + p 2 and q < max {px + q2, p2 + qx).
Exercise 9.4 (a) If Xt is an ARMA (1, 1) process given by Xt = 4iXt_x + 6 et_x +
| 0 | < 1 and et is a WN (0, cr2) then express X, as an infinite linear combination of
{£t, £t~i» &t-2*• • •}•
(b) If | 6 | < 1 then express £t as an infinite linear combination of {Xt, XM, . . . Xt_h
Exercise 9.5 (a) If {X,} is zero mean second order stationary nondeterministic time
series, then using Wold’s representation theorem show that fc-step optimal mean square
predictor is given by Xt+k\t = £ Wuket-i> where £ \j/j < 00 and e/s are i.i.d. random
1 i=o 0
variables. Also show that the minimum value of the mean square error is given by
*-1
cr2 £ ipj. (b) In case of AR(1) process X, = oOCt_x + £„ | a \ < 1, show that the optimal
mean/=0square predictor is given by Xt+k\t = orXv
k In case of MA(1) process X, = £t + &£t-X,
| (5 | < 1, show that the optimum one-step ahead predictor is given by
x l+tu= px, - 8I 2X
Exercise 9.6 Show that the real valued function K(h) defined on the set of integers Z,
1 if/r = 0
K(h) = pifh = ± 1
0 otherwise
is an autocoveriance function iff | p \ < 1/2.
Exercise 9.7 Let {St, t = 0, 1, 2 ,...} be the random walk with constant drift pydefined
by S0 = 0 and
St = p + St_x + Xt t = 1, 2, . . .
where Xb X2, ... are i.i.d. r.v.s with mean 0 and variance or2. Compute the mean of St and
the autocovariance function of the process {S,}. Show that {AS, = (1 - B)St} is stationary
and compute its mean and autocovariance function.
Exercise 9.8 Show that in order for an AR(2) process X, = 01X,_1 + <j)2Xt_2 +£„ £t ~
WV(0, a2) to have an infinite moving average representation, the parameters (0b 02)
may not lie in the triangular region determined by the intersection of the three regions 02
+ 0i < 1, < 1 and [ | < 1. Show that 0, = P(1)(1 ~ P(2)) <t>2 =
1 - P 2(1) 1 - P 2(D
Also prove that p 2(l) < l/2(p(2) + 1).
Exercise 9.9 Let X, = 6Xt_x + £„ t 1, 0, 1, ... be an AR(1) process where £t are
_ °~21^
i-i.d. /V(0, cr2) and | 6 \ < 1. Show that autocovariance function is given by y (k) =
1 -0 2
the spectral density is given by /(A) = | 1- 6e a |2. Also show that the log-likelihood
Oi-dimensional p.d.f.)
Ln = log p{xx< . . . , xn) of Xh . . . , Xn is - 1/2 {n log {2kg2)- log (1 - Q2)
n
+ ct- 2 [ ( \ - o 2) x f + Z { Xj -eXj_x)2}).
7= 2
P Q
Exercise 9.10 Let {X,} be an ARMA (p, q) process given by Z a rXt_r = Z /3y£,_y,
r=0 s=0
Oq * 0, Pq * 0 and {£,} is a \V7V(0, ct2). Assume that the roots of the AR and MA
polynomials are within the unit disc. Show that the variance of the prediction error is E
| Xt - Xt |2 = |V « o |2 <?2 and Xt = E(Xt \ X,, s < t - 1). Also show that
G2 «0 2;rexp<1/(2 7i) I log/(A)dA where/( A) is the spectral density of {X,}.

Po
♦♦Exercise 9.11 Let {X,} be a MA(1) process Xt = 6 xet - | 6 X\ < 1 and e( is as in
exercise 9.9. Show that the log-likelihood ratio Ln = - 1/2
1 - | 0, |2("+1) l-IO iP
n log {2 k g 2 ) + log
l - |0 i |
+ er~2
i w- i - 1e, |2(n+1)
2 0.x,
if | 0, | < 1 and K* = 2 0,*-; 1,2____, n and = - 1/2

7=1
n log (2 tz<j2) + log (n + 1) + <7~ 2 Yf ------ 5-r 2 Yj s if I0 , I= 1

7=1 ' « + 1 I 7=1
where 1} = Xj + X2 + . . . + Xy, y = 1, 2, . . . , wand the spectral density is given by
/U ) = - 0, |2.
** Exercise 9.12 Let X, be a MA(2) process given by Xt = et- 2et_x + £r_2where £/s are
as in Exercise 9.11. Show that the spectral density
/(A ) = f i | l
and the log-likelihood

n{n + 1)(2n + 1)
L„= - 1/2 | n log {2 k g 2) + log dn + cr~2 Z Yf!dn
i r‘
( n \2 r « » ^1
+ (2 + n) Zj Yj + (2 - n - n2) S /7 ? |
\7= J I '- 1 2=1
where K, = Z &X*,y = 1, 2,. . ., n and = -jr {n + 1){n + 1){n2 + 5n + 6).

l 12
** Exercise 9.13 Let {A',} be an ARMA (1,1) process given by

Xt = 6 xXt_x + 6 2£t - £t-\,
where | Ql | < 1, | 02 \ < 1. Show that the spectral density is given by

/(A) = {a 1H
i |0, - eu |2| - ea \2
and the log-likelihood
|0i - <h I2 Nn(9,)

L „ = - 1/2 n log (2/rtJ2) + log
1 - of
+ <7-2 Z o', - 0,ry.,)2 - 10, - 02 12 Z Q-u-'\Yj-

2=1
(1 -0 2 ) + ! 0 ,- 0 2 I2 (Nn(02))-' , where | 0 21< 1 and
Yj = Z X*0f *,j =1, 2 ,... n, y0 = 0

y'=i
L„ = - 1/2 In log (2 n o 2)+ log ^1 + j + o'2
j~1
where = X„ = X, + (1 - 0,) Z X*,;' = 2 ,...,
Exercise 9.14 (Explosive AR(1) process) Consider the Stochastic process generated
by Xt - aXt_x + et where | a | > 1 and et is HW(0, o2). Show that this may be written as
a B(oTlB~l - 1) Xt = et and hence that Xt = orlB~l(oTlB' 1 - l)-1£f where B is the backshift
operator and Brx is the forward shift operator. Deduce that the original process can be
modelled in terms of future £,’s rather than past ones.
Exercise 9.15 Suppose e2, ... are independent r.vs. all having the same mean p and
variance cr2. Define Xx = eh
Xn = aXn_i + en for n > 2,
n
where - 1 < a < 1. Show that n~l Z Xt pl( 1 - a).
i=i
Exercise 9.16 Let X0 = 0.
_ n
Define X„ as in exercise 9.15 and X„ = n~' Z X,. Prove that
V«(X„ -/i/(l - a)) A/V(0, ct2/(1 - a ) 2), if - 1 < a < 1,
a 2 12), if « = - 1.
What happens when a = 1?
**Exercise 9.17 Let Xh X2, . . . , be i.i.d. r.v.s with mean and variance <72.
_ n
(a) Find the joint asymptotic distribution of Xn = n~x Z Xj and sample autocovariance
y(l) = n~l Z (Xf - X„)(X,+] - Xn) [Hint: Find the asymptotic distribution of aXn + bZn
i=1 n
for all real a and b, where Z„ = n~] Z X,X/+1. Then apply Cramer-Wold’s device and
_ 1=1
express y (1) = Zn ~ X„ .
(b) Now assume also ju = 0 and EX ?< 00 ■Define the sample autocorrelation of
lag 1 based on first (n + 1) observations as
n ( » 'i
p( 1) = n-' J , ( X , - X M ) - X i
i=1 /
/ K
n -'lX ? - X f
1 Z
(Note that the limiting distribution of p(l) does not depend on the assumption jj, = 0)
(i) Let Zf = (X„Xf,XiXi+ly. Show that -
V P3 oN
01
where = <T2 and Z = P3 Q
1
0 0
, o. 1° 0 * 4J
(ii) Show that -<fnp(\) (0, 1).

N
Exercise 9.18 Let {Xn} be a stationary time series with zero Means and autocovarience
y(m). Show that
Var(Zn) = y(0) f [ 1 d.F(X)

J n ^ n sin (A/ 2 )J
where F(A) is the spectral d.f. Deduce that Xn is mean square consistent (m.s) iff F(0)
i "-1
- F(0-) = 0 and show that y(0) {F(0) - F(0 -)} = Jim —^ y(k).
Queueing Theory
10.1 Introduction
Congestion is a natural phenomenon in real systems. A service facility gets
congested if there are more people than the server can possibly handle. A footpath
(side-walks) gets congested when there are too many pedestrians using the footpath
at the same time. Queueing for milk, kerosin oil etc are examples of queueing
system. In all these situations the uncertainties related to the system characteristics,
such as arrival of customers, time needed for service, the processes occurring in
the analysis of these systems are better represented by Stochastic Processes. “A
system consisting of a servicing facility, a process of arrival of customers who
wish to be served by the facility and the process of service, is called a queueing
system”. The four common characteristics of such systems are:
1. Input process Some factors needed for the complete specification of an

input process are the source of arrivals, the type of arrivals and the interarrival
times. The best can be done in this situation is to represent the input process in
terms of r.v.s Zj, Z2, . . . , Z„,. . . Let the successive customers arrive at time tQ,
t\, t2, . . . The interarrival times are then Z, = tt - tt_\ with d.f.F and {Z,} is a
renewal process.
2. Service mechanism The uncertainties involved in the Service Mechanism
are the number of services, the number of customers getting served at any time,
and the duration of service. Random variable representations of these characteristics
seem to be essential.
3. Queue discipline All other factors regarding the rules of conduct of the
queue can be pooled under this heading. In this context, rules as first come first
served (FCFS) or first come first out (FIFO), Last come first served and Random
selection for service are important. In many situation “Priority” disciplines need
to be introduced so as to make the system more realistic.
3. Number of queues When there is only one server in the system, specifications
of the above three factors give a complete description. However, in problems of
net-work flows and job-shop scheduling, one has to deal with more than one
server in series and/or in parallel.
This book will mainly deal with (i) negative exponential service times
distribution, (ii) basic queue discipline will always be assumed to be first come
first served and (iii) one server.
It is convenient to simplify the description of a queueing system by a notational
representation (designed by Kendall in 1951). This can be written as :
Input distribution/service time distribution/number of servers in that order.
Some standard notations used are G (or GI if input) for an arbitrary distribution.
M for Poisson (if arrivals) or negative exponential distributions (also called
Markovian) (for interarrival or service times); D for constant length of time (for
interarrival or service times), and Ek for the Erlang distribution (Gamma with
one of the parameter, k an integer). For instance, suppose the arrivals are Poisson,
service times are negative exponential, and there are 5-servers, then the system
is denoted by M/MIs. In this notation, M /Mbo, MIDI 1, are M/G/l, M/G/l/R etc.
same as M/G/l with fourth descriptor R denoting that the system has a limited
holding capacity R.
Queueing Process
The following r.v.s which arise in the study of Queueing theory provide important
measures of performances and effectiveness of a stochastic Queueing system.
I. The number of customers Q(t) at time t waiting in the queue including
the one being served, if any, is also called Queue length.
II. Busy period 7(7,) (initiated by i customers) which means the duration of
the interval from the moment the service commences with arrival of an unit at an
empty counter to the moment the server becomes free for the first time.
III. Waiting time W(t) in the Queue i.e. the duration of time a unit has to
spend in the Queue. Also the waiting time Wn of the nth arrival.
IV. The vartual waiting time W'(t) i.e. the interval of time a unit would have
to wait in the Queue, were it to arrive at the instant t. Q(t), the number in the
system at time t and its probability distribution, given by
nn{t) = P(Qx(t) = n/Q( 0) = i)
are both time dependent. For a complete description of the queueing process we
need to consider transient or time-dependent solutions. It is often difficult to
obtain such solutions. In many practical situations one need to know the behaviour
in steady state i.e. when the system reaches an equilibrium state after in operation
for a pretty long time. It is easier to find n n = lim Pn(t) provided the limit
exist. It is necessary to know the condition for the existence of the limit in the
first place when the limit exists, it is said that the system has reached equilibrium
or steady state and the problem then boils down to finding the steady state solutions.
There are certain useful statements and relationship in Queueing theory which
hold under fairly general conditions. Though rigorous mathematical proofs of
such relations are somewhat complicated, intutive and heuristic proofs are simple
enough and have been known for a long time (see Morse, 1947). It has been
argued in (Krakowski- 1973) also by Stidham (1974) that conservation methods
could very well applied to supply proofs of some of these relations. Conservation
Queueing Theory 229
principles have played a fundamental role in physical and engineering sciences

as well as in economics etc. Similar principles may perhaps be applied in obtaining
relations for queueing systems in steady state. Some such relations are given
here. The most important one is L = AW, where A is the arrival rate, L the
expected number of units in the system and W the expected waiting time in the
system in steady state. A rigorous proof of the relation has been given by Little
(1961) and so the relations is known as Little's formula. This result, of great
generality, is independent of the form of interarrival and service time distributions,
and holds under some very general conditions.
Denote the expected number in the queue and the expected waiting time in
the queue in steady state by L q and Wq, respectively.
These are related by the formula
Lq = ?lWq
A relation which holds for G/G/l queue in steady state is A = (1 - or
TZo = 1 - p, where p = A/p, the traffic intensity.
Under some restrictive conditions for a system with Poisson input the Pollaczek-
Khinchine formula holds. We shall prove this formula letter, while some of the
other relations will be proved in some special cases.
10.1.1 A generalized Queueing model

Under certain assumptions on the arrival and service processes, a large class of
queueing system can be modelled as a generalized birth and death process.
Arrivals can be considered as ‘birth’ and service completion as ‘death’.
1. Arrivals Let the number of customers in the system be n at time t, during the
following infinitesimal interval (f, t+At) no customer arrives be 1-A„Ar + o(At),
and more than one customer arrives be d?(Ar)* Also assume that the customer
arrivals during (t, t + Af] are independent of the time since the last arrival.
2. Service Probability that there is one service completion in the interval
(r, t + At) be jinAt + o(At), the probability that there is no more than one service
completion is o(Ar). Assume that the service completion epoch is independent of
the time it started. Let
PnU) = P[Q(t) = n I 0) = 0],
Then pnn_,(/, t +At)=
= p nA t —
> ),
pnn(t, t + At) = [ l - A„A t+ o(Af)

= 1 - A„A/ - p„At + o(Af),
Pn,n+i(;’ * + At) = [A„A
= XnAt + o(At), and
£
j*n-\,n,n+\ Pnn-jJ (?> t ^ 0 = o(At').
For transitions occurring in nonoverlapping intervals (0, t) and (t, t + At)], by

Chapman-Kolmogorov equations
pn(t + AO = [1 - A„A/ - p„At +

+ [A„_ \At + o(At)]pn_i(t) + [î„+1A? + o(AO]pn
which on rearranging and letting At —» 0, give the difference differential equation
p'n(t) = -(A„ + P„)Pn(t) + Xn_xp n_\{t) + P n+\P„+\{t)
and po(t) =-A 0p o (0 + ViPi(t
f 1 if n = 0
with initial conditions pn (0) = \
0 if/i ^ 0
Since for irreducible (i.e. when all states communicate) and positive recurrent
(starting in any state, the mean time to return to that state is finite) Markov
process lim pn(t) exists, it is independent of the time parameter and
t—
Pn(i) —^ 0 as t —> °°.
Writing lim p n(t) = n n and setting lim p'n(t) = 0,
/—
»oo t—
>°°
® “ (^7i Pn)^n ^n-lPn-l + Pn+l^n+\

A
0 = -A 07ro + P i n x =» tt, = — n 0.
P\
_ , . AqAj . . . An_!
By induction n n = ------------------ n 0 .
P\Pi • • Pn
t 4. 2 AqA] . . . A„.!
Also X Tt • — 1 —A7Tft —
y=o ; u n+! P\Pi • • Pn
It should be noted that pn is a nonzero solution only if
eo A A
1+ X - — —— < oo i.e. limiting distribution of Q(t) as t —> <» exists
A I + 1 îyU2 • • • Pn
only if
i + £ A0A i. . . x n_\ ^
( 10. 1. 1)
PoP\ " P n
By this method we can analyze the Q(t) processes of some simple queueing
systems and solve realistic examples based on them. But there are other complex
realistic models which cannot be solved by this method.
10.2 Standard Queueing Models
10.2.1 The Queue MIMI1 (Erlang’s model)

Consider a queueing system in which arrivals occur one at a time in a Poisson
process with parameter A. These customers get served at a single counter and
their service times are independent and identically distributed r.vs. Let the common
distribution of these £.vs. be negative exponential with mean p~l. This implies
that as long as there are enough customers for service, the service completions
Queueing Theory 231
occur in a Poisson process with parameter p. For waiting times of customers, we

shall assume that the customers are served in the order of their arrival (first come
first served). We assume that the service do not stop as long as there are customers
to be served.
Hence A„ = A, pn = p for all n in the last generalized queueing model.
The difference differential equations for pn(t) take the form
Po(t) = -Apo(0 + HPi(t)

(nP0 = - U + p ) p n (t) + (t) + p p n+i (t), 0.
Let o - — = ^ ean Service time = Traffic intensity of the system (also

^ ~ JU ~ Mean interarrival time
called loadfactor or utilizationfactor).The limiting distribution of Q(t) (queue length)
exists i f l + p + p2+ ...< ° o i.e. if p < 1 (from 10.1.1).
Let n n - lim pn (,t), then p 0 = [1 + p + p 2 + .. .]"* = 1 - p and ^ = (1 - p)pn
for n > 1 (by 10.1.1).
Therefore the steady state solution (assuming the existence of a steady state
of MIM/l Queueing model) is given by a Geometric distribution. Writing
Q(°°) = G,
E(Q) = p/1 - pand V(Q) = -■

(1 - P r
Let W(t) be the virtual waiting time process and it is the sum of the service
times of those customers who are actually waiting and the remaining service
time of the customers who are already in service. Here the service time distribution
is negative exponential.
Hence W(t) = 0 if Q(t) = 0
( 10.2 . 1)
and W(t) = Z\ + 7^ + . . . + Tq^ if Q(t) > 0
where Zx is the residual service time of the customers being served and r 2, . . .
> T Q( the service time of those waiting at time t and the distribution of the
remaining service time Zx (by Markovian property) is the same as that of the
original r.v. which is negative exponential. Therefore, if there are n customers in
the system at time t, the waiting time W(t) is given by the sum of n random
variables which are i.i.d. with the negative exponential probability density (note
that interarrival times are i.i.d. exponential in a Poisson process) x > 0 and
the probability distribution of the sum is the gamma density g(x) = e~tixjJLnxn~{!
(n - 1)!, x > 0. (The total waiting time is gamma for Poisson process.)
Note W(t) is a Markov process.
Writing W(oo) = w , let F(x, t) = P[ W(t) < x | W(0) = 0]
and F(x) = P(W < x)
(we get from (10.2.1))
m o = poo(o ( 10.2 .2)
Now let Q(t) = j > 1. Transition probabilities are

Pijit) = P(Q(t) =j I 0) = 0-
The distribution of W(t) is given by
dxF(x,t) = P(x < W(t) <x + dx\ 0) = 0)
= Z PoiU)^ t 0 l ^ dxio< x< x) (,0 .2 .3 )
where Poj(t) —> 7tj = (1 - p)p7 as t —> «>.

The distribution of the waiting time thus has a discontinuity at the origin
given by (10.2.2) and is continuous in the range 0 < x < To obtain the limiting
distribution of W(t) as t —» °o taking limits inside the summation in (10.2.3)
which is justified by dominated convergence theorem, we get
F(x, t) —» F(x), where P(Q = 0) = F(0) = 1 - p and
00 (
dF(x) = jE (1 - p)p"e~^x
= A( 1 - p)e~iM-X)xdx ( 0 < x < ° ° ) (10.2.4)

is valid for p < 1, F(jc, t) —> 0 as t —> if p > 1.
These results imply, as in the case of queue length that if p > 1, the waiting
time increases indefinitely as t —»<*>, whereas if p < 1, it tends to be distributed
according to the negative exponential law in the range 0 < x < °o, the probability
of not having to wait being 1 - p. From equation (10.2.4) we get the moments
of W as
E(W) = p /p ( 1 - p) and V(W) = ; ■ ^ 2 .

p 2(l - p Y
In practical problems the distribution function of the waiting time W furnishes
useful information regarding the system
F(x) = P(W < x) = P(W= 0) + F(0 < W <x)
= (1 - p) + f A(1 - p ) e - (»-X)>dy
Jo
= (1 - p ) + ^ 1 - p ) r ( p - A r ^ - ^ d y
p - A JQ
= (1 - p) + p[l - e^ ~ X)x]
- i _ pe- ^ x
It is the probability that the waiting time of the customer in the system is no
more than x. If one wants to consider the total delay of the customer in the
system (time spent in the system of Sojurn time), then the length (Queueing
time) of service has to be added to the waiting time.
Let G(x) be the probability that the total time spent by the customer in the
system is no nore than x.
Queueing Theory 233
Then we have G(x) = P[W + V < jc], where V is the service time which is
negative exponentially distributed with density jxe~fW. Hence
G ( * ) = f F(x -v) jLi e-^ dv

Jo
-f (I - ,,
pe~^vdv e-H{\-p)Xe-H(\-p)ve-nvdv
- rJ o Jo
= 1 _ e-!*x + e-V(l~P)x Xe~Xv)d v

Jo
=i _ + e-Mn-PU[e-Ax _
= 1 _ £-/41-P)*
The expected total stay of the customer in the system is
£ (1 V + V ) = — i — - , A E ( W + V ) = 1
1*0- P Y H (l-p) 1
but EQ = So EQ = AE (W + V) .
Alternately, consider the customers Q , C2, C3, . . . , Cn getting service in the

order of their arrival. Let Vn be the service time of the customer Cn. Clearly, the
number of customers in the system at the nth departure epoch (at which time Cn
leaves the system) is the number of customers who have arrived during Wn + Vm
where Wn is the actual waiting time of C„. If Qn is the number of customer left
by the nth departing customers, A being the arrival rate, we can write
E{Qn) = XE(Wn + Vn),
when arrival process is Poisson W„ and W(t) have the same form.
Therefore LQ= XWQand E(Q) = XE(W+ V) i.e. Little’s (1961) form
where W = W(°o).
10.2.2 Transition solution

Here Q(t) is a birth and death process.
Let Pij(t) = P(Q(t) =j/Q(0) = 0 for t >0and pt] (
At t = 0, continuity and differentiability both hold, but from the right, and hence
Pio ( 0 = -Apco (0 + PPn (0
(10.2.5)
p'u (0 = - U + P) p ,j (0 + (0 + PPu+i (0
is the forward Kolmogorov differential (j > 0) difference equations of the birth
and death processes arise in Ml Mil Queue.
Let G(t,z)= ' L p u (t)zj ( \ z \ < l )

7=0
^ oo
and G(0, z) = z l (i > 0) then p'j (t) z j .
Multiplying (10.2.5) successively by 1, z, z2 , . . . and adding, we obtain
= -(A + /i) l,Qpij ( t ) z J + A , ^ p i H (t)zj

oo
+ 2 0 Po+l ( 0 z' + AtPiO(0 .
Hence z-=^ =[Az2- (A + /t)z + - ju(l - z)/?,0(t)10.2.

(
Laplace transforms of G{t, z) and (t) and are
G*(9,z)= [
(0
J e~0 lG(t, z)dt (10.2.7)
and A*(0) = J[ e~0'pjj(t)dt (Re (9) 0>) (10.2.8)

f0
respectively. Let us state a theorem from complex analysis which will be used
soon.
Roue he's theorem If /(z ) and g(z) are functions analytic inside and on a closed
contour C, and if | g(z) \ < \ f(z) | on C, the / (z) and f(z) + g(z) have the same
number of zero(s) inside C.
Integrating (2.2.7) by parts yields the relation
e~e,8 G/St = e~0lG

(t,z)]jT z)dt
=- z‘ +0G*(0, (10.2.9)
Multiplying both sides of (10.2.6) by e~01 and integrating over (0 < < °°), we
obtain
9zG*(9, z) - zM = [A
-M( 1 - z)Pio(9),
z,+1 - i w ~ z ) p h ( e )
or G*(9,z) ( 10.2 . 10)
(A + n + 9)z
The denominator on the right hand side of (10.2.10) has two zeros, namely
A + ju+ 9 —^(A + + 0 )2 - 4
1 = 1 (0 ) = ( 10.2. 11)
2A
Queueing Theory 235
A + p + 9+ ^/(A + p + 0 ) 2 - 4A p
r?= tl(9) = ( 10.2 . 12)
2A
where the square root is taken so that its real part is positive. Clearly
| £ | < | r] |, and moreover ^ + 7] = ^ + ^ + ^ = (10.2.13)
For | z | = 1 and Re (0) > 0, we get

|(A + p + 0 ) z | = |A + p + 0 | > A + p > | p + Az2|,
so that by Rouche’s theorem (A+ p + 9)z - p - Az2 has only one zero in the unit
circle; this is clearly z = £ Now since the Laplace transform G*(0, z) converges
in the region | z | = L Re (0) > 0, the zeros of the numerator and denominator on
the right hand side of (10.2.10) must coincide and therefore
^ h-i
Pw(0) = (10.2.14)
M l - T)
Substituting (10.2.14) in (10.2.10) we obtain the result
zM ( l - z ) g M / ( l - g )
G *(0,z) (10.2.15)
A(z - Z)(ri - z)
for the transform of the transition probabilities of the process Q(t). This is due
to Bailey (1954). The probabilities Pij(t) can be obtained by inverting transform
(10.2.15). Alternately p{j (t)'s can be obtained explicity by combinatorial method
(due to Champernowne (1956)).
10.2.3 The basic process X(t)

Let A(t) and D(t) be the number of arrivals and departures, respectively, during
(0, t]\ their probability distributions being given by negative exponential law.
Consider the process N(t)= X(t)=A(t)-D(t), where A(0 and D(t) are independent. Let
kj(t) = P(X(t) = j ) J = -1, 0, 1, 2, .. .
= e - (U^ ' p j n l j ( 2 ^ J / I t ) (10.2.16)
(from elementary properties of Poisson process)
oo
where /. (jc) =
J
Z ( x/ 2 )2n+j /[n\(n + y)!] is the modified Bessel function of
n=0
index j.
e - ( ) ' p -j/2 (j 2 )
= P~jkj(t).
When A = p, this gives k_j(t) = kfit) (p = 1) and X(t) then is a symmetric
stochastic process.
The P.G.F. of X(t) is given by
Z k j( t ) z j = exp{[Az - (A + p) + p/z]t] (10.2.16)

j = -°°
Letting z -> 1, P(X(t) < «>) = Z *.■(*) = 1 (10.2.17)

j-~°°
which shows that X(t) assumes only finite values. The d.f. of X(t) is given by
A' n
*0(0 = X M O = f>
J )——oo
2Z7
0>.m
Hrr>0 ?-m<j THnl (t)t+m (10.2.18)
= P{A{t) - D{t) <j).
For j > 0, | e e,kj(t)dt= T) >I(A( tj-

(10.2.19)
10.2.4 Combinatorial methods

Note X(t) increases or decreases by unity at random points of time and that the
intervals between successive points of increase and the intervals between successive
points of decrease are r.v.s distributed according to negative exponential laws
with means A-1 and iu~\ respectively, all intervals being mutually independent.
A typical realization (or sample path) of X(t).
X(t)
X(0)-----------
_____ I-------
o ----------------------------------------------------------------------
A typical realization (or sampfe path) of X(t)
The process Q(t) shares the same properties as X(t), with the important difference
that Q(t) has an impenetrable barrier at 0, such that when Q(t) reaches zero, it
remains at zero until process X(t) takes the first upward jump (of unit magnitude),
and at this point Q (t) = 1. The process Q(t) then repeats itself. It can be proved
that
Q(t) = max{ sup [X(t) - X(r)], Q(0) + X(t)} (10.2.20)
0<r<f
This representation is typical of many of the processes which will be encountered
henceforth and its regorous proof depends on storage process (due to Pyke and
Gani (I960)) and additive process (separable infinitely divisible process)
Q(t) = max [X(t) - a(t), Q(0) + X(t)] (10.2.21)
where , * (*) = 0>n/£, I Mr ) - D(r)].

Since A(0) = D(0) = 0, <7 (/) < 0 a.s.
From (2.2.20) and some combinatorial arguments, reflection principle and
the value of Kj(t) gives
Queueing Theory 237
P(Q(Oj I Q (0) = «) = 2 I P{AU) = e, D(t) = <j \ 0) =

f m
A (t)= t, D(t) = m)= - p>+'K_H _2 (t).
Hence Pu(t) = P(Q(t) =j \ 0(Q) =
= {<2(0 j| 0(0) = /} -
<
= [Kh -(0 Pi+lK-H - 20( ) - { - P>K_H _
= + p Jk_J_l_\(t) + (1 - p)pJK_H _2 ( 0 (10.2.22)
Using = p'jkj(t) we have
+ P '" ‘W ' ) + d - P)PjK-j-i-i(0. (10.2.23)
When p = 1, this becomes
P#y(0 = *h(0 + kj+i+xit), (10.2.24)
where (t) = e~1Xilj( 2 Xt).
By (10.2.22), we get
£ PiJ (o = £ (r) + £ (o + (i - p) .£ p ; z C
7=0 7 7= 1 7=0 7=0 u=—oo
= S.M O + 2* p~v~‘~'kv+ S M l - P ' U' M
= U =I—oo M O = l (by (10.2.17)) (10.2.25)
Thus in finite time there is no possibility of an unduly long queue. Now an

alternative derivation of limiting distribution Q(t) as t —» ©o can be given
\
exp[ 2 -yfXpt]
kj(t) ~ e - ^ ' p i ' 2 /;(* ) as * —> oo
2 (ut )1/2 (A p)l/4 Jinx
e-(4*-J7i)2t pj/2
as r —> oo.
2(7T/)1/2(A p )1/4
, ^ /.s _ V exp{[V X - V p ]2t} 4, ^ u/2

If P > 1. Ky (0 - ^ (0 ~ ^ 1/2 / 1 1/4 ^ P (10.2.27)
2 ( ^ 0 1/2 (Ap)
" co, , 8)
0 if p > 1
Therefore as (-> » for all 7 (10.2.29)
1 if p < 1
By (10.2.26), (10.2.29), (10.2.23) and (10.2.24), we get
0 if p > 1
lim pij(t) = (10.2.30)
t —>°o J
(1 - p ) p i if p < 1
This implies that independently of its initial length /, the queue grows unduly
long at the end of an indefinitely long period of time if p > 1, whereas if p < 1,
the Queue-length attains statistical equilibrium and tends to have the Geometric
distribution. The limiting distribution is stationary in the sense that if the initial
Queue-length 2(0) has this distribution, so does this Queue-length Q(t) for t > 0.
For we have
P[Q{t) <j] = Z P[Q

(t)<j| 0) = i] P[(2(0
l=U
= I [KH (t)- p J+l K ( t ) \

/=0 J J
= 1 - p 7+1 (after simplification).
10.2.5 The busy period; zero-avoiding transitions

Suppose that initially the system contains i (>1) customer, and let Tt be the next
subsequent epoch of time at which the server is free; Tt is called the busy period
initiated by i customers.
Then Tt = inf {t | Q(t) = 0, 2(0) = /}. (10.2.31)
Suppose that Tt has d.f. Gt{t) = P(Tt < t) (0 < t < °°).
Define the zero-avoiding transition probabilities of the Markov process
Q(t) as °Pij(t)= P(Q(t) =j; Ti > t \ Q (0) =>1).
It is the probability that there will be j customers busy throughout the interval
[0, t].
By (10.2.20), (10.2.31) can be written as Ti = inf [ t \ i + X(t) < 0} and hence
°P ij(t) = P[i + X(t) > 0 (0 < r < /); i += j] (10

= P{i + oU) > 0, ( + XU) =j].
By (10.2.20) and (10.2.32), we get

P(Q{t)< j IQ
( 0) = i)= P(X(t) - a(t) X(t
= S P ( v + a ( t ) > 0 \ v +X( t)= °X

(10.2.33)
U=H-1 1+ 1 J V7
which is a relation connecting the zero-avoiding transition probability with the
ordinary ones. In particular (putting j = 1 in (10.2.33)), we get
Pi0 (t)= £
1+ 1
From (10.2.33) and (10.2.22)

Queueing Theory 239
= PIQU) <j -11 Q(0) = i - l ] - P [ Q ( 0 1) (Bailey 1954)

The Laplace transform of °p,,(t) is given by
°p*i(9) = [ e e‘
Jo Jo
°pij(t)dt = f e e,kJ+
e e'kH ( t ) di(t)dt
t-p '
r£Ujp_ - <gf )
if J ^ i
M n - ^)
By (10.2.19) °p* (0) = (10.2.35)
- i d l if j i
Mri - €)
where r) are given by (10.2.11) and (10.2.12).
The distribution of busy period Tt is given by
dG,(t) = °pn(t) pdt (10.2.36)
= i/tk_i(t) dt (after simplification)
= gfa) dt which is the density function of Tt(i > 1, t > 0).
(i = 1 is due to Kendall (1951) and i >1 due to Good (1948)
The Laplace transform of this p.d.f. is given by
g * (0 )= f
Jo
e-d,gi{t)dt=
Jo
f (10.2.37)
£<(£•-> - n"1)
- M n-(, 10.2.35)
fl if p < 1
P(Ti < oo) = lim g *( 0 ) =
e~^° [p 1 if p > 1
Since £{0) —> 1 or p"1according as p < 1 or p > 1 (follows from 10.2.11). So

the busy period will terminate eventually if p < 1, whereas if p > 1 there is a
positive probability that it will continue indefinitely. When / = 1 we have the
busy period T initiated by a single customer, the Laplace transform of its distribution
is g* (0) = and from (10.2.37) we see that gt{t) is the /-fold convolution of
g\(t). So Tt is the sum of ii.d. r.vs. each having same distribution as Tv
10.3 The Model M /M /c

Here the input process and the service time distribution are the same as M/M/l
model but there are c servers where 1 < c < ° ° customers are served on a ‘first
come first served’ basis (FCFS). Here again the number of customers Q(t) present
in the system at time Ms a birth and death process, the transition densities are
computed as follows.
Here for each of the c channels use i.i.d exponential service time distribution
with mean rate p. If n (< c) channels, are busy, the number of services completed
in the whole system is a Poisson process with mean np and thus the time between
two successive service completions is exponential with mean 1/(np) whereas if
n (> c) channels are busy, the time between two successive service completions
is exponential with mean 1!cp.
Pnn+\(dt) P(Q(t + dt) = n+ 1 | Q(t) = n) = Adt + o(dt)
Pnn-\(dt) p ndi + o{dt)
Pnnidt) 1 -(A + pn)dt + o(dt)
Pnmidt) P(Q(t + dt) = m | Q (t) = n) = o(dt) if m * n - 1, n, n + 1,
(10.3.1)
np if 0 < n < c
where Pn (10.3.2)
cju if n > c
Thus Q(t) is a B.D. process with constant arrival (birth) rate Am= A and state
dependent service (death) rates as given above.
Substituting (10.3.1) in Chapman-Kolmogorov equation and proceeding as
the model MIM/l we find the forward Kolmogorov equations of the process to
be
Pnm (0 = ~ ( A + Mm )Pnm ( 0 + &Pn,m-\ ( 0 + Mn+lP
Using (10.3.2) and writingpn{t) = P[Q(t) = n \ Q(0) = 0], i.e. forward Kolmogorov
equation of birth and death process becomes
P o (t) = -A /?0 (O + P P \(t)
Pn (0 = - U + np)p n (t) + Apn_j (t) + (n + l)jupn+i (0(0 < n < c)
Pn ( 0 = ~(A + c p) p n ( 0 + Apn_x( 0 + c/upn+1(t)(n > c).

These equations have been solved by Saaty (1960) by an extension of Bailey’s
method of generating functions. The transient solution for c > 1 is too complicated.
We give only the steady state solution, putting the values of A„ and Pn in generalized
Queueing model as t —» «>, the limiting distribution nn of the Queue length Q
(= Q(°°)) is obtained.
A.A. . . A
7Zq q i p Y 7t0,n = 1,2,. .. , c
71n
P( 2 p ) ...........(np) n\
(10.3.3)
and AA. . . A A"

71n n ° M(2m ) ■■■CM{(CM) • • • (CM)) c'.ju'cn~rMn' c ^
n o= p nl(c\cnc)n0 (n > c). (10.3.4)

c\c
Queueing Theory 241
The condition X n n = 1 gives

n= 1
c-l n c -l
n : 1 = 1 + 2 (A/p)" /n! + I = I + — I (p /c )« .
0 n=l ^ c c\cn~c n=l rt! c! n=0 ^
For existence of steady state solutions, the series Zq (p/c)n must converge
and for this to happen the relative traffic intensity pic must be less than 1.
c -l nn c
Then = 2 ^ +
0 n~o n\ ' c!(l - pic)'
Also nn = (p/c)n~cnc for n > c.
Note
(1) nn satisfy the following recurrence relations
[1! n p n n_x, n = 1, 2,. . . , c - 1
]
[He p7tn_i ,n = c ,c + 1,. . .
(2) The probability that an arriving unit has to wait is given by
P(Q > c) =2 n„ = p'7[c!( 1 - P)]7I0 = jic/(l - p).

n=r
The Mean Queue length is given by
nn0p n
E(Q) = X n n n = X 7i0 n / n \p n + X
n=0 n *=0 U ^ #i=c+l c!cn_C
c-l oo
= p n 0 X l/n\ pn + n 0 /c \( p) c X (n + c){p!c)
n-v 1
c-l
pic (
= pn0 2 l/n !p " + n 0 lc \ p c
_ ( l - p / c ) 2 (1 -
_ „ , c p ^ o
H
1
to
1
Let 0 ' be the number of customers actually waiting in the long run. Then
Q' - Max (0, Q - c).
Therefore P(Q' =0) = P( Q < c ) = n 0 Z V j ' - P i
and P(Q' =J) = P(Q =j+ c)> l).

cJc\
c+1
E(Q’)= J +, (fi - c )n n = n2,( p / c ) ' =
c !(1 - pic) n 0
pic
2 *o
(1 - pic)
10.3.1 Distribution of W, the long run waiting time

of a customer
If a customer on entering the system finds j customers ahead of him, then he will
be served immediately if j < c, whereas j if > c, he will have to wait till the
(j - c + 1)th customer completes his service. Therefore the probability that he
does not have to wait is given by
F(0) = P(W = 0) = P(Q <c) = n 0 I p n!n\

n=0
If all the servers are busy, then the intervals of time between successive
departures are independent r.vs. having a negative exponential distribution with
mean \/c\i and the total time until the (j - c + l)th departure is the (j - c + 1)-
fold convolution of this distribution and hence
o° e-c»x (CLLx)n~C
e Cfix(cjux)
dF(x) = P(x < W < X + dx) = E 7T„ ---- 7-------rr---- cfxdx
ai=o (n - c)\
= cp 7i§e (cfi X)xdx (0 < jc < oo), A = lip.

Hence, W has a discrete probability at the origin and a continuous frequency
density in 0 < x < <*>. The expected waiting time is given by
In a single server queue looking at the limiting probabilities (/r0 = 1 - p, nn =

(1 - p)pn, n > 0) it is clear that the traffic intensity p is the probability that the
server is busy at any time (or the fraction of time the server is busy). Is this true
for the multichannel system MIMIcl
Consider the probability that a particular channel is busy at sometime in the
queue M/Mlc. It should be noted that when the number of customers in the queue
is > c all the channels are busy, and when there are k < c customers in the queue
the probability that the channel under consideration will be busy is k/c. Thus, the
probability that all the channels are busy (by (10.3.3) and (10.3.4)) will be
{ c (c - 1)! + c (c - 2 ) ! + '
.. + l / c p \ n 0
Queueing Theory 243
= plcn0 n=i f) n\ n=r ^ n~

= P*.
Again the probability that a particular channel is busy at any time (in the long
run) is the traffic intensity p.
10.4 The Concept of Imbeded Markov Chain in

Queueing Models
In M/A//1, M/MIc, Af/A//°o, Queueing systems the Queue-length Q(t) is a Markov
process because the specification of the Queue-length at time y is adequate to
predict its length at time y + t on account of the Markovian property of the
Poisson distribution. In the systems MIEk/\ and Ek/M/l, Q(t) is not Markovian,
but its properties could be obtained from those of a second (another) process
which is Markovian. When the input and service time distributions assume
general forms, the process Q(t) is no longer Markovian. The correct specification
of the Queueing system is then given by the vector process [Q{t), u(t), v(t)],
where u(t) is the time which is elapsed since the last arrival, and v(t) is the
expended service time of the customer being served at time t. Though rather
complicated, with the help of this vector process it is possible to do something
about the process Q(t) by making use of its ‘regeneration points'
A set 5 of time-instants is said to be a set of regeneration points for a stochastic
process X(t) iff, for all t > t0 we have
[Distribution X(t) | X(t0)] = [Distribution X(t) | X(y) for all y< t0] whenever t0
e 5. This implies that the development of the process during t > t0 is independent
of the history of the process during (0, t0). For a Markov process the whole
parameter space (range of t values) is a set of regeneration points. If a process
has a set of regeneration points, then it is called regenerative. Suppose that
there exists a denumerable 5-set of regeneration points {tn, n = 0, 1, 2,. ..} such
that t0 < tx < t2 < . . . and put Xn = X(t„); then Xn is a M.C. This chain is said to
be imbedded in the given process X(t).
Note
1. In a single server queueing system, the set of points at which the counter is
free is clearly an 5-set.
2. Let Sx consists (for a given x) of those instants of time at which the
customer at the counter has already been served for a period x.
3. In M/GI1 system, the instants at which the customers leave the system are
points of regeneration. In the general case if there exists an increasing sequence
{^} of regeneration points such that the transition probabilities associated with
{Qn+\ | Qn)y where Qn = Q(tn) can be calculated in a simple manner, then the
analysis of the imbedded M.C. {Qn} will yield valuable information regarding
the process Q(t).
10.4.1 The system with service in phases: The Queue model MIEkl 1
In this system customers arrive in a Poisson process with mean At, and the
service time V has the Erlangian distribution
dV(v) = Y J e' ^ kv (k/u)k v k ]dv (0 < v < °°) (10.4.1)
with E(V) = p~l and traffic intensity = p = A///.

Since the distribution (10.4.1) does not have the Markovian property, Queue-
length process Q(t) in this system is not Markovian. Note that the service here
can be interpreted as being offered in k consecutive phases, the times required
for these phases being mutually independent r.vs. with negative exponential
distribution kfie~kfivdv(0 < v < °°) and the service of a customer being completed
at the end of the kth phase. The arrival of each customer introduces k phases into
the system, and the phases are completed successively in a Poisson process at a
rate k. If Q\(t) is the number of phases in the system at time /, it is clear that Q\{t)
is a denumerable Markov process.
The transitions probabilities in an infinitesimal interval (f, t + dt) are given by
Poo(dt) = 1 - Xdt + o(dt)
Pu(dt) = 1 - (A + kp)dt + o{dt) (/ > 1)
Pi,i-\(dt) - k p d t + o(dt) (i > 1) and Pij+k(dt) = Xdt + o(dt) (i > 0),
Pij(dt) = o(dt) (i > 0, j * 0, i - 1, k).
This stochastic process is a time homogenous birth and death process, slightly
more general than Feller’s process. The forward Kolmogorov equations are
P ioi0 “ ^Pio ( 0 “*■

P'ij (0 = -(A + k ^ p u (t) + i (0 (1 )
p'ijO) = -(A + kp)pij(t) + k/upij+l(t) + Apij_k (t) (j > k)
Let G(z, 0 = 2 Pij(t)zj (\ Iz ^ 1) be the G.F. of {/?,-, (?)}

dG
Then -r—= E pU (t) z j . Multiplying the last Kolmogorov’s equation
dt j
successively by 1, z, z2, . . . and adding, we get, after some simplifications, the
differential equation
z -gj- = [kp - (A + kp)z + Xk+X - kp(l - z)Pi 0 (t).
Taking Laplace transforms with respect to time on both sides, we then get
,i+i k p ( \ - z ) p l ( 9 )
G * U ,0 ) = (| z 1 < 1, Re(0) > 0) (10.4.2)
( 6 + A + k p ) z - k j L i - A , z &+i
In the region | z | = 1, Re(0) > 0, we have
| (0+ A + kp)z \ = \ 0 + X + k p \ > X + k p > \ XzM + kp |,
so that, by Rouche’s theorem, the denominator on the right hand side of (10.4.2),
has only one zero in the unit circle. Let £ = £ ( 6 ) denote this zero. The arguments
used in case of Ml MIX Queue then give
Queueing Theory 245
/+1 z|,+l- ( l /(I - g)

Pw(9)= and G*(z
km - T) (9+ A+ k n ) z ~ k p - Az*+I
Since G*(z, 9) 1/0as z —» 1, we get Z 1-

7=0
Therefore Qi(0 remains finite in finite time.
10.4.2 Limiting distributions

Let us assume that the limiting probability distribution {7ij} of Q$t) exists,
where lim Pu(t) = /r, > 0 (y > 0) and n 0 = lim p i0(t) = lim 9p*n(G)From
l— ■ J 6—>0
(10.4.2) we have
lim G(z, t) = lim 0G*(z, 6 )
»oo 0_>O
*/*(! - z)7Tq
* ju - (A + */z)z + Az*+1
(\z\<D (10.4.3)
= p(z), say.
We invert the ztransform to find the distribution of the number of ph
the system.
M l - z)
In case k = 1, P ( z ) = -p - -+- A z L - (Af *+ ■ (since
p)z 1 + p z 2 -(1 + p)z
p = XIp) - -■■■-■—-— - = -j——■— which is the G.F. of geometric
(1 - z)(l - pz) 1 - pz 5
distribution pk = (I - p) p k, k > 0. (P(l) = 1 (10.4.3) and L’Hospital rule gives
kp n 0
P{ D = 1 n o = l - A/p = 1 - p if p < 1.
kp - kk'
Expand P(z) by partial fraction expansion and then invert each term by
inspection. Denominator of (10.4.3) is equal to (1 - z ) [kp-X{z + z 2 + . . . + z*)].
If z\, Z2* • • • Zk are the remaining zeros of the denominator assuming distinct
when (p < 1), the denominator = kp{ 1 - z)(l -!z$ . . . (1 - zJzk).
Then P(z) = 7 ------ 7— -— --------7— r. By Partial fraction expansion
(1 - z/zi) . .. (1 - z!zk )
P(z) = ( l - p ) 2 where A; = . 1 (10.4.4)

=1 (1 - */*,■)’ 1 (1 - zi/z )‘
Inverting (10.4.4) by inspection, the final solution for the distribution (the
limiting distribution of Q\(t)) of the number of phases in the system, we obtain
pj =d - p) 2 Atz ; j j = 1 , 2 , . . . , *
which is a weighted sum of geometric distribution [expanding (10.4.4) in power
series in z, convergent in the region | z | < z,-, | Z; | > 1. Z/ are the roots of the
equation z* + z*~l + . . . + z = plk and Xz\ . Z2 • • . zk = £p(-l)*+1
P(Z) = (1 - p )SA
, (z/z,- ) ' , | z | < min | 1/w, |
k
= (1 - p)/tf (1 - zm/ ) where mt = l/z„ | ra;- | < 1].
Let W be the waiting time of a customer who joins the system after an
indefinitely long period of time. If the queue is empty, W = 0; but if the system
contains j > 1 phases then W is the time to complete these j phases.
Therefore P(W = 0) = 1 - p, P(x < W < x + dx)
= I (1 - p ) Z Atz~Je~klix ( kp) j / (j - 1) \ x j~ldx

j =i
k
= ( l - p ) I AjZ~] kpe~k^ x~x'Zi)xdx, 0 < x < °°
i=1
is a continuous p.d.f. which is the weighted sum of k negative exponential terms.

This concept of phase type service is due to Erlang (1920). Also note that the
Queue-length Q(t) and the number of phases Q\(t) are related by
Q(t) = [(G,(0 + k - m i
where [ x] is the greatest integer contained in jc.
10.5 The Queueing Models M IG I 1 and GI/Af/1

We have discussed queueing processes which are either birth and death process
or non-birth and death process. They are in either case Markovian and the theory
of Markov chains and process could be applied in their studies. We now shall
consider models where the distributions of the interarrival time or the service
time do not posses the memory loss property i.e. are not exponential. Here the
process Q(t) will no longer be Markovian; however, the analysis of the process
can be based on an associated process which is Markovian. Two techniques are
generally being used for this purpose. D.G. Kendall (1951) used the concept of
regeneration point (due to plam 1943) by suitable choice of regeneration points
and extracts, from the process Q(t), Markov chains in discrete time at those
points. This is known as the technique of imbedded Markov chains. The second
important technique (due to Keilson and Kooharian 1960) known as supplementary
variable technique, involves inclusion of such variables. Let Rif) be the remaining
service time of the customer currently being served. The vector (Q(t), R(t)) is a
vector Markov process. For a complete investigation into the behaviour of Q(t)
the vector process has to be studied by some means.
10.5.1 Queues with Poisson input: Model MJGI1

Assume that the input process is Poisson with intensity A and that the service
times are i.i.d. r.v.s V^, V2, . . . Vn with an arbitrary distribution with mean l//i.
Denote the service time by V, its d.f. G(u), p.d.f., if exists, by g(v) = G'(v) and
its Laplace transform by
Q u eu ein g T h e o ry 247
G * (s)= e~ su d G ( v ) (1 0 .5 .1 )
Jo
L et tn, n = 1, 2 , 3 , . . . (t0 = 0 ) b e th e nth d e p a r tu r e e p o c h , i .e . t h e in s ta n t at
w h ic h t h e nth u n it c o m p l e t e s h is s e r v i c e a n d l e a v e s t h e s y s t e m . T h e s e p o i n t s tn
a re t h e r e g e n e r a t io n p o in t s o f t h e p r o c e s s [ Q ( t) } , Q (0 b e i n g t h e n u m b e r o f
c u s t o m e r s in th e s y s t e m . T h e s e q u e n c e o f p o in t s { t n } f o r m s a r e n e w a l p r o c e s s .
Q (tn + 0 ) , t h e n u m b e r in t h e s y s t e m im m e d i a t e l y a fte r t h e n th d e p a r t u r e h a s a
d e n u m b e r a b le s t a t e s p a c e 0 , 1, 2 , . . . W r ite Q (tn + 0 ) = <2„, n = 0 , 1, 2 , . . . a n d
th e n { Q n} is a M .C . ( w h ic h w i l l b e d e m o n s t r a t e d s h o r t ly ) .
L et X n b e th e r.v. g i v i n g t h e n u m b e r o f c u s t o m e r s th a t a r r iv e d u r in g t h e
s e r v i c e t i m e o f t h e n th c u s t o m e r w h o s e s e r v i c e t im e is d e n o t e d b y Vn.
\Qn - 1 + X n+1 i f > 1

T hen Qn+1 -
[*„+i if Qn = 0
i.e . Q n+{ = Q n - H {Q n) + X n+l, H is t h e h e a v y s i d e f u n c t io n g i v e n b y
f 1 i f jc > 0
» (* ) =
0 i f jc < 0
N o w t h e s e r v i c e t i m e s o f a ll th e c u s t o m e r s h a v e t h e s a m e d is t r ib u t io n s o th a t
X ni X f o r a ll n > 1.
T hen w e h ave P { X - r| s e r v i c e t im e o f a c u s t o m e r is i>}
-Xv U v y _ r
P(Xn = r \ V n) P ( v < V n < v + dv)
r‘ 'Jo
MOO -Xv (Au)r
and so kr = P(X - r ) - ‘ dG {v),r= 0 , 1 , 2 , . . . ( 1 0 .5 .2 )
Jo
g i v e s t h e n u m b e r o f a r r iv a ls d u r in g t h e s e r v i c e t im e o f a c u s t o m e r . T h e tr a n s it io n
p r o b a b ilit ie s p tj - P ( Q n+\ = j \ Q„ = /) are g iv e n b y
kj-i+i, i> 1 ,j - 1
Pu
0 i> 1 J ( 1 0 .5 .3 )
Poj = P ij = 0
E q u a t io n ( 1 0 . 5 . 3 ) s h o w s th a t [ Q n, n 0> } i s a M .C . w it h
*0 *2 *3
*0 *2 *3
0 *0 *1 *2
0 0 *0 *1
248 In tro du ctio n to S to ch a stic P ro ce ss
A s e v e r y s t a t e c a n b e r e a c h e d fr o m e v e r y o t h e r o t h e r s t a t e , t h e M .C . { Q n} is
ir r e d u c ib le a n d a l s o s i n c e p ti > 0 , t h e c h a in is a p e r io d ic . It c a n b e s h o w n th a t i f
A
th e tr a ffic in t e n s it y p = ~ < 1, th e M .C . is r e c u r r e n t, N o n n u ll a n d h e n c e E r g o d ic .
B y E r g o d ic t h e o r e m o f M .C . t h e lim i t i n g p r o b a b i l it i e s n ,• = lim
j n — >oo
p l\J , y = 0 , 1,
2 , 3 , . . . e x i s t a n d a re in d e p e n d e n t o f t h e in it ia l s t a t e i. T h e p r o b a b ilit ie s
n - (7T0 , 7i\ . . .), 2 7i j = 1, a re g i v e n a s t h e u n iq u e s o l u t i o n o f t h e e q u a t io n s

j
n - 7i P ( 1 0 .5 )
L et # ( .? ) = 2 kfSj and 7T(s) = 2 7t j SJ .

7=0 7 7=0 J
(Aw)'
T hen £ (* ) = 2
7=0
s 7'
fJ o ■</G(w)j (1 0 .5 .5 )
r
Jo
)vd G ( u ) = G * ( A - A j ) ( | 5 | < 1) (1 0 .5 .6 )
H ence E (X n) = K '( 1) = - A G * ( 0 ) = A /// = p .

M u l t ip l y t h e (/: + l ) t h e q u a t io n b y sk in t h e i n f i n i t e s y s t e m o f e q u a t io n s in
( 1 0 .5 .4 ), k = 0 , 1, 2 , . . . a n d a d d in g o v e r k w e o b t a in , o n s i m p l i f i c a t i o n ,
, {l-K'(\)}(\-s)K(s)
K(s) —s
= ^ — — - ( 0 < P = ^ ,(1)< (1 0 .5 .7 )
_ (1 - p ) ( l - s ) G * ( A - As)
G * (A - A ^) - 5
w h ic h is t h e P o l l a c z e k ( 1 9 3 0 ) - K h i n t c h i n e ( 1 9 3 2 ) ( P .K .) f o r m u la .
Note
( 1 ) In t h e M/G/ l Q u e u e b y e x t r a c t in g a p r o c e s s a t d e p a r tu r e p o in t s tn w e get
an a s s o c i a t e d p r o c e s s Q n = Q (tn + 0 ) w h ic h is a M .C . H e n c e t h e t r a n s it io n
p r o b a b ilit ie s p i; o f th e M .C . g i v e d e p a r tu r e p o in t p r o b a b ilit ie s o f t h e Q u e u e -
le n g th . W e h a v e j u s t s e e n that w h e n p < 1, th e lim it in g p r o b a b ilit ie s nn - lim p \ m)

m—
>°° 7
( w h ic h a r e a l s o s t a t io n a r y o r s t e a d y s t a t e p r o b a b i l it i e s ) e x i s t a n d c a n b e o b t a in e d
b y a p p l y i n g e r g o d i c t h e o r e m o f M .C . A l s o in t h is c a s e p in = lim p e x is t and
o n e c a n s h o w th a t 7in = p n f o r a ll n i .e . t h e l i m i t i n g d e p a r tu r e p o in t p r o b a b ilit ie s
a re e q u a l t o th e c o r r e s p o n d in g s t e a d y s t a t e p r o b a b ilit ie s o f t h e q u e u e - l e n g t h
(sy s te m s iz e ).
( 2 ) A n o t h e r in t e r e s t in g p r o c e s s c a n b e e x t r a c t e d . T h e p r o c e s s Y(t) = Q n, tn <
t < tn+ j g i v e s t h e n u m b e r o f u n its le f t b e h in d in t h e s y s t e m b y t h e m o s t r e c e n t
d e p a r tu r e . T h e n { Q n, t n ) is a M a r k o v r e n e w a l p r o c e s s a n d { Y(t), t> 0 } is a s e m i -
M ark ov p rocess. Q n b e in g its i m b e d d e d M .C ., a n d t h e w a it in g t i m e in a s ta te
b e i n g d is t r ib u te d a s t h e s e r v i c e t im e V.
Queueing Theory 249
10.5.3 Examples
Example 1 The Model M/EjJl: Here the service time Vis Erlang (&) with p.d.f.
(fik)k v k le
dF(V) = g(v) = , 0 < v < oo
( * - 1)!
juk
Its Laplace Transform is G*(s) = y
From (10.5.6) and (10.5.7) we get
-k
fik p(l - s)
K (s) = G*(A - As) = = 1+
A - As + ilk
(1 - P ) ( l ~ s )
where p = A/p and n(s) =
1 - s ( 1 + p (l - s)/k )k *
Expanding in powers of s and comparing coefficients of sn on both sides, we
get n,j, and equivalently pn.
We can find the moments of pn from n(s). This model could also be considered
as non-birth and death Markovian, by considering that the service comprises of
k exponential phases each with mean 1/(kp) and the Chapman-Kolmogorov
equations can be written and solution obtained there from.
Special case: Ml MU Queue: putting k = 1, we get (for MlMl I Queue),
(1 - P ) ( l ~ s ) 1-p
7t(s) =
(1 - s - sp( 1 - s)) 1 - sp'
Example 2 The Model Ml Dll: As k —> °o the Erlang Ek distribution tends to

1/p i.e. Ek —> D, as k —> ©o.
Therefore G*(j), the Laplace transform of Ek —>e~s/fl, so that
K(s) = G*(A - As) ^ and =^
Expanding right hand side in powers of 5, the coefficient of sn gives

35= 1 - p , nx = (1 - p ) ( e p - 1), . . . (10.5.8)
n- 1
{kp)n~k (kp )n-k~l
n n = (1 - P ) n > 2.
k-\ (n - k)\ + (n - k - 1)! J
10.5.4 Pollaczek and Khintchine results waiting time:

Pollaczek-Khintchine formula
Assume that the steady state exists (p < 1).
Let W and Wq denote the waiting times in the system and in the Queue
respecively and W(t) and Wq{t) denote their d.f.s Then
pn = nn - P {departing customer leaves n in the system}
- Jof P {departing customer leaves n in the system\ t < W < t + dt}

P ( t < W < t + dt)
- —t^ — d W (t),n > 0, (10.5.9)

f
Jo
Thus {e-*'dW(t)}\ I ao" .

- S * ■ * ■ -£
-r
where W*(0) is the L.T. of W(t). Now
e - A,(1 - s ) d W ( t ) = (10.5.10)
P(s) = I ’ pns n= S n ns n = n(s)

n-0 n—
0
(1 - p ) ( 1 - 5 ) G * ( A - A j )
by (10.5.6), (10.5.7) and (10.5.10).
G*(A - As)- s
(1 - p)(cc/A)G
Hence IV* = , where a = A- As
G * (a ) - (1 -
_ «(1 - p ) G * ( a ) ._ *(1 - p ) G * ( s )
(10.5.11)
a-A{\- G * ( a ) ) ’ ° r (5) j - A(1 - G*(j))
This connects the Laplace transform of the distribution of the service time V
and waiting time W in the system.
Now W = WQ + V, so that W *(s) = Wq ( s )G * ( s )
( W q and V are independent), where Wq is the Laplace transform of W q . Hence

from (10.5.11) we get
W{o = ------— — —------

Q s - A(1 - G * ( a ) )
The relations (10.5.11) and (10.5.12) are also known as Pollaczek-Khintchine

(P.K.) formula. To distinguish it from the other P.K. formula (10.5.7) it is called
PK. transform formula.
10.5.5 Expected waiting time and expected number in the system

Using (10.5.7) and (10.5.12), moments of Q ( t ) and W q can be obtained. We have
G*<*)(0) = J ^ . G* ( 5 ) U = (-1)*
ds
From (10.5.12)
^ - A + AG*(i) - s{ 1 + AG*(I) (j)}

U£(,,(j)/(1 - p ) (10.5.13)
[ s - A - AG*(s)]2
Since Wg1*(0) is 0/0 form as s — >, 0by L’Hospital’s r
hand side of (10.5.13) becomes
Queueing Theory 251
(1 - AG*(1)Q)) - (1 + AG*(I)(5)) - 5(AG*(2)(5)) _ -AG*(2>(0)

2(5 - A + AG*(s))(l + AG*(1)(5» j=0“ 2(1 + AG*(1)(0))2
Using Little’s formula or otherwise
E{Wq ) =IV*(1)(0) =TH~

—T = ~ ^ E(V—-
2 2(1 - p ) 2(1 - A //I)2
E ( W ) = E (W q+ V) = 2 ( 1 - ' p ) E ( y 2 ) + ^
£<2(0 = AE(W) = p + 2( iA! P ) £ ( v2)
and £(fi*(r)) = A(£WG) = A2/[2(l - p)]E(V2).

Kleinrock (1976) called this P.K. Mean value formula.
By differentiating Wq (a) k times with respect to s and putting s = 0, we can
find the &th moment of Wq. It can be proved that
V a r ( ) = — A_ — j [4(1 - p ) E ( V 3) + 3A{£(1/2)}2 ]
and C2 > 1, where C2 = Var (WQ)/E2(WQ).
10.6 System Where Customers Arrive in Stages:

E k/ M / 1 Model
Here we have Erlangian arrivals, and the service time distribution p e '^ d v
(0 < v < <x>) with mean inter-arrival time A-1 and traffic intensity p = /Jji.
(Xk}k
[The Erlangian input dA(u) = —— — e~Xku uk~xdu, 1 < k <°° is a &-fold
( k —1).
convolution of the negative exponential distribution and can be considered as
representing an arrival process which occurs in k successive stages such that the
time required for each stage has the negative exponential distribution independently
of the others, and the arrival takes place only at the completion of these k-stages,
when this happens, the process starts with the first stage of the next arrival.]
Let Q\(t) be the number of stage completed at time /, so that if Q\(t) =jk + v
(0 < k < k - 1), then the Queue length is Q(t) = y, thus Q(t) = 6 i(0
k
The successive stages enter the system at a Poisson rate kX, whereas the
departure of each customer removes k stages from it. Thus Q\(t) is a denumerable
Markov process whose transitions during (/, t + dt] are given by
P u(dt) = 1 - kX dt + o(dt) (0 k),
Pii+i(dt) = kXdt + o(dt),
Pu-k(dt) = pdt + o(dt) (i > k),

Pij (dt) = o(dt) (i > 0, j * /, * i * i - k).
The forward Kolmogorov equations are
p'w0( = ~ k A p 0i( t) + ppik (0,
Pijit) - - k X p IJ(t) + kApij_\(t)+
( 10.6 . 1)
Pij (0 =-kAp,j(t) + kXpiH (t)+ ppi.

P ijit) = - (kA + p) pjj(t) ++ p p iJ+k(t)
From these equations, an expression for the transform G*(Z, G) can then be
derived exactly as M \E k\\ Model. However, Jackson and Vickols (1956) obtain
the limiting probability distribution u} of Q\(t) by solving these equations after
setting p'ij(t) = 0. Thus u- s satisfy the equations.
kAu0 -
, , (10.6.3)
kAuj = kAUj_{ + puj+k (1 < j < k - 1)
(kA -l- p)Uj = kA uj+i + puj+k (j > k)

To solve (10.6.3), multiplying (10.6.3) successively by 1, z, z2, . . . and
summing, we get
k- 1
a t / ( z ) + /i U(z) - ? Uj Zj = ^ 2 uH z J
k-\
= kAzU(z) + U(z) - I uj z j
which gives
(1 - z k ) ' L u Jz }
U(z) = (10.6.4)
k p z M - (1 + k p ) z k + 1
= Z ujZJ (| z \< 1), the G.F. of {uj }.

Now for | z | = 1 + d, d > 0 and small, we have
k+\
1 + kpz ^ (1 + 8)-* +k p( l + 8) (1 + 3)-* +(1 - k p d )
(1 +*p)z* 1+ kp + 1+
[1 - kd + U2k(k +l)d 2 + . . . ] - (1 - k p
” 1+ 1+
= i , ‘ ( e . - w * - < „ v p < i l.
1 + kp
Queueing Theory 253
By Rouche’s theorem it follows that the equation k pz M - (1 + kp)zk + 1 = 0 has

exactly k roots within the circle | z \ = 1 + d ; the remaining root Z0 (say) is such
that | zo | > 1. Now, Zq1 is the only root within the unit circle of the equation
zk+l- ( I + kp)z + kp = 0 (10.6.5)
(z k+l + kp)
or z = AT(z), where K(z) = Here K(z) is a p.g.f. with the mean
1 + kp
K \ 1) = -- > 1 since p < 1.
7 1 + kp ^
From the theory of Branching process it is known that 0 < z0 * = if0 (say) < 1
iff K \ 1) > 1. Since if0 satisfies equations (10.6.5), the relation kp( 1 - if0) =
f o(l - & o ) holds (10.6.6)
Since the p.g.f. U(z) is analytic in the region | z | < 1, it is clear that k - 1 zeros
k- 1
of the polynomial 2^ Ujzj and the zeros of the denominator on the right-hand
side of (10.6.4) must coincide in this region, we must thus have
k- 1
A[ k pz M - (1 + k p ) z k + 1] = l)(z - z 0 ) S (10.6.7)
where A is a constant. Substituting (10.6.7) in (10.6.4) and simplyfying

k- 1
XzJ
U(z) = A—----- . The condition U( 1) = 1 gives A = (z0 - 1)/k, so that finally
Zq - z
( k-1
1 - if p
UM = } I z^ ( |z |< D
v'=° 1 - Z&0
( k- 1 N\
1*' O-Sfo) Z A z% b )J
7=0
k-1 j oo k- 1
1 - if n
I z] Z + Z Z if r
0 u=0 0 j=k D=00
*-l
= 1/Jk Z z '( l - if£ +1) + 1/Jfc Z z ' if£ ’*+1(l - i f J)
7—0 7-^
k
= l/Jfc Z z-'d - if£ +1) + 1/;t ? zy if£ '* +1(l - i f o) (by 10.6.6)
'/=o ' ' u ' ./=*
Hence, the limiting distribution of Qj(r) is given by
i/*(i - i f ^ 1) (o<y<*-i)
p ( l - i f 0) i f ^ 0 ’z k ) .
Using the relation Qx{t) - [Q\(t)/k], we get the limiting distribution of the
Queue-length
(by (10.6.6)),
jk+k-\
Vj = £ uv= p ( l o(H) (j z 1).
Limiting distribution of waiting time is given by

dF (0) = V0 = 1 - P
dF( x) = S vj e -^xiij x j-ld x /( j - 1)!
= A(1 - i f £)exp [~px(l - i f *)]dx (0 < jc < <*>).
10.7 Machine Interference Problem
10.7.1 The model with finite input source

In this world of machine and automation, employing the right number of repairmen
to look after a certain number of machines is a problem of some significance.
The number of repairmen should be decided so as to minimize the loss due to
idle machines and repairmen. We shall consider only a simplified model of such
a situation.
Suppose that there are M < °° machines, each of which calls for service after
being in use for an interval of time that has a negative exponential distribution
with mean A"1. When there are m machines working at time t, the probability that
any one of these machines would call for service in the infinitesimal interval
(/, t + At] is mXAt + o(At). Let the service times of these machines be negative
exponential r.v.s with mean f f x. Suppose there are S repairmen (called operatives).
We shall call p = AJp, the servicing factor. First, we shall derive some behaviourial
results of this process which can be used in decision problems.
Let N(t) be the number of machines not working at time t. Define
Pn(t) = P i m = n I 0) =
The stochastic process N(t) can now be modelled as a birth and death process
with parameters:
The forward Kolmogorov’s D.E.s of the process take the form

Po(0 = - MAp0 (t) + HPI (t)
PnU) = - [ ( M - n)A + np]pn(t)l) A /v ,( f )
+ (n +l)ppn+l(t), 1<
Pn(t) = - [(M - n)A + sp]pn(t) + (M - n + 1)Ap„_, ( )
+ s p p n+l(t), s < n < M
Queueing Theory 255
Pm (t) = -sfipM (0 + AP m - \ (0* As t —» « \ MXtZq = p7Ti,

[(M - rt)A + = (Af - rt + 1)A + (rt + 1) ^7Tn_b 1 < ft < J,
[(M - n)A + n/i]7Tn = (M - n + 1)A 7^.! + sp 7Tn_b s < n < M
spnM - nM_], where n n - lim p n (t ), n > 0.

/ —>oo
Solving recursively, we get

' M^
p n^ 0 if 0 < « < 5
7T„ = i
All
j ^ - p nUQ \ i s < n < M
n i SIS
A/
and can be obtained from Z 7in = 1. We get
/ 2=0
i-i
s M
\ M '
J-l
M!
7T0 = pn + Z
n ) ! s i s n-j Pn
II
"=-v (A/ -
o
3
v»J
When 5 = 1, we get
Ml
7r„ =
(M -^y.p n °
Ml
and Uq - 1+ ■p 2 + . . . + M lp M
(M- 1)\P + (M - 2)1'
The probability that the number of machines in working order is r (< Af) is
given by nM_r.
Two measures of effectiveness for the system can be defined as follows:
Machine availability = 1 - E (N)/M
nn M
Operative utilization = Z — - . + Z n n.
n- 0 S /i=a+1
Clearly, the machine availability represents the fraction of total running time
on all machines and the operative utilization represents the fraction of time any
operative would be working. When s = 1, the operative utilization has the simple
form 1 - 7T0.
Other measures of effectiveness of the system are coefficient of loss for
machines, defined as
Average number of machines actually waiting
Total number of machines
and
^ . -, Average number of idle operatives
Coefficient of loss for operatives = ----------- :-^ --------
Total number of operatives
10.8 Bulk Queues-Elementary Results (Non-Birth and Death

Queueing Processes)
Bulk Queues: This section some non-birth and death processes which are
Markovian. Namely discusses we shall consider Group Arrival System
1 and processes with Bulk Service Rules. Consider MÎMIX queue with Poisson
input with parameter A. In this model the number of units arriving at an instant
is a r.v. X with p.f. ck = P(X = k), k= 1 , 2 , . . . This is same as saying that in an
infinitesimal interval (r, t + h) the probability that a group of k arrives is Xckh +
o{h). Service takes place singly and independently with exponential distribution
having mean Up.
It is still Markovian but is a non-birth and death process, as transition may
occur to states not necessarily neighbouring states. We denote this model by
M(x)/M/l. We obtain
P o(t) = - A p 0(t) + pp^ (t)
n
Pn )0 = - ( A + p)p„(t) + + 2
k- 1
Then the P.G.F. of X is C(s) = 2 with mean arrival rate A£X = AC'(1),
k=1
XEX
and the traffic intensity p = ■.......= A C '(l)/p. Assuming steady state solution
exists, we get
0 = - Ap0 + Xpx (10.8.1)
0 = -(A + p ) p n + PPn+\ + k Ck Pn-k ,n > I (10.8.2)
Let P(s) be the G.F. of {pn } = 2 p ns n .

Multiplying (10.8.2) by sn for n = 1, 2, 3 . . . and adding to (10.8.1) we get
0 = kP(s) - ju(P(s) - p ()) + (p/s)(P(s) - p0 ) + A 2 2 ckPn_ks n

n = 1 Ac=l
= - A ( s ) - p ( P ( s ) - p 0 ) + ( p / s ) ( P( s) ~p0) + A Z c ks'c( z p n_ks 1-*'

k=l ^k
=- AP(s)- p(P(s) - p0) + (pis) (P(s) - Po) + Ac(s)P(s).
Therefore P d - s)p0
P(s) =
- s) - As(l - C(s))'
To get p0 we use P(l) = 1. By L’Hospital’s rule
-PPo
1 = t!£!i p (si = -* j + ac'(1) or p 0 = 1 - A
M(l-s)(l-p)
Therefore P(s) =
p d - s ) - A s ( i -C (s)Y
Particular case: M/MIl Since cx = 1, ck = 0, k > 1 and P(s) = (1 - p)/(l - ps).
Queueing Theory 257
10.8.1 Bulk service rules

There are a number of policies/rules according to which batches for bulk service
may be formed. The bulk service frequently used in the literature are as follows:
1. Bailey (1954) considers that units are served in batches of not more than b
(say). If after the completion of a service, the server finds more than b units
waiting he takes a batch of b for service while other wait. If he finds r units
(0 < r < b), he takes all the r units in a batch for service. Jaiswal (1960) and Neuts
(1967) consider the same rule with the restriction that r * 0 (1 < b) i.e. the
service facility stops until a unit arrives. This rule is called usual bulk service
rule while Bailey’s rule is called modified type rule.
2. The rule with a fixed batch size k has been considered by Fabens (1961)
and Takacs (1962). In this the server waits until there are k units.
3. Neuts (1967) considers that if, after completion of a service, the server
finds less than a units, he waits for service, if he finds a or more but at most b,
he takes in the batch for service b units, while others wait. The batch takes a
minimum of a units and maximum of b units. Since rules (1) and (2) can be
considered as special cases of this rule, this is called general bulk service rule.
4. U.N. Bhat (1963) considers that the number taken in a batch is a r.v.X. The
corresponding Markov model is denoted by /\.

Exercise 10.1 Find the expected number L in the system MlMl Ilk and expected Queue
length Lq.
Exercise 10.2 Find the steady-state probabilities ^ for the finite-waiting-space multichannel
queueing system M/M/c/k, where the number of channels is c and the maximum queue
allowed in the system is k(k > c). Also deduce expressions for L, Lq, E(W), Wq by using
Little’s formula. Find the expected number of busy channels.
Exercise 10.3 Use Bayes theorem to find arrival point probabilities qn in the system
MlMlelk. Find also the waiting time distribution function in terms of q„ s Show that in
MIM/c/oo, qn = Kn for n > 1.
Exercise 10.4 (Erlang’s loss system) Find the steady-state probability nnfor the queueing
system M/M/c/c and also find the probability that all the channels are busy in this system.
Describe a practical situation represented by the above model.
Exercise 10.5 Find the steady-state probability nn, given that
(N - n)A, 0 <n<N
0 n> n
np, 1< n <c
cp n <c
Describe a practical situation represented by the above model.
Exercise 10.6 A machine shop company has N machines. These machines break down
periodically and the company has repairmen to service the machines when they break
down. When a machine is fixed, the time until the next breakdown is exponentially
distributed with mean y. The shop always has enough of a work backlog to ensures that
all machines in operating condition will be working. The repair time for each repairman
is exponentially distributed with a mean —. Find the average number of machines operational
at any given time, the expected “downtime” of a machine requires repair, and the expected
percent idle time of each repairman.
Exercise 10.7 Show that the two expressions for the effective mean arrival rate, in the
M/M\/k model are equal, that is, A' (1 - nn) = p(L - Lq).
Exercise 10.8 In the M/G/\ queueing system, let Qn be the number of customers left
behind in the system by the nth departing customer. Using the fact that in the steady-state
the expectation E(Qn+l) = E(Qn) and E(Q*+l) = E{Ql), deduce that the steady-state
expectation E(Q) is given by
p 2(l + C2 )
E(Q) = p +
2(1+p) ’
where Cv is the coefficient of variation of service times, and p is the traffic intensity.
Exercise 10.9 (Takacs integral equation) In Af/G/1 Queueing system, show that the busy
period distribution function G{x) is given by
GO) = f £ {(kv)ne-Xuln')Gw (
Jo n=0
where V(v) is the service time distribution and G('"’ denotes the /t-fold convolution of
G(jc) with itself. Hence deduce the functional equation
G*(s) = V*(s + A - AG*(s))
where G* (s) and V* (5 ) are the Laplace-Stietijes transforms of G(jc) and V(v), respectively.
Also verify the expression for G*(s) obtained in the text for the MIM/\ system.
Exercise 10.10 Show that, for a MIG/1 queue, the sequence of times at which the server
passes from being busy to being free constitutes a renewal process.
APPENDIX I
Sample Function of Brownian
Motion
Let {X(0, t e T) be a real valued stochastic process on a complete probability

space {Q, and J?is the Borel a -field of the real line. From the definition
of a stochastic process, it follows that the mapping X(r,.): Q —>R is ^-measurable
for every t e T. In other words, {co: X(t, co) e B] e £?for each Borel set B c R.
However,
{co: X(t, co) e B, t e T) = n {tw: X(t, co) e B]

te T
need not be in J?in general, unless T is countable. This then implies that functions
supX(r) and inf X (0 may not be ^-measurable because they involve uncountable
te T te T
intersections and unions like {supX(f) < x} = n {X(r) < jc} .

te T te T
Therefore, a large number of important functionals of a continuous parameter

stochastic process may not be random variables. A remedy for this situation is
the separability concept introduced by J.L. Doob.
Definition LI The Process {X(r), t e T) is said to be separable iff there is a
countable dense subset T0 of 7\ called the separating set and a set A of probability
zero called the negligible set, such that if co £ A and t e T , there is a sequence
tn e To, tn —» t, with X(tn co) X(r, co). Following results are well-known and
easy to prove.
1. If a stochastic process {X(r), t e T) is separable and continuous in probability
(i.e. X(t) X(r0) as t —>t0), then any countable dense set T0 c T can be used
as a separating set.
2. Let A be the negligible set and T0 be the separating set of a separable
process {X(0, t e T}. If co € A, t0 is a cluster point of T, and lim X(t, co)
te T §
exists, then lim X(t, co) exists and the two limits are equal.
t-^to .te T
3. If {X(0, t e T] be stochastic process which is continuous in probability is

separable, then every countable set dense in T is a separating set.
A basic property of separable processes is that many sets whose definitions
involve uncountably many values of the parameter t become measurable. So
separability allows us to handle events and random variables that involves
uncountably many values of the parameter t of a stochastic process. However, if
we wish to integrate the sample functions, a new concept is needed. Suppose that
for {X(/), t e /}, / an interval of R we ask the question whether almost every
sample function is Lebesgue integrable on /. Using Fubini’s Theorem
I I | X(f, (0 ) | dtdP = f I \ X( t yco)\dPdt = | E[ | X(t) \]dt and that

Jq J/ Ji Jq J[
implies J | X(t, co) | dt < <» for almost every to provided J E[ \ X(t) | ]dt <
and X(t, cu) should be measurable in both variables t and co.
Definition 1.2 A stochastic process (X(r), t > 0} is said to adapted to the family
of a fields JF(t), t > 0 if for s < t s) c <5F(t) c & and X(t) is ^ ( t ) - measura
ble for each t.
The stochastic process is said to be progressively measurable iff for all
/ > 0 the map (s, co) —>X(s, co) of [0, t] xQ —> R is measurable relative to
& [0, tJ x ^ \ t ) and & ( R ) .
motion {W(t), t > 0} has the property E(W(t) | W(r)y 0 < r < s)
A J B r o w n i a n
= W(i) a.s. for any t > s. A process having this property is called a continuous
parameter martingale. For discrete parameter martingale the reader may refer
An introduction to measure theory and probability, A. Basu Prentice Hall, India.
For Brownian motion we define a continuous parameter martingale a little more
generally as follows.
Definition 1.3 Let (X(r), t e T] be a stochastic process, and let {JF{t), t e T]
be an increasing family of a-algebras such that for each /, X(t) is J^t) - measurable.
[X(t), J^lt), t e T} is said to be a martingale if i > s implies E(X(t) | s)) = X(s)
a.s. and E(X(t)) < ©o. The process is said to be a submartingale (supermartingale)
if the last equality is replaced by > (respectively, <).
For any Brownion motion t > 0}, we can take 3T(t) to be the smallest
cr-algebra with respect to which {X(s), s < t] is a martingale. To prove this
merely note that for t > s,
E(X(t) | j r (s)) = E{(X(s) + X(t) - X(s)) | j r {s)]
= X(s) + E(X(t) - X(s)) = X(s) a.s.

If {X(r), t > 0} is a Brownian motion, then
E(X(t) - X{s))2 = | r - 5 |.
Therefore, by Chebychev’s inequality, we have
P( I X(t + h) - X(t) I > e) <1A1 0 as

Thus, a Brownian motion is continuous in probability. It follows that every
Brownian motion has a separable and measurable modification.
A separable Brownian motion has some important sample function properties.
The most important is the following.
Theorem 1.1 With probability one, every sample function of a separable Brownian
motion is uniformly continuous on every finite interval.
Appendix I: Sample Function of Brownian Motion 261
To prove theorem 1.1, let us first state and prove an extremely useful condition
due to Kolmogorov which guarantees almost sure sample continuity of a separable
process, (see Definition 1.1, Appendix I).
Theorem 1.2 (Kolmogorov) Let (X(r), t e T] be a separable process and let T
be a finite interval. If there exists a, /3, c > 0 such that E | X(t + h) - X(t) \a <
c hl+p then
sup | X(t) - X(s) | -> 0 a.s. as h 0
t,seS
\t-s\<2n
so that almost every sample function is uniformly continuous on T.

Proof From the Markov inequality, we get
Pi I X(, + *) - m i a e) ^ w f ., o.
£a
Therefore, P(| X(t + h)- X(t) |> h rc hl+/}~ar

Let 0 < r 0. Then
P( I X(t + h) -X(t) I chx+
Let Z v= sup ( |X((k 1)/2U) - |)

0 < * < 2U-1
Hence P(ZV > (l/2u)r) < 2 Z ' P( | X((* + l)/2°) - X(k/2U) \ > (l/2 u) r)
< c2u(l/2 u) l+5 = c2-lSu
Since X 2 - ^ < °o We get X P{ZV >(
1/2" )r ) <
u=0 & u=0
By Borel-Cantell Lemma, there exists N((0) almost surely finite such that
Zv(co) < \!2vr for all v > N(a>) and
lim X Z „ = 0 a .s . (A.I.l)
n—
>oo v=n+1
If | t - s | < 2"”, then we can find k such that 0 < 2" and 11 - k/2n \ < 2~",
\ s - k/ 2n \< 2~n.
If t e S= 2{k/", ^ = 0, 1.2" - 1; n = 0, 1, 2 ,
t must be of the form
m
k/2n ±X 2~i( tv = 0,1)
Thus,
n+m oo
|X ( 0 - X ( * / 2 " ) |< s z„<
u=«+l
X z„
u=n+l
Since, | X(r) - X(s) | < | X(t) - X(k/2n) \ + \ X(s) - X(kl2n) |, we have
sup | (t)- X(s) | < 2 X Zv

X (A.I.2)
t,seSv=n+l
From (A.I.l) and (A.I.2), we get

sup | X(t) - X(s) | —>0 a.s. as n —><»
/, S G5
\t-s \< 2 'n
Proof of Theorem 1.1

For a Brownian motion W(t)
E I W(r) - W(5) I4 = 3 I t - Is2 and E \ W{t + h) -W(t ) |4
which varifies Kolmogorov’s condition of Theorem 1.2 with c = 3, a = 4, )3 = 1.

Since Brownian motion is separable and continuous in probability, the diadic
rational set S is dense in T = [0, 1].
Hence sup | W(t) - W(j) | = sup | W(t) - W(s) | a.s.
t,s eS t,seT
\t - s \ < h \t - s \< h
Hence every sample function of a separable Brownian motion is uniformly

continuous on every finite interval T.
Although the sample functions of a separable Brownian motion are almost
surely continuous, they are highly irregular. For example, with probability one,
the sample functions of a separable Brownian motion are nowhere differentiable
and are of unbounded variation on every interval. Roughly, (W(t + h) -W(/)) is
of order 0 ( 4 h ) , which is incompatible with either differentiability or bounded
variation. This behaviour of the Brownian motion is made precise by the following
theorem.
Theorems 1.3 (Quadratic variation) Let T = [<a, b] be a closed subinterval
of (0, <»).
Let n n = [a = t^n) < f1(n) < . .. < t $ n) = b], n = 1, 2, 3 , . . . , be a sequence
of partitions of T such that
A„ = max (tin)- ►Oas n—> <».

1< k < N ( n ) K
Let, {W(t), t > 0} be a Brownian motion, then

N (n) .
E ( W(t[n) - ))2 - as -
oo
If S A„ <oo then the convergence is also almost sure.
Proof To prove quadratic mean convergence (L2), we write

N (n)
Sn = I (W (4n)) - W(t[nJx))2
N (n)
= Z [(VT(^n )) - W(t[n
_l))2 - (; - ;<">)]
Since Sn is a sum of independent r.v.s each with zero mean, we have

N (n)
ESn= 0, E S 2= 2 ^)- W
V
W
K
E
N (n)
=2 2 (t(kn) - t (kn_ \)2 < 2A n( b - a )
[Since E(W(t) - W(j))4 = 3 11 - |s2 and

£(H '(/)-lV W )2 = ( ( - 1)]
Since A„ —¥ 0 as n—» Sn
0 as n —»
By Chebychev’s inequality we find
P( | S„ I > c) < ES2 / £ 2 < 2(/> - a) An/ e 2
Since 2 A„ <
we have for £ > 0,
n=l
2 P ( |S „ |> £ ) < o o .
n
By Borel-Cantelli lemma, | Sn | > £ i.o. has probability zero and hence Sn —» 0

a.s. as n —» «>.
Note
1. The condition < « for a.s. convergence, can be replaced by the
n
condition that Tn is nested, i.e. Tn+X z> Tn for each n (see Doob, p. 395).
2. Taking t (kn>=k /2 ", * = 0 , 1 , . . . , 2", A„ = max 2" 1/ 2" -4 0
and 2 1/2" < therefore

n
2n
lim Z a.s. and in L2,
n —»«> /c = I
z
*=1
max
1<j<2n
The numerator on the right converges a.s. to t, while the denominator tends to
zero because Brownian motion is uniformly continuous over bounded intervals.
Therefore
2n
1
lim Z w{— t
1
n—>°° &=1 1 2" ,1 \ 2n )

i.e. the total variation o f Brownian motion path is infinite with probability one.
Theorem 1.4 Almost every Brownian path is nowhere differentiable.
Proof We follow Dvoretzky, Erdos, and Kakutani (1961,4th Berkely symposium)
and shall prove stronger result that W(t) has (a.s.) no point of increase or decrease.
Fix p > 0 and suppose that W (t) has derivative W'(s), | VT'(.s) | < /3, at some
point s e [0,1]; then there is an n0 such that for n> n0
\W(t) - W(j) \ < 2 P \ t - s \ \ f \ t - s \ < 2!n, (A.I.3)
Let W(.) denote functions on [0, 1].
An = {W(.); there exists s such that
| W(t) - W ( s ) \ < 2 P \ t - s \ , i f \ t - s \ < 2In]
Then An T A c {set of all sample paths of W(t) on [0, 1] having a derivative
at any point which is less than P in absolute value}.
If (A.I.3) holds, and let k be the largest integer such that kin < s, then the
following is implied
k +1 k +1 k -l
Y„ = max W\ w\ W -W\ W\ ^ \ - W \ <6f3/n
n n n
Therefore, if
Bn = {W( )\ at least one Yk < 6pin].
Then An c Bn = Z ] {W(-); Yk < 6p i n },

k=\
n-2 k+2 k +l
P(B„)< 2 max) w ~W
k=\
w\ ± |- w\
k- 1
< ^^(max) W - - W - w - - w - w\ - < fM
= n 3 -™Ll2dx
=n iy = /w)
4lm
Therefore P(A) = lim P(An) < lim P(Bn ) = 0, which implies the theorem.
n —>oo n —>oo
The law of the iterated logarithm (P. Levy)

It is one of the most precise and well-known theorems regarding oscillations of
a Brownian motion. It has gone through many refinement and generalizations
since its proof by Khintchine (1924).
Theorem 1.5 If {X(0, 0 < t < °o} is a separable Brownian motion with
var(X(f)) = /, then
X(t)
lim sup . ■■■■■:= = 1 = 1 (A.1.4)
t-*°° <y]2t log log t
X(t)
and lim sup = 1 = 1 (A.1.5)
0 <yj2t log log (1/0
(A.1.5) is called the local law.

Proof We first prove that the probabilities given in the statement of the theorem
are well defined. By separability the sample functions of {X(0, t > 0} are continuous,
since ^ 2 t log log t is continuous and bounded away from zero for t > 3, then for
any interval (a, b) c (3, °o)
sup v■/.-:= = sup . .-= a.s. for any sequence {tn} which
te(a,b) f 2 t log log t n ^ 2 t n log log tn
is dense in (a, b). So the supermums are r.v.s.
To prove (A.I.4) it is sufficient to prove that for every e > 0,
r X(t)
lim sup . ==T > 1 + e = 0 (A.I.6)
^ 2 / log log
/-> o o
and
P lim sup . ■— > 1- £ = 1 (A.1.7)

/->« <yj2t log log t
We first prove (A.I.6). It is sufficient to show that for e> 0 there exists t0 such
that P[X(t) (<1 + e)J2t log logt of r all r0] = 1
Let a > 1 and let us denote
Ak= [sup [X (0 | 0 < t < a k\> (1 + 2) ^ 2 a k

By Lemma 8.1
P(Ak ) =-^2/(3ra*) f
"(l+e/2^2ak
_______ exp
log logrt*
___________ y
< J 2 l { n a k)
* ( \ + e / 2 ) y j 2 a lc log log a k (l + e / 2 ) ^ 2 a k log log a k
x exp (~y2f(2ak)) [ dy
--------------========= exp {- (1 + e/2)2 (log log a k)}

(1 + e/2) yj7tlog log a*
_____________ 1_____________ = 1
(1 + £/2){/r(log k + log log a)}x'2 (k log a )l+e/2
Hence Z P(Ak ) < oo so by Borel cantelli Lemma

P(Ak i.o.) = 0.
Hence with probability one, for all sufficiently large k, (k > &0(co)), and for all
t e [ak~\ ak],
X(t ) < (1 + e ll) J l ^ klog logo*
< (1 + el) ^2? log log f-y/(2a* log log a* )/(2a*~' log log a*"1).
Choose a > 1 such that
1/2
log k + log log a
(1 + £/2)V^z < 1+ for all* > 2 .
log ( / : - ! ) + log log a 4
Then for t sufficiently large (depending on co)
X(t) < (1 + s ) ^ 2 t log lot / a.s.
[Note: If we had taken a < 1 but sufficiently close to 1 we could have proved in
the same manner the first half of the local law, namely
X(t)
lim sup —======= > 1 + e = 0 for every e > 0.]
/->o ^J2t log log t
To prove (A.I.6), let us take a > 1 (but this time very large).
Let Bk= [X(ak)- X(ak~') (>1 - ell) V2 a *log log a* ],
Since X(t) has independent increments, the events Bk are independent. Since
X(ak) - X(ak~')d ( 0, ak - ak~'),
N
P(Bk) =(ln(akak-'))-m __________ exp [ - y 2 l(l(a k - ))]dy

J ( l - e / 2 ) ^ 2 a k log log a*
Let 2 = y/(akak~')u2=. y

4 a k {1 + 1
ck= (1 - ell) ^/(2 log log a* )(1 - 1 and 1
then P(Bk) = ( I n ) - '12 f e~z2,2dz > ( I n ) - '12 [ * exp (~z2ll)dz

Jck Jck
> (ln)~l/2 e\ p
Let t =(1 - elV

/ I - 1/a. If a is large 1 - l/a is close to 1 and
Now 1/2 {t ^ 2 log log ak + 1}2 = r 2(log k + log log a) + 2 log log a* + 1/2.
Since for large *, &-/*< Ox for any k > 0, 6 > 0, we have for large k
r ^ 2 A o g \ o g ~ (1 - r 2) log log a k
Hence for large k,

Bx,1d \ = exp {l/2{r-y/2loglog~^"+ 1}2
< e x p { ((r2 + l ) / 2 ) { T V 2 I ^ t o ^ + l}2
(log k + log log a ))} = (log a )(r2+1)/2/:(f2+1)/2
( 1/2
lo g a)r2+1
Hence, P(Bk )> k~P, where 0 < j8= ( r 2 + l)/2 < 1. Therefore,
2n
V J
E P{Bk) = °o and by Borel’s zero-one law P(Bk i.o.) = 1. Since the distribution
k
of process (X(/), t > 0} is synmetric, we obtain from (A.1.6)
P[-X(ak *) < (1 + 5) ^ 2 a k 1 log log a k 1 for all large k\ = 1 for any S > 0.
Hence P[X(ak) > (1 - e/2) ^ 2 a k log log a k - (1 + 5) ^ 2 a k~{ log log a k~{
for infinitely many k] = 1.
But
(1 - e/2) ^ 2 a k log log a k - (1 - 5) ^ 2 a k~x log log a k~]
= ^ 2 a k log log a k {1 - e/2 - A},
1 + 5 logloga k- 1
where A = -< e/2 if a is large for all k > 2.
-sfa V log log <
Then P[X(ak) > (1 - e) ^ 2 a k log log a* i.o.] = 1
*(«*)
lim sup ■ ------------------ > 1- e = 1
or
yj2ak log log a k
Since
X( ak ) i:____
lim sup > 1- e lim sup . X(t)—-— > 1 - e
k-+°° ^ 2 a k log log a k £-»«> ^\2t log log t
(A.1.7) is proved.
Note: If one select 0 < a < 1, with a close to zero, then similarly we can prove
X(t)
lim sup . > 1 - e = 1. (A.I.8)
/->o ^]2t log log | 1/f |
Corollary 1.5.1 The function X(t) has (a.s.) arbitrarily large zeros.
Proof The r.v.s {-X(0} are again Brownian motion to which the law of iterated
logaritham (A.I.6) applies. Hence (A.I.6) holds for -X(t) as well as X(t). This
means that X(t) has both positive and negative values for arbitrarily large t, and
continuity does the rest.
Corollary 1.5.2 For each t0, the function X(t) - X(t0) has (a.s.) a sequence of
zeros converging to t0 from above. In particular t0 = 0 is not an isolated root of
m -
Proof follows from local law of iterated logarithm for Brownian motion.
The Zeros of X(t)

For any continuous function, the zero set is a closed set. Let T(co) = {t e [0,1];
X(t, co) = 0} be an infinite set a.s.
But Lebesgue measure of T(co) is a.s. zero because
mL(T(co)) = mL{t: X{t) I{o}(X(t))dt,

Jo
where mL(A) is the Lebesgue measure of A.
So that £ w , r n « ) i i = £| £ i m i X u n d , P[X(t) = 0]dt = 0

■ Jo
r
where interchange of E and mL is justified by (Fubini theorem) and joint
measurability of X(t, co), hence of /{0}Wf, co)).
Theorem 1.6 (Breiman) For almost all co, T(co) is a closed, perfect set of
Lebesgue measure zero (therefore, uncountable).
Corollary 1.6.1 For any value a, the set T {a) = {t: X(t) = a, 0 < t < 1} is, for
almost all co, either empty or a perfect closed set of Lebesgue measure zero.
Definition 1.4 A real valued function/on R is locally Holder continuous of
order a if, for every L > 0,
sup{ \ F ( t ) - f ( s ) \ / \ t - s \ a- , \ t \ , \ s \ < L , t * s } <°o (A.I.9)
Theorem A.I.7 The Brownian motion is locally Holder continuous of order a
for every a < 1/2.
Proof We shall first show that if a stochastic process (X(r)} be such that
E | X(t) - X(j) |r < c 11 - s| 1+”, 0, y> 0 (A.I.10)
then, there is a modification X of X such that
e [{suP ( i* (o -* (* ) i / i * - * m < -
S*t
for every a e [0, rj/y]. In particular, the paths of X are Holder Continuous of
order a. For each positive integer m, let Dmbe the set sm= {2~mi}, i is an integer
in the interval [0, 2m). The set D = u Dm is the set of dyadic numbers. Let
further Ambe the set of pairs (5 , t) in Dmsuch that 11- s | = 2~m\ there are 2msuch
pairs. Now set Kt = sup | X(s) - X(t) |. By assumption
(M)eAt
E[ Kj ] < S E\ X( s ) - X( t ) \ Y < 2 i - C 2 ' l(]^ )= c 2 iri
1 (s .t)e A j 1 1
For a point 5 (t) in D, there is an increasing sequence {^} ({r„}) of points in

D such that sn(tn) is in D„, sn < s(tn < t) and sn = s(tn = t) for some n onward.
Let s ,t e D and | s - 11= 2~m\ either sm = tm or (sm, tm) e Am, and in any case
X(s) - X(t) = I [X(sM

) - X(s, )] + X( sm)
+ i-m
.2 )]
Here series are actually finite sums. It follows that
I X(s) - X(t) I < S £

i= m + 1 m
Set Ma = sup{( | X(t) - X(s) \/\ t —s \a , s, t E D}.

s*t
Then, we have
Ma < sup{2(w+1)a sup \ X ( t ) - X ( s ) \ \ s , t e D , s * t ]
m \t-s\< 2~ m
 1 and a < r]/y we get, with C\ = 2a+l C,
(E | Ma\y){ly < C, 2 2ia(EKj )llY < C\ 2 2i{a-W y)) < 00.
For y< 1, the same arguments holds to EM a instead of (EM a )1/y. It follows,
in particular that for almost every co, X is uniformly continuous on D and it
makes sense to set
X(r, co) = lim X(jc, co)
s-*t
seD
By Fatou’s lemma and the hypothesis, X(t) = X(f) a.s. and X is clearly the
desired modification.
Now back to Brownian motion (increments being normal), for every p > 0,
E | W(t) - W(s) \2p = Cp \ t - s
So Brownian motion has a modification which is Holder continuous of order
a for a < (p - \)l2p - 1/2 - 1/2/7. Since p can be taken arbitrarily large, the result
follows.
Note Condition A.1.10 is the same condition which appears in theorem A.1.2 to
prove much weaker property of Brownian motion.
Corollary 1.7.1 The Brownian paths are a.s. nowhere locally holder continuous
of order a for a > 1/2.
Proof If | W(t, co) - W(s, co) | < C | t - s \a for 0 < 5 , / < 1 and a > 1/2 then
? )- W
O
C (t,,, (0))2 < C2 sup I tM - |2“-' (A.I.10)
1 /
Now by theorem A.I.3 and the note after that
if sup | A„ | —> 0 where {An} = {/rt+1 - tn] is a subdivision of [a, b],
n
Z( W(tM ) - W(ti ))2 (b - a) a.s. as -» °o. (A.1.11)
i
Hence the right hand side of (A.I.ll) would converge to 0 as n —> if W(t)
is locally Holder continuous which is contradictory. Note Theorem A.I.7 and its
corollary leave open the case a = 1/2. It can be proved by the law of iterated
logarithm for Brownian motion (Theorem A.1.5) that the Brownian paths are not
Holder continuous of order 1/2.
APPENDIX II
Second Order Stochastic Analysis
11(A) Second Order Calculus

A second-order process {X(t), t e T} is said to be continuous in quadratic mean
(q.m.) or mean square (m.s.) at t = t0 if E \ X(t0 + h) - X(t0) \2 —» 0 as h —> 0. If
a process is q.m. continuous at every t e T, we say that it is a q.m. continuous
process.
From general property of m.s. convergence E | X(t0)\2 < °o and moreover
lim EX(t) = E( lim X(t)) (m.s) if E I X(t) - X(t0) I2 -> 0 as t -> t0. The q.m.
continuity of a second order process must, of course, be closely related to questions
of continuity of the covariance function of the process.
The following theorem summarizes succinctly the relationship between q.m.
continuity and continuity of the covariance function.
Theorem II.l Let {X(0, t e Tj be a second-order process on an interval T, and
let T(/, s) = E ( X tXs) denote its covariance function.
(a) {X(0, t e T } is q.m. continuous at every t e T iff T (...) is continuous at the
diagonal point (t, t).
(b) If {X(0, t e T ] is q.m. continuous at every t e T , then T(. . .) is continuous
at every point of the square T x T.
(c) If a non-negative definite function on T x T is continuous at every diagonal
point, then it is continuous everywhere on T x T.
Proof If T is continuous at (f, r), then by Hermition symmetry of T
E | X(t+ h) - X(t) |2 = T(r + h, t + h) - T(r, t + h) - T(t + h, t) + T(r, t)
= [T(t + h ,t + h ) - T(/, 0] - [Hr, t + h) - F(ty t)]
- [T(r + h, t) - T(/, t)] —? 0 as h -» 0.
Conversely, if (X(r), t e T ] is q.m. continuous at f, then
T(t + hyt + h') - T(r, 0 = E(X(t + h) X(t + h')) - EX(t)X(t)
= E((X(t + h ) - X(t))X(t + h')) + EXt ( X ' ^ - X , )
From Cauchy-Schwartz inequality we have
\V(t + h, t + h ')- V ( tyt)\<[E\X(t + h) - X{t )I2 E\ X(t + h') |2]1/2
+ [E | X(f,) |2 E | X(r + V) - X,|2]1/2 0 as K h’ -> 0.
(b) If {X(0, t e T )is q.m. continuous at every t e T, then

\ r ( t + h, s + h') -T(t, 5)| = | EX{t + li)X(s + h') - EX{t)X(s)\
= \ {E( X‘(t + h) - X(t))X(s + /(')} +
< J{E\ X( t + h ) - X(t)\2 E\ X( s + |2}
+ V l£ |X ( 0 |2 E |X (5 + h') - X (5)|2} O a s/i,/i' 0.

(c) Since every nonnegative definite function on T x T is the covariance
function of some second-order process on T, part (c) follows immediately from
(a) and (b).
Corollary II.l Let X(t) and Y(t) be two second order stochastic process (t e T ),
Let t0, s0 be two limit points of T. Then if
X(t) -» X (m.s.) as t —> t0 and Y(s) —> Y (m.s.) as s s0, then
EX(t )Y(s ) —^ EXY as t —> to , 5 —^ sq .

Definition A Second order process
I I . l { X(t ), r T} is said to have a derivative
E
X(t) in q.m. (m.s.) at t eT if
lim ±{X{t + h ) ~ X(t)) in q.m. = X(t) (A.II.l)
Theorem II.2 Let (X(r), t e T ) be a second order process. If X(t) has a derivative
at all t E T, then the derivatives
d 3 3^
T(r, s), T(r, s), T(r, s) of the covariance function T(f, s) exist
ot os otos
and are finite on T x T.
Proof If X(t) has the derivative X(t) (q.m.) at t eT, Then
X(t -f- u) —X( t ) ■n

— ------ — -» X{t) as u —> 0.
Therefore lim ( I ± L ± ! L J ± z I i h l l ] = ^ £ f A V t “■>7-*«>

«-^0
= fif lim — ■- —— X (j)l = £X(0X(5). (A.II.2)
Hence the derivative — T(L s ) exists and

dt
fr(t,s) = EX(t)X(s) < oo.
at
Similarly ~ -T { t, s) = EX(t)X{s) (A.II.3)
Finally i!51o H i r (r ,5 +")" i ra’s)}

Appendix II: Second Order Stochastic Analysis 273
X(s + u) X
- (s)^
= lim E(X(t) (from A.II.2)
u—Ô
(A.II.4)
= EX(t)X{s),
which shows that -~r- T(f, s) = EX(t)X(s). (A.II.5)

otos
Let {X(t), t E T) be second order process with mean function m{t) = EX(t) and
covariance function F(t, s). Suppose that X(t) has derivative X(t) on T. Then
m(t + h) - m(t) X(t + h) - X(t)

lim = lim E\
h->0 h /!->0
which exhibits validity of interchange of the operations of differentiation and
32
taking expectation. Further (A.II.5) can be written as -r-r- Fx (f, s) = r \ (t , s).
otos
Next
T(t + h, t ' ) - r ( M ' )
=E ' hl xm n o
shows that the existence of the partial derivative ^dr ( t , /') depends on that of
the derivative of X(t). We shall need the concept of the second generalized
derivative of F(t, t'), defined as the limit (if it exists) as /i, h' —» 0 of the quotient
-hh|7(r(/ +h ,t'+ h ')- r< t + h,t') - r(r, t' +

Above concepts are due to Slutsky (1928) and the following results are due to
Loeve (1945-48, 63).
Theorem II.3 A second order process {X(0, t e T } has a derivative at t e T iff,
the second generalized derivative of F(t, t)') exists and is finite at (r, t).
Proof For fixed t e T, let us put
V X(t + S ) - X( t )
Then Ys Xt (q.m.) as 5 —> 0, where X(t) is a stochastic process iff E( Ys Ys>)

converges to a limit as s, s —> 0 (by Theorem A II. 1). But
£( W = - L E{{X(t + 5) - X( 0) (X(t + s') - XU))}

SS
s, t + s') -ru sr
+ r t)}; (a.ii.6)
SS
the limit of which (if it exists) as > 0 is the second generalized derivative
of F(t, t') at (/, t).
The Stochastic Integral

Quadratic-mean integrals arise even more frequently than q.m. derivatives.
Let (X(0, a< t < b) be a second order stochastic process.

Let a = t0 < ti < . . . < tn = b
be a subdivision of [a, b],
tk- i t k(k= 1,2
and consider the Riemann sum
Sf l = (A.II.7)
If, as n in such a way that max |f* | —> 0, S„converges (q.m.) to

1< k < n
a limit, we say that this limit is the Riemann Integral of X(t) over [a, b] and
f*
denote it by I - I X(t)dt . Clearly, / is a r.v. If I exists, we say that X(t) tis
"a
Riemann integrable over [a, b]. A simple and typical use of this integral is the
l cb
estimate Li = ------I X(t)dt of the constant but unknown me
b ~ a Ja
order stationary process {X(t), a < t < b).
Theorem II.4 A second order process X(t) with covariance function T(r, s) is
Riemann integrable over [a, b] iff the double integral
pb fib
T(r, s) dt ds exists. (A.II.8)
"a Ja
fib fib 1 fibfib
If this is the case, then E X{t )dt \ Xr( j) r f j| = T(r, s) dtds
"a "a ) Ja "a
Proof Now Sn defined by (A.II.7) converges to I (q.m.) iff
E(SnSm) ~ * E ( II), (A.II.9)

where Sm corresponds to a subdivision a = 50 < 5j < . . . < sm = b of [a, b] and
Sf_i < r}j < Sj (j = 1 , 2 , . . . , m). Then we have
_ n m
ESnSm = £ 2 (tk - t, .,
k=17=1
and if the limit of this double sum exists as
n —» oo, m —> oo, max Itk - tk_11—> 0 and max \s , - 5,_. I —>0,
k K 11 1<y<m 7 711
then it is the double integral (A.II.8). This proves the first part of the theorem;
the second part follows from (A.II.9).
Remarks
1. In the last theorem instead of Riemann integral Lebesgue and Stieltjes
integrals can be considered along similar lines.
2. If m(t) = EX(t), then
fib fib fib

EI X(t) dt=\EX(t)dt=\ m(t)dt.
"a "a *a
Let {X(0, t e T ] be a second-order process. A r.v. Y is said to be derived from
a linear operation on {X(0, t e T ] if either
(a) Y(co) = £ CkX(t k ,(0) or
k= 1
(b) Y is the q.m. limit of a sequence of such finite linear combinations.
Denote the collection of all such r.v.s derived from a given process {X(t), t e T ]
by Xx- Two elements Y and Z are not distinguished whenever E | Y - Z |2 = 0.
Equality between second-order r.v.s will always be understood to be up to zero
q.m. differences. The space Xx becomes an inner product space if we define the
inner product (T, Z) = E Y Z . The inner product automatically defines a norm
|| Y || = ^[(yTy) and a distance function d(y, z ) = || Y - Z ||. A sequence {Yn] in
Xx is said to be a Cauchy sequence if ||Tm - Yn || —> 0 as n, m —> «>. Since
II - Ym ||2 = E | Yn - Ym|2, a Cauchy sequence in Xx is a mutually q.m. convergent
sequence, which by completeness must converge in q.m. to some r.v. Y in Xx- By
definition Xx is a Hilbert space.
Let/(/) be a complex valued piecewise continuous function defined on a sub
interval [a, b] of T. If for each (0 e Q the stochastic process X(t, (o), a < t < b
is piecewise continuous, then f(t)X(t, cd), a < t < b is also piecewise continuous.
Since the ordinary Riemann integrals of calculus are well defined for piecewise
continuous functions, I f ( t ) X ( t , co)dt is well defined for each co gQ . We

Ja
define the q.m. integral

f* f ( t ) X( t ) dt as an element of Xx whenever
Ja
n- 1
lim q.m. Z f(t'k ) X(tl )(t[n) - f!"?), where tl are any sequence of points
n— >°° k=0 n n k ki
satisfying t[n_\ < tkn < t[n) exists and is indepandent of the choice of the partition
points tkn.
Theorem II.5 The quadratic mean (q.m.) interal f ( t ) X( t ) dt exists if and

Ja
fib fib
only if I I f ( t ) f ( s) T( t , s) dt ds exists as a Riemann integral.
J a "a
Proof is similar to that of Theorem II.4 and hence is omitted.
Remarks
1. The q.m. integral can easily be generalized to include infinite integrals by
taking q.m. limit as one or both of the end points goes to infinity.
2. A common way in which q.m. integrals arise is when a process { Y ^ t e T ' }
is generated from a second process {X(t), t e T ] according to the formula.
y(r) = J h ( t , s ) X{ s ) d s , t e T ' (A.II.10)
Then (AIL 10) often admits the interpretation of a linear system with input
X(r), output Y(t), and impulse response h(t, s).
3. From applications point of view one might prefer J f ( t ) X ( t , c o )d t

to be defined as the Lebesgue integral of sample function (X(0, t e T). The
existence of such a Lebesque integral is insured if the process is a measurable
process and if J \ f ( t ) \E | X(t)\dt < °°.
11(B) Continuity, Integration and Differentiation of Second

Order Process
Definition II.2 A stochastic process {X(t), t E 7) is said to be sample continuous
(or{X(0, t e T] has continuous sample path), if for each c o e Q , X(t, ft/), t E T is
a continuous function of t. For example Brownian motion is a sample continuous
stoohastic process but Poisson process is not. Let (X(f), t E T] be a continuous
parmater second order process satisfying (i) the mean function mx(t) = E(X(t)),
t E T is a continuous function of t and (ii) t), s E T, and t E T, is jointly
continuous in 5 and t. We say that this process is differentiable if there is a
second order process {T(7), tE T), satisfying assumptions (i), (ii), (iii) lim X(s, co)
At
in finite, (iv) X(s, co) —>X (t, co) as ^ 1 1 and (v) the function X(t, co), t E T has only
finite number of points of discontinuity on any closed bounded subinterval of T
such that for t e T
X ( t ) - X ( t 0) = f Y(s)ds, t E T
then the process Y(t) is called the derivative of X(t) and is denoted by X'(f), t E
T. Thus
X ( t ) - X ( t 0)= f X'(s)ds,tET. (A.II. 11)
It can be proved that if a second order process is differentiable then it is

differentiable in q.m.
It follows now that X(t) has continuous’ sample functions and that
J - X(t) = X'(t) holds excepts at point of dicontinuity of the X'(t) process.
It also follows that m x(0 ~ mx(t0) = m'K(s)ds ancj hence

mx(0 = f t mx( t \ t e T.
To find the covariance function of X'(t) process it is necessary to consider
cross covariance functions of two second order proceses {X(0, t E T] and
{Y(t), t E T) defined by E(X(/)T(5)) = Txy = Txy(t,s). Suppose that X(t) is
differentiable and satisfies conditions (i) and (ii). We shall show that
r yx,(s,t) = -$-r yKa(s,t)eTnd e (A.II.12)

at
and Tx'y(s, t) = y - r xy(s, r), 5 e T and t E T (A.II. 13)

If moreover {X,} is second order stationary and twice differentible then

r v (*,/) = - r X”( t - s ) .
Proof Choose
t0 e T and write X(t0) = f X'(u)du.

•''o
Then we have Y(s)(X(t) - X(t0)) = f X'(u)duY(s).

*to
Thus E [ Y ( s ) [ X ( t ) - X ( t 0)}]= f E[Y(s)X'(u)]du (A.II.14)

J'o
Now mx (t) - mx {tf) = I mx>{u)du,

•''o
and hence my(t)(my(t) - mx(t0)) = I my(s)mx'(u)du (A.II.15)

J'o
Subtracting (A.II. 15) from (A.II.14) and rewriting the resulting expression in
terms of coveriance function, we conclude that
Fyx(s,t) - Tyx( s, t 0) = f Yyx'(s, u)du (A.II. 16)
Now the cross covariance function r yx>(s , 0 is jointly continuous in 5 and t

if the second order processes {X(0, Y(t), t e T] are q.m. continuous. Thus for
each fixed s we can differentiate (A.II. 16) with respect to t and obtain (A.II. 12)
as desired. Now (A.II.13) follows from (A.II. 12) by symmetry.
Now (A.II. 12) and (A.II.13) imply r ( j , 0 = 4 - r ** ?) = 4 " ^ (J. 0 and
at at
d2
Fxx' (s, t ) = - £ ^ r xx( s ,t),s e T and r e f .
Since {X„ r e T} is second order stationary, mx>(t) = 0 and
r , u . , ) = ^ 7 r , (, . o = ^ r.(.-«) = - r .
Orthogonal Expansions
As on page 275 the Hilbert space generated by the class of second order r.v.s
X(t), t e T will be denoted by %x = X- A family ^ o f elements (random Variables)
in Xx is said to be orthonormal (0 - AO family if any two distinct elements Y and
Z of & satisfy
E\\Y\\ = ( E | K | 2)1/2= 1 = (E |Z |2)1/2 and (K,Z) = E(YZ) = 0 (A.II. 17)

Suppose that (X(r), t e T] is a q.m. continuous process, where T is an interval,
finite or infinite. Let T' be the set of all rational points in T. For every t e Y, there
exists a sequence {/„} in T' such that tn —> t as n —> ©o.
Since {X(t),te T) is q.m. continuous, we have for every 7, X(t) = lim q.m. X( t n).
n—
><»
If follows that every element in Xx is a linear combination of {X(7), t e T' } or
the q.m. limit of a sequence of such linear combinations. In short, the countable
family {X{, t E T ) is dense in Xx- It follows that every O-N family in %x is at most
countable. For a q.m. continuous process {X(7), t e T}, let {Z;, j*> 1} be an
O-N family in %x. If Z e Xx^ then
E \ Y - 2 < Y , Z j > Z j \ 2 =E\ Y\ 2 - 2 | < Z, Z; >|2 > 0 for every n > 1.

7= 1 7=1
Therefore, o o > £ | y | 2 > £ |(Z, Z; ) | 2.

7= 1
OO OO
Hence 2 (Z, Z,-) Z, is well defined and Y - 2 (Z, Z.) Z, is orthogonal to

7=1 7= 1
every Z;. So by completeness of {Z;} in every
Y 6 Zi has the representation Y = 2 (Z Z.) Z, (A.11.18)

7= 1
Suppose that {Z;, j > 1} is a given complete O-N family in Xx and set
0j(t) = (X(t),Zj). (A.II. 19)
Then from (A.II. 18) we get
X(t) = 2 0, (t)Zj , t e T (A.II.20)

7= 1
since 0r(O’s are the cross covariance function of the q.m. continuous X(t) and the
r.v. Zr are continuous functions.
Further, the set of functions {Qj(t),j > 1} is also linearly independent, i.e. for
every n
2 Cjtpj (t ) = 0 for all / G T implies Cj = 0,y = 1, 2, . . . , n.

n
The linear independence of {Mt), j > 1} is due to the fact that 2 cjty (t ) = 0
7= 1
n
for all t E T implies 2 c , 0, is orthogonal to X(t) for every t e 7, Hence, also
7= 1
orthogonal to every 0; which implies cj = 0 for every j. It follows from (A.II.20)
that
E(X(t)X(s)) = r(t, s)= 2 0; ( /)0( 5) r, 5 e T (A.11.21)

7= 1
Conversely, Suppose that {0/7), 7 e T,y' > 1} is a linerly independent family

of continuous functions such that
r ( r , 5 ) = 2 0/ ( 7) 0/ ( 5) 7, 5 E T.
7= 1
Then by Karhunen general representation theorem (1947) there exists complete
O-N family of r.vs. {Z;, y > 1} in Xx such that
X(t,co)= E <pj(t)Zj{(o),t e T .
7=1
Thus (A.II.20) and (A.II.21) represent each other. The representation (A.11.20)
permit uncountable set of r.v.s {X(0, t e T] to be presented by a countable
number of orthonormal r.vs. {0,(0}.
Though it is usually difficult to express the r.v.s {Z; } explicitly in terms of
{X(0> t e T) but in case when {0,} are orthogonal, i.e. <Pi(t)0j(t) dt = 0

whenever i * j the task is easier and that leads to well known Karhunen-Loeve
expansion (1963).
Consider a q.m. continuous process {X(0, a < t < b}. Suppose that there exists
an expansion of the form
X (t , (o) = E 0, (t)Zj (co), a < t i(t)<l>j(t)dt = AjS'j (A.11.24)

"a
Now, from (A.II.21), the covariance function T(.) must satisfy
T(f, s) = E 0 (00/ (s) for each (t, s) e [a, 6] x [a, &] (A.II.25)
Now from Cauchy-Schwartz inequality and the fact that T is continuous on

[a, b] x [a, b] we have
n
sup 2 <Dj(t)<t>j(s) < SUP £ |0 ; (O |2
a<t,s<b 7= 1 ^ 7 a < t , s < b 7= 1
< sup r ( r ) < o o , ( r ( / , o = r ( 0 ) (a .ii .26)

a<t<b
Therefore, the convergence in (A.II.25) is bounded and
f
Ja
T(/, 5)0y (5)^/5 = A7 (00/ (0, a < t < b (A.11.27)
This proves that if an expansion (A.11.22), satisfying (A.II.23) and (A.II.24)

exists, then {0,} must be solutions to the integral equation (A.11.27). We shall
show that under our assumptions, such an expansion always exists.
The integral equations of the form (A.11.27) are well known and we shall now
summarize some of the elementary facts about integral equation of the form
(A.11.27) which are necessary for Loeve-Karhunen expansion of second order
process.
Rudiments of Integral equation

Consider the integral equation
f
* a
T(f, s)(j>{s)ds = A0(/), a < t < b (A.II.28)
where fa, b] is finite closed interval and T(f, s) is continuous on [a, b] x [a, b].
The A * 0 for which there exists 0 satisfying both (A.11.28) and the condition
rb
I |0 (/)|2 dt < °o, is called an e/gen va/we of the integral equation. The
J a
corresponding 0 is called an eigen function. Let us list the important facts.
1. Any eigen value of (A.11.28) must be real and positive. The fact that A is
real follows from the Hermition symmetry T(L s) = F(s, t). The fact that A is
positive follows from the non-negative definitness of T .
2. There is at least one eigen value for (A.II.28), if T is not identically zero.
The largest eigen value Ao is given by
fib fib
A0 = max J J T ( t , s ) Q ( s ) Q ( . t ) d s d t , (A.II.29)
- 1/2
where ||0|| = |0 (O |2 dt
Note: This fact is not easily proved. It depends on both the nonnegative definiteness
of T and its continuity (see Taylor 1961, p 334-336).
3. Let 0O(t) denote the normalized eigen function corresponding to Ao, then
0o(0 is continuous on [a, b], since
i ^
0o (t) = —— f V(t, s)0o(s)ds, a < t < b (A.11.30)
A 0 Ja
and continuity of 0Ofollows from continuity of T(r, s) is t.

4. Let Ti (r, s) = T(r, s) - AO0O(t) 0 o(s). Then s) is both continuous
and nonnegative definite. The continuity of Tj is obvious from 3.
To show nonnegative definiteness, let
Y(t) = X(t) - 0o(0 [ X (r)0o(r)r/r (A.II.31)

Ja
then E(Y(t)Y(s)) = t)s- e \

(X
E Ms) 0o{u)du
fib fib
~E X ( s ) M t ) X(T)dT + E 0o(f)0oU) X{T)<j>0 (u)drdu
"ra Ja Ja
= r (t, s) - Ao0o(O0o(^) = HU, s) (A.11.32)
Hence, 1^(7, s), being a covariance function, must be nonnegative definite.

5. We observe that
and obtain Ab then
J
a
(/, s)<p0 (s)ds = 0. Therefore, if we repeat step 2
f r,(f, 5 ) <p\(s)ds = a <t < b.

Ja
Therefore J* 0i(O0o(O dt —J* 0o(0 j"^ J* fî (f>
=irl 0i(i)[J r,(j*r)^»(r)A ds

=jj- J 0i(s) J r,(i,r)0o(O^ ds = 0
Hence, 0] is orthogonal to 0 O. Also

fib fib _
T(r, s)0i(s)ds = I r ^ r , s)<t>l(s)ds - Ao0o(O
*a "a ~a
= Aj^jCO, a < t < b (A.II.33)

i.e. Aj and are eigen value and eigen function of (A.II.28).
6. Repeating the above procedure we get a nonincreasing sequence of eigen
values Ao, Ab . . . and a corresponding sequence of eigen functions <fo, 01? . . .
which are orthonormal, i.e.
r _
0, (00/ ( =
7. The sequence Aq, Ab . . . may terminate after a finite number of terms, in

which case we have
(A.II.34)
T(r, i ) = X A„0n(r)0„(i) (A.11.35)

n=0
If the sequence Aq, A,, . . . does not terminate,

fib N mb
oo > I r (t,)d> X A„ I |0 „ (O |2
J„ n=0 Ja
= X A„ for every N (by (A.II.34)

n=0
and hence as A„ —» 0 as n —> <*>.
8. If the number of eigen values is infinite, then
lirn sup r(f,s)- 2

k=0
=0 (A.II.36)
i.e. lim Z Ak<pk(t)<t>k (s) =r(f, (A.11.37)

k=1
uniformly on [a, 6]~.
This result is known as Mercer s theorem (see M.Loeve 1963). Since uniform
convergence implies convergence in mean square,
pb pb 2
lim r(r,j)- Jt=0
Z At0 *(O dtds = 0 (A.II.38)
*a "ft
9. In general, the O-N family {<pis not complete in
*b
At the most we can say that, given f l / ( o p dt < oo, we can write
Ja
/(?) = /o (0 + lim Z 0*(r) |

W->°° £=Q "a ■r fin q.m. fA.II.39)
and / 0 satisfies f r(*, 5)/o (s)ds = 0 for all t G [a, b] (A.11.40)
If follows that {0^.} is complete in L2 [ay 6], iff (A.II.40) implies
"a r
* |/ 0(O|2 * = 0.
Now we are in a position to state and prove the celebrated.
Theorem (Karhunen-Loeve expansion, Loeve 1963, pp. 478-479)

Let {X(r), t G [a, b]} be q.m. continuous second-order process with covariance
function V(t, s).
(a) If { 4ik} are the orthonormal eigen functions of
f
«a
T(f, s)(j>{s)ds = A0 (f), a < t *} the eigen-values, then X (0 = lim £ uniformly for

Jt=0
a <t <b (A.11.42)

where {A^} is a sequence of r.v.s satisfying
rb
Ajc = I 0*(r)X (r)^(q.m .) (A.II43)
and E( AkA j ) = S kJ (A.11.44)

(b) Conversely, if X(t) has an expansion of the form (A.11.42) with
"af <h(t )<Dj(t) = 8
k} and {Ak} must be eigen functions and eigenvalues of (A.11.41).

Proof (a) First note that (A.II.4A) is well difined, because X(t) being q.m.
continuous and hence 0„(f) is continuous on [a, b] by result (3).
rb _
Now 4^ k Ak - [ X{t)Qk(t)dt,
E(X(t)Ak) E(X(t)X(s))4>k(s)ds,
!)=vt 1
= —r= = f r(f, s)<t>k(s)ds = -j= = Ak0k(t) =
Ja
Hence E{x(t)iQ VI7<t>k{t)A k = iQV4
=£=0
i V * r * ( ') f c ( o < A T = k2=0 * * lfc (o i2.
Therefore, E\X(t) - ^ JX^<l)k (t)A k \2 = T(t, t) - Z (/)|2 —> 0 as

k=0 k~0
n -$oo uniformly in r by virtue of Mercer’s theorem (Result 8, equation (A.II.36)).
(b) Suppose X{t) has the expansion (A.II.43). Then we have
IV. s ) = t)X
(X
E (j)) = £ Ak<k ( f ) 4 (j)
k=0
= lim E( Xn(t)Xn(s)\
where Xn(t)= S J ^ < k O ) A k .

k-0
fib fib oo _
Hence I T(t, s)0, (s)ds = I Z Xk(j>k {t)<))k {s)<))i {s)ds
Ja Ja k=°
= A;-0y(f), a < t < b.
Let us give an example of how an integral equation of the form (A.II.41)
might be solved.
Let F(t, s) = min (r, 5 ), the covariance function of Brownian motion and
consider
f min(f, s)<t>(s)ds = A0(f), 0 < t < T (A.II.45)

Jo
or I s<p(s)ds + t
Jo I 0 (s)ds = A0(r)
Differentiating with respect to t yields
f <Ks)ds = A 0 '( O ,O < f < r (A.11.46)

Differentiating again, we get

- 0(f) = X 0"(r), 0 < t < T (A.II.47)
Now the boundary conditions are 0(0) = 0 and 0'(7) = 0.
Equation (A.11.48) with boundary condition 0(0) = 0 yields
0 (0 = A sin -1
Vx
f 1 J
Applying the boundary 0'(T) = 0 in (A.II.47), we find cos -7= T = 0 . Hence
I Va j
the eigen values are given by
A* = ------ ~ 2^ ~-.-,k = 0, 1 ,2 ,... (A.II.48)

(k +1/2)2
The normalized eigen functions are given by
<t>k{t) = sMin [it + 1/2) n ()] (A.II

By Mercer’s Theorem we have
T(f, s) = min (t , 5)
= 7 J o ?r27 * + ~ i/2 )2 sin[(* + 1/2)* (5/r)] sin K *+ l/2)^(f/7)]

uniformly on [0. 7]2.
Wold Decomposition Theorem (1938)

Any stationary process X(t) can be expressed in the form X(t) = U(t) + V(t),
(a) where £/(f) and V(t) are uncorrelated proces.
(b) U(t) is regular (or indeterministic) with a one-sided linear representation
oo oo
f/(/) = 2 C,Et_i with Cn = 1, 2 c,2 < oo

1=0 1 ' 1 U i=0 '
and £, is white noise W.N. (0, a 2) and uncorrelated with V(r), i.e.
E(£,V(s)) = 0 for all r, s.
The sequence {c, } and the process are uniquely determined.
(c) V(t) is singular (or determinstic) i.e. can be predicted from its own past
with zero prediction variance (See Note 1, page 286).
Proof Let X(t) denote the projection of X(t) on Xt-i 0-e- the linear subspace of
% spanned by the r.vs. X(s), 5 < t - 1 i.e. limits of Cauchy sequences of linear
combinations of XM, Xt_2, . . . and the all such linear combinations. In fact X{t)
is the linear least square predictor of X(t) given X(t-\), X(t-2), . . .). Define the
process {£,} by et = X(t) - X(t). Then £tXXt-\ i-e* £t orthogonal to every
element of %t-v For anY PaiL £r} with, say s < t. We have et±Xs c Xt-b an<^
€s e Xs (since both X(t) and X(s) are in x s)-
Hence es and et are orthogonal i.e. E{es £t) = 0 for s * t. Now consider the
projection of X(t) on Xt (£X the subspace spanned by £„ Et_1 ,.. and this projection
can be written as
1/(0 = £ Ci£t-i (A.II.50)

j=0
where e,’s are orthogal,
Ci = E[X(t )et^ V a l (A.II.51)
with = ||£, ||2 = E e }.
Thus Co = E[ X( t ) et] l a l = £[(X (f) + = £(£,2/cr2) = 1
(v £,±X (/), £, = X ( /) - X ( 0 )
Now write V(0 = X (0 - 1/(0 = * ( 0 ~ £ £••£,_,■ (A.II.52)

y=o
Thus for 5 < r
W H ) = £ [ ( * « - t/(r))£rj = £[*«<*] - E[U(t)e,] = 0
(since £[£/(r)£v] = &eCt-s and by (A.II.51)
E[X(t)e,] = <r}c,-,)-
For s > t,
V(t) e X, and es± x , ( " e,e
Hence £s is orthogonal to V(t) for all s, t, and f/(s) is orthogonal to F(r) for
all v, r,f/(.y) being linear combination of es for all s.
Therefore || X (f)||2 = || U(t) ||2 + 1| V(t) ||2
and we then obtain
f L cfI= \\U(t)\\2 < II X (t)||2 = £ (X 2 (0) < « .

l ,=0 )
For orthogonal subspaces £j and £2 we define the direct sum £j © £2 =
{x + y : x e £b y e £2}. Let %t (v) be the space spanned by V(t), F(/-l), . . .
Then V( t ) e Xt = Xt-i ® •*/?{£,}, where sp{et] is the space spanned by et.
Thus Vt e Xt-\ • Since V,±£„ and by repeating the same argument we find that
VI e X'-S’
for all i > 0 ( V V,eXt-i © jp {£,). < Vt, £,-\ =
oo
Hence Vt e X-<*> = n Xt >and is therefore determined by the “remote past”
/= - o o
of {X,}. But the projection of V, on Xt-i s* the sa

Xi-i(V), since we may write Xr-i(^) © i(£) (by A.II.52).
Hence V, e iX
t-(V) i.e. X,(V) =-X
t i(V) for all t and clearly V, e Xt-s <X)
all 5 > 0.
oo
We can write Vt e X-oo(V) = n (V), i.e. V, is determined by its own

/ = - oo
‘remote past” and can be predicted with zero prediction variance. To establish
uniqueness we observed from
X(t) = £ + Vt (A.II.53)
.,=0
that if {£,} and {Vt} are any sequences satisfying (A.11.53) and having the desired
properties proved above, then Xt-\ c sp {es , 5 < t - 1} © sp {Ve, s < t - 1}
from which it follows, {£,} being white noise (orthogonal) and E(etVs) = 0,
that et is orthogonal to Xt-v Projecting each side of (A.II.53) onto %t-\ an<^
subtracting the resulting equation from (A.II.53), we find that the process {£,}
must satisfy £, = X, - X, = X, - PXt-\ X, • By taking inner products of each side
of (A.11.53) with et_j we see that cj must also satisfy (A.II.51). Finally, if (A.II.53)
is to hold, it is obviously necessary that Vt must be defined as in (A.II.52).
Note
1. For a general stationary discrete parameter process {X,} (whose spectrum
need not necessarily be continuous) let us consider the sequence of prediction
variances (J2 = E[Xt+m - Xt+m]2, m > 1. By stationarity of X ,, a 2 does not
depend on t. Clearly %s c %t, s < t and hence a 2 < c \ < . .. < cr^, < . . .
(obviously we would expect that variances to increase as we predict further
ahead).
If a 2 > 0 the process is called regular or nondeterministic where as if crj2 = 0
process is called singular or deterministic. If a 2 > 0, it follows that <7% > 0 for
all m. On the other hand if a 2 > 0 then X/+1 is its projection on Xt i.e. Xt+i= Xt
for all t and all the Xt spaces are same. Thus, Xt+m is its own projection on Xt f°r
all m > 1 and hence 0 = <72 = a 2 = . . . a 2 = . . . We therefore see that the
{ a 2} are either all positive (corresponding to X, regular) or all zero (corresponding
to X, singular). In the later case X, can be predicted “prefectly” i.e. with zero
prediction variance, from its past values.
2. If X -00 = {0}, then Vt = 0 in Wold’s decomposition and X, = Ut is “purely
nondeterministic”. If o j = 0, then Ut - 0, and Xt - Vt is then a deterministic
process.
Appendix III
Stochastic Integral (Ito Integral)
Introduction
In recent years, it has become apparent that physical systems, classically modelled
by deterministic differential equations, can be more satisfactorily modelled by
certain stochastic counterparts if random effects in the physical phenomena as
well as measuring devices are to be taken into account. In this context, an
ordinary differential equation - ^ - = /( r ,* ) would be replaced by a random
differential
dX(t)
= F (f,X (0 , Y(t)) (A.III.l)
dt
where Y(t) represent some stochastic input process explicitly. It is often seen that
it is not possible to interpret (A.III.l) as an ordinary differential equation along
each sample path. For example
dX(t)
= f ( t , X( t ) ) + g((t,X(t))N(t) (A.III.2)
dt
with N(t) being a Gaussion white noise process. Equation (A.III.2) has been
popular in the engineering literature especially, since Gaussion white noise
approximates the effect of the superposition of a large number of small random
disturbances, a situation encountered in engineering systems. However, the
irregularity of the sample paths of N makes (A.III.2) intractable mathematically.
Just as a solution of (A.III.l) satisfies the deterministically integral equation
X(t) = X(r0) + I f ( s , X( s)) ds a solution of (A.III.2) should be a solution of
J'o
the random equation
X(t) = X(t 0)+ f f ( s , X( s ) ) ds + f g(s, X(s))N(s)ds (A.III.3)

J'O •''O
but unfortunately the last integral in (A.III.3) cannot be defined in any meaningful
way. To deal with this difficulty, the integral in question is replaced by an integral
of the form
g(s, X(s))dW(s). (A.III.4)

'o
where W is the Wiener process and with the motivation being that, at least
formally, W(t) = N(t) and so dW(t) = N(t)dt. However, even equation (A.II.3)
with the second integral replaced by (A.III.4) is not unquestionable. The natural
interpretation of (A.III.4) would be as a Stieltjes integral along sample paths.
But this is not possible, since the sample paths of the Wiener process are not of
bounded variation. Furthermore, if different choices of the T/ are made from the
subinterval [/M, rj of the partition t0 < tx < t2 < . . . < tn = t, the natural
approximating sums
Z s ( T ,,X ( r,) ) [W O ,) - W(?M )]. (A.III.5)

f =l
converge in the q.m. sense to different values of the integral. If, in (A.III.5), r,
is taken as t,_l, the Ito integral results. The impact is that the integral inherits
many of the probabilistic properties of the Wiener process at the expense of the
corresponding calculus differing from what one would expect in the Stieltjes
case. The main instrument of this calculus is Ito’s formula, which can be thought
of as a probabilistic chain rule; it yields a stochastic equation for any sufficiently
Cb
smooth function of a solution. Stochastic integral I g(t)dW(t) of Ito satisfies
Ja
=0 (A.III.6)
and g{t)dW (A.III.7)

«ir
*a
and viewed as a function of the upper limit of integration, forms a martingale.

Note that W(t) is a martingale with E[W(b) - W(c)] = 0 and E\ W (b) - W(a) |2
= b - a. This illustrate tracability that motivates the choice of the Ito integral
from a mathematical point of view.
The difficulty alluded here from g(r{) being a random variable which may not
be measurable with respect to the a -algebra generated by W(rM). From our
discussion it seems reasonable to start by defining stochastic integral for random
step functions. However, there are infinitely many (indexed by parameter A)
such definitions for random step functions leading to distinct q.m. stochastic
rb
integrals. Specifically, consider the integral W(t)d W(t) and for any fixed A,
"a
0 < A < 1, the random step function approximation of W(t)
f nx(t) = AW(t[n>) + (1 - X)W (t[n_ l),t<
knJ < t (A.III.8)
for which f rf(t)dW(t)= (A.III.9)

Ja
where 0 < a = t (nn) < . . . < t (nn) - b is a sequence of partitions of the interval
l a , b l A t [ n) = t[n) - t [ % d n = max At[n\ and A W {k n) = W{t[n)) - W (t[n„\)
results in the stochastic integral (by taking q.m. limit as dn —> 0)
Appendix III: Stochastic Integral (Ito Integral) 289
f W(t)dW(t) = ^ [ W 2 ( b ) - W 2 (a)] + ( A - l / 2 ) ( b - a ) (A.III.10)

"(I
For example, when A = 1/2 the Stratonovich stochastic integral exists. In this
case the form of (A.III.10) indicates that the corresponding stochastic calculus
will be analogous to the usual Riemann-Stieltjes Calculus. However, a disadvantage
of this and any A > 0 case is that expansion
A W ( t [ n) ) + (1 - X) W( t {kn] ) (A.III.l 1)
is not measurable with respect to the a -algebra gen

which prevents the martingale property as well as (A.III.6) and (A.III.7) from
holding. The corresponding step functions f„ (t) are said to be anticipating in
this case. The nonanticipating case, namely, when (A.III.11) is measurable with
respect to the cr-algebra generated by W(t["j) and hence independent of the
increments AW^n), occurs only when A = 0. This leads to the Ito case and with
the random step functions f n ( t ) = t[nJx < t < t[n) in (A.III.9), one
obtains the special case of (A.III.10),
f
*a
W(t)dW(t) = \!2{[W 2 (b)- W 2 {a)} - (b - a)}
and it can be shown that the integral interpreted in the Ito sense satisfies
\
(a) W(t )dW(t) - 0 and
(b) E W(t)dW(t) = \! 2 (b 2 - a 2).
Definition of Ito Integral

Let W(t) be a standard Wiener process defined on the probability space (Q, JF,P)
and let - <*> < t < <»} be an increasing family of sub cr-algebras of &
satisfying the following conditions (i) c if tx< t2, (ii) W(t) is measurable
for each t and (iii) For s > 0, W(t + s) - W(t) is independent of^T If the integrand
Cb
f does not depend on co, then the integral 1( f )
= I f ( t ) dW( co?t) can be
Ja
treated as second order (q.m.) integral as defined before (see Appendix II, theorem
II.5).
Interpreted in that way, only the property of a Brownian motion as a process
with orthogonal increaments is made use of, and not its other properties. I f / is
random i.e. it depends on co and, then the integral has to be defined anew. It turns
out that its definition now depends in a crucial way on the martingale property
and
E(Wt+s \ ^ ) =aW
, nd E(W,+S - W,)2 = s a.s.. s > 0, of (A.III. 12)
We make the following assumptions on the integrand/.

(a) The function / i s jointly measurable in (tu, t) (with respect to & in COand
the Lebesgue measureable sets in t) (A.III.13)
(b) For each r , / = /( ., t) is measurable with respect to and /satisfies
f E\ f ( t ) \2 dt <oo (A.III.14)
~a
Denote by £the class of random step functions satisfying (A.III.12), (A.III. 13)
and (A.III. 14).
The stochastic Integral is now defined as follows:
1. If a = t0 < tx < . . . < tn = b be partition points of the interval [a, b] are
independent of (O and
fi((o) =f (t h .) = /(<w, t) if t x < t < tM and i = 0, . . . , n - 1 (A.III.15)
satisfy conditions (A.III.13) and (A.III. 14), then we define the stochastic
Integral by
f /(© , t)dW((D, t) = I U (co)[ W(co, tM ) - W(a>9 t i )] (A.III. 16)

Ja /=0
2. If/satisfy (A.III.13) and (A.III.14), then we shall show that three exists a
sequence of random-step functions [ f n( c o , r)} satisfying (A.III.13) and (A.III.14)
such that
[ E |/ ( .,0 - / „ ( ..0 |2 * -> 0 a s n -» o o (A.III.17)

Ja
cb
It will then follow that I /„ ((0 , t ) d W ( c o , t) converges in mean square as —>
Ja
and this q.m. limit is the same as any sequence of random step functions which
satisfy (A.III.17).
Therefore, we define
b *b
f(w,t)dW (CO, t ) = lim /„ in q.m., (A.III.18)
J n —>°o J
"a * a
where { f n( c o , r)} is any sequence of random step functions satisfying (A.III.17).

We assume that both a and b are finite. If not, the stochastic Integral is defined
as the limit in q.m. as a - <*>and b —> or both.
Theorem III.l Let { W t , t e T} be a Brownian motion, and let/((U, t) satisfy

(A.III.13) and (A.III.14). Then,
(a) there exists a sequence of random step functions {/„} satisfying (A.III.13)
and (A.III.14) such that
IIf-nI2I = f \f(.,t)2I
E -> Oasn
(b) For each n , / ( / n) = f f (co,t)dW(( 0 ,t) is well defined by (A.III. 16),

"a
and {I (fn)} converges in q.m.
(c) If {fn} and {/„*}are two sequences of random step-functions satisfying
(A.III.13) and (A.III.14) such that \ \ f - f n || and \ \ f - f n || both go to zero as
n —> oo, then lim /( /„ ) = lim I ( f *) in q.m.
n—»°o n—>°o
Proof (a) Suppose E[f(., t ) f ( . ys)] is continuous on [a, b] x [a, b], that i s ,/
is q.m. continuous on [a, b], then an approximating sequence of random step
functions {/„} can be constructed by partitioning [a, b], sampling /(g>, f) at
partition points tfn), defining f n (a), t) = /(cu, tjn)), t ^ < t < t^ , and refining
the partitions to zero (taking max (tj*j - t (p ) —> 0 as n -» oo). Since f is q.m.
continuous, E |/(., t) -/„(., t) |2 —» 0 as n —» °ofor every r e [a, 6]. By dominated
convergence theorem, we have E |/(., f) - / „ (., t) \2dt —> 0 as n -» oo.
More generally; if /m erely satisfies (A.III.13) and (A.III.14) and is not

necessarily q.m. continuous on [a, 6], we construct a sequence of approximating
random step functions in the following manner:
By setting/(r) = 0, t £ [a, b \ f can be considered defined on the entire line
(-«>, oo). Then by (A.III. 14) and Fubini Theorem
A OO MOO
E\ |/ |2 dt= | / | 2 < oo (A.III. 19)

J —oo J —oo
so that, J 1/(0 |2 dt < °° a.s. (A.III.20)
Now J I/O + h) - f ( t ) \2dt <

4 J 1/(0 I2*; (A.III.21)
hence (A.III.20) and (A.III.21) imply
f | f {t + b) —f (t) \2 dt —> 0 a.s. as h —> 0 (A.III.22)

J —oo
by the dominated convengence theorem for Lebesgue integration along sample

paths. Now from (A.III.20),
+ ) -/ (f)
h \2 d <4E J |/(0
So by (A.III. 19) applying dominated convengence theorem in the probability

space we get
E + h) —f {t) |~ dt —> 0 a.s. h —> 0 (A.III.23)

Define the function 07(r) = [jt] / j j an integer > where [ ] denotes the “greatest
integer” function.
Since Qj(t) —> t as j —» «>, (A.III.23) can be replaced by
E f |/(5 + Qj (/)) ~ f ( s + t) \2ds —> 0 as7 —» «> for any fixed r (A.III.24)
J —00
As this expression is bounded by 4E \ |/ ( s ) |2 ds < the dominated

J—
00
convengence theorem together with Fubini theorem, yields
J - 0 0 J a- 1
f U + 0 /(0 ) - / O + 0 -r•'rj-i J-
If ( s + 0 ,(0 )
- / O + 0 |2 dsdt —> 0 as7 —>00 (A.III.25)

Therefore there exists a subsequence {jn} such that
f E \ f(s + 0jn( t ) ) - f ( s +t) |2d t - ^ 0 as j n —> °°, for almost all 5 (A.III.26)
Jo- 1
Fixing such s e [0, 1] and replacing t by t - s in (A.III.26), we have

/• b+s
E| f ( s + <t>jn
Therefore, since 0 < s < 1, (A.III.27) implies
f E \ f ( s + <l>jn ( t - s ) ) - f ( t ) \ 2 d t ^ 0 (A.III.28)
Jo
Taking f n (t ) = / ( s + <t>in (t - 5)), we have f n e e and (A.III.28) states that

II fn I~
fI - > 0 a s n - > » .
(b) Assume that {/„} is a sequence of random step functions such that
|| f - f n || —> 0 as n —> 00 and define I(fn) by (A.III.16) as
/( /„ ) = S /„ . - , t)n))] (A.III.29)
Writing A j W (n) = W (a> ,t$ - W(co,t)n)), we have
E\ /(/„ ) |2 = ? 2 E ( f f at A* W(">)
J
If k>
j , then Ai W(") is independent of ) while )is
J^,(n(-measurable. Hence E i ^ j ^ A j W ^ A t W ^ ) = 0 if k ± j
and E \ I ( f n ) \ 2 = l E [ \ f nj \2 ( A j W ^ ) 2]
= Z E \ f n.\ 2 E[(AJ W ^ ) 2 |
Cb
Z E \ f nj\2 ( t ) H - t (; ) )= \ (A.III.30)
J Ja
Now, l ( f m+n)- /(/„ ) = I ( f m+n - f n ) and ~ fn is again a random step

function. Therefore,
Cb
E|K f m+n) - / ( / „ ) |2 = E| n(., 0 - / „ (.
fm
+
Ja
<2 [b E \fm+n(-.,t) / ( . , r ) | 2 dr + 2 f £ | / „ ( . , * ) - / ( . , r ) |2 dr
•'rt Ja
—> 0 as n —> oo (A.III.31)
Hence {/(/„)} is a mutually q.m. convergent sequence, and there exists a
second-order r.v. 1(f) such that E \ I ( fn) - 1(f) |2 —>0 as n —> <».
(c) Suppose {/„)} and {/„'} are random step functions such that
|| f n -/II —>0 as at -> °o and \\f' - f \ \ -> Oasn oo.
Then, II/. || < V 2 (||/. - / | | 2 + ||/ n' - / | | 2)1/2 -> 0 asn -> ~
Therefore, E\ l ( f „) - „I(f ') |2 —>0 as n » °° and
—
lim /( /„ ) = lim /(/„') in q.m.

n—>°o n—>°°
Theorem III.2 Let/(<tf, t) and t) satisfy (A.III.13) and (A.III.14).
Then f
J a
f ( . , tt)dW(t
) dW( t )) f
"c ■ "ra E(fg)dt (A.III.32)
Proof It is enough to prove (A.III.32\0 for/ = g, because
E [ I ( f ) W ) ] = \ [E\ 1 ( f ) + Kg) |2 - 11 ( f ) - 1 (g) |2 ]
+ i.I [£ | /(-«/) + /(g) |2 - £ | - 1 (g) |2 ]
= ± [ E \ I ( f + g ) \ 2- E \ I ( f - g ) \ 2]
+ E[E\ I ( - i f + g) I2 - g) I2(A.II
(A.III.32) follows from the special case E\ 1(f) |2 = f E\ g \2 dt . Now if /is

Ja
a random step function, we have already proved the last equality in (A.III.30)
that E\ 1( f ) \2= f E |/(., r) |2dr.

Ja
I f /is not a random step function, let {/„} be an approximating sequence of
random step functions, then
E \ I ( f ) \ 2 =E\ / ( / - / „ ) + /( /„ ) |2
= E | / ( / „ ) |2 “ )])
+ £| / ( f - / ,) |2.
Since E\ I ( f - /„) |2 —> 0 as n —)we have
E \ I ( f ) \ 2 = lim £ | /( /„ ) |2 = lim f
E\ f ( . , t ) |
Now we want to define definite Ito Integral and in this vain let us state and
prove the following theorem.
Theorem III.3 Let {W(r), ^ } be a Brownian motion and let/satisfy (A.III.13)
and (A.III. 14). Define a process {X„ a < t < b) by
X (gm ) = [ f(co, s)dW(a), s) (A.III.34)

Ja
Then {Xt, a < t < b) is a martingale, i.e.
a.s. t > s, E ( X t | &1) = Xs a.s.
Proof First suppose that the integrand/in (A.III.34) is a random step function.
For t > s, let t\, t2, .. ., tn be jump points of/between s and t. Then we can write
X(w,t) - X {(0 , s)= J* f ( 0), s)dW((o, t)
= fo(o»\mo), /,) - WHO, S)] +Ma»lW((0, h) - W(O), r,)] + . . .

) IW(©, t)- W(co, /„)] (A.III.35)
where f 0 is S f - measurable and f k is ^.-measurable, k = 1, 2, ..., n.
By successively taking conditional expectation with respect to ,
_j , . . . , we find E[Xt - Xs \ = 0. Since Xs is obviously
measurable, this proves the theorem when / i s a step function. I f / is not a step
function, [fn] be a sequence of step functions convenging to /in q.m.
Define Xn(a),t) ((o, t)dW(a>, t) (A.III.36)
Then for each n, {Xn(co, t), a < t < b] is a martinglae and hence
E[Xt - Xs | = £[*(., f) - X„(., t) \ j r \ ~ E[X(., s) - X„(., s) \
Since E\ X(., t) - X„(., t) |2 —> 0 as n —> «>, applying Jenson’s inequality for
conditional expectation, E[X (., f) - X„(., r)] - E [X(., s) - X„(., s)] —> 0 in q.m.
as n —»
Hence, £ [X ,-X J | ^ ] = 0 a.s.
One immediate consequence of the martingale property is that a stochastic

integral does not behave like an ordinary integral. Note that if the stochastic
Integral, I W(s) dW( s) behave like an ordinary Riemann Integral, surely it

Jo
must be equal to -(IV (r)2 - W(0)2) = ~ W 2 (t). However, - W 2(f) is not a
martingale. What W(s)dW(s) will be clarified by the so called Ito’s

Jo
differentiation rule.
Ito Formula
Our aim here is to give a feel for simple ITO formula and not to give the most
general formula. In usual calculus we know that for a continuously differentiable
function/(r) on (0, <»).
m=Jorf'W*+m
or /(0 -/(0 )=
Jo
f
f\s )d s .
Can we replace t by the r.v. W(t) and we can write
f(w (t))-f(w m =
Jo
f r (w(s))dw(s)?
The answer is no, but some interesting and useful facts can be proved. Why does
not the formula work in ordinary calculus? This is because of Taylor’s expansion
which says f ( t + At) - f(t) =f' (t) At + f "(t) (At) 2 and (A t) 2 term is too small
and can be ignored. To derive the calculus formula we assume that/has continuous
second derivatives (i.e ./e C2) and £ e (t,t + At). Take a sequence of partitions
n „ of [0, t] with || n „ || = ()max_i (tj+{ - t j ) -> 0, n „ ={0=f0<fi< . • .</*= t}.
Of course the points tk depend on n.
Then
n- 1
fU)- /( 0 ) = .2 [ f ( t M ) - f ( t , )] (Talescopic sum)
n-1 n-1
= 2 /'(< , )[//+i ~ t i ] + 2 1/ 2 / " ( |; )(fI+, - ti )2 (by Taylor’s expansion)
1=0 1=0
As n —>oo, the first term —> I / ' ( s)ds by definition of Riemann integral and
Jo
n-1
| Second term | < 1/2 C 2 (f/+1 - t{)2, where C = sup \f"(s) |
1=0 0<s£t
^ 1/2 Ct || Iln || —> 0 as n —> °°.
For Brownian motion on the average (W(t + At) - W(t) ) 2 behaves like At (not
like (Af)2) and hence can not be ignored.
Ito formula (1) Let/be a real valued function with continuous second derivatives.
Then
0)) + J o
f(W
= f ' ( W( s ) ) dW + ^ J ' f"(W(s))ds.
Remark Left side is the process whose value at (a), t) is f(W(co, t)) and f( W (0))
is a r.v. and/'(VT(j)) is the process I f(co, s)dW(s), where/(ft>,$) = s)).

Jo
This Ito integral makes sense because s —> W((0 , s) —» / (W(a), 5)) is continuous
function of 5 and hence I / 2 (CO, 5)^5 < Further it is adopted. It is jointly

Jo
measurable, because (ct), s) —> W(co, s) is so. The last term is usual pathwise
1 f'
integral I £(**>» s )ds where g(co, s) = / " (W(a), j)) i.e. first keep/" (jc) ready,
given (co, 5 ) calculate W(to, 5) and evaluate/ " at the point jc = W(co, s).
Proof of Ito formula (1)

Fix a sequence {!!„} of partitions [0 = t (an) < t{n) < . . . < t[n) = t] = IT* so
that || n„ || —>0. First fix a t > 0 and then fix F^. Of course the tjn) depends on
n. Then
n m t ) ) - / ( w ( o )) = 2 [ / ( w ^ V ) - /( w ( r ,(n)))]
= 2/'(W '(r< "))) [ H ^ ) - W(r1<n))]
+ 2 l /2 /" ( ^ ) tW ( 4 " ) ) - W(/J(,,))]2 = / + //. (A.III.37)
where lies between points W(aj, /■'” ) and VV( . t-" /). Note that first term I
of (A.III.37) is equal to f f„(s)dW (s), where /„ (tu, tjn))) if

Jo
tjn) < s < t\*i is a random step function for each n. Note that f n(co, s) ->f(co, s)
=/"(VT (ft), 5 )) for each co and each 5 . Then the first term converges in probability
to the r.v. I f ' ( W(s) dW( s). This is obtained under weaker condition
Jo
I |/(ft), t) \2dt < 00 a.s. instead of the condition I E|/(ft), 0 |2 dt< 00. Here
Ja Ja
the stochastic integral is defined by
fib fib
I f ( a) , t ) dW( co, t ) = lim I f n (ft), t)dW(a), t) in probability sense,
*J a n— >00 I
where f n(co, t) is defined by
Appendix 111:Stochastic Integral (lto Integral) 297
f IH<o, 0 |2 d/ < n
fn («0, t) = /(© ,O if *a
0 otherwise
This can be proved as follows :
Cb
Now /( /„ ) = I f n (co, t)dW(co, t) is jointly measurable, for fixed co it is
•b
measurable and f E\ f n (co, t) |2 dt < oo, so that I(fn) is well defined. Now,
rb
for any co such that I | /(co, t) | 2 dt < min (m, n), we have
"a
sup \ f m(CO,t) - f n 1(0, t)I = 0
and that implies
"a r
It follows that for every £ > 0,
.o-r "a
f m(co,t)dW (co,t).
P(I / ( / „ ) - / ( / , ) | > e) min(m. n) 0
as m, rt -4 oo which proves that {/(/„)} converges in probability to /(/), say.

Back to the Proof of lto formula (1),
n- 1
/ / = - 2 l/2[/" (W (f,)) - / " ( | , )][W(rI+1) - W |(t,)]2 + 1/2
M-l
2 / " ( W ( / , ))n v (ri+1) - W(/, )]2 (A.III.38)
i=0
n-\
The first term of (A.III.38) in modulus < cn (co) Z [W(tM ) -
i=0
W(t t )]2, where
cn (co) = 1/2 sup sup |/" (W (fl ) ) - / " ( * ) |, between W(co, t{) and W(co, ti+x).
i x
Note that by continuity of/"(jc) and W(t) paths it is easy to see that cn (co) —> 0
n- 1
a.s. as || fl„ || -» 0 and Z [ W(ti+{) - W(t{)]2 -4 t in q.m. The product
/=0
n-\
Cn(CO) I . m t ^ - W i t t ) ] 2 - ^ > 0 .
i=0
1 n ~ l
nd term of (A.III.38) = j Z r ( W ( t , ) ) { l
n- 1
+j x n m t ' M t M - t , ] (A.III.39)
The first term of (A.III.39) in modulus < C\(co) " l ( W /i+1) - W(t, ) ) 2 t
1=0
where c^co) = [sup/" (* ), x e {W (ct>, 5), 0 < s < t}].

n-\
.2 m t M ) ~ w ( t i ))2- t —> 0 in q.m.
1=0
Therefore the first term in (A.III.39) —£> 0.

The second term of (A.III.39) (being the Riemann sum) converges to
I ’ f"(W(co, s))ds in probability. Now for each fixed t the two random variables
Jo
on either side of ITO formula (1) are equal a.s. So the two processes are equal
a.s. because both sides are obviously continuous path processes.
ITO formula (2) Let / (x, t) be a real valued function on R x (0, 00) with
continuous second partial derivative with respect to x and has continuous first
derivative with respect to t.
Then f (W(t),t)=f(W(0),t)+| £(W
, f' 3 f ( W ( s ) , s ) J, 1 f ' 3
Jo — 3 7 — ^ 5 + 2 j 0 ------- ------------
Proof Proof is exactly the same as formula (1)—only use Taylor expansions for
functions of two variables and a little more messy computation than before.
Hence the details of proof is left to the reader.
APPENDIX IV
Some Important Martingale
Theorems
Lemma A.IV.l Let {X„} be a (super) martingale with respect to ^ , =

SB (Y0, . . . , T„) and Ti
s a stopping time with respect to J%.
Then for all n > k,
£(X„/|r=ltl)(<) = E[XkI{T=k)] (A.IV.l)
E X0)(>)
( = E(XTa„)(>)= E(X„), n > 1
Proof E(XnIl=
k,T) = E{ E[XnIlT=k]]}
= E[I[T=k]E(Xn \ j r ) ] (<)
= E[I[T=k]Xk].
£ ( X Ta„) = So E[XTIlT=ki] + E[X„/,r >n)]
= Zo E[XkI[T=kl] + E[XnI[T>n]]
(> )= 2* £[X „/[r=il] + £[X n/ (7>nJ]
= £(*„)•
For a martingale, EXn = EX0, which completes the proof of (A.IV.2) in this
case.
For a supermartingale we have to prove ) > E(XTa„). We first assume
£ | Xn| < oo, for all n >1.
n
Define X* = 2 [Xk - E( Xk \ ^ _ i)} .T h e n {X*yJ% n > 0} is a martingale
with X0 = 0. Then by (A.2)
TAn
0 = E[X*TAn] = E £ { X k ~ E ( X k|^_,)}
TAn
>E Z1 {x, - x,.,}
k-
= £[X Ta„] - E ( X 0)
Therefore £[X0] > E[XTtJ .

The general case can be obtained by truncation considering the new super-
. , * f X„ if X„ < c
martingale Xn = < , where c is a constant.
[ c i f Xn >c
Theorem (Optional*stopping theorem) A.IV.l

Let {X„} be a martingale and T a stopping time if
(0 P[T<oo]= 1
(ii) E| [Xr |] < co
(iii) n—
lim
>°o v
EX„/(7->n)
[7 ] = 0.
Then E(XT) = E(X0).
Proof For all n >1,
E(Xr) = [EXT/(r<n)] +
= E[XTa„] - E[Xnl(1>n)] + E[XTI(T>n}]
Now E[XT
A
n] = E(X0) by lemma A.IV.l and lim 0 by
n—>°° v '
assumption (iii)
0 < \ E { X T) - E [ X Tf T<n)] |
= | E[XTl(T>n}] |
<E[\Xt \ I(T>
)]
Thus E( Xt ) = lim E[XTAn]= E[X0 ].
n— >°°
Let us state and prove the other optional theorem.
Theorem (Optional sampling theorem) A.IV.2

Suppose {Xn} is a martingale and T is a stopping time. If P(T < <») = 1 and
E[sup| X TAn |] < <*>, then E(XT) = E(X0).
n> 0
Proof Let W = sup | XTAn |. Since P(T < <»),

w>0
XT y?0 XJ ^T=J> ~ j?o XTajI(T=
and that implies \ XT \ <W, and therefore E \ XT | < E(W) < °o

By lemma A.IV 1 we need only to show lim E[XTAn ] = E( XT). We have
n—
>eo
| E[XrA„] - E[Xt ] I < E[
| (XTa„ - Xr) I l(T>n)] <
Now
E [| W |] > E[ WI (T<n)] = i E [ \ W \ \ T [T = j]
oo
2Q£ [| W\\T = j ] P [ T = j ] as n - > ~
= [E\ W\ ].
Appendix IV: Some Important Martingale Theorems 301
Hence lim [IW I l ,T<n)

E ]= E\ IW< «. and therefore lim] =
n—>°° v n—>oo '
By A.IV.2 the proof is complete.
Under the assumption that the second moments of {X„} are uniformly bounded
let us state without proof the martingale mean square convergence theorem.
Theorem (Martingale convergence theorem) A.IV.3

Let {Xn} be a martingale with respect to a - fields {3%} satisfying E[ X„ ] < c < °°
for all /i > 1. Then {Xn} converges as n —> to a limit r.v. XTOwith probability
one and in mean square, that is
P( lim X„ = X oa) = lan d £ [|X „ - X oo|2] = 0
Also E(X0) = E(Xn) = £(X J for all n > 1.

Let {Xt, t > 0} be a continuous parameter martingale adapted to the increasing
sequence of sub cr-fields 3^, i.e. 3 ^ C .Jîf s < t, 3*] a 3 r for all t > 0.
A r.v. T having possible values in [0, <»], is called a Markov time (Stopping
time) relative to {3Jj if, for every t > 0, the event {T< /} is in ^ fo r equivalently
[T > t] E 3^, t > 0. For continuous parameter process, it is not sufficient to
require { T = /} e 3^ for each t.
Let A be a closed set and define T(A), the entry time to Ay to be the random
time
T (A) = inf [; > 0; X(t) e A].
Then T(A) is a Markov time with respect to 3^= 37{X(u)\ 0 < u < t), provided
X(t) is a continuous function of t. But unfortunately the . time to a set A is
not necessarily a Markov time with respect to {3 rt ] if X{t) is not continuous or
A is not closed. We can rectify this defect by enlarging the cr-fields 3 Suppose
X(t) is a right continuous (i.e. lim X(s) for all t > 0) process with left hand limit
sit
(i.e. X(t-) = lim X(s) exists for all t > 0).
sit
Define 3 rt+ = n 3 ^+e . Then each X(f) is -measurable, and 3 r+ c ,

* f>0 1 s t
if ^ < r. Also let 3 £ be the completion of 3^+ (i.e. the smallest a-field containing
3^+ and every subset A of Q for which A e ^ a n d P(A) = 0).
Then for every Borel set A, the entry time
is a Markov time with respect to {3^+ }.

Both the martingale optional sampling and convergence theorems are valid in
continuous time. If (X(r), t > 0} is a supermartingale with respect to {3^}, then
£(X(0)) > E[X(T a r)] > £[X(/)], t > 0, for all Markov times. T. The inequalities
are reversed for a submartingale and equality is achieved for a martingale. If
P(T < oo) = l, then X(T a t) -> X(7) as t ->
Like discrete parameter case we obtain the optional sampling theorem for
continuous parameter martingale.
Theorem (Optional sampling theorem) A.IV.4

Let {X(/): >0} be a martingale and T is a Markov time. If
P (T < oo) = l and £[sup| X(t) |] < «>, then
/>o
E(X(0)) = E(X(T)).
For more discussions about both discrete and continuous parameter martingales
readers are advised to consult excellent books by Doob (1953). Brieman (1998),
Chow and Tiecher (1990) and many others.
Note: Martingale Convergence theorem A.IV.3 is valid under much less restrictive
condition sup E | Xn | < «> (see J.L. Doob).
Solution to Exercises
Chapter 1
Exercise 1.1 Let X - 4acandY= 0< < 1.

Then FY(U) = {bP 2< u) =P(b< J u ) =Vm fo
and Px (u) = P(4ac < u) = f P(ac 1 or •—> u
Av 4
cu f i r u
Hence Fx(u) = I 1 • dv + I —dv = ~ 1 - log *j , 0 0) = P(X < Y) = f

Jo
P(X - K < 0 | m < K < w + da] dFY(u)
36
Exercise 1.2 P(X P(X = f c | w < A < 4 + du) P(u < A < u + dw)
Jo
= f e- ^
Jo * ! V/t
4 = u " - 'e -udV un =V F T T = (1/2)*+".
Exercise 1.3 Joint probability distribution of X and T is
jc!
pp y (ui - ^P y - y =£ ± f-£ _ V
y! U -p j (*-y)!
\y
Therefore P(y) = — - A*(l -/>)*

y! V l - p J -t=y ( x - y ) !
= £-r- t —
>! U
f y 2-P)x=o a -
= i— (Ap)y,y = 0 , 1 , 2 , . . .
( n
Exercise 1.4 The joint Ch.f. of Xtr . .. Xtn = E exp / ?
7=1 7
Now I 0 ,X , = (0! + 02 + . .. + 0„) + (X,2 - X,j)(02 + . . . + 0 J

7=1 J
Since {X(/)} has independent increments,
exp i 2 0,-Xf/ = E exp *' S 0, Xri

7= 1 7 7= 1
x E exp i 2 0,(X ,2 - X „ ) E [exp (idn( Xtn - Xtn_x))]

7=2
( n \ f
0 X 0 j,t\ 0 ^ 0/» (^2 ~ *i) • • • 0(0n^ n ~ tn- 1)).

^=1 ) V 7=2 7
Exercise 1.5 Let /0 < < . . . < * „ < fn+, e T, the index set and let A be an
arbitrary subset of the real line. Let {an} denote the set consisting of the single
point an
P[Xtn^ £ A | X,. —an, Xtft_i —dn-\"> • • •»X/q —a0]
= P[(X,„+1 - X/fj) G A - {an} | X,„ = «„, X,„ - X/n_, =
~~ &n- 1* • ’ •’ Xtl —XtQ— —CIq]
= P[(Xtn+l- X tn) e A - { a n} \ X tn = an]

(due to independent increments)
= E[X,n+1 e A \ X tll= a H]
Exercise 1.6 Suppose T has a geometric distribution given by

P[T = k\ = pqk' \ k = 1, 2, . . . , 0 ;c + y ]= I P[T = k] = q x+y

£=.v+y+l
So P [T > x)P [T > y] = q xq y= = P [T > x + y]

Conversely let pn = P[T
then qn+x= P[T> n +1] = P[T >

l ]/>[r> = = ...=
So pn = P[T > n- 1] -P[T>n=
n- 1 n rt-l/i\
= =<h 0 - q\)
which is geometric.
Exercise 1.7 Suppose that T has a exponential distribution given by

P[T > x] = e~**, x > 0, A>0.
Then P(T > x)P(T > y)e'Xx ■e~Xy = = P( T>
Conversely let g(x)= P[T > *] then g(x + y) = g(x)g(y), T is continuou
g(0 ) = 1. This is a Cauchy functional equation and the solution is given by
g (x) = e~Xx for some A > 0.
Exercise 1.8(a) E [,e~ex{T

)] = E[E{e~ex(J) = f)]
E[Q-sx(T) T=t ] dF{t)
but E [0 -“ ( n |
Hence, £ [ 0 - “ <r >] = ( 1)
(b) If T is negative Binomial then
p+k- r ( Ap
E[ e ~sx(T
)] = £ (iT—f ° ^ Pp (1 - P ) k =
k=o yA + s j ^ Ap + s
which is the Laplace transform of a gamma distribution with parameters P and

Xp.
Conversely, if X(T) has a gamma density with Laplace transform
P
E[ e - SX(T)
p +s
(by (1)) ( 2)
Put = *•-". Then s = A[eu'a - 1],

A+s
So fJo e~udF(t)=
H + X[ e ula- 1]
(by (2))
P
pe -u/a
where 0 < p = -j- < 1.
1 - qe -u/a
This is the Laplace transform of a geometric distribution given by

P+k- 1
P(aT=P+k) = pP (1 - p) k where/? = -—.
Exercise 1.9 P[Y *y] = Jo

f P[Y < y | X = x\f(x) d)c, (since Y \ x is uniform)
= f" f ( x ) d x + |f - f ( x ) d x
Jo Jv
0 if y < 0
Hence fr(y) = J ^f-dx ify > 0 ( 1)
P(X - Y < u) = f
J U
P [ X - Y 0
Thus Y and X - Y are independent and identically distributed. But X = Y +

(X - Y). Let 0(t) be the characteristic function then
0x(t) = M *) 0x-y(O = [<M 0]2 (3)

We have to find 0 y(r).
dy (from (1))
■f^fr e"ydy dx
e',x- 1
= if f ( x ) dx (by Fubini Theorem) (4)
" Jo
fix)
So it (fry (t) + i(j)y(t) - fJo
Jo
[xeltx] dx (differenting the expression in (4)).
Hence t(f>y(t) + <t>y(t) = f

Jo
f ( x ) [ e ltx] dx = (j)x( t ) (5)
Therefore, by (3) and (5) we get t(j)y(t) + <t>y(t) = <f>2 (t) which is a linear
differential equation. The solution is (t>y(t) = 1 with initial condition

1 - At
<t>y(0) = ///=> A = Ha. So <px(t) = 1 a=—

(1 - Hat) 2 V
which is the ch.f. of gamma distribution given by
f(x) = a 2xe~exy a > 0.
Exercise 1.10 EY(t) = EZ(t) = 0, and since sin2 t + cos2 t = 1 for all t e T,
E Y 2(t) = E Z 2{t) = 1. Since Y{t) and Z{t) have a joint normal distribution, it is
enough to show that E{Y(t)Z{t)) = 0. Y{t) and Z(t) are independent normal having
zero mean and variance unity for all t E T. Therefore X{t) has same distribution
as X(t + h) for any h. Here the bivariate process X(f) = (Y(t), Z(t)) is strictly
stationary.
Exercise 1.11 EX(t) = 0, Cov (X(t), X(s)) = E(X(t)X(s))
= E(A2) cos Xt cos (As) + E(B2) sin (Xt) sin (As)
+ E(AB) cos Xt sin (A^) + E(AB) cos (As) sin (Xt)
= <j2 (cos (Xt) cos (Xs) + sin (Xt) sin (As)) = cr2 cos X(t - s)
Obviously (X(r)} is a covariance stationary process.
Exercise 1.12 E Z t = 0, cos (Z„ Zy) = S cr2 cos XAt - s), t * s.

7=1
Therefore {Z„ t > 0} is covariance stationary. Since given upto second order
moments is not enough to determine joint distribution of a stochastic process,
we can not say whether or not the process is strictly stationary.
In 2n
Exercise 1.13 E{X(t)} = f
2n J c
sin <ot dco = H2 n cos cotlt = 0
In
E{X(t + j)X(j)} = ^ Jq sin co(t + s) sin cos dco
_ j_ c2n
[cos cot - cos co(t + 2s)]dco
Jo
In In
1 1 . 1
y- - sin cot sin co(t + 25)
4n t t +2s
= 0 i f * , j = 1, 2, 3, . . .
But E(X(t + s)X(.y)) = f ( t , s) * 0 if t, s are not integers, w here/is a function

of t and s but not of t only.
So (X(r), t = 1, 2, 3, . . .} is covariance stationary.
Now Cov(X(/), X(t + 5)) depends on t and 5 not on | / 1only if we consider the
process (X(r), t > 0}. Hence it is not even covariance stationary and therefore can
not be strictly stationary.
Exercise 1.14 Use trigonometric identity to evaluate
IT if n = 0
E(Xn) = £(cos nU) = \
r
E(XnXn+v) = I cos nu cos (n + v) udu
* -n
r
= [cos (2n + v) u + cos uu] du
Jo
= 0 if n > 1.
Therefore, {Xn} is covariance stationary.
Similarly calculate the third moment to determine
E(XnX n+v Xn+v+[) depends on n.
Hence {XJ is not strictly stationary.
Exercise 1.15 Now Y(t) = Yn for (n - 1)T < t < nT
Where Yn)s are i.i.d. with P{Yn= ± 1) = 1/2
Since U and Y(t) are independent.
E(Y(t - U ) \ U = u) = E(Y(t- u)) = 0.
Consequently, m(t) = EX(t) = E[Y(t - u) | U - u] = 0
Therefore Cov (X(s), X(/)) = E(X(^) X(t)).
The time axis divided into successive intervals, it follows from the independence
assumption on the r.v.s Yn that Cov (X(j), X(t)) = 0 if | t - s | > T. Let | t - s | >
T. Since U is uniform in (O, 7), it follows that E(X(s), X(0) = 0 and consequently
E(X(s)X(t)) = E(E[X(s)X(t) | u]] = 0. Now first \ t - s \ < T and 5 = nT. Then t and
s will lie in the same interval if U + [t - s | < T that i s U < r - | r - 5 | and this
event occurs with probability (T - | t - s | )/T. Therefore
1- \t-s\/T if\ t - j | < T

E(X(s)X(t))
0 \i\t-s\> T
Hence X(t) is wide sense stationary.
Exercise 1.16 E(X(t)) = E(Y)E(Z(t)) where Z(t) = (-\)m

E(X(t)X(s)) = E(Y2)E(Z(t)Z(s)) = E(Z(t)Z(s))
Now Z(t) takes values +1 and -1 with probabilities given by
P{Z(t) = 1) = P(N(t) is even) = e A/ S ^ - e Xt cos htX

n=o y£n)\
and P(Z(t) = - 1) = P(N(t) is odd) = e~A' sin htX

Therefore EZ(t) = e~2Xt and hence
E(X(t)) = 0 ■e~2A' = 0. For t > s
P{Z{t) = 1, Z(s) = 1) = e~x v cos /zsA e~X(t~s) cos /iA(r - s)
= P(N(s) is even and N(t - s) is even)
P{Z(t) = - 1, Z(s) = - 1) = e~Xs sin htZe~X(t~s) cos hX{t - s)
= P(N(s) is odd and - s) is even)
P(Z(7) = - 1, Z(s) = - 1) = P(N(s) is even and N(t - 5) is odd)
= e~Xs cos hsX e~Xit~s) sin /iA(/ - s)
P(Z(t) = 1, Z(s) = - 1) = P(A(s) is odd and /i(f - s) is odd)
= sin h{sX)e~X(<t~s) sin hX(t - s)
Therefore E(Z(t)Z(s))= e~2X(<t~s) when t < s and
E(Z(t)Z(s)) = e2A('~v) when t > s.
Hence Cov (.X(s), X(t)) = E(X(s) X(t)) = e~2Xjt~xj.
Thus, X(t) is covariance stationary.
Exercise 1.17 If H(t) is known then we have
f N\
P(Xt -
k
To find H(t), partition (0, t] into n subintervals of equal length A* = tin. By
assumption, the average number of germination in (0, At] = N(l - p)XAt. So the
number of nongerminating seeds at time Aris N{\ - p) (1 - AAt). Thus the number
of nondegerminating seeds at time t is approximately N{\ - p)( 1 - AAO”. Hence
for sufficiently large rc,
P(T< t) - {1 - (l-AAt)"l(l - P) = { l - d - A f / n r i d - p )
By letting At —> 0, we obtain
' N^
H(t) =(1 - p)(l - e~and [(1 - p)(l - e~Xl)]k
'/
,-A/\ 1n-k
[i-d -rtd -n i
Chapter 2
Exercise 2.1 From the directed graph we see that the state space decomposes
into two classes C\ - {1, 2, 3} and C2 = {4, 5}. Also for i e Cj and j E C2 we
see that i -/> j or j -A i. Therefore C\ and C2 are closed sets of states.
1.0 ( 2)
The state spaces decomposes into C = {1, 2, 3} and A = {4}. Hence C is

closed and A is not.
Exercise 2.2 State space of the M.C. is given by S = {0, 1, 2 , . . . N}. *„+1 = 0
if Xn = 0, Xn+l = min {N, Xn + Jn+X- 1} if Xn > 0, where Jn+l = number of new
jobs arriving during nth service period. Then the transition matrix in its cannonical
form is given by
1 0 0 0 0
Po Pi Pi Pn-i Pn
0 Po Pi PN-2 Pn-i
0 0 0 Po P\
State “0” is absorbing and others are transient.

Exercise 2.3 (Bartky’s sampling inspection scheme)
f N "
P(Xk = j ) = Pj = p j+'( 1 - P ) N-J~'.
J + 1y
Obviously the conditional distribution of Sn+X given S2, . . ., Sn is a Markov
chain with states space {0, 1, . . . , a, “> a'} and the transition matrix is given
by
1 0 0 0 0 0 0
P- 1 Po Pi • Pi 0 0 0
P=
0 0 0 0 P-i Po 1 -Pi- 1
0 0 0 . 0 0 0 1
States “ 0” and “ > are absorbing and other states are transient.
Exercise 2.4 Let Xn be the number of customers present at time n. Since the
number of customes present at time n + 1 equals the number of customers

present at time n less the one served at time n (if there is any), plus the number
of customers arriving in time interval (n, n + 1) if the total does not exceed m
and equals m otherwise, we have Xn+] = min {Xn + 50)A/l - 1 + %n, m], where
Sjj is the Kronecker’s delta. Since Xn+x depends only on Xn and and Xn and
%n are independent, {Xn}„>0 is a M.C. with state space {0, 1, . . ., m\. Put
P{ + p{ - pM + . . ., 1 < I < m. Then
P ( x n+, = j\X „ = 0) = P(£n = j \X n = 0 ) = Pj if o <y < - 1

Poj =
P(£n ^ m )= Pm + Pm+\ + ■• • =Pm
'O' =
P(X„+i = j I Xn - i) = P(^n = j + l - i )
= Pj-M i f f - 1 < y '< /M - 1
For 1 < j < m, Pij = P(x„> m + 1 - y) = Pm+H ify = m
0 otherwise
So the transition matrix is given by
0 ' 0 1 2 m- 1 m
1 Po P\ Pi Pm-\ Pm
2 Po P\ Pi ••• Pm~\ Pm
0 Po P\ Pm-1 Pm-\
m- 1 0 0 P\ Pi
m 0 0 Po P\
Exercise 2.5 (Storage mode 1)

Like the previous exercise (Queueing) it is easy to see that
x "+i = Min (X"+ I + + 7]„+1 - 1, 1).
Hence X" is a M.C. with state space {0, 1, . . . , m - 1}.
Poo = / w +1 = oI X"= 0) = P[(r]n+] 0) u (r?n+1 = 1) | X''= 0]
(since X'n' and r]n+l are independent) = Po + p\.

Similarly
Poj = / W +l =;• I 0) = P(itf+l = j + 1 I x ; = 0) = p7+1, 1 <7 < m - 2
P0,m-1 = 0?7„+i > W I X"= 0) = Pm, where Pm = X p* plp 1 < f < w - 1

k=m
are calculated like Exercise 2.4 (Queueing). Hence the transition matrix is given
by
0 1 2 m- 1 m
0 > 0 + Pi P2 P3 Pm-1 pm
1 Po Pi Pi Pm-2 Pi 1
P= : 0 Po Pi Pm-3 Pm-2
m- 1 0 0 0 Pi Pi
m 0 0 0 Po Pi
Exercise 2.6
= i’n-1 1 ^ = 1 A^n+1 = 'n+1' • • .)
P(Xn- «= *2i-b Xn = *'n>^n+1 = *rt+l> •

P(Xn —ifj, —/w+i, . . .)
“ ifi+\ ’ Xn+2~ in+2» ••• |^/2—*n» Xn-l = ln-OP(Xn —im Xn-\ —ifi—\)
P(Xn+\ — Xn+2 = in+2»•••> I Xn —in)P(Xn —in)
= = V i | Xn = /„) [since PCA^j = /n_b Xn+2 = /„+2, • • . |
Xn = Xn_x = i„_!) = P(X„+1 = /„+!, Xn+2 = Jw+2, • • • I Xn = /„)].
Exercise 2.7
fcxl
f n r n
i i
pkxk ,&xl
= A= 1
V 1 V
is an eigen value of the stochastic matrix P of order k.
If A is an eigen value of the stochastic matrix Pkxk = (ptj) and u kxX = is a

\ uk
left eigenvector corresponding to A, we have Aw, = Z u tp ih 1 < j < k , whence
2= 1
S |A«y| = 2 | 2 UiPt | < 2 |U|| 2

j=i 7 =i /=i i=i 7=1
=2 \ I=> |A I 2
ju |m7 I < 2 |m, I => |A I < 1.
2=1 7 =1 2=1
Exercise 2.8
Clearly X0 = r,X n = 4 +1 + X„+1, n > 0 . . . (1)
The probability that among i susceptables k of them (k < i) have contacts with
infectious individuals equals p kq l k (q = 1 - p). Hence
P(Xn+] = j \ Xn =i) = P ( 4 +1 =X n - j \ X n = i)=
. p ifi>j
P(L+i = i - j \ X „ i) = i w ;
0 if i < j
and obviously, the distribution of X„+, is independent of the values assumed

by the r.v.sXm
, m < n.Therefore {X„, n>
0}
2, . . . , rand transition matrix
0 1 2 0 k
1 0 0 0
p q 0 0 0
P2 2 pq 42 0 0
( r\
pr r pqrX p r 2q 2 .. qr
v2.
The duration of the epidemic is the smallest number r such that ^r - 0 i.e. the
moment t when there are no further infectives i.e. the smallest number T such
that Xr_i = Xr using the equality X„ = £n+1 + Xn+l. Hence the number of cases in
the epidemic is §) + X0 - XT= §) + r - Xr
Exercise 2.9 By chain rule of conditional probabilities
P(Xm — in, X k—ik, Xm—

P(Xn= i„,Xk = = im)=
P( Xm = im)
PjXn = in |X* = i^Xjn = = 1 =

P(Xk =
P(Xm = im)
P(X„ = i„ |Xm = iJP fX n, =|X,=^ )P (X , = ik)

P(Xm = im)
= P(X„ = in I Xm = im)P(Xm = im
X k = ik)/P
= P(X„ = inI Xm= im)P(Xk = ik| Xm=
Exercise 2.10 (a) P(Xn = in | X„+1 = in+]----- - Xn+k = in+k)
~~ P{Xn—Xn+1= in+\, . . ., Xn+k —in+k)/P(Xn+j= irt+j, . . , Xn+k —
= [P(Xn+k = in+k | X„ = . . ., Xn+k_| = w ,) P ( X n+*_, = in+k~\ |
= t'n* • • ■’ Xn+k_2 = in+
)k-2 ’ ' - ^ ^ n+l = '"+1 I =
x P(X„ = t'n)]/[P(Xn+^ = in+k| X„+1 = /n+], . . ., Xn+k_\ =
Xn+1 = t’n+1. • • •> Xn+k_2 = in+k-■■■P(Xn+1 = /„+])]
By Markov property and cancellation, the last expression

= P(Xn+\ - hi+1 I Xfi = in) = in)/P(Xn+1 = ^+l)
= P(Xn+1 “ in+b Xn ~ in)fP(Xn+1 = ^+l)
- P(Xn = in I Xn+\ - in+1)-
(b) We shall prove this for m = 2 only. Since P(AB | C) = P(A | BC)P(B | C),
P ( X , i k+2 = = *'U + ! I = Xj' ^ - j - ^)
~ P{ X nk+2 ~ Xnk+2 I Û+l = ^ - J - ^)

x P ( X nk+]= x n M \ X n. = x p l < j < k )
By Markov property the last expression

= P (X nk+2 - x nk+2\ XnM = x nk+l ) P( X nk+1 = x nk+x | X nk = **)
= P (X nk+2 = *,7*+2 » = ■***+! | X nk - Xk)
The proof for general m is exactly similar and hence is omitted.
Exercise 2.11 It is easy to see that
ft m = \
are stochastic matrices. We have to prove that

i n
lim — 2 P m = Q (say), exists. (0)
n->o* n m=\
Since the elements of the stochastic matrices are bounded by 1, every sequence
of Stochastic matrices contains a convergent sub-sequence. To show existence
of Q it is sufficient to show that the convergent subsequences of the averages on
the left all have the same limit.
Suppose for some convergent subsequence nx < n2 < . . • .
! nk
lim — X (1)
p m =A
*->°° ft m=\
Multiplying both sides on the left and right by P we get

n k +\
lim — X P(2)
»oo nk m=2
Since lim Pnk+l/nk = 0 = lim P/nk (elements of powers of Pare bounded by 1).
k—
By (1) and (2) A = AP = PA (3)
i " 'i f 1 " s\

Therefore A = APn = PnA = A - X P m - X Pm
l " m=] <n ”’ ' J
for all n > 1.
Hence, if B is any limit matrix of a subsequence of averages in (0),

A = AB = BA (5)
Since A and B are arbitrary limit matrices here, they can be interchanged giving
B = BA = AB. (6)
By (5) and (6) A = B. So there is only one limit matrix.
Thus, by (3), A = QP = PQ and taking limits n —» °o in (4), we get Q = Q2.
Exercise 2.12 Suppose that C is recurrent but is not closed, then there exists
i e C such that £ p tJ < 1 and hence there exists k £ C such that pik > 0, i.e.
i —>k. State i being recurrent we have i <-> k, hence k e C. This is a contradiction.
Hence C is closed. Conversely, suppose that C be a closed class of states. Since
the state space of a finite M.C. contains at least one recurrent state, the sub-
Markov chain with state space C has atleast one recurrent state. But recurrence
is a class property, hence C is a recurrent class.
**Exercise 2.13 (a) We proceed by induction on r. For r = 1 this is true by the

very definition of f n. Assume it holds for r = m - 1.
Write Pi (A) = P(A | X0 = i) and Tt = min (n > 1 : Xn = i)
Pi(Xn = i for at least m - 1 values of n > 1)
= £ Pi (Ti = k, XT.+; = i for at least m - 1 values of j > 1)

*>i '
= £ Pi(Ti = k)Pi(XT+j = i for at least m - 1 values of j > 1 I Tt = k)
k> l '
Now Pi(XTj+j = i for at least m-1 values of j > 1 | Tt = k)
= Pi(XTj+j = i for at least m - 1 values of j > 1 | XT. = /, Tt = k)
= Pi(XT.+j = i for at least m - 1 values of j > 1 | X T. = i)
(by strong Markov property of M.C.)
= Pi(Xi = i for at least m - 1 values of j > 1)
= [fu]m~l (by induction hypothesis).
Therefore Pi(Xn = i for at least m values of n > 1)
= 2 = = [/„]'"
*>1
(b) Proof is entirely similar to that of (a) using Ti;- instead of T{ when
Tij = min {n > 1; Xn = j given that X0 = i }.
(c) Let A = {The Markov chain returns to i infinitely often}
= n {The Markov chain returns to i at least r times}
r= l
= lim {The Markov chain returns to i at least r times}

316 In tro du ctio n to S to ch a stic P ro ce ss
B y (a )
f l if/,, = 1
E x e r c is e 2 .1 4 (a ) F ir s t p r o v e th a t ( b y in d u c t io n o n n)
P I[ v ( j ) > n ] = f iJ[fj j 1r (1 )
For n = 1 , Pilv(j) > 1] = / i; ( b y d e f i n i t i o n )

For n = 2 , Pt[v(j) > 2 ] = fljf jl
S i n c e Pt [ u ( y ) > n + 1] = / / 7{ / 7/ ] n, i n d u c t i v e l y w e o b ta in ( 1 ) . W e h a v e
Pi [ v ( j ) = n ] = 2 Pt [v(j) = k ] - 2 P,[vU )2k]

k>n k>n + 1
- Pi[v(j)>n]~P,[v+ 1]
- fi jU ij V A 1 -/,./]• (2)
(b ) fi(k, i) = 1 Pk [v(i) > n] = 2 A,
~ f ki <oo if o <fu< 1, i.e . i f i is tr a n s ie n t .

1 -fa
F r o m e q u a t io n ( 2 )
pk ( v ( i ) < CO) = f kl ( 1 - A ) 2 [A ] " - 1

«=i
= ^ r r r L = 1 > f / y < 1 a n d A , = 1.
1 Jij
(c ) O b v io u s fro m (b ).
E x e r c is e 2 .1 5
2 P (n)= 2 2 f (k)p in- k ) = 2 /- a ) 2 p (n~k)

n= \ n= 1 / = 1 n=k
H ence I /<*> > 2 p<;> > 1+ 2 (A:)

U ( 1)
n —\ M=1 JJ
J V
fc=l
f o r a ll ra' < m. T h e la s t i n e q u a l it i e s h o ld b e c a u s e
1+ t p ™ * 2 P)?~k) = 1 + 2 "
n—\
f m-m'+k \
An-k)
and 2 (/<*>) 2 > 2
*=1 n=£ JJ &=1 (^ } £ ^
f m -m ' \ ( m’ \
1+ L
n= k
V
m
Dividing both sides of (1) by 1 + Z p-n) and letting first m —> °° and then
n=1
m —> «> yields the result.
The last part is trivial and is omitted.
Exercise 2.16
Pa = for i = 0, 1,. . ., k
k+ 1
i + 1 •r • ■
P" = I 7 T ,fy<1
= 0 if j > i
From the relation Xn = Xn+l + (Yn - Xn_\)+it is easy to see that Xn is a Markov
chain. Since Pkk- U k is an absorbing state.
From the transition matrix
1 1 1 1 3
k+ 1 k+ 1 k+ 1 ’ ' k+1
0 2 1 1
k+1 k+ 1 " k+1
0 0 3 1
k+ 1 " ' k1
0 0
0 0 1 ,
It is clear that j —>k but k -/> j for j < k and the chain is finite. Hence all the states
0, 1, 1 are transient and k is positive recurrent.
Sincepu > 0 for all i = 0, 1 ,2 , . . . , k all the states {0, 1, 2 , . . . , k] have period
one though the chain is not aperiodic (since not irreducible).
Exercise 2.17 (Polya urn Scheme)

Here P(S0 = a) = 1. After the nth drawing there are a total of a + b + n balls in
the box. If i of these are white, then the probability of drawing another white ball
is -----
a + b\-----
+ n . Hence np . . +i = ------ }------ . Also
nri,j a +if£ Sn
+ n= i then Sn+1 =l i means
nn+
a black ball will be drawn in the next n+ 1th drawing. Since there will be
a + b + n - i black balls in the box
a +b +n - i
n P i , j ~~ P { S n + 1 ~ i \ S n ~ i ) ~
a +b +n
It is obvious from the nature of transition probabilities that the past history of
values of Sm for m < n does not influences the conditional distribution but does
depend on n. Obviously np i} - 0 if / ^ /, i + 1 and it is an nonhomogeneous chain

since transition probabilities depend on n.
Exercise 2.18 The states space of the M.C. is {1, 2, 3, 4, 5, 6}. The transition
matrix is
1 ( 6l 1 1 1 1 n
62 6 6 6 6
2 0 1 1 1 1
6 6 6 6 6
3 0 0 2 1 26 26
6 6
4 0 0 0 4 2 26
6 6
5 0 0 0 0 5 2
6 6
6 0 0 0 0 b
We obtain, for example, the third row of the matrix as follows. Suppose the
system is in state 3, i.e. the maximum of the numbers occurring on the nth toss
is 3. Then the system remains in state 3 if a 1,2 or 3 occurs on the (n + l)th toss;
hence p 33 = - On the other hand, the system moves to state 4,5 or 6 respectively,
if a 4, 5 or 6 occurs on the (n + 1) the toss; hence /?34 = p35 = p36 = ^ The system
can never move to 1 or 2 since a 3 has occurred on one of the tosses; hence
p 31 = p32 = 0. The other rows are obtained similarly. The matrix is not regular
since state 6 is absorbing and there is a 1 on the main diagonal in row 6.
Exercise 2.19 (See Exercise 2.26)
Let Yn be the content of the dam at time n and Xn is the input at the nth period.
Then
yn
+ 1= min (*. y
n+-1
Hence, clearly {Yn} is a Markov chain. The transition matrix is^
0 q + qp qp2 CP* ■■■ dP Pk

pk-x
1 q qp qp2 ... qp*-*
p k-2
2 0 q qp ■■ qp™
- 1 0 0 q P
Exercise 2.20 The transition matrix of the M.C. is given by

0 0 0 0 0 1 ^
1 J_
0 0 0 0 0. . .
2 2
1 1 1
0 0 0 0
3 3 3
1 1 1
0
M - 1 M - 1 M - 1
1 1 1 1
M M M M J
This is a finite irreducible chain and the chain is positive recurrent. Hence a
stationary distribution exist.
The stationary distribution is given by
(7Tj, 7thy . . .
» n N t)
' 0 0 0 0 0
0 1 1
0
1 !
0 0 0 0
2 2
0 0 0 0 0
I I i
X
3 3 3
1 1 1 1
0
M- 1 M- 1 M- 1 M- 1
1 _1_
I m M )
' K\ '
n2
Thus, tci = n M!M, n 2 n M-1 nM

M- 1 Af
= 7r2/2 + 7T3/3 + . . . + k m _ xI M - 1+ ti m /M ,
TTyv/ = 7T{ + 7^2 4- 7T3/3 4- . . . +
Hence
/Tj = /Tm/M, 7^2 = 7lx + 7tM_x/(M - 1), 7T3 = /^ + Km-iHM - 2), . . . , /%_,
= ^M-2 + ^2^2, n M - 7lM_i + 7T].
Therefore taking 7TM= 1,
1= jj, n M. x= Kn2 =
distribution is given by
n' M { M + \ y 2n ( M + D M ' 7*3 M ( M + i y " ,7tM
Exercise 2.21
f 0 1 0
p = 1/2 0 1/2
v0 1 0
The chain is irreducible as the transition matrix P is so. We have

1/2 0 1/2
0 1 0
1/2 0 1/2
In general P2n= P2, P2n+l = P, so that p i2n) > 0 , p (2n+l) = 0 for each
states are periodic with period 2.
We find that f l\l) = O ,//^ = 1 so that f u = S f^\n) = 1, i.e. the state 1 is
recurrent. Since the chain is irreducible, the other states 0 and 2 are also recurrent.
Being a finite chain all the states are positive recurrent.
Exercise 2.22 Here {Xn} is a homogeneous two-state Markov chain with
p ^
transition matrix P = ( Q . It can be shown by induction or otherwise
VP <7 )
that
1/2 + 1/2(q - p ) m 1/2 - 1/2(q - p) m

1/2 - 1/2(<7 - p ) m 1/2 + l/2(g - p ) m
' 1/2 12
/ '
and lim P m =
m—>°° 1/2 1/2 ’
Since the initial distribution is given by

P(X0 =0) = a =1 - P(X0 = 1),
P(Xm = 0, X0 = 0) = P(Xm =0 | = 0) P(X0
and P( Xm = 0 , X 0= 1) = (1 -« )P ,(0m)-
By Bayes Theorem
P(xo =o I Xm =0)
= ______________ P(Xm = 0 |X q = 0)P (X q = 0)______________

=0 ) P( X0 =0) + = 0 1X0 = D /’CXo = 1)
« /4 m)
(m)
“Pm ) + (! ~ a)P 10
a[1/2 + 1 /2 (9 - p ) m}_____________
_
a{ 1/2 + 1/2(9 - p )m
} + (1 - a){ 1/2 - 1/2(9 - P)m >
a {l + (9 - p ) " }
1 + ( 2 a - l ) ( p - q ) m'
Exercise 2.23 Let /v(z), z e S, denote the indicator function of the set {y },
defined by
Since /^(X,,) = 1 if the chain is in state y at time n and Iy(Xn) = 0 otherwise,
N( y ) = 2 / , (
n= 1 y
Now P(N(y) > 1 | X0 = jc) = pxy. Let Ty be the first passage time to the state y.
By Markov property P(Xn+l = xn+u,Xn+m = xn+m =
= pXn Xn+l , . . . p Xn+m_\ ,xn+m- Hence the probability that a Markov chain starting at
x first visity at time m and next visit yn units of time later is
P(Ty = m \ X 0 = x) P(Ty= n \ X 0 = y).
Thus,P(N(y) > 2 | X0 = *) = 2 2 P(T n =

m-1n=1 ' J
= 2 P(Ty =m
| X0 +
m=1 7 n=\ J ' •'
By induction and Markov property

P(N(y) > m+ 1 | X0 = x) =P(N(y) > m \ X 0 = x)>1|
= PxyPy^' Pyy = PxyPyy (2)
Now E ( I , ( Xn) \ X 0 =x ) = P( Xn = y \ X 0 =x ) =.
Hence by (1) E (N (y )|X „ = x) =£ ^ 2
(Monotone Convergence Theorem)
= I E ( /V(X „ )|X 0 = x) = L l ; > (3)

n=l rc=l
Since P(N(y) = m \ X 0 = x) = P(N(y) > m \ X 0 = x ) - P(N(y) 1 | X0 = It

follows from (2) that
P(N){y) = |mX 0 = * ) = p ,yPy""‘(l - 1.
Let G (X ,y) = £ (N (y )|X 0 = x) = 1 mP(N(y) = m \ X0 = x)

m=1
= J , mPxyPyV 0 - Pyy )
Substituting t = pyy in power series

1
Z mt n
m=1 (1 - o 2'
we conclude that
Px
G( x, y) = < oo if 0 < pyy < 1, i.e. y is a transient state.
1 -Pv
Hence by (3)
lim p {xny = 0 if x e S and y is transient (4)

If S is finite and all states are transient, then by (4)
0= I lim £ p[y = lim £ = lim P(X„ e S \ X0 = x)

yes yeS - yeS - /i—
= lim 1 = 1 which is a contradiction.

n—
Hence the Markov chain must have at least one recurrent state.
Exercise 2.24 By Chapman-Kolmogorov equation
.(n+m) _ y (m) (n)
jj j t s ? Jj P j j
I -""I ( 1)
- p ' - ’r ) ? *p'"j' h . p»
-= Pn(W>J n) p\T = i
JJ JJ + Pj j ( l ■p(r
J£S
+ d -p\V)-
Exercise 2.25
Parent aa x aa aa x Aa Aa x Aa AA x aa
mating AA x AA AA x Aa
Progeny
mating
aax aa
AAxAA 1 ■
1/4 1/8 0
aa x Aa)
0 1/2 1/2 0
AA x Aa]
Aa x Aa 0 1/4 1/4 1
AA x aa 0 0 1/8 0
The latent roots of the matrix P, i.e. the solution of
| p = A/ | = 0 are given by (A - 1)^ A - —j(A 2 ~ A/2 - 1/4) = 0
or A! = 1, A2 = 1/4 + 1/4V5, A3 = 1/4 - 1/4^5, A4 = 1/4.

4
From spectral decomposition theorem p r = X A/"A,.
So the proportion of heterozygotes in rth generation is given by
hr = >4(1/4 + 1/4V5)r + £(1/4 - 1/4V5 V + C (l/4 )r
and determine, A, B and C from the values of hQ,h\, and h2 which are 1,1/2 and
1/2. This gives
A = 2 /V 5 (l/4 + 1/4V5),
B = -2 /V 5 (l/4 - 1/4V5) and C = 0.

Thus we found
hr = 2/V5 {(1/4 + l/4 V 5 )r+1 - (1/4 - l/4 V 5 )r+l}.
Exercise 2.26 Let k be the finite capacity of the dam and Zn be the storage at
time n i.e. the amount of water in the dam before the input Xn flows into it.
Then if Zn + Xn> k, an amount Zn + Xn- k will overflow, but if Zn + Xn< k, there
will be no overflow, the dam now contains a quantity min (k, Zn + Xn). At time
n + 1, an amount of water m{< k) if Zn + Xn > m or Zn + Xn if Zn + Xn < m is
released from the dam. The release is thus Yn = min (m, Zn + Xn).
Hence Zn satisfies the recurrence relation
Zn+\ = min(k, Zn + Xn) - min(m, Zn + Xn) (n = 0, 1, 2, . . .) if m > 1
= min(/:, Zn + Xn) - 1 if m = 1.
Thus, {Zn} is clearly a Markov chain. Let {gj} be the probability distribution of
Xn with P.G.F
G(z) = ^ 0 8jZJ'I z i < 1

and transition matrix of {Z„} is denoted by P = (p = P(Zn+l = j \ Z n = i). {Zn}
is a finite state M.C. with states 0, 1, 2 k - m and the transition matrix
(m > 1)
0 1 2 k- m- 1 k- m
0 Gm Sm+ 1 8m+2 •* 8k- 1 K
1 Gm-i 8m Sm+ 1 •• 8k-2 hk~\
P= m G0 81 82 8k-m -\ hk-m
m+ 1 0 &o 8 1 • 8k-m -2 h k -m -1
k- m 0 0 0 8 m -\ hm
(Assume m < kl2) where G,- = g0 + Si + ■• • + 8b ht = g( + gi+l + . . . ( / > 0).

If m = 1.
0 1 2 k- 1
0 <?o + Si 82 8 k- 1 hk
1 Go 8 1 8k-i hk-1
to •••
II
0 8o 8k-3 hk-2
k- 1 0 0 So hx
where ht = S g ,. Let us assume that gf > 0 for all j, the chain is then irreducible
j= i J
and aperiodic, so that the stationary distribution n t{i = 0, 1, 1) exists
where n l = lim P(Zn = i). The /r, is the unique solution of the equations
n—>°o
k-l
n j = 2 7T, Pij (j = 0, 1, 1) together with
' i-o J
^0 + + • • • + ^ -1 = I-
In case of geometric input = j) = gp7 = gr 0 < q < 1, p = 1 - qj =
0, 1, 2, . . . when m = 1, the transition matrix is given by
0 1 2 k-2 * -1
0 q + qp CIP2 qp3 ■■ w k-' Pk
1 q qp qp2. qpk~2 Pk- X
P=2 0 q qp ■• <7P*-3 Pk-2
k- 1 0 0 9 p
Exercise 2.27 Loss suffered in the first year is Y • /?/100 = Y{ • Ux. Thus X x =
Ki - YxU{ = Y\( 1 - f/j). Loss suffered in the second year is K2(l - Ui )R2I 100 =
K2(l - UY)U2. Therefore X2 = K2(l - Ux) - Y2( 1 - UY)U2 = Yx( 1 - Ux) (1 - U2).
n
Hence by induction on n, the crop yield in the nth year is Xn = Yn n Zh Z, =
i=i
n
1 - Ui and Xn = YnLn. Since Ln = n Z, = Ln_xZn, Ln is obviously a Markov chain.
/=i
Exercise 2.28 Let pjk = P{/: white balls in first urn after n + 1 interchanges | j
white balls in first urn after n interchanges}.
Note that if there are j white balls in the first urn after n interchanges then
except for j - I J , J + 1 states other states are impossible after n + 1 interchanges.
Therefore pjj_x = P (Replacing a white ball in the first urn by a black ball in
the 2nd urn) = j/N •jIN = j 2/N2.
Pjj = P (Replacing white ball in the 2nd urn or replacing a black

ball in the 1st urn by a black ball in the 2nd urn)
N~j
=j / N -
N N
Pjj+l = P (Replacing a black ball in the first urn by a white ball in the 2nd urn)
_ N -j N -j
N N { N )
if k = j - \, j =
Here Pik = 2(j/N)\ J if* = y , j = 0 ,1 ,2 ,... N

N
(1 ~ j / N ) 2 i f k = j + 1,7 = 0, 1 , . . . N - 1
0 otherwise
Pjl > 0 for all /, j e S so that there are no transient states, the chain IS
i
irreducible and aperiodic finite chain.
Hence lim = n . > 0 for all i, j = 0, 1, . . . , N.

n-> oo lJ J
N N
Ttj can be solved by solving the equations £ itiPij - Kj and £q 7tj = 1.
(Nl)4
The solution is given by n } = ---- r-------------=--------.
J U!)2((N -y )!)2(2
Exercise 2.29
(if N)q if k = i + 1
4r p +^ -i-q if k = i,i= 1 ,2 , 1
(a) Pik = N - i p if = i - 1
N
0 otherwise
pn = 1 if i = 0 and i = /V. Equivalence classes are {0}, {N}, {1,2, . . . , i V - 1).

Hence {0} and {N } are absorbing states.
i2/ N2+ ( N - i ) 2/ N 2
if
i(N -- ii) .,CI . . . . , . , ^ A,
(b) ------t— if k = i + 1, i - 1 i = 1, 2 ,.. . N
N2
0 otherwise
(c) Equivalence classes are {0}, {1,2, . . . , N - 1}, {N}

Exercise 2.30 In fact we shall show that a 2 x 2 stochastic matrix is the two-
step transition matrix of a Markov chain iff the sum of its diagonal terms is
greater than or equal to 1.
1- a 1- a
Let P = P and A be two stochastic
\-q <7 1-/3
matrices such that P = A2. We shall prove that p + q > 1.
Proof p = a2 + (1- a) (1 - /?), q = 0 2 + (1 - a) (1 - 0).

So p + q = a 2 + 2(1 - a) (1 - 0) + 0 2 = (a + 0 - I)2 + 1 > 1. Conversely if
p + q > 1, then there exists a stochastic matrix A such that P = A2.
1st Proof We have to show that there exists a solution of
a 2 + (1 - a)(l
(1 - a ) ( a + =1 • • ■( 2)
if p + q > 1.
( 1 - P ) ( a + P ) = 1 - <7 • • • (3)
p 2 + (I - a)(l - P) = q . .. (4)
From (3) and (2) 1 - 0 = (1 - a)-j— Use this in (1) to get
a 2{l - p - q) - 2a{ l - q } + { l - p - q + p2} = 0.

The solution of this quadratic equation in a is
(2 - p q- ) a = 1- q± - q ) 2 + (2 - - + p - l - p 2 )
= 1 - q ±(1 - p) - 1
Since p + q - 1 > 0, this solution is real. Choose the positive root, then
1- q (1 + <? - 1
> 0.
2 -P~q 2-p-q
1-p + (1 - + <7 - 1
Similarly >0.
2-p-q + 2-p-q
2nd proof Let P and A be as above. Now if then the eigen values of P
and A2 coincide. Since A and P are stochastic, 1 is an eigen value of P and A. The
other eigen value of P is p + q - 1 and that of A is a + 0 - 1 . So p + q - l =
(a + 0 - l)2 > 0. Now since 1 > 0 and p + q - I > 0, P = A2 has a solution
A = ^fP (all eigen values are non-negative).
Exercise 2.31 1st Solution Since the M.C. is finite and irreducible, it is positive
recurrent. Since it is aperiodic p ^ —» 7Tj > 0 as n —> ©o. Hence there exists a
N(i, j) such that > 0 for all n > N (/, j). Since the M.C. is finite,
N = max N(i •j) < ©o.
iJ
Hence for all n > N p {J ] > 0 for all /, j e S.
Second Solution If p-"0 > 0, then p{m+n(i(7)) > q for all sufficiently large
positive integer n and where d(j) is the period of state j. Now same type of
arguments as the first solution applies.
Exercise 2.32 We have p (/ -} = Z ///V/y

k=0lJ 77
k) •
OO OO fl oo oo
(by Fubini Theorem)
(Since j is transient and f l} < 1). Also p ^ = 0 if i * j , p ^ } = 1, and Z p - < <».
Exercise 2.33 Since P is doubly stochastic 1 = Z p~n) for all n > 1 (by
induction). Fix A > 0.
oo yv
Then 1 = Z p,(.n) > Z p.(.n) > N min p \ S i n c e chain is irreducible,
i= 0riJ 1=0r i J 0 <i<Nr i J
0 < lim [ min p -^] does not depend on j and is < for all a. Take N —»
0</</V 7 A
then lim p- w) = 0 exists for all i j E S and hence the chain is either transient or
rt—
>oo •'
null recurrent.
Exercise 2.34 Let P^(z) be the generating function of {pj)n)},

(a) na = lim (1 - z)Pu(z) (by Cesaro-Taubor Theorem)
7 z—>1— 7
F y(z)d-z)
= lim —J---- „ . .
h i- 1 - (z)
and this limit exists always (may be zero) where F,y(z) is the generating function
(n ) ]
of {/,.
(b) By Fatau’s lemma
2 n u=2 limI 2 p,(,m) < lim 2 2 p,(,m) = 1.

y 7 j n—>oo /i m=l 7 n —>°° m=l y 7
( C ) z /r
k
ikP kj =Z
k
lim - V (m)
lim
n—
2_
n m= 1 k
(by Dominated Convergence Theorem and Fubini Theorem)

lim I £ p<m+1) = lim £ ±p< m) + lim - [ p ^ +l) - PiJ] = %

=
w->oo n m=\riJ n—>°° m=\ n lJ n->oo n lJ J J
Similiarly 'Z p ik7ikj = /r,; .
By induction tt,/ = X n ikp {^ . Sum this for m = 1, 2, . . . , n.

1 n
Then n^ = — X X^ tt,-* . Taking n-ôo and appling Dominated Convergence
Theorem 7TI7 = X n ikn kj .
(d) If j is positive recurrent then Ffy(l) = ftp X n f - ^ = F,' (1 ) < oo.
By (a) n, j = f u -1 = f iJn j , where , =*

" nn J jj " nn J jj
Exercise 2.35 Let 0, 1, 2, . . . , A are the states of the M.C. and the M.C. is
irreducible. Therefore the chain is positive recurrent and hence has a stationary
distribution. But itj = 1 N jc,-p^

for ally is a solution of the equations x y = X
N N
since S = 1. Moreover itj > 0 and Xq Uj - 1. By uniqueness itj = ^ + y
is the only stationary distribution.
Exercise 2.36 We apply induction on r. For r = 0, P[Nt = 0] = 1 - f u. Assume

that
P[N{= r] = (1 - f u) {fH)r. P[Nt = r + 1] = X
n
P[Nt = r + L Xn = i for the first time]
(omitting the condition X0 = i)
= I P[N, = r+ 1 I X„=i ] P [ X = ifor the
n
=I P[N, = r]f<n) = (1 - f u K f „ y
n n
= ( 1 - f a ) fa fa = 0 - fu ) f a +'( "
E(N,) = £ rP(N, .= r) = (1 - / „ ) £ r f '

r= 1 r=l
/« /«
= d -/» ) (•••A < l).
(i-A )2 1- f ‘
Exercise 2.37 First Solution By induction (or by first entrance) it follows that
f ) n+l)= k*j
2 ./ 4 '0 for al1 n - !• Hence
r(«)
< » + »/-;■ *" = /« * , 5. < ’ = / « - ? » kj
and since the chain is positive and irreducible
f ‘J = L
Hence u ij = 1 + X p ik X n f W (Fubini Theorem)

^ lJ k * j ri K n=0 JkJ
= 1 + £ j pik Pkj = k L Pik + S j Pikflkj (1)
—Pij j^ j Pik H Pkj ]•
Second Solution Given a statey, the mean first passage times {jUy, i * j ] can
be regarded as mean absorption times in a new Markov chain obtained from the
original Markov chain by makingy an absorbing chain. If P = (P,-,) is the transition
matrix of the original M.C., define a new transition matrix P ' = (p ki) as follows:
p'ki = 1 if k = y, i = y, = 0 if k = y, i *y, p'ki = p*/ if * * y \«is any state. The
resulting chain contains a single absorbing statey, while all other states are non
recurrent since they are inessential. The behaviour of the new chain before
absorption is the same as behaviour of the original chain before visiting state y
for the first time. In particular, the mean first passage time from i to y in the
original chain is the same as the mean time to absorption.
Exercise 2.38
n n
< f jk for all n > 0.
Therefore to
f i n) _ An) y r i m) (n-m)
Jjk -Pjk- mt 0 Pkk
Hence /* * !y p c)
Combining (1) and (2) we get (a).
(b) If j —> k, then p ^ > 0 for some n. Hence by (1)
fjk > sup p i p > p i p > 0 i.e. f jk > 0.

n
Alternately, if f jk > 0, then by (2) X p ^ > 0 => for some n, p ^ > 0. Soj —» k.
(c) Similarly, k —>y iff > 0. Soy <->k iff fjk f kj > 0.
<d> I " ? - 1 r> - . y r »y pub™)
s.
m=l p r + i
(e) From (d) taking j = k
Thus, if f kk< 1 then £ p ‘n) = -r— -
Exercise 2.39
= 2 a t = 1.
jeS
(b) By dominated convergence theorem
= 7i j Z a , = n j ,
1 ieS 1 J
where 7tj = 0 if j is transient or null recurrent and > 0 if j is positive recurrent,

(c) A stationary distribution is such that
71 j =Z 71, p-,n) for all n > 1.
J ieS riJ
Hence
Therefore the n-step unconditional probabilities are the same for all n > 1 and
the M.C. is “Stationary” i.e. the joint distribution of {Xt[+h , Xt2+h ,. . ., Xtn+h }
for positive integers tx < t2 . . . < tn is independent of h, if h is a positive integer.
Exercise 2.40 First Solution Check that
is a solution of x k = X x kpj k with
Now p00 = e~x > 0, p0i = e~xk > 0 if 0 < p, q < \ and 0 < A < °°
Similarly every other pjk > 0 for all j , k e S.
Hence, the M.C. is irreducible.
Since p ]k > 0, p {jk > 0,. . . > 0 for all n > 1. So the M.C. is aperiodic.
Since 0 < ^ | xk | < °o, the M.C. is positive recurrent. Moreover xk > 0 and
^ x k = 1. Hence it is the stationary distribution of the M.C.

Second Solution Again irreducibility and aperiodicity of the chain is obvious.

Let g(z) = 2 jck z k be the generating function of {xk}. We try to find a
k-0
solution of x k = S XjPjk . Then
k
„-x 2 p nqj-nA k-n /(k - n
* (z ) = *?o 7=0 n=0
= e~x I I 2, ( z*) k-n/(k - (by Fubini)

n=0 7=0 7
V'V
= r (I' z)A S ( zp)nq' - n

7=0 7 n=0
V'V
(by Fubini and £ (zA)* n /(k - n)\ = e zX)

k=n
= e Mz~l) 2 x. (zp + q p = 2 x , [1 + p(z - l)]7.

7=0 J 7=0 '
Sog(z) = e~A(l~z)g[l + p ( z - 1)]. By induction

g(z) = e-Mt-z)e-M
i-z)P
g[[ + p2{z _ 1}]
_ e - M \ - z ) ( \ + p + p 2 + ...+ p n ) g \ \ + p n (Z - 1)1.
Now 1 + p + . . . + p" —> and 1 + p n (z - 1) —> 1 as /i —» for | z | < 1.

Therefore
g{z) = exp [-A(l -z)lq\ g( 1),
where 0 < | g (1) | < <*> since otherwise | g(z) | = °° or 0.
We choose g(l) = 1. Then
g(z) = e-k/q[x- z)] = e x,qe{X,q)z = £ e~*lq ( XlqV l j \ zJ .
Therefore, Xj = e~Xlq(X/q)j/j\
is a solution of the given system which is accidently also a stationary distribution.
Hence the M.C. is positive recurrent.
Exercise 2.41
P[Y„ =j] =<?«-/ 2 P[Sm =j], where and = W, + . .. +

7 m=0
Wm. Hence
where
Poo = m^ 0 =j] and = (x t)J
is the generating function of {p^}.
Poo(*t) = [1 - G C tf)]" 1 ( 2)
1 - G(t)
and Z qmt m =
m=0v
Z0 1 - n=0
2 00
tm =
1- r
(3)
In fact P[WX > j]= P[WX 1] - P[WX>j] =- qj.
Hence G{t) = £ (9m_, - £ - % t mqm

m- 1 m=lm=l
= ' m-0
\ ~ m=\
2 + 1.
Thus, 1 - G(r) = (1 - OlI /i=0

Z ' m<7m iI => mZ=0 t ”qn = -5-r—
1 —t •
Since W„’s are i.i.d. r.v.s, G ” ( t)= Zq t kP[Sm = k].
Poo(x t)= Z p & H x t y =

7= 0
£7=0 mZ= 0 P[Sm =j ]( xt ) j (by Fubini)
= z z , p[ s m=j ] ( x t y
m = 0 7=0
= z z (xtyp[sm =n
w=0 7=1
= I G ' (* 0 l (v G(r) < 1, | r | < I).

m -0 1 - G(jtr)
oo n 1 _ f2(t\
Therefore, Z f" Z ^ /’[ f „ = i l = — ... y * . . .
«=o 7=0 " ^ (1 - r)(l - G(-tt))
Exercise 2.42 Let ak be the probability of absorption in state 0,

Then Og =1, aN = 0 (since N is also absorbing)
a* = Pk cck-i + 0 - A* - ak + Akak+l, 0(1)
=> Pk/htek -i ~ a*l = [a* - at+il. 0< k

So a ,- a 2 = p ,(l - a,), - a 3 = p2(l - a,). • • •
a N-I — 0 — P ^ f_ i(l - Ctj) ( 2)
where A = ^ Po = 1-
Summing over (2) gives a x =(pj + . . . + pN_x) (1 -
AM
f N- 1 \ f AM \ -i 2 pi - 1
7=0
or «i = Oi
/—I N- 1
I'-1 / V 7
.2 A
7=0
f am
z pj 1-1
7=1
From (2) OCi - 1 = TV-1
- 1 2 pj
7=0 ^
2 P,7
j=0
7=1
2o a
(X; = 1
TV-1 (v p 0 = 1)
2 P7
7=0 ^
AM /A M
Putting ( = k , a k = Z Pj / S pi
J=k J / 7=0
By summetry, if (3k is the probability of absorption at A, starting at &, then

j80 = 1, /3yy = L and the same reasoning works. From the explicit formulas one
shows that P (absorption at 0)+P (absorption at AO = 1 for every k.
( k- 1 ( AM \
Hence, J8* = I
y=0 Pj •
Exercise 2.43 First Solution Define Z„ = 1 if = and 0 otherwise. Then

£(Z„) = P{Z,= 1) = P(Xn = 1 1 X0=i)= p\'". Since i is posi
So
E [N„m i
-E Z Z„ —> 1 1
n m=1 n Vu Vi
Second Solution Define a renewal process as follows: Let {Wn, n > 1} be the
inter recurrence times to state i. By Markov property {Wn, n > 1} are i.i.d. non
negative r.v.s with distribution P[Wn = k\ = / j k). Now since i is positive recurrent,
E(Wn) = p, = Z n/„(n) <

fl=l
Let A^l) be the renewal counting function for state /, that is i.e. A^(i') =
max [m : WO + . . . + Wm < n]. Then by discrete renewal Theorem
E( Nn(i)) _ j _
lim
n—>°o Vi’
Exercise 2.44 First Solution Assume that the M.C. is transient and put
p*j=T^p(j \ Since j is transient, p* < °o.
By Problem 2.38
l-/oo > 0 since the M.C. is irreducible.
2. fko ^ Pko by 2.38 (a) for k*.0
3- Pko =fkoPoo by 2.38 (e) and 2.38 (d), and = > °-
Let k * 0. Then 0 < cc<fk0 < p*k0 (by 2)
= fko Poo (by 3) =fkofooPoo - • • • =fko[foo]nPoo for all ai > 1.
Since p^0 = ----- — (by 3) > 0 and / 00 < 1 by assumption, we can choose
1 ~/oo
n so large that [/oo ]” < •
Zj kOPoO
Hence 0 < a < f k0 [fooT p oo < ol!2 which is a contradiction. Therefore, the
M.C. is recurrent.
Second Solution
Let <2,v = P [visit j.o.

i | X0 = = p \n n [
i]
[m=\ n-m J
(by Monotone Convergence Theorem)
= lim p \ u [Xn = j \ X 0 =/]}

m—>°° [n=m J
~ Jl™, ^ P{Xn ~ j' - k for some n > m \ Xq = i} (by Markovproperty)
= lim X P[Xn = j for some n > m I Xm = k} P{Xm = k I X0 = /}

k
= lim X P {visit j in sometime I Xm = k] = lim X/t,/?J.m)

m— k k J lK
Hence = lim (1)

J m—>°° k J
It is easy to see that

Qij=fij (Q
n2)
Given 0 < a < f i0for all i*0 =>from (1) with 0, j = 0,
Hence &o * Hm £ p \ p f k0 > a Hm S PT > 0.
l2io
Qoo = —f~
> 0 since Qi0 > 0 and f i0 > 0. (3)
JiO
Since Qoo is either 0 or 1, by (3) Q00 = 1 => 0 is recurrent and the chain being
irreducible, it is recurrent.
Exercise 2.45 (a) Assume that j is transient. Let S* be the Communicating class
of j. If there is no k such that j -/» k and k +j then all states of S are in S* so that
S is positive recurrent (since the chain is finite). Since j is transient, there will be
a k such that j -A k and k + j.
Conversely if j —> k and k —>j, then > 0 for some n and p ^ = 0 for
all m.
Hence 1 - fn = P(Xm * j for all m > 0 | X0 = j) > > 0 for some n and some
k * j. Hence fy < 1, i.t.j is transient.
Exercise 2.46
Solution (a) If j e S is absorbing, then pjj = 1 and pjt = 0 for all i * j. Since S
is denumerable, record S such that the absorbing states are numbered 1 , 2 , . . . .
The remaining states are supposed to be transient. Clearly
E Ec
(I 0^| E
P=
Q )E C’
(b) If /, j G Er, let ml} - number of visits to j starting from i.
1 if X* =7
Then let Zk
0 if Xk * j
Clearly m,. = X Zk and ,V„ = E(m„) = X E{Zk) = X = XQ

Jk = 0 1 1 0 0 J0
ije E c.
An)
(c) = X f \ p l) X X,c Pik Pkj ( 1)
o=0 o=0
= 2 Z p (ikn)pkj = X Nlkpk] = (NR)i j (by Fubini)

keEc„ = 0 '* 1 keEc 1
If j e £, then
p f = i f t p T m)= ? j u m) • 1 so that 4 " +1) = /4"+,) - pT j e
_(/*+1) _ y y „(«) _ , y „(n) „

P ij ~ ? P ‘kPkj ~ k^£ , Pik Pkj + £ E Pik Pkj
An)
= PiP + \ kj• Hence
iP == X pik
keE' keE1
(/ 0 '
(d) =>
C, vfl(/ + Q + . . . + <2n-1) GB,
f / 0" f7 0N
—> =
kRN oy 0,
since RN = # and Q* —» 0 (since j is transient if j G £ c ).
Exercise 2.47 Since P is symmetric ptj = pjt. By Chapman-Kolmogorov equation
=L PikPkj = L PkiPjk
Hence, P{2) is symmetric. Similarly, by induction on n P (n) is symmetric for all
n > 1.
Now 2 pa = 2 p n = 1 so that P is douldy stochastic. If the M.C. is irreducible
kGS j GS
and infinite then it is not positive (by Ex. 2.33). Since ptj > 0 => pJt = ptj > 0. So
the M.C. consists of closed classes only. Every closed class can be treated as a
separate M.C. So if every closed class is infinite then by previous argument the
original M.C. can not be positive. But if the M.C. consists of finite number of
states only then there will be no transient state, not all states are null recurrent
and therefore there will be some positive states. But if the M.C. is irreducible
and has finite number of states then it will be positive recurrent and same thing
is true for every commucating class.
Exercise 2.48 Let the initial distribution be {nh i e 5}, then by Ex. 2.13.
(1) 71j = Z 7 1 , for all n> 1, (2) p\n) = P( Xn = = 7th
ieS J 1 J
Hence
D(m) _ P(Xn+m ^l^Xn = 0 _ 7tj_

(by (2))
R Ji -
Pi^n+m = V') ftj P\ r
independent of n (=> the new chain is stationary),
(a) > 0 is obvious.
2 R (m) = - ± - 2 (by (D)

ieS 71 i ieS n .l
(
r >(m +n ) 7 tt ( m+n) 71 i n (m) U k (n)
Ru = ^~ Z (Chapman-Kolmogorov)
keS 7lk Pik V
71 j Pkj
/
= 2 for all m, n > 0.
ke S J
Hence /?/z is a transition matrix and since the initial distribution is given, /?(m)
is the m-step transition matrix of a discrete time Markov Chain.
Note Let N be any large integer. Define Yn = XN_n for n = 0, 1, 2, . . . , N then
{Yn, n = 0, 1,. . . , /V} is a Markov chain with transition matrix R. It is called the
reverse or dual chain of {X„}.
(b) Since the M.C. is irreducible, so is its dual. Moreover
lim fl<m) = 7b. lim P<m) = TLi 0.

m -> o o lJ 71 j w -> o o J1 7ti 1 J
Thus the chain is positive and the stationary distribution is given by the initial
distribution {7r„ i e S}.
Exercise 2.49
Pn = Pn-i(0 - c)p + c) + qn_j((l - c)p), where </„_! = 1 - p n_x
= <7Vi + 0 - <?)P-
Therefore /?„ = Ac""1/?! + ^ = Ac""1/? + p , n > 1 (since /?! = p).
From initial condition p = Ap + p so that A = 0. Therefore
pn = p = P(X0 = 1) for all n > 1.
Therefore E(Xn) = p, Var (Xn) = p - p2 = p(\ - p).
E(Xn_\Xn)= 1.1 ((1 - c)p + c)p = ((1 - c)p + c)p = +cp
and Cov (Xn- 1, Xn) = 2(p - 2 + cp) cp
Similarly Cov (X„_2, X„) = c2p (1
and in general
Cov (X„_*, Xn)=ckp(\ - Pf
or n
2 Var (X„) = np(l - p ) , I Cov(X,,X*)

*=1 7<* 7
= p ( l -p){(c + c2 + . . . + cn- 1)
+ (c + c2 + . . . + c""2) + . . . + c }
= p d - p ) T ^ \ ( n - l ) - c-
n
Hence, Var(S„) = 2 Var(X*) + 2 2 Cov(X,,X*)
k =1 j< k
cjn - 1)
= np(l- p) + 2p(l - p) - c 2( 1 - c"-')/(l - c)
1- c
—^ A + i----- (.*. p < c < 1).
Exercise 2.50
i n
First Solution IfP = />2then p (jln) = /?,, for all n > 1. Hence /?,, = — Z p (hm) —> tt,
z/ 7 7 n m=o lJ7
1 n
and pij = — Z^/?^”0 —» ;r7 (since the chain is irreducible). So p{j = pjj = Ky
Assume p0o = 0, then pi0 = 0 for all * e S and hence {0} can not be reached,
so the M.C. is not irreducible.
Hence p0o > 0 and {0} is aperiodic. Since the chain is irreducible, all states
are aperiodic.
Second Solution If P2 = P, the p^n) = p i: for all i e S and hence lim pjfn) = pjj.
n—>°°
Assume j as transient or null recurrent then pl} - 0 for all i and j and so P is not
a stochastic matrix. Hence j is positive and the chain being irreducible, the M.C.
is positive recurrent. Hence p \ p —> n } and p {^ > 0. So the M.C. is aperiodic
and p l} = Pjj.
Chapter 3
Exercise 3.1
n k{n) = pnM (n - 1) + qnk_t(n - 1), 1 < k1
K \(n ) = P7h(n
- 1 ) + q 7 l0 (n - 1)5,,i (1
TC^n) = <5„0, na{n) = 0 where <5„„ i= 1,0 are Kronecker’s deltas. Multiplying both
sides of the above equations and summing over all values of n we obtain
Gk(s) = ps Gk+l(s) + qs Gk_x(s), 1 < k< a - 1 (2)
G0(s)= l,G fl(*) = 0. (3)
Since (2) is a linear difference equation with one variable k, taking Gk(s) = k k(s)
as a trial solution we find from (2)
A* = PsAk+l + Â*"1
or /xvA2 - A -i- gs = 0. The roots of the quadratic equation are
Ai ( 5 ), A2 (5) = - ~ where 0(s) = (1 - 4 p g j2)172 (4)
0 1 f
Since 1 - 4pqs~ > 0 or s < — ==■, we assume here 0 < 5 < — ■== for two real
2 ^p q 2j p q
roots A](5 ) and A2(s).
The general solution is given by
Gk(s) = 4[A,(j)]* + B[A2(s )]*

where A and B are functions of 5 and must be deduced from the boundary conditions
(3). Hence we get
A,""*( 5 ) - A?"* ( 5 )
Gk (s) —( q / p Y (5)
A." (s) - A? (s)
[1 + 0(5)]"-* - [ 1 -0(A)] a-k

= ( ql p) k (2ps)t
[1 + 0(5)]" - [ 1 -0 (5 )]"
(substituting A)(s) and A;(.v) from (4))
( a —k\
z [0(^)]r {l ~ ( - l ) r }
r=0
= (q/p)k (2ps)k
z
r=0
[0(S) Y { i - ( - [ ) ' }
V' y
Since no contribution comes from even r. setting r = 2j + 1, we get

( a - k - 1)/2 ' a-
S [0U )]2;
1=0
Gk(s) = (c,/p)k(2ps)k if a - and a are odd.
( a - 1)/2 f a
2 [<K*)]2'
2j + 1,
(n -k-2)/2 ra-
2 W * )]2'
7=0 V27 + ly if a - k and a are even.
(2/«)* (a-2)/2 ( a N
2 [«>(5)]2;
2 7 + l.
G(5)
Now G* (s) = where G(s) and VG) are polynomials in 5 with degree of
V(J)
G(s) does not exceed that of V(s) by more than one. So we assume the degree of
U(s) is less than the degree of V(s) by m (say). Let the equation V(s) = 0 has m
distinct roots sh s2, . . . , sfn. Write Gk(s) in partial fraction as
Pm
Gk (s) - P\/(s\ - s) + + ...+
v^2 “ s ) (Sm ~ s )
with coefficients pk are given by
Pk = lim (s* - s)Gk (s) = , /: = 1, 2, . . . m.

s-*sk v (sk)
Hence the coefficients of s n is power series expansion of Gk(s) are given by
(«) = -^7■
/J+l + . . . + P'
n+1 (7)
Making auxiliary variable c = 2 (p ^ )1/25 wethenhaveA1(5),A2(^) = (^ /p /e ±'0,
so that equation (5) yields

sin(a - /:)#
Gk (s) = ( q/ p)kl2 ( 8)
sin
The roots of the denominator are 6 = 0, — . These roots are related

a a
to the roots of F(s) = 0 by the relation
7 = 0, 1------ ,a.
2( pq) U2 cos | — j
For y = 0 and j = «, we get the corresponding values for 9 as 6 = 0 and 6= n

which are also roots of the numerator of (8). If a is even, there is no s}-corresponding
to j = a!2. So sr j = 1, 2, . . . , a - 1 are real and distinct.
sin(a - k)6 .
So Pi = lim ( 2 — / a T (s i ~ 5)
sin (a0) J
kl2 sin {(a - k) j n/a] _ kl2 sin(kj n/a) sin (jn/ a)

-(qlp
. ( dQ^ " (q P)
2 a( pq) v2 cos
a cos j n
I * ) .., ■(£
Now from (7) we get
rt-1 fl
2 I cos”" (jn/a) sin (j n/ a) sin (/:j n/ a)
./ = 1 Jo
Letting we get the probability 7r*(n) that a player with initial capital
k > 0 will be ruined in exactly at the nth step against an infinitely rich opponent
n k(n) = 2n p i n - M q W 2
f
•'o
cos'7 sin (nx) sin ( k nx ) dx
By an elementary way this is also
p ( n - \ - k ) / 2 q ( n + k )/2
k/n
(n - k)/2
Exercise 3.2 If the particle starts at a, absorption occurs at time 0 with probability
1, and if it starts at any other point, absorption cannot occur at time 0. So
1 if 7 = a
( 1)
0 if j * a
Since the random walk takes positive jump or negative or no jump at all at the
first step
C = P f J : L + (1- P - fja " 1) + <7
( j = - b +1 ,___
The boundary conditions are given by
= 1 a n d /';", = 0, 1, 2, . . . (3)
(2) is a difference equation of the first order in n and second order in j with
initial conditions and boundary conditions (3).
Multiplying equation (2) by sn and summing over n - 1, 2, . . . we obtain
Fj„(s) = s[ pFJ+[a(s) + (1 - p -q)Fjn(s) + qFH a(s)] (4)
with boundary conditions
Faa(s) = 1 and F_M (s) = 0 (5)
(4) is a difference equation in one variable j.
Substituting a trial solution Fja(s) = A'(s) we get from (4)
A7' = s[pAJ+] + (1 - p - </)A7 + g A 7" 1]
or psX1 - A [1 - s(l - p - q)] + qs = 0 (6)

The roots of the quadratic equation (6) in A are given by
w S 1 , X 1 - 5(1 - p - q)± [1 - j ( l
A , ( s ) , A 2 ( j ) = ------------------------------------------ y ]Ts-------------------------------------------
Assuming j is real and positive and the function under the radical sign is also
positive leads to
______ 1_______
[ l - i ( l - p - <7)12 > 4 or 0 < s <
i - (4 p - V?)2
The general solution of (4) is given by
Fja(s) = A[Xl(s)]J + B[X1(s)]J
where A and B are functions of 5 and are to be determined from the boundary
conditions (5).
[ A, ( j ) ] ^ - [ A 2 ( j ) ] ^
This gives ( 8)
u , ( 5 ) r ft - [A2(5)]fl+i
putting j = 0 in (8)
[*,(*)]* - [ A 2 (J)]*
[ A ^ ^ r " - [ x 2( s ) r +b
Exercise 3.3 Since the jumps are independent, the particle’s position at time n
depends only on its position at time (n - 1) and the nth jump. During the interval
(n - 1, n), the state j can be reached by one of three mutually exclusive and
exhaustive ways, viz. a jump of + 1, 0 o r-1 from states j - 1J J + 1, respectively.
Pij = PP\ni-\ +( l - p-q)Pj"~') + (0 < j )
At the barrier states we have
p (ln0) = ( \ - p ) p (l n0) + qp(ir ' )

Since the chain is finite irreducible the stationary distribution exists and are
the limiting values of occupation probabilities (see theorem 2.9). Then nk satisfies
the equations
7lj = p7tj_\ + (\ - p - q ) 7lj + qnH (k= 1 , 2 , . . . , )
fin ~ pfta-\ 0 ~ q) fta

Ko = (1 - p ) 7l0 + q7lx
Solving these equations we get
7ij - (p/q)’ n 0(j = 0, 1, . . . , a) and can be calculated from 2^ n } - 1.
Hence, we obtain the stationary distribution

1 - p /q
Ui ( p/ q) 1 (y = 0, 1,. . . , a).
1 - ( p / q Y ' +'
Writing n} (jc) = 1 -X x ] where x = p!q, by L’Hospital’s rule

a+1
1 - a :
lim n j = lim n. ( a :) = ----- r for all j = 0, 1, . . . , a.

—8i 7 .v— ->i 7 a+ 1
Exercise 3.4 For 6 > 0 we have

0(0) > P(e' > 1 + <5)(1 +
Hence, since P( e x > 1 + 5) >

0, lim 0(0) =
0 -> ° o
Similarly for 0 < 0, 0(0) > P(ex > 1 - 8)1( - 8)e
Hence, since P( e x < 1 - 5) > 0, lim 0(0) = (2)

6-8-°°
Since 0"(0) = E(X2e ex) (since m.g.f. exists for real 0 differentiation under
expectation is valid), it follows that
0"(0) > 0 for all real 0. (3)
Now (1), (2) and (3) imply that 0(6) has a unique minimum at 6 = 6" say.
Since 0'(O) = E(X) * 0, 0* * 0 and 0(0*) < 0(0) = 1. Also 0(0) is strictly
decreasing over the interval (-<*>, 0*) and is strictly increasing over the interval
(0*, °o). Since 0(0) = 1 and 0(0*) < 1, there exists an unique real 0O^ 0 such
that 0(0q) = 1.
Exercise 3.5 Under the conditions of Exercise 3.4 there exists real 0O^ 0 that
0(0O) = 1. Substituting 0Ofor 0 in Wald’s Identity, we obtafn E( ee°SN) = 1.
Therefore
P( SN < - b ) E ( e e°SN \S,N < - b ) + P( SN > a ) E ( e e°SN\ SN > a)
= E ( e e°SN)= 1.
Since 0(5,, < a) = I - P(Sn < - b), solving we get P( SN £i - 1

E\ - E0 '
Exercise 3.6 Let E* = E(eeSN \ N - n).We shall first
£*. If
0 > 0, E* <
E (e0SN \ S N > a , N = n) (1)
and E(eeSN | 5.v > a . N = n, e0SN-' = pea8) = e"ep E êex” \ e xn6 > 1 ) (2)
P)
Solution to E x e rc is es 343
< 1(0) w here 1(0) = su p p E ( e xe \ e xe > 1I p )

p e ( e - {a +l,) l°l
N o w s in c e 0 ( 0 ) e x is ts , 1(0) > °o F rom (1 ) and (2 ) w e get
E* < ) if 0> 0 (3 )
and h en ce E{eeSN) < e ael {6) if 0 > 0
S im i l a r l y w e g e t *E< (0) if e< o (4 )
and h en ce E ( e es»)< e - bel(Q) if e< o
N o w to p r o v e th a t W a ld 's id e n t it y c a n b e d i f f e r e n t ia t e d u n d e r th e e x p e c t a t io n
s i g n a n y n u m b e r o f t im e s w it h r e s p e c t to 0 at a n y v a lu e 0 in t h e d o m a in
| 0 ( 0 ) | > 1, it is s u f f ic i e n t to s h o w th e f o l l o w i n g t w o r e s u lt s h o ld .
Result 1 A ll d e r i v a t i v e s o f ( e 0SN ) [ 0 ( 0 ) ] ~ * w ith r e s p e c t to 0 e x i s t s in th e

d o m a in | 0 (0) | > 1.
Result 2 F o r a n y p o s i t i v e in t e g e r r a n d a n y f in it e in te r v a l / in w h ic h | 0 (0) > 1.

th e r e e x i s t s a f u n c t io n H ( S N, /V) s u c h th a t
H (Sn,N )> 0Sn {0(0)}-"]
fo r a ll v a l u e s o f 0 in / a n d E (H (S N, N ) ) < °°.
N o w e x i s t e n c e o f 0 ( 0 ) fo r a ll 0 i m p l i e s t h e e x i s t e n c e o f a ll d e r i v a t i v e o f 0 ( 0 )
( D u e o f a n a ly t ic it y o f 0 ( 0 ) ) . T h is i m p l i e s r e s u lt 1.
Since {e6Ss [0(0)] *} is equal to the sum of a finite number of terms of
the type N ri eeSN [0(0)]“** Result 2 is proved if we can show that any given
integers r, and r2 there exists a function Hnr2{SN, AO > \S% N n ees" [0(0)]"* |
for all 0 e / and E{Hrxn (SN, N)} < <».
Since | 0(0) | > 1 in /,| N r2ees" [0(0)]"* | < S„ N ri~eI5"*00 (5)
where 0O> | 0 | in /. Let 0! > 0O. Then for c > Max (a, b) we have
| | < Ce\SNI01 . (6)

Hence it follows from (5) and (6) that
| S r^ N r2eeSN [0 (0 )]"* | < C/V'2/ ,|5a/|< CNr2 (e5"01 + e~SN°])
for all 0 e /.
Put Hr]ri(SN, N) = CNn ( es"e' + e~SNe').
Then E( Hr. , A S N ,N)) = C

£ P ( N = n ) N r2[E(eSN0' \ N = n)
1 " n = l
+ £ ( e - Xw01 | N= (7)
Hence by (7), (3) and (4) we obtain
E [ H n n { S N , N ) } < C [ e e 'a l { d x) + e - b6'1 (-# ,)]
S in c e E ( N r) < °° fo r a ll in t e g e r s r > 0 ( b y T h e o r e m 3 .1 ) t h e R e s u lt 2 h o ld s .
D i f f e r e n t i a t i n g th e W a ld ’s id e n t it y a t 6 = 0 , t h e d e r iv a t iv e o f e SN° [<j)(0)]~N
at 0 = 0 is g iv e n b y
d' (0)N
Sn ~ J ^ W = Sn ~ E ( X ) N (8)
T a k in g e x p e c t a t io n w e g e t E (SN) = E (N )E ( X ) and h en ce
FS k,
E(N) = ~E{X)
D i f f e r e n t ia t in g W a ld ’s id e n tit y t w i c e u n d e r th e e x p e c t a t io n s ig n at 6- 0 , w e get
W - N0>'(6)l<l>m2 - N $ (d)0! ^ ~ } 24> } ees» [0 (0 )]-". (9)

[ [0(0)r J
E(X) =
S i n c e 0 ( 0 ) = 1, 0 '( 0 ) = 0 a n d 0"(O ) = E X 2, p u tt in g 0 = 0 , (9 ) b e c o m e s
S 2 - N ( I ) " ( 0 ) = S 2n - n e x 2 .
T a k in g e x p e c t a t io n E [ S j j - N E ( X 2)] = 0.
T h e r e fo r e E N = E ( S N2 ) / E ( X 2 ) if EX = 0.
Chapter 4
Exercise 4.1
L e t C i b e t h e fir st t im e a c u s t o m e r m a k e s a p u r c h a s e a n d 0 < S x < S2 < . . .
b e th e arrival p o in ts (r e n e w a l p o in ts ) o f th e c u s t o m e r s . T h e n , b y th e to ta l p r o b a b ility
r u le
P (C \ < x ) = i P (C , £ JCI C , = S t ) P(C\ = S t).

k= 1
B y th e a s s u m p t i o n s , P ( C X = Sk) = (1 - p ) k~xp and P ( C X < x | C x = Sk) = F((ky
T h u s, P ( C X < x) = p (1 - p ) k~] F,{kJ = Fx ( x ) , sa y . S im ila r ly , i f Cm is t h e
k=\
mth t im e th a t a p u r c h a s e is m a d e .
P(C, < C, - C, < y) =2 \ <x , C 2 - C , < y \

(C
P G
k,m
w h ere G km = { Cj = Sk, C 2 = S w }, 1 < k < m. W e c a n e a s i ly p r o v e th a t t h e p o in t s

Cj f r o m a r e n e w a l p r o c e s s w ith in te r a r r iv a l d is t r ib u tio n F x(x). I f th e tim e s c a le
is c h a n g e d b y a f a c to r 1Ip, th e in te r a r r iv a l d is t r ib u tio n b e c o m e s F {(xlp). The
a c t u a l f o r m u la fo r F ^ jc ) r e d u c e s to F(x) i f F ( x ) is e x p o n e n t ia l.
Exercise 4.2 F r o m e x e r c is e 4 .6
Wt ( x ) = P ( Y ( t ) < x ) = [1 -F (t + x-y)]dH (y).

Taking Laplace transform on both sides we get
e -s, ji»oo e - s * d W i (jc) = ji«oo r mt+x

dJ e - s ( x +,-{{ _ F (
Differentiating under the integral sign and making change of variable along
with substitution r = x + t - y yields
e - ' ' Lw{s)= [1 e~sydH(y).
Exercise 4.3
(a)E(X-x\X>x) = i; [1 - F ( t ) ) d t
1 - F(x)
(b)E(Y, + Z,) = E(Y,) + E( Z, ) = 1/A + I P ( Z , > x ) d x

f
Jo
= 1/A + J e~x 'dx= 1/A + 1 (1 - e - A').
Therefore, lim £ (K, + Z,) = 2 /A.

/ — >oo
Exercise 4.4
1. (a) By Fatou’s Lemma lim H(t) = lim

/->oo
Z Fin) (t) > nZ- 1 F ('I)(«>) - <*>.
n-\
(b) 0 < //(r) is obvious, and since F(t) < 1,

E (0 < 1
1 - F ( t ) ~ 1 - F(t)'
Fit)
To prove H(t) < ----- =rr-r if F(t) = 1 for t > 0, then the result is obvious. So let
I - t it)
Fit) < 1 for all t > 0.
Proof 1. H(t) = F ( t ) + \ F(t-

fJo
(since F ( r ) t)
Proof 2. Since F ^ f t ) = J F ^ ' " (t- x)dF(x) and F

shows F (n\ i )< (F(r)]". °
Fit)
Therefore Hit) < Z [Fit)]n = -j--- pr-r, since F(t) < 1 for 0 < t < <».
2(a) E \ N 2(/)] = E i r ( F (")( 0 - F ("+l,(/))

n= 1
= lim \E (2n - l ) F (n)(0 + N 2F(N)( >0

n—\
= E (2/i - 1) F (n)(/) + lim N 2F iN)( 0

n- 1 /V—
Since FÛ) < (except if F is degenerate at 0), by applying the ratio
test Z n 2F{n) (t) < °o.

17 = 1
So n2Fin\t) —» 0 as n —» oo for all finite t > 0
Therefore F[jV2(0] = £ (2n - 1) F {n)(t).

n= \
(b) H*H( t ) = f Z
: F(n) (t - u) d Z {F(n) (0} (by Fubini’s theorem)
Jo n n= 1
OO OO /• t oo oo
=E £ F<n) (? - u)dF(m)(t)= E E F(n+m,(r)

n= \ m= \ J q n=l w =l
(by definition of convolution).

Puting n + m = k and applying Fubini Theorem
H *H (t)= i, £ Fa)(?) = £ E F(t)(0 = £ (* - 1)F(*)(0

n= l k=n + 1 &= 2 n = l k =2
So H(t) + 2 H * H( t ) = I F a ) f+ E 2(fc - l ) F (t)(0

t= l i =l
= E (2 k - 1) F(k){t) = /F (r )
k=1
i.e. / / 2(r) = H(t) + 2 f H ( t - u ) d H ( u ) .

Jo
3. Note F*(r) = 0 for / < 0 so that X* > 0.
Since ~ [ F * ( r ) ] = 1 [ 1 - F(/)] > 0
by the Fundamental Theorem of Calculus, F*(t) is (absolutely) continuous and

nondecreasing.
We have only to prove that F*(°°) = 1.
F* (oo) = lim )=- lim r[l - F(/)] + J a: <afF ( a: ) | (1)

/—>oo F (—
>oo
(integration by parts)
But f[l - F(t)] —» 0, since
0 = lim
/->°°
\ ju - f
Jo
xdF( x) = lim f
J/
xdF( x)
> lim t \ dF(x) = lim t [1 - F(t)] > 0.

I
Hence (1) implies F*(°o) = f xdF(x)/ju = 1.
Jo
F* (0 is differentiable and hence continuous. So it is right continuous. Alternately
•t+h t+h
F*(t + h) = — I [1 - F ( x ) ] d x = + C. f dx
Jc r1 JO
(by mean value theorem of integral calculus) = C t + h
Therefore lim F*(t + h) = Ctlju = ± f u - F(x)]dx = F*(t)]

hiO t* J o
(b) Let / ~*(x)= f

Jo
e~s'dF*(t)
/• o
e~s,[\ - F(t)]dt a . i e~sl F(t)dt (2)

■H W V Jo
Integrating by parts for 0.
/(*) =J Jo
e~stdF(t) = e~s'|F
(t)o + J
Jo
f e~s,F{t)dt (3)
(2) and (3) = -- - - - - - - — f ( s ) = — [I - / ( * ) ] .

JUS jusJ JUS J
(c) Since F* is the d.f. of X* > 0, F(X*) = - / * ' (0).
But F(X2) = /"(0) also since X > 0, finite or not.
-sf'(s) - 1 +f(s)
By (3) F(X*) = -/* '(()) = lim -
V—>0 + /US2
s f" (s )
= lim — ---- - (by L Hospital's rule)
.v—
>o+ 2 fis r
F (X 2) ju2
2/u
4(a) First Proof Let h = f be the Laplace-Stieltjes transform of

1 ~
H(t) = E Fv') ( t ) , f and f * be as above. Then Laplace-Stieltjes transform of
n=I
[d - F*)*H] = (1 - f * ) h = h - — / / ( I - / ) (by 3(b))

flS
= li — — + -— — = Laplace-Stieltjes transform of
jus jus v J
[H - t/ju + F*] (by 3 (b)).
By uniqueness theorem of Laplace-Stieltjes transform
(1 - F*) * H = H(t) - t/ju + F*(t)
i.e. f (1 - F*(t - u))dH(u)= [ Q(t - u)dH(u) = H(t) - tlfi + F*(r).

Jo Jo
Second Proof
F*(t) = -
F Jo
f (1 - F(jc))djc = - f (1 - F(r - «))rf(r - w) (put jc = r - u)
J/
= - - | 0 - F(t - u))du= 1 £ (1 - F(t - =^
= tin — — (4)
Let G(t) = H(t)-tinThen Hit) = Fit) +F*Hit)

becomes by (4) (1 - F)*G(t) = F(t) - F*(t). Convoluting with t/fi
F** G= — * (1 - F)*G = - L *
n
So F* * H = F* * G + F* * t i n = — * F
V
or (1 - F*) * H = H - F * * H - H - t / n * F = H - t l n + F*(by (4))
(b) 0 <Qit) = 1 - F*{t) I and f Q(t)dt E(X*) = /t2/2

Jo
by (3c). By the Key renewal Theorem and part 4(a)
lim {//(f) -
; -> ° o jJ. J Q
tin }+ Hm Jh_
iF -
f Q(t)dt =
i.e. lim {H(t) - t/ju] + 1 = jLi2l2ju2.

/ —>oo
Vi - 2 / t 2
or Mm [ H i t ) - t i n } = -^V
2 /^ 2 n2
(c) The above relation means that
LLi
H{t) tends to its asymptotic value t/ju + — z 2 ± l = / / / I + C
2 n-
Proof of 4(b)
Define A(t) = H(t) + 1 - j i xt
= E[N( t )+l ]-t / j u (1)
Since E(N(t) + 1) < <*>, by Wald’s equation
MO = P l{E[Sm+l ] - t }
= n~'E( Y(t)) where = - t,
the excess life time.
By the renewal argument, conditioning on the time X{ = x o f the first renewal
we shall get a renewal type equation for A(t). Distinguishing two cases
x-t if .r >
we get E[K(f)|X, = * ] =
fiY ( t - x) if
Therefore n A (f) = X,=

Jo
-r
(x - t)dF(x) + ju I A(t - x) dF(x)
Jo
f (2)
Now f
Jt
(x-t)dF(x)= f
Jo
ydF (t + y) =
Jo
f {1 - F ( t + y)}dy
MOO MOO
dF(u) ( 3)
Jo Jt+y
is a decreasing function of t.
Since the integrand is nonnegative, by Fubini theorem interchanging the
orders of integration we get
/•o o I moo | moo moo moo
[1 - F ( t + y ) ] d y \ d t = \dF(u)dydt
Jo [Jo J Jo Jf Jf+y
moo moo f m u-t 1 /•= » /too
= J J |J dy\dF(y)dt= J J (
= [
Jo Jo
f (u - t )dt dF( u) = 1/2
Jo
f l/2(cr2 +(4)
Since (x - t)dF (x) is bounded, the renewal type of equation (2) has an
Jo
unique solution given by
fiA(t)
f ■f
(x - t)dF(x) + jJ. I A (t - x) dH(x), H( x ) = S F (k) (/).
Jo k~ 1
Now taking Q(r) = J (x - t)dF(x) by Blackwell’s renewal theorem and key
renewal theorem and by equation (4)
lim juA(t) = 1/2(<t2 + ju2).

/ —>00
or lim {HU) - tin) = lim {A(t) - 1} (by (1))

/ —» o o t —>°°
T T 9 9
<7~ + JU- , <7~ - jU^
2 jU2 ~ 2 "^
Exercise 4.5 P (MO > n) = P(5„ < 0 = Fin>(t)
ENk ( t ) =X n*[f»(S„ < /) - P(S„+, <?)]

n=l
I n* [ F <n) ( t ) - P(n+1)(r)]
n= l
= I [ ( n - l ) ‘ - n f ] F (,,tl)(l) (1)
n—1
by rearrangement of terms provided we can prove that the above sum is convergent.
If F(x) is degenerate at a point c > 0, then both sides of (1) are clearly finite
for every fixed t. If F(x) is not degenerate, then there is a y > 0 such that 0 < F(y)
< 1. Hence for every m> 1, F (m\my) < 1 because 1 - F^m)(my) = P(Sm > my) >
P(Xj >y , 1 < j < m ) = [\ ~F(y)]m >0-
Furthermore, for any z > 0,
i.e.
F(n)(z) =
r F(k)(z - x ) d F (n-k>( x ) <
F in)(z)< [F (m>(~)]rf. wheren>md. Now if x 0 is arbitrary we choo

fix) m > I so that m y > x. Then, with d the integer part of n/m
F ("\x)< F (n\my) <
Now (1) shows that E N k(t)<

Exercise 4.6
/•/+A'
P(t +x >
Sn > r, S„tl > / +x) = J P(Sn+i + | S„= v)
/• t +.V f t+ .X
=J P( Xn+\ > t + x - y) dF{n)(y)= J
Hence P(Y(t) < x) = Z F(t + x - y)]dF{n)(y)

n= \ r [i
^ oo ^
[1 - F(t + x -y )]dH (y) Since Z F{n) (y) = H (y) and 0 < t < v
> n=\
Exercise 4.7 Starting with total probability iaw, we first note that P(Z(t) < z,
N(t) = 0) = 0 for z < t.
Hence, P(Y(t) > x, Z(t) <z) = Z P(Y(t) > x, Z(t) < z, N(t) = n)
n=\
= £f
/I = I J
P ( Y ( t ) > x , Z ( t ) < z , N ( t ) = n \ S n =y)dP{S(1)
Note that conditional probabilities are equal to zero if v £ (t - z, r), while for
y e (t - z, t), the meaning of such conditional probabilities is that the process
N(t) has no points in the interval (v, t + a), i.e. Sn+] - Sn > t + x - y. Therefore by
(1)
OO »t
P(Y(t)>x,Z(t)<z)=Z [1 - F ( t + x - y ) ] d F $
H-l J[_~
[1 - F(t + x ~ y)]dH(y)
(interchanging the order of Summation and Integration by Fubini Theorem).
Exercise 4.8
X ( s ) X ( s + t) = Z Vi Vi HYi_l < s < y i} ! {Y{_l <s + t < y , )

i j =1
where Yi = 7, + . . . + Tr Let Zv is the waiting time from s to the arrival of the next
customer. If / * j E | VlV] | = E [| Vi || V; |] < <72 (by Cauchy-Schwarz) and
Thus term-by-term integration is possible and we get

K (t) =E[ X( s ) X( s + t)] =a2

I<s<yt,
1= 1
= crP {no transition in (s, s + 7]}

= o2P [Z, >t] = cr( 1 - F,(0).
Also X(0 = Z V//|y._1<,<y.j
and hence £X(0 = 0. Since K(-t) = K(t), we have o ^ l - Fj(| t |)) = K(t).
Exercise 4.9
(a) We have Y, =Z //s <f) • + X„ - /) +, / > 0

n—1
where all terms but one are equal to 0. We can thus raise the sum to any power
termwise. Let H (t) = Z F{n) (t) be the renewal function. Deviding the range
n—\
of integration into [0, t) and [/, <*>], after an elementary computation we get
E Y r' = E ( ( X x - t ) +y - 1 + f
Jo
(5),
where Q(t) = J (u - t )r~l dF(u) .
A OO
Since I Q(t) dt = —EX[ < 00, by the key renewal theorem

Jo r
We get EY,’-'— EX[ as r ^ 00.
& 1 rfi 1
(b) Since EXf = a 2 + j i 2, we get (b) from (a) for r = 2.
Exercise 4.10 From the integral equation of renewal theory
H(t) = F(t) + where H(t) = I F{n) (t).

n —1
Also H(nd) = Wo + u\ + • • • + un(n > 0).
The actual number of partial sums equal to nd is Yn = Z / (Sn = nd) and un

7=1
= EY„. From discrete convolution formula we get the result.
Exercise 4.11 From Doob’s renewal theorem it is enough to show that
, t > 1 >is uniformly integrable for all r > 0. Let N(t) + 1 = v(t) = min
{n: S„ > t). So it is enough to show that , t > 11* is uniformly integrable
(u.i.) for all r > 0.
Consider v(t + 5), t, s > 0. In order to reach the level t + s we must reach the
level t. When this has been done the process starts afresh. Since Su(t) > t the
remaining distance of the process to climb is at most equal to 5, and thus, the
required number of steps to achive this is majorized by a random variable distributed
as
v(s), i.e. v(t + s) < u(t) + min {k - v(t): Sk - Sv{t) > 5}
= v(t) + v l(s) (1)

where v^s) is distributed as v(s).
By induction v{n) 1} are distributed as u(l).

Since ENr(t) < 00 for all r > 0 and t > 0, Evr( 1) < <*>for r > 0.
Hence by Minkowski’s inequality and (2)
(Evr(n))Ur < n (Evr(l ))1/r (3)
Since u(t) < u([t] + 1), we have for t > 1,
£U) < _ + 1)
/ “ 1 '
Therefore from (3) we get
(E(v(t)/t)r)Ur < 2 (£(y(W + l))/([f] + 1

< 2 (£(u(l))r) l/'' <
Since the last bound is uniform in t {(v(t)/t)p, t>t] is uniformly integrable for
allp < r.Since r is arbitrary {{v(t)lt)r, t 1) is u.i.
Exercise 4.12
N(t) l
(a) Since ----- - a.s. — as -* ( 1)
t H
and
N(t) —> — as —» oo ( 2)
(by Exercise 4.11) for any r>

0, it is enough to show that
is uniformly integrable. Due to monotonicity of {N(t), t > 0}, it is enough to
N(n) - nljuY
show that
^ J , n > 1 [ is uniformly integrable. Hence for r > 2,
(2) implies
N(t) - t!ju
, t > 1 \ is uniformly integrable.
V7
(b) can be proved in a similar way.
Exercise 4.13 Conditioning on the time X] = x of first renewal we get
1 if x > t +y
P(Yt > y \ X ] =x) = 0 if t + y > x > t
Av(t - x) if t > x > 0
where Av(t) = P(Yt > y).
Also Av(t)= f P(Yt > y \ X ] =x)dF(x)

Jo
= 1 - F(t + y) + f A ^ t - x)dF{x)
Jo
is a renewal type equation and the solution is given by
Av(t) = 1 - F(t + y) + f {1 - F(t + y - x) }dH(x).

Jo
Since jjl = E{Xx) (1 -F(x)) dx < oo,

- Jo
f
•'O
I 1 - F(t + y)}dt = I
t
{1 - F(y)}dx < oo and {1 - F(t + y)} is
nonincreasing function of t. By key renewal theorem we get
Iim P(Y,
/ — > oo t —> ° °
> y) = lim
i; = 1 - F(y)}dy,y > 0
Now the set of events {Yt > x and Zt > y } is equivalent to {Yt_y > x + y}.
Therefore
lim P(Zt > y, Yt > x) = lim P(Yt_v > x + y)
In particular,
= /i-' r
•'.V
;i - F ( z ) } d ( Z).
lim P{Z, > z] = lim P(Z, > 0) = J {1 - F(x)} dx
Define Kx{t) = P{S( = Y( + Xt > x}. Therefore

1 if y > max (x, t)

P{St > x \ X l = y} = K x(t - v) if y < t
0 otherwise
Then by the law of total probability we get the renewal type equation
Kx(t) = 1 - F(max (*, t)) + f Kx(t - y) dF(y).

Jo
By key renewal theorem and integration by parts
lim P(St > x) = lim K x (t ) 1 n

oo / — > oo
V .Jo
1 r ,
V Jv
Therefore lim P(St < x) = jlH-

t —>oo
i
/; ydF(y).
Exercise 4.14 Now Yt is the cost incurred at the ith replacement cycle, then
C\ with probability 1 - F(T)

Yi =
C2 with probability F(T)
and E(Yi) = C ^l - F(T)) + C2F(T). Since the expected length of a replacement

cycle is
[1 - F(x)]dx.
E (min (.Xh T)) -
Jo
f
We have the long run mean cost per unit is given by
Ct [l -F (Q ] + C2F ( r )
6(T) =
|Jo [1 - F(x)]dx
Differentiating 9(T) with respect to T and equating to zero we get

(-C ,+ ( C2) f ( T)
!/: [1 - F ( x ) ] d x \ - { C ,( l - F(T)) + ( C j - C i ) F ( T ) ( l - F(T))}
f cT
[1 -F U )]& =0
ll
which yields the result.
Exercise 4.15 Let U(t) be the interoccurrence distribution for N(t). Then
1 - U(x) = P(N(x) = 0) = P(Ni(x) = 0, 0) = [1 - F(x)]2 (1)
Let K|(r), Y2(t) and Y(t) be the excess life at time t for the processes Nu N2 and
N respectively. Then, because the processes N\ and N2 and N respectively. Then,
because the processes N] and N2 are composed and we must have
Y(t) = min {Y2(t), Y2(t)}
and P(Y(t) > x) = [PiYiit) > x)}2
Letting / —» °° and by exercise 4.14, we obtain
[1
if ( 2)
where
-f Jo
[1 - U(y)] dy, the mean of U(y).
Differentiating (2) and using (1) gives
- [ l - F W ] 2 = ill-t/(K )] = ^ -
j; [1 - F ( y ) ] d y \ [ l
i.e. 2v r
1 - F(x) [1 - F ( y ) ] d y (3)
n 2J . v
Letting G(x) = 1 - (x )and differentiating (3), we get

F
2 v c(x)
(4)
dx
Since G(0) = 1, the solution of differential equation (4) gives
G(jc) = 1 - F(x) = e ~ where X - l v I / x 1.
Hence all of the Nx(t), N2(t) and N(t) are Poisson, because of the fact that sum
of two Poisson processes is a Poisson process.
Chapter 5
Exercise 5.1
f (s) = 1 - p ( l - s)^ is a Probability generating function iff the coefficients in the
expansion o f / (5 ) are nonnegative and sum to 1, since (3> 0 ,/ ( l ) = 1.
f \
( P )
By the binomial expansion f ( s ) = 1 - p I M ) "
n- 0
V J
( f P '
\
2 , 1 - p > 0 (since p < 1).

1
1
II
^3
( ~ s ) n
n= 1
V )
R\ n -\
Also = P ( P - l ) . . .(P -n+ l)/n'.= (-D
n n\
and (1 - (5)(2 - fi) . . . (n - 1 - P) > 0, since (3 < 1
' p\ t 1X2„
Coefficient of s" is - (-I)” = p — ^— P(l - f 3 ) ...( n - 1- /?) > 0, since
/? > 0, for n > 2.
P
Also for n - 1 we have - p (-1) = pP > 0. H e n c e /(s) is a probability
u
generating function. To compute the iterates, we use induction.
For n = 1,/,(s) = /( s ) = \ - p(l- s / .
Assume for n= m f m(s) =1 - p l+P1(1 - s)^m, then
/m lW = f[fjs)]= 1 - p { ^ '+/?+ + r _ ' (1 - S)P" )P
= i - P{p« +p+- +r ~
'i(1 _ _ s ) p”+'
Exercise 5.2 E l x n = E\ lim E I x„ (since X,7> 0, by monotone

n=1 N —> °o M=1
convergence theorem)
N N N
= lim
yV—
Z= i E( Xn ) - limN —Z (EX i) " = lim S #»"
>oon - \
h N —>°° n — 1
.V+l
= lim m - tn , since a?i < 1.
1- m 1- m
Exercise 5.3
/ (5) = as2 + bs + c, a > 0, b > 0, c > 0 and a + b + c = 1.
Hence m = f '(I) = 2a + b = 2a + (I - a - c ) = a - c+ 1. (1)
Now only root of / (5) = 5 are s = 1 and s = * ~ ^ ~ ^ = ~
Since the extinction probability is less than 1, it is d a , for otherwise from (1)
m - 1 > 0 => a > c => a/c> 1 => da < 1. Therefore, s - 1 cannot be the probability
of extinction and d must be d a .
Exercise 5.4 qn = 0n(O) = 0„-i(0(O)) = (^(po)* 0 < p0 < 1.

Exercise 5.5 Part 1 can be proved by induction or by a careful argument on
branching process with more than one ancestors.
„(«.*) = A2:=0 ;=0

2 P[X„ = k]uk= X 2
fc=0 7=01
=*)«*x
= k=0
Z P[X/1 = ifc]MA & P V * 1
= 2 P[Xt = *][«&-/ (x)]k =(h (*)]

*=0
if we can show that Z p {k" l)x j = [<j)n-i(x)]k. ( 1)
Proof of (1): Writing n - I = m (1) becomes Z = [Qm(x )]k • ( 2)
For m = 0 = <5;v => Z 7 = x* = [0b(*)]* since <fo(x) = x.

Assume (2) is true for m, then
| 0< +i-V = | 0 | 0 /’[X«+i =y.X B = /| *o = *]U
= 2 I P[Xm+i= j \ X m P[Xm
=
= l p [ X m = l \ X 0 = k]
= j t p t f ' M x ) ] 1 =[<t>m«Kx))]k =[<t>m+l(x)]k.
„ d2(jh (u,x)
1 E ix - x ' ' — l f c j 7 - 1 - '- -
Now, 90, „ (u, x)l du |„=1 = 0'[w0n_, U)]0n-/(*) |u=)= 0'[0„_, (*)]0h-/
90/,„(«.Jc) .
Hence [u=l. v=l = 0 "d )[0 n- /( ^ ) ] ^ - /( ^ ) ^ - /( ^ ) U i
9 udx
+ 0'[0„_/(*)]0UU ) |*=i
& -,(!)-1 + 0/U) «U;U)
= 0 " ( l ) w n- ' + m n" ' 0 / ( l )
= m "-1 (0 /'(l) + 0 / ( 1) ),n>l.
o 2m n~l ( mn- 1)/(w - 1) if * 1

3. Var (X„) =
ncr if m= 1
cr2m; '(/r^ - l)/(m - 1) if m * 1

Var (X ;) =
/(72 if m = 1
cr2/n"" (m' - l)/(w - 1) if 1

C o v ( X „ , X; ) =
/CT2 if m = 1 and
CovjX^Xt) a 2m n- l ( ml - 1){m - 1)
VVar(X/2)Var(X/) (m - 1)cr2 ^]mn+l~2 (mn - l)(m/ - 1)
= ^ m n~l {ml - \)!{mn - 1) if m * 1
= *Jlln if /w = 1
—> 0 as n —» oo
Thus, /?„/ —» 0 if m < 1 as n —> If w > 1, then mn —> °°, m~n —> 0 and hence
Rni —■> V1 - ni 1 .
Exercise 5.6 (i) For Poisson distribution with parameter Athe generating function
0(z) = <?A(z"l) satisfies the difference equation A0(z) = 0'(z); (ii) Similarly for
Binomial distribution with generating function 0(z) = (q + pz)" where parameters
are a? and p satisfies np0(z) = (q +pz)(p'(z)\and (iii) negative Binomial distribution
with (Parameters n and p) with generating function </>(z) = (pz)n/(i - qz)n also
satisfies np$(z) = (1 - qz)<S>\z). All of them are desired form of differential
equation. Now from the relation <j>n(z) = 0(0„_i(z)) differentiating with respect to
z, we get <p'(z)
= 0'(0„_i(z)) 0„'_i (z) (1)

Apply the assumed differential equation (c + dz)Q'(z) = a + Z?0(z) to equation
(1), yielding 0n(z) = [(a + 60(0,,_i(z)))/(c + d(t>n^ (z ))l <t>'.\ (z) = [{a + b0n(z))/
(c + d<()n_x(z))) (t>n-1 (z). Multiplying the last relation we get 0n'(z)[c + d 0„_i(z)]
= [a + 60„(z)]0„'_i(z) (2)
Now any probability generating function G(z) is differentiable any number
of times within the unit circle and G(/:)(0) = k\ pk, k> 1. Hence,
P(Xn+{ I )=
k+
= (0) (3)
Also note that qn = P(Xn = 0) = 0n(0). (4)

Now differentiating k times both sides of (2), using (3) and (4) and rearranging
we get the desired recursive formula.
Exercise 5.7 Clearly E(Xn!Xn_{) = mXn_x a.s. Also EXn = mn. Hence Yn =
Xn/EXn is a martingale. Also Var (Xn) — o2lm2n~2 if m * 1
Hence Var ( Yn) — a 2/m2 if m * 1
< cr2 if m > 1
Thus, sup Var (Yn) < oo Hence the martingale {Yn} converges a.s.
n
Exercise 5.8 Let ph k = 0, 1, 2 be the probability that one red cell produces k
red cells in one minute span. It is given that p0 = 1/12, p x = 2/3, p2 = 1/4.
We note that the 0th generation starts with one red cell.
1 9 1
Then P(s) = — + ' | 5 + J ‘s2’ wêre P(s) is the probability generating function
of the number of off springs per individual.
Now P'(s) = ?- + \ s , P ,(l) =l ( > l ) - > m > 1.

5 2 o
Hence probability of extinction is the root (less than 1) of the equation
5 = P(s)
or s —— 1 +2 —s + —
1 s2
12 3 4
or 52 1 1 n
T "3 n -0
or 3s2 - 4j + 1 = 0
or (3 5 - 1)(5- 1) = 0
or 5 = 1 or 1/3
Hence probability that the blood culture will eventually die out is 1/3.
Chapter 6
Exercise 6.1
For / > 0, P„(X(t) =1) = l/2pM(0 + l/2p
and Pn(X(t) =1) =1/2.
By Chapman-Kolmogorov’s equation
P\,\{t + s) =P\,i(s)pul(t) + pi,_i(i)p-i,i(0 (2
From (2), we get the functional equation
a(t + s) =a(s)a(t) + (1 - a(s)
= 2a(s)a(t) + 1 - afs) - a(t) (3)
Put a(t) = 1/2[1 + h(t)].Then equation (3) becomes
h(t + s) = h(s) h(t) (4)
This is the Cauchy-functional equation whose only continuous solution is
h(t) =<rA', A > 0
From (4) and (5) we obtain that a(t) =(1 + e~^
Exercise 6.2 This is a two state birth and death process. The Chapman-
Kolmogorov equation for a homogenous M.C. is given by
Pil(t + h) = p ik (0 pkj(h), k = 0, 1
For i = j = 0, we get the equation

Pooi* + *) = Poo(0 PooW + Po\(t) P\o(h)
= (1 ~ a h + o(h))p00(t) + Po\(t)(3h + o{h)
Taking limit /z —> 0, we get
dpoo(t)
= ~a Poo(0 + ( 1)
dt
Since we have
PooM + Poi(0 - 1- (2 )
equation (1) becomes
<fpoo (0 = P - (a + (t)
dt
(3)
Solution of equation (3) subject to the initial condition poo(O) = 1 >s given by
Poo0 = Poo(O) + f - v)/3^

Jo
= eâ+p)'+ p e-(a+P)t l/(a + P) (e(a+p)‘-l)
P e -(a+P)<' o
a +p a +P
« + _JL _ , > o
a +P a + P'
Similarly Pio(0 = ^ P -<a+0)t

a+P a +P
Also p0(t) = p0(0)poo(?) + (1 -Po(0))Pio(0
- P r+(Po(0)-----
a + P a+ P
_P _
+ e~(a+P)' if p0(0) = 1.
a +P
Exercise 6.3
Poo(0 = a + [
+P a +P
IT if X(t) =
j given X(0) =
Let fy (0 =
lo if X(t) * gj iven X(0) = I
Then E(YjXt)) = p,,(0- The expected time spent in state is given by
Pij(t) = £ J Yij(r) dr = J dr] = J dt
(by Fubini Theorem)
Then /!„(>) - £ P„ « ) * = _[ +— ,-(01+0)1 dt

p
Pj
+ al(a+ p ) 2[1 - e-<a+P)‘]
a +P
In a similar way the other functions Po\(t), /Li\o(t) and yUu (r) can be determined.
Exercise 6.4 Put qn = (v + /A), qu+l = v + /A, <7^ = 0 if /: i + 1

in Kolmogorov’s forward equation.
Clearly p00(t) = e~v\ p'0l ( t) = -(v + A)
=- (v + A)p0i(t) + ve~v‘. (1)
(1) Can be solved with the aid of an integrating factor. Let h(t) = elv+M,pl)](t).
Then (1) may be expressed as ~~jj~ = or h(t)
= /i(0) + f ve*Mdu = ¥^- { e h -

Jo
1).
Therefore, p0(lr) = e^v+X),h(t) = -y ( e 'w - e-(t,+A)' )

A
= (1 (2)
Now pq2 0) = - (v + 2A)/702(0 + (v + A)p0i(0
= -(u + 2A)po2(0 + (i; + X ) ± e ~ vt(\ - e - At) (3)
Similarly, as (2) we get
A ) 2 ( 0 = e T-(.v+
2X
)t j e ( v +2X
)u(V etp
R- W ( l
U + A) f '
_ g -(u+2A )/ I
(e 2 Xu „X u
eMi) du
Jo
w+2A)» + A ) J g 2h 1 <?A' - 1
2/1
_ u(t> + A) |-e-vt _ 2e-(v+X)i + A)']

2A
i 2 £ ± i ) €- „ [ , _ e -A,|2
2a
We shall prove (3) by induction on n. Assume that
1,(0+ 1 ) . . . ( » ■ > ( » - i w e. „ (1 _ e. „ . hold
Pon(t) = (4)
n! An
Then,
P™ (0 = -(u + (n + l)A)p„n+i (r) + (i> + nA)pon (f) yields
Po,n+\(t) = e-(vHn+')X)l e(v(v + nX)pm{u) du

Jo
_ v(v + A ) . . . (v + A
n ) ^_(V+{n+l)X
)tf e(n+i)A»
n\A" Jo
(substituting the value of pon(u) from (4))
Letting x = eXu yields
J e(«+DA«[i - e - XuT du = j ^ * " ( 1 - 7 } dx
Xt i \ n+ 1
(( X \ \ ndx
j 1 I (* ~ 0 " +1 1 (eM - 1)
4 j; - i) - - r
A n+1
Ji A 1
v(v + A ) . . . (y + hA) , _ n+,

Therefore PO,n+l U )
(n + l)!A n+1
Exercise 6.5 Consider a branching process starting out with i particles. Let
be the times until these particles split apart or disappear. Then
. . . , each has exponential distribution with parameter 0and hence X\ = min
(£b . . . , £;) has an exponential distribution with parameter g, = iG. Whichever
particle acts first has probability p of splitting into two particles and probability
1 - p of disappearing. Let p/; (f) be the probability that starting at i X(t) jumps to
state j after waiting a random length of time. Thus for i > 1
Pl i+i(0 = P and Pu-i(L-i) = 1 - P

State 0 is an absorbing state. Since A, = qiPi i+\(t) and = qt • we
conclude that the birth and death rates are A, = iOp, and = /0(1 - p), i > 0.
Exercise 6.6 Suppose there are initially j particles present. Let (£b . . . , be
the times at which these particles split apart or disappear and let rj be the first
time a new particle enters the system. We interpret the description of the systems
as implying that r] is independent of £b £2, . . . , r\ are independent exponentially
distributed random variables having respective parameters 0, 0, . . ., 0, A. Thus
Tj = min (£b . . . , rj) is exponentially distributed with parameter
qj =j 0 + A and P(r, = 77) =
The event [X ^ ) = y + 1] occurs if either Tj = 77 or Tj = min (£b . . . , £y) and

a particle splits into two new particles at time Tb Thus
« ■ 7 5 T T 1(l - ■ ^ 1>') ■- 7 5 T T + 7 5 T T P-
Hence we conclude that the process X(t) has birth and death rates
X q- j P j , J+\ = jOp +A and Hj = q,pu y_, =j 9( l - p).
[Note that equation (1) follows from some standard properties of independent
exponentially distributed r.vs: If . . . , be independent r.v.s having exponential
distributions with respective parameters . . . ,0 n. Then £(d = min (£b . . . , £„)
has an exponential distribution with parameter 6\ + . . . + 6n and P(t;k =
Q
min (£h . . . , £„)) = -2------- , k = 1, 2,. . ., n. These results can easily be
■+ 0n
verified in the following manner.
P(min(^, . . . , & ) > * ) = P(£i > x, . . . , £n > x) = > x) . . ./>(£„ > x) =
.. r =6?-(0i+ Hence £(1) is exponentially distributed. Set
rjk = min (%j : j * k). Then T]k has an exponential distribution with parameter
fik = Z 6j, and and rjk are independent.
Thus = min , 4 )) = />(& ^ Vk)
= f e ke -(Sk+Mxdx = -s - ^ = -s----- ^ s
J0 Ok + Pk 0\ + - - . + 6 n
Exercise 6.7 Now Kolmogorov’s forward equation becomes
Po(t) =- Ap0(r) + p p x(
and p'j ( t )= XpH (t) - (A +jp)pj(t) + + j )
Consider the P.G.F. of {Pj(t)} u) = p s ( ) u1. We get
dg(t, u)/dt = S p'(t)uj

7=0
= -Ap0(0(1 - u)p p x{t)(\ - u)

- Apj(0( 1 - u)u + (1 - +...
= -A(l - u )7 =l 0 PJj (t)u> + jU(l - u) 7Z

=0 J
= -A(l - )g(t,u) + p(l

u
So g(t, w) satisfies linear partial differential equation
dg(C«)
— ^ ------ /l(l - u) ,—
9g(^ C«) , ,1, - u)g(t, u)
— = —A(
Suppose that T(0) = N, then g(0, u ) = uN ( 2)

is the initial condition. Solving this by Lagrange method (Recatti) and using the
initial condition, we obtain
g(t, « ) = { ! - ( ! - »'e)u N
) exp { - £ (1 - n)(l - e~>*) J
Note that the first term on the right of (3) is the RG.F. of Binomial distribution
with p = exp(-jur), whereas the second term is the RG.F. of the Poisson distribution
with mean a (t) = (1 - Therefore Y(t) = Y0(t) + Y{(t), where Y0(t) is a
Binomial component and Y\(t) is the Poisson component and they are also
independent for all / > 0.
Expanding the P.G.F., we get
min (N,t)f AT a y * - e
Pj(t) = exp {-AJp(l *“)} I
=0 ks a . (T a j!
for j > 0 and
E(Y(t)) = Ne-* + £ ( 1 - e - f ) .
P
This formula was first time obtained by Palm.
Exercise 6.8
(a) The system of difference-differential equation in this case is
p'n (r) = -n(A + p)p„(r) + A(/i - 1 ( )+ + l)p„+l (?), > 1(1)
Po( t ) = ppx(t),pn(0) = <5„, (2 )
Let G(z, ?)= J=0

S p„(?)z".
There = 2 npn(t)zn~]
dz "=i
and — - 5 o' friz"

dr ~ nt o Pn( )z '
Multiplying (1) by zn and summing over n = 1, 2, 3, . . . and adding (2), we get
= - U + /?) % npn (?)z n A£ 1 )p„_, (

Ot n- 1 n-\
+ a? n2,
=\ (n + 1)/Vn ( ?)z" + p\ (?)
- (A + ^ )z ^ + Az2^ p + / J r p
az dz dz
2i dG
= [p - (A + p)z + Az2]
dz
= (z - l)O z - M) G(Z, 0) = z (3)
is a partial differential equation of Lagrange type.
(b) and (c) The characteristic equations are

dt _____ dz______ dG
1 (1 - z)(Az -jU) 0
Hence G(z, t) = c{ and —■= (1 - z)(Az - û) is aRiccati differential equation.
Putting 1 - z = 1 / m this reduces t0 ^ ~ (A* ~ + A = 0 which is a first order

linear differential equation with constant coefficient and is easy to solve. After
changing back to z, we obtain
C, = e{^ X)' Az “ f1 so that G(z, t)

z- 1 ( z- 1
where the function \jf is to be determined from initial condition.
G(z, 0) = z =V Hence i//(k) = “
i) + Z(A - )
Hence G(z, r) =
( A e ^ ' - / t ) + Az(l - e ^ ' )
1- _ ; f 1- 1
Put £ (0 = ju (4)
then G(z,r) = g( 0 + z{i - g ( 0 - n(t)} if A * n (5)

1 - zt?(t)
and | ( 0 = ?7(0 = and G(z, = — if A = ju ( 6)

1 + At —zAt
By expanding G ( z ,t) with respect to z and collecting coefficient of
Po(0 = £ (0
if A * (7)
Pn u ) =(i - $(0)(i - r 1, « > i
A?
Po(0 = 1 + At
and if A = ( 8)
(A O " -1
Pn (0 = n>1
(1 + At)n+1
(d) if A = Jta nd if
p,p0(t) = T * ^ =^
(e) From the definition of |(t) and r/(t) we see that
E if ju < A
and
lim r\(t) = A
1 i f jU > A
S i n c e f o r A * jU, th e p „ (r) c o n t a in s e it h e r 1 - %(t) or 1 - r](t), f—

lim p n ( t )
»oo
= 0
f o r n ^ O . S i n c e p 0( 0 = lim p 0 ( t ) = 1 ifp > A a n d ~ if p< A. T h u s u lt im a t e

Hoo
e x t i n c t i o n is a c e r t a in e v e n t u n l e s s t h e r a tio A /ju = b irth r a t e /d e a th ra te e x c e e d s
one.
(0 By d if f e r e n t ia t in g G ( z , 0 w it h r e s p e c t t o z, w e o b ta in
f e i f A * p an d
£ ( * (0 ) =
[ 1 if jli = A
1} if A * p
Var (X(r)) = A - p
2 A/ i f A = jU
Alternative derivation
m(t) = E(X(t)) = S /*/?„ (0 and mo(r) = EX2(r) = S rt2p„ (*)•

n=l n=l
Multiplying both sides of (1) by n and summing over n = 1, 2, 3,. . . and adding
Z np'„(t) = -(A. +iu) £ rt2p„(?) + A 1 n(n-l)/>„_i(/) + jU I

n= 0 n=l n=l n=l
Now T, n(n - l)p„_i (0 = S (n - l) 2 p„_, (t) + £ (n - l)p„-i (t)

n- 1 n=1 n=\
= m2(0 + m(t).
Hence £ n(n + 1)pn+

x( /) = £ (n +l) 2 pn+l (t) - £ (n
n=l n=\ n- 1
= {m2(-t) p i( ( ) | - {m(r) -p i
= "i2<0 - '"(0
oo
and I (0 = m'(t)
n=\
Therefore m\t) = - (A + p)m2(t) + A{m2(0 + m(f)} + p{m20) ~ w(0}

= (A + p)m(t)
Solving, we get m(f) = Ce{X~ ^
Initial condition gives m(0) = 1 = C.
Therefore m(/) = if A * p
m2{t) can be found in the same way.
0 if A p
^ U L ea-n)>{ea - ^ t _ i} j
A — /LI
Also Var [X(t)] = > 0 {since ^ — - > 0 for all t

X- n
2 Xt if A = p
oo \iX>JJL
Hence lim Var(X(t)) = oo if A = p
t—>°o
0 if A < p
(g) In classifying the states, we observe that state 0 (since X$ = 0) is an absorbing

state. Moreover {1, 2, 3 . . .} form a communicating class, so that we can
consider only state 1.
Since p x(t) (= p\\(t) essentially since = pn(0)) is given above
f P{{ t ) d t = \ a-i(o)(i
J0 Jo
= (^-A)3
I
o - Ae(A-">' ]2
[H
Putting v=n -Xe{A we obtain
l/A if A > p
i; Pi({) dt = Up if A > //
l/A if A > p
< oo.
The last case is obtained-either by continuity from the cases A > p and A 0 ( 1)
The forward equation reduces to
p'j(t) = kj-\Pij-\(t) - XjPy (t) (2)
From (1) and (2), since k 0 > 0

p'u(t) = - k t pa(t) which with initial condition ptj (0) = StJ, j > /, gives
Pu(t) = exp [- k^].
Since (2) is a linear differential equation, it is also easy to see that
Pij(t) = Pij(0)e Xjt + f kj _xe Xj{t S)p ij_l (s)ds

Jo
r xi < ' - • ' ) Pii-](s)ds if j

u '
Jo
Putting j+ 1
P. 7+1 ( 0 = A , f e A,+l(' x)pu(s)ds.

Jo
(b) Let n(0, i,j) = f e”e'p,y ( ) , 00.Integratingbyparts

Jo
f e~e‘p'j(t)dt =<re' p y ( 0 L +ef

Jo Jo
= 8jj + , /,;)
Using this in forward equation (2), we obtain
- 8i
j + 8n(8, i,j) = - 1) - A,7r(0,
So that
[9 + A7]tt(0, i j ) = + Xj_in(9, - l ), j > 0 (3)
If j < /, n(0, i,j) = 0 (being birth process).
Let i = j , then we get from (3)
n(0, i, i) = 1/(0 + k t) (4)
From (3) and (4)
A,
n(9, i j + 1) =
(9 + Aj)(9 + A,+i)
Proceeding successively, we obtain
A. A/+, . . . Aj_j
7t(9, i= J >i (5)
(0 + A,)(0 + Ai+1) . . .(9
Resolving the RHS of (5) into partial fraction, we get
8(17 , ij)= X — i l Z l , B ( * . (> y> = lim [ 0 + A * ]? r (0 , i ,y ) .

*=/ a + A;,
This gives the expression of B(k, /, 7). But the inverse Laplace transform of
[0 + k k]~l is given by exp [- t k k], t > 0 and hence we obtain the desire Pij(t).
(c) Pij(t) can also be written as
'H - ^ L
Pij(O = n Xk I exp (- if > i
J \ k=i k=i
- exp (-Xjt) if j = i
where y/'i^k) = rn- i (Xr - Xk). Putting Xk = kX, we get

r*k
P i j U ) = i(i + 1) • ■• (j 1)AJ '
- Xkt
x Z
* = ' A ' " ( I - *)((+ 1 (-1)1 ...(j-
fj-i^
_ e ( 1-
y j-
**Exercise 6.10 (Feller and Lundberg)

j
Define S(t,i j ) = (t)
= S pik (t) (since p,,(r) = 0 if / < / and r > 0)

k-i J
Since p^(0’s are transition functions, S(t, /, °o) = 2 /?,*(*) < 1. Then either
k-i
S(t, /, 7) = 1 or < 1 for all f.
Using forward equation we obtain
S'(t, i, j) = Z p'ik (t) = Z {Xk_xpik_x(t) - Xk p ik (r)}

k - i ‘ in* ' k-i
=- .jP
A i(?) for j > i ( 1)
Let cr((9, /,y) be the Laplace transform of S(f, /,7). Noting that S(0, i,j) = 1,
it follows from (1) and integrating by parts that
9(7(9, i j ) -1 = - Aj 71(9, i, j)
J Xk 1 . We see that
By previous exercise 7r(0, /, j ) = n
k=i 0 + Xk Xj
n-i
n 1+ e if k=i
Z l/Xk < 00
k>i
Jim Ajiz(9, i,j)
if kZ= i 1/A* = 00
Hence
(i) lim 0<7(0, /, j ) < 1 if the series Z 1fXk < 00
7—>°° k>i
(ii) lim 0cr(0, i ,j) = 1 if Z 1/Ak = ©©.

Consequently by monotone convergence theorem lim S(t, i,j) = 1 or < 1 for all
j —>°°
t implies lim
;-> °o
0 If°° e~dtS(t, i, j) dt = 1 or < 1 . Hence lim S(t, i j ) = Z
J 0
00
pik(t) =
*= *
lim 0 a (0, i, j) < 1 or = 1 according as the series k.=i

j —>OQ
Z 1/A* converges or diverges.
Let X(0) = x and Wn be the nt\\ jump time Wn = xt + . . . + xn where xk is the
interarrival time between (i + k - 1) and (/ + X:)th births. Since the interarrival
times rk are independent and exponentially distributed, the Ch.f. of Wn is
iO
¥n(0) = n 1 - -t— and clearly (from the theory of infinite product)
k=i J
lim y/n (0)
n—4 oo
= 0 if Z AIs.1
K
= ©©. Therefore, as n - > ©o, w - > oo a.s.
**Exercise 6.11 Let rk be the duration of occupancy of the £th state. If rk are
positive, by definition of Markov process they are independent with distribution
function P(rk < x) = 1 - exp [- Xkx\ and Erk = 1!Xk. Define r k = rk if xk < 1 and
Tk = 0 for rk > 1. Then Exk < Z Exk> Z

k=] k=1
P(xk > 1) < Z E( xk). Since xk s
k=1
are nonnegative r.v.s, Exk2 < E( xk) and therefore convergence of the series
Z E(xl) implies convergence of the series kZ= i Exk2 and hence of the series
k=i
Z Var (r £). Therefore by Kolmogorov’s three series theorem for independent

k -i
r.v.s Z xk converges a.s. if MXk < ©©. On the other hand,
l P ( T k >1)= Z (1)
k=i k=0
and £
k= 1 k=0
E(
k=0
t I) = £ e ' Xk + £ (1
Hence ^Zq 1/A^ = ©©but the series (1) converges, then the series (2) diverges.
Thus in the case of divergence of the series 1/Xk, one of the series (1) or (2)
diverges and consequently Z■xk diverges with probability one. For linear process
£ i a k =4 -2 i — OO. Hence after a finite interval of time linear process does

k- 0 A 1
not move out to infinity.
Exercise 6.12 (a) From Kolmogorov’s forward equations for birth and death
process we get for the unconditional probabilities pt{t) the system of equations
p'i(t) = - p,(r)(A, + Hi) + p,+i(f)A/_i + (O/Xf+i, i ^ 0 and p_,(0 = 0.

We assume that pt(r)’s are independent of f.
Then the above system of differential equations degenerates into homogeneous
difference equations
- (A, + Pi + Ai+I + = 0, i > 1
~ôPo + P\P\= 0-
Suppose that p* > 0 for k > 1. Then pj = (A0/pi)po and by induction, we find
AqAi . . . AqAj . . . A*_!
Pit = - p0, and Z p* = Po 1 + E
P 1 P 2 • • - Pit ' *=°J P 1P 2 • • Pk
(b) It follows from our assumptions that
Pk.k+ i(Af) = (m - &)(AAf + o(kt)y fc = 0,
£pA/ + o(At), 1<k <s

P * .* -i(A r) =
s p k t + o(At), m>k>s
PkMri&t) = o(A0, r > 2;

that is we have birth and death process with finely many states . . .Em with
A* = (m - &)A for k - 0, I, . . . , m\
pk = kp for 0 < k < s, p k - sp for s < k < m.
AqAj . . . Aj^_i
From part (a) by virtue of equations pk = ------------------p0, k = 1, . . . , m
P 1P 2 • • •Pk
^0 • • • ^k-\ ^
and E p k = po 1 + Z the stationary distribution is given by
k=o K ™ *=1 P\ . . . P*
the formulas p* = — p0 for /: < s

p;
1) (s + l)( A
Pit =
s £-.S u .
\* m
k ( k - l ) . . . ( s + l)( A_
Po = 1 + Z - + 2
k=0 J i = .t + l P
**Exercise 6.13
1 (a) 1 > pij (s) = Eoe~Av pjjn) > 0, 5 > 0 is obvious

(b) %QPij (5) = e(Ây)"/ /i!pj"'1 (by Fubini The
= 2 e - Av(Ay)"/n!= 1 for all s > 0.

n=0
(O') V 5 (,-) _ V V V „ - A ( .s+») ( ^ ) n ( ^ O ” (n ) (m )

(C) h P i i ( ' ) P i k {j -==0o n„ ==00 mmt= 0o e n! n!m: m ! P ‘JP *
_ £ £ g - A ( J + / ) i ^ £ ) ^ _ . £ ^ £ ) ^ L n (n+m)
n'. /n!
(by Fubini and Chapman Kolmogorov)
_ e-xu+i)£ J (/) (putting I - m + n)

n-0 l=n n\(I -«)! k
! rn
_ g-A(j+l) J“ ejk_
o(,)
• 2 (As)" (A/)'-"
1-0 l\ /i=0
A
= S P1
* [A (j + /)]' -Pik (s + /).
(d) Pij (5) - dy = - 1) + Ay X £-/lv (Ay)n-1/rt! (<5,/s are Kroncker’s delta)
< - A y + (As)2 . c + As/: as 0 < s < s0.

So lim | p ik (s) - 3,y | = 0 (by Dominated Convergence Theorem).
2. Same argument as in d applies to prove that A - X[P - I]
\ av
or
I îi ~ ~ Q ii = Xpa ’ * ~ J
Therefore 2^ atJ = A^ 2^ (py - 3ty)j = A(1 - 1) = 0.
So that the M.P. is conservative.

3 (a) Suppose i —>j in M then there is an n > 1 such that p \p > 0.
Now e~^ — p i{}n) > 0 for all n > 0, A > 0 and 5 > 0, so that
n\ lJ
Therefore i —>j in M .
Conversely, if i —>j in M then there is some s > 0 such that Py (s) > 0.
Now
e - Xt (Ay)"
and />'"1 > 0 for all n > 0, some 0 and A > 0
and there will be at least one n > 1 such that
e - Av(As)"
p '"'' > 0 =>p-(n)
'; " ' > 0 for some 1. So —»y in
n!
(b) and (c) We have to prove
(i) Urn p0 (.s) = Urn and (ii) J p,; (s) =jj- p\p
Proof of (i) Let p-n)—» coexists since the M.C. is aperiodic), then | Py(s) - a,
-Av
(As)"
< X | p)n - a j | and for every e > 0, there exists N(e) such that
n=0 n\
e -^ ik sY
for n > N | p {tJn] - a] \< ell. Since —> 0 as j —» oo, we can choose
.v so large that e Av U fT < J _f > yy A|so I (») _ a I < 2 for all

/i! 4/V IK'7 71
f ^,-Ay
(A s)”
> 1. Hence I p;/(5) - a , I as
5 —> oo.
r°° °o p {n) p -
Proof of (ii)
f/) J (5) ds = Zq ■J e”'1' (As)” ds (by Fubini Theorem)
= 1 £ n<n>
A n=0^/7
From (i) and (ii) we get (b) and (c) since j is transient in M iff
iff py (s) ds < 00 iff y is transient in A/.

n Jo
Similarly let M be positive and lim pJs) = m = lim 0,.. > 0.
.v—>00 J 1 n —>°° ■"
**Exercise 6.14
4. Let M be irreducible and positive with stationary distribution {7ry, k e S}.
Then | pl} (s) - n } | < Z -pn - e

//=0 n\
Let M be transient. Then lim pi} (s) = 0 = lim pjj since p;Jn) < f f , 0 0. For°o= Z p ^ }< Z p"
1 ' n n
1 < oo.
1- p
5. Let A be infinitesimal generator, Z an =0. Define a matrix P = —7 + / where

./
AT> sup | au | < 00 and P a stochastic matrix since P > 0, 2 p i; = -p-Z al} + 1
= 1. So P determines a discrete M.C. M (with some initial distribution).
Define p/7(s) = Z^ e~Ks P//0 • Then A = /C[P - /] = A by part (2) of
Exercise 6.13, and the definition of P. Hence M and M are the same by
Chung’s theorem. Hence M is positive <=> M is positive <=> M is positive by
Foster’s Theorem (By Exercise 2.36) ( Z j c = x} has a solution such that
0< Zi | jc,-1 < 00 (by definition of P) <=> Zi jc, = x; has a solution such
- f +
that 0 < X |jt,-1 < °° <=> %XiOjj = 0 has a solution such that 0 < X | | < °°)
This proves 5 of Exercise 6.14.
Exercise 6.15
-(a + A ) p y
1. | A —A/ | = y - ( a + A) = 0 or
P y -(a + A )
- ( a + A)[(a + A)2 - YPi - j8 [a y - yA - + y{ y2 + + Py\ = 0

or - A3 - 3A2a - A(3cr - 3 PY)- a3 + /J3 + y3 + 3a/3y
or A3 + 3A2a + M i a 2, - 3/3y) = 0 (.-. + y = a)
3fy
or A| A + ^ - l/2 ( /3 - y ) V 3 A + —— + 1/2 (/? —y )V3 = 0
So A = 0, - p + /a o r - p - io where p = — ^ and <7= —(/? - y)V3 .

2. Let P(0 = /? + Se~ip+l(*)t + Symmetries of A tell us that
Pn(0 = p2i(t) = p33(0
P d ( 0 = P 2 3 ( 0 = P 3 l W
Pi 3(0 = P 2 l ( 0 = P 3 2 ( 0
Also A2 is of the same form (has same symmetries) as A and so on.

Hence p n (t) = Z A,([° t n /n\ - Z A^} r" /rc! = p?2 (0. and so on. The given
n n
M.C. is finite and clearly irreducible (since p > 0 or /3 > 0) and hence positive
recurrent (If not obvious, use skeleton chains to reduce it to discrete time).
Hence lim P(t) = R. By Exercise 2.14 every row of R satisfies Xjc,ai7 = 0.
t ~.»°o I J
3
Since x { = x2 = x3 = 1/3 is a positive solution, X x x = 1, it is the stationary

/=l
distribution of P(t). Hence every element of R is 1/3. In particular R is real.
Note Actual computation also shows this.

To determine P, let S = S' + /S", T = T' + i T " where S', S", T'.
T" have real entries. Then P(t) becomes P(t) = R + e~pt[(S' + T')
cos (err) + [S" - T") sin (ot) + i[S" + T"] cos (at) + i[T' - 5 '] sin (at)
(0 )
Now P{t) is real, hence S" = - T" ( 1)
and S' = T (2)
But P(0) = I and hence I = P + S' + 7* (3)
Also P (0) = A, and hence A = - p[s' + T'] + o [ S " - T " ] (4)
From (2) and (3), S' = T ' = 1/2(/ - R) (5)
From (1), (4) and (5) 5" = - T" =■

— + £ ( / - /?)
This yields immediately (with convention above)
Su = V = 1/3
rv p 2p - 3a . 3
j,/// —~ j>t a// _—— + —- — 2------ r 0, since p =
2a 3a 6a
S' = V = - 1/6 if/ * j
P // T">// _
*5 // + l ~ ~ 1ii+ 1 ~
2cr 3(7 = ^ 4— - 1/3 V372 ( < 7 = i(jB - y ) V 3
Using these values in (0) the desired result follows.

4;r
3. Let A = then sin (cr/z) = 0, cos oh = 1
(P -r)V I
Let p„(/i) = 1/3 + 2/3e-ph
p„(/i) = 1/3 - 1 if i it j
which depends only on
ph = 2 P±J_ P=- =|( ^ r )

P-Y
All continuous time Markov chain with infinitesimal generator of the given
form and with P+ r constant have the same skeleton (X(/i), X(2h), X(3h), ...},
P-r
where h is defined above.
Since we look for solution of Elfving’s problem, we need that
p q r
/>(!) = r p q ,p * 1 (6)
q r p
All properties of transition function matrix P(t) (namely semigroup, non

negative, stochastic) are obviously satisfied if (6) has those properties. So
p = 1/3 + 2/3e'p cos cr
and q = 1/3 - H3e~p [cos cr- V3 sin cr]
(Note if p = q = r = 1/3 then cos a = sin cr= 0. Hence there is no solution for
the problem).
e p cos a - 3 P - 1
Hence (7)
e~p sin <7= (p + 2q - l)
, r- p + 2q - \
so that (7= V 3 — -------- :— .
j p —1
This determines infinitely many <7’s if we can find p also. If p can be determined
then every continuous time Markov chain with that p and one of the possible a
values is a solution to Elfving’s problem. Now from (7)
e~2p = e~2p (cos2 <7 + sin2 o)
= 3 ( p 2 + q2 + pq - p1
The right hand side is minimal for p = q = r = 1/3. This is excluded. Hence
e~2p > 0. But g > 0 , so p 0 so p < I - q. Moreover one of these
inequalities is strict since otherwise q = O.p = 1.
So p2 < (p + q){ 1 - q) or 3[p2 + ^2 + ^ - p - ^ ] < 0
or e~2p < 1. Hence p is well determined.
Exercise 6.16
(a) is completely similar to Feller-Arley process.
(b) Following Feller-Arley the P.G.F. of {pn (0)o
G(f, z) = [£(*) + z{ 1 - Z(t) - ri(t)}]{ 1 - zri(t)}.

By expanding with respect to z, we get

o(0 = &0, Pn(t)
P =(1 - £(0)(1 - »?(?)[r?(?)r\ n > 1
where £(?) = 1 -,r)(?) = 1 - 1/W, |l + J epptdt

ut2 ” I
and p= r---- A?. E(X(t)) £ (?) = Gj (?, z
Z n- 1
= (1 - £(?) - ?7(?)}{1 - ??(?)} + ??(?){1 - ??(?)}

{I - r](t)}2
!-£ (? )
= ^ = e ^ " 2'2
1 - r(?)
0 if Ji > 0
Hence E (X (? ))^ 1 if A = fj.
00 if jc > 0, // = 0
(c) The zero state, corresponding to extinction, is obviously an absorbing

state. The probability that extinction has occurred at or before time t is the
coefficient of z° in G(r, z). Since T is the time till absorption,
P(T<t) = P[X(t) = 0 ] = p 0(t) = &)
1—e = 1 - e - f /2.
When A = 0 ,p 0 (O|A=0 =
f V
I11 + Jo
E(T) = I* [1 -- rPo(t))dt= f = ifA =0

Jo Jo
Chapter 7
Exercise 7.1
1st Solution Let X(t) be the number of shocks upto time ?. Hence
Axk-'e~Xx
P[T< ?] = I e - A'(A?) " /n\ = f dx (i)
n=k Jo Tk
(integrating by parts).
k
Xk t k~ - 1 „ - A /
Differentiating (1) gives f ( t ) = ,t> o.

r*
2nd Solution
Let Xt be the interarrival time till the ith shock, then P[T < t] = P S xt< t
t-1
So the Laplace transform is 07 (s) = [0Xj(s)]k = • By uniqueness
theorem of Laplace transform the result follows.
Third Solution
Let T = Tk. Then by induction on k,
A*-1 uk~2 e"~Xu

““ . ^k e t+k-1
A'
P[t < Tk < t+ dt]f du = dt
Jo n k - 1) Tk
Exercise 7.2
P[Y(t) = m] = 2 P[K(r) = / n | X ( f ) = * ] f ’[A:(0 = *]

A:=m
( l\
= 2 m„k-m (A/)*/*! = (
k-m p q
V"V
since each arrival is registered with probability p independent of other arrivals,
(k\
and probability of m registered arrivals among k arrivals is p mq k~m,
q=l -p.
Exercise 7.3 F(t + h) - F(t) - P (failure in ( , / + /i) and n

P (failure in (t,t + h) | no failure in (0, t) ). P [no failure in (0, r)] = [g(t)h + o{h)}
[1 - F(t)]. Dividing by h and letting h40, we o
jE jjp j = g(t) =>

- log [1 - log c + J * (r
From the initial condition F(0) = 0 => c = 1.
Hence 1 - F(t) = exp J^-J /( x) d x
Note For g(x) = U 1 - F(t) = e h . If 1 - F(t) = e~t<x then
F’(t) =e-'a a t a-'.So g(t) = { F

_ = a t a~\
Exercise 7.4 Let r = W* - W. Then r is the interval time and P(r < s) =
1 - e-Av. Now P(N = k) = P[y(lV*) -Y(W) = k]
= E{P[Y{W*) - Y(W) = k | T= 5]}
'o
e (fis)klk\ Xe ds =
K- Jo
J ds
= A/(A + /i)(/i/(A + /*))*.
Exercise 7.5
(a) Let 0 < /0 < / ] < . . . < tm. Then
Now, {£,} are the i.i.d., X(t) being a Poisson process, X(tk) - and
- X(tk_2) are independent and Y(tk) - Y{tk_\) and Y(tk_i) - Y(tk_2) are also
independent.
( b ) ( 5 ) = Zo Eexp [iu| X(/) = ) =n]
= Z [0{ («)]B <rA' (Af)'7n! = e - /U exp[Af0{ (n)]
(c) £y(0 = £[EW 0 | X(0)] = £[X(r)£(©] = Af£(§)

Var WO) = £ [Var WO | X(r»] + Var (£ WO | X(01)
= E (X(t) Var (£)) + Var [X(f) £ ($ ] = Ar Var (£) + Ar (E(£)2)
= A? [Var (£) + (£ (£))2] =
Exercise 7.6 Since the interarrival times Tu T2, ... in a Poisson process are
i.i.d. with common distribution exponential with parameter A, £(^(7]) | Tx - t )
= Ar. Hence
£ W ,) T ,) = £ [r,£ (V(7,) | 7-,)]
= 2/A.
Var (/vcrWi) = £(M7",)£|)2 - £2 W , ) r , )
£ i £i__4_ 24 _6____ 4 _26_

A2 A2 A2A2 A2A2 " A2 '
Exercise 7.7 Since EX(t) = Ar, it follows that

EY{i) = E(X(t + 1) - X(t)) = X(t + 1) - Xt = A.
Due to independent increment of Poisson process X(r) the r.v.s X(s + 1) - X(s)
and X(t + 1) - X(t) are independent if | t - s \ > 1. Consequently
py(s, t) = 0 for | t - s | > 1. Suppose s < t < s + 1. Then
Cov (Y(s), Y(t)) = Cov (X(s + 1) - X(s), X(t + 1) - X(t))
= Cov (X(t) - X(s) + X(s + 1) - X(f), X(5 + 1)
- x(o + x(i + i) - x (j + i)).
Again by independent increment property
Cov (X(t) - X(s), X(s + 1) - X(t)) = 0
= Cov (X(t) - X(s), X(t+ 1) - X(s + 1))
= Cov (X(s + 1) - X(t\ X(t + 1) - X(s + 1)).
Also Cov (X(s + 1) - X(0, X(s+ 1) - X(0) = Var (X(s + 1) - X(0) = A(j + 1 - t).
Therefore Cov (K(^), Y(t)) = A(5 + 1 - 0 - By using symmetry
if 11 - s | < 1
PrC M ) =
if | / - s | > 1
Therefore Y(t), - oo < f< o o isa second order stationary process having mean
A and covariance function
p Y(t) decreases linearly to zero as 111increases from 0 to 1 and remains zero for
all larger values of 11 1.
Exercise 7.8 Now Y{t) = X(t) - Xt is a martingale.

Set X(0) = 0 a.s. Then )C(Ta) = a => E(Y(Ta)) = F(X(Ta)) - XETa = E(Y(0)) by
the optional sampling theorem (see Appendix IV). Also, Y(0) = X(0) - A. 0 = 0
a.s.
=» E(X(Ta)) = AE(Ta)=>a = AE(Ta) = >± = E(Ta) (1)
Also U(t) = Y2(t) - X t is a Martingale. Since Ta is a stopping time by optional

sampling theorem (see Basu [1998], Chapter 11).
E(U(Ta)) = E(Y2(T„)) - = 0) = 0
E(Y2(Ta)] = XE(Ta) = a( by ( l ) ) (2)
Also
Y(Tn) = X(Ta)- XTaandE[Y2(Ta)}= E(X(T„) = E[XE(Ta)
(••• X(T„) = a, E(T„) = a/X => X(T„) = XE(Ta). Therefore
E(Y2(T„)] = X2E(Ta- E ( T a)]2
=> a - X2Var ( T„) =>
Again V(t) = exp [- (t )+ Ar(l - e-0)], r > 0 is a Martingale.
X
G
Hence
£ [T O ] = E(
exp [- GX(T„) + Arfl(l - e - % = E(V(0)) =
=> 1= E{exp [- 0X(Tn)e] xp (- a T a)} where 1 - e-0)
1 = e-0a E[exp (- ccTa)] £[exp (- )] = e"e.
Again X ± a _ e-e = ee ^ = eae

a+X a +X
£[exp ( -a])=
a +X
Ta has a gamma distribution with parameters a and X.
Exercise 7.9 Let N(t) = Nx(t) + N2(t), A, + A2, A, > 0, > 0 be a Poisson
process, where [N{(t)} and {;V2(/)) are independent renewal processes.
k\
Then e~Xl (Xt)klk\= I
m=0
A 1m ^2
l k-m
= P(N(t) = k )= X P[M] (0 = rn] P[N2

m= 0
by independence of N\(t) and /V2(0- By inductive reasoning, since this is valid

for k = 0, 1,2, ... and all k.
e~XtIT [n J =■
P[yVl(f) = 7,11 P[Nl(t)
Putting k - m = n we get
g 1 e-^‘ = />[Af, (/) = m] P[/V2 = «].
Summing over n we obtain

(2.J t y
\ = P [ N l (t) = m] X P[/V2 (r) =«] = />[ A, (r) = m
n=0
Exercise 7.10 This problem can be looked upon as a birth and death process
with birth rates and death rates
A if 0 < ft < m - 1 [n/Z if 0 < ft < Aft

An = , = L
0 if ft > fti 10 if ft > m
respectively. The arrangement is as follows:

Let I be an interval of length h. Let {X(0, t > 0} be the number of busy
channels at time t with ptf h ) = P[X(t + h) = j \ X(t) = /].
Then
pnM+\(h) = P [a new call arrives during 1] + o(h) = Ah + o{h).
pn ,7_i (/i) = P [exactly one of n conversation in progress terminates during /] +
o{h)
1r r* if r
=
1)1 LJ o
I fie px dx
Jl J h
+ o{h)
= /i(1 - e ^h) e (n + o(h)

= njuh + o(h).
Taking h —> 0 we get An - A and jun = nA.

Let pn(t) - P[X(t) = n] = P[n channels are busy at time f], n = 0, 1, ... , m. Then
for a Birth and death process the basic differential equations become
= - Ap0(t) + npi(t)(1)
p'n (t) = (A + np) pn (t) + Ap„_, (0 ++ 1) (t) = 1, ... ,
We know that lim pn(t) = n n exists and are independent of initial conditions
t->0o
pio(0) = 1 and /?„(0) = 0, n * i. Also lim p'n (t) = 0.
Either we can derive explicit solutions of the differential equations (1) and (2)
which is known from D.G. Kendall’s result or just taking limits t —> °° in (1) and
(2) and putting p'n (t ) = 0. We get
Ano = p,7i\
(A + np)nn + Ann_x + (n + l)pnn+\
Let an = nfiUn - Ann_\. Then from (4) we get
an+x = a for all n > 1. So a x - a2 = ...
But from (3) we get a x = 0 and an = 0 for all n > 1.
Therefore, from (4) we get nfinn = Ann_x for ft > 1 or n n - — • - 7 l n-X(jU* 0 ).
P
By iteration we get
r a y /• m m
n - I — I /ft! 7r0for all n > 1. But 1 = ^ n k = /r0 ^ 1!
Therefore, 1
/T0 =
IT
So the stationary probabilities are given by
ju) n\
ft = 0, 1
X | , ,,
*=o I ju) k\
Exercise 7.11
Let EY(t)
-rJo
P(Y(t) > x) dx - I < oo ( 1)
Notethat P (K (f)> * ) = fJo*o P(K (0 > * | S, = y) dF(y)
- f
Jo
P(Y(t-y)> x)dF (y) +
J/ +JK f
where for y < /. Since the origin is shifted to y, so that point t becomes t - y, while
for y > r, we observe that
P(Y(t) = y) = 0 if r < y < f + x and = 1 if y > t + x.
Substituting this recursive formula in (1) we obtain
/[1 -fW ] [1 - F ( y ) \ d y (2 )
■ f
Since F(x) is continuous, the right hand side of (2) is differentiable;
differentiation and the solution of the resulting differential equation give the
exponential distribution, since F(0) = 0. We know that N(t) is Poisson iff F(x) is
exponential. In fact,
oo > /
■a P(Y(y - t ) > x ) d F ( y ) d x +
JO Jr+.v
dF(y) dx
T[( P(Y(y - t) > x) dxdF(y) + f

Jo
[1 - F(t + x)] dx
(since / does not depend on t)
= f / dF(y) +
+ If [1 - F ( y ) ] d y (y = t + x)
Jo Jr
= lF(t)+\
I [1 (F( 0) = 0)
Hence /(I - [1 F- F(y)]

( t) )dy= and / — (1 - F(t)) = 1 - F(t) with
dt
initial condition 1 - fXO) = 1 gives 1 - F(t) = e~t/!. Hence F(t) = 1 - e~tn
Therefore, the Renewal Process Y(t) is Poisson.

Exercise 7.12 (a) Let p{t) be the probability that an injured man is still in the
hospital at time t. Let T be the time spent in the hospital and S be the time up to
the accident. Let Nt be the Poisson process.
P[X(t) = k] = 1 e~Xt (Xt)m! (P (t))k [l

m- k
\ kj
= e~hp{l) [Mt)p(t)]k/kl (1)
Then p(t) = P[T + P[T > t - s | S = s] P[s < S < s + ds] .

Jo
But since the accident has happened according to an exponential distribution
we have by the theorem on order statistics and Poisson process that S is uniformly
distributed over [0, t]. Hence, since T and S are independent and
P(t)=f P [T> t-s] — =\f [l-F(r-f)]rfs =ia (

Jo 1 1 Jo 1
We have from (1) and (2) P(X(t) = k) =e~XaU)

k = 0,1, 2, ...
(b) E(T) < oo f [1 - F(t)] dt < oo.

Jo
Then P [ lim X(t) =k]= e~mT) Ik!
/ —>oo
because lim a(t) = E(T) < ©o (integrating by parts).

t —>oo
[0 if r < L
(C) F
(t=
(l if t>L
c \t if t < L
Therefore a(t) = [1 - F(x)]dx =
Jo u if t>
{e~M ( At)k if t < L

So P[X(t) = k] =
\ e - x' ( AL) if t > L
Chapter 8
Exercise 8.1 Since Brownian motion X(t) is an independent stationary increment
process to derive the joint density of X(t{), X(t2), . . • , X(tn) (tx < t2 < • . • < tn)
subject to the condition X(0) = 0 a.s., we only have to know the probability
density of Zx - X(rj) = x x, of Z2 = X(t2) - X(t{) = x2 - x x, etc. and finally that of
Zn = X(tn) - X(tn_x). Since the Jacobian of the linear transformation is unity,
making change of variables the joint density is given by
f(x\, ■■■, xn) = p(xu t\)p{x2
where p(x, t) = t xp ( - x 2/2t)
Now consider the case n = 3 with the condition X(tx) = X(r3) = 0 and more
specifically suppose that t\ = 0, t3 = 1, t2 = t(0 < t < 1). By (1) the joint density
of X(t) and X(l) is
f(x,y) exp - 1/2 + ( y - * ) 2/(l - / )

2n^Jt(\ - t)
/ U ,0 ) 2njt{r^7C
Xp ( 2r(l- / ) ) '
The marginal density of X(l) = Y at the point 0 is
fy (0)=L i ^k^T e
)xp(2^1-0) =vfe
Therefore, the conditional density of X(t) given X(l) = X(0) = 0 is given by
fix I X(0) = X(l) = 0) =f(x, 0)/fy(0) = - .......- L -
J 2 n t( l - t )
Exercise 8.2 This can be reduced to the preceding exercise 8.1 as follows.
Since X(t) has stationary increments, the r.v. X(t) subjected to the conditions
X(ti) = A and X(t2) = #, has the same density as the r.v. A + X(t - t{) under the
conditions X(0) = 0 and X(t2 - t\) = B - A. This clearly has the same density as
the r.v. A + X(t - tx) + —— — (B - A) under the conditions X(0) = 0 and
h ~h
X(t2 —t j) = 0.
Exercise 8.3 (a) We must compute the probability p that the particle will cross
zero level at some point in the time interval (a, b). Let px denote the probability
that the particle located at A at time a will cross the zero level in the time interval
(a, b).
Then p = f pzp0[\X(a)\ = MdA
Jo
where Pq(A) is the conditional probability of A given X(0) = 0.
Now Pa = P{ min X{t) < 0 | X(a) = A}
a<t<b
= P{ min (t) <- A |
X
a<t <b
(by spatial homogeneity of Brownian motion)
= P{ min X(t) < A | X(a) = 0)}
a<t <b
(by symmetry)
â<t<b-a ~^ (by temporal homogeneity of X(t))
= P0[Tx < b - a ]
where T\ is the first passage time across the boundary A of Brownian motion.
Hence
p= P0[Tx < b ~ a ] P 0[\X(a)\ = X \ d l

I
-T (2na)-'l2e - A /2aP0[T
= (2lna)v2 f (2tt)-,/2 A f r- 3m/ 2 e^ - A 2 H a ,- A 2l i t

dtdX
Jo Jo
= X~xa 1/2 J t 3/2 jj A exp[-A 2(r x + a 1)/2 ]^ A | dt
- n~xa~xl2 J0 t~m {atl(t + a)} dt 2 _ 22 t + a

^putting u2 =A
2at
b-a
= 7Txa x/2 f t 1/2 (t + a) 1Jr
Jo
((b -a )!a )m
= 47T 1 Js/(1 + s 2 ) (putting t = o r )
Ml
b-a
= 2n 1tan 1
Setting a = tan"1 I —- — gives cos a - (a/b)xl2.
Hence, p = 2/7TCOS"1 (alb)112.

(b) Note that if 7 < a, there is no zero in the interval (a, t) and conversely.
Therefore
P[T < t ] ~ \ - p - \ - 2lncos~x(alb) = 2In sin"1 (a/b)xl2,
since 7T/2 = cos"1 Q + sin"1 6.
**Exercise 8.4 Since X(t) is stationary and Gaussian,

E(X(s) X(t)) = E(X(s + T)X(t + r)) or I\s, t) = I\s + T, t + T).
Taking
T - - s. I\s, t) - r \ 0, t - s) = R(t - s), say if t > s > 0.
Since E(X(t)) = 0 and Var (X(t)) = 1 for all t > 0,
Pis) = AO, 5) = tf(j).
By the condition of Gaussian Process being Markov (see Theorem 8.4)
no, s + t) = no, s) r\s, s + t)/i\s, s)

= no, 5)no, t)
i.e. p(s + t) = p(s) p(t). (1)
Since X(t) is stochastically continuous, p(s) is continuous at 5 = 0 (see Appendix
II), then because p(0) = 1, we have
p(t + h) - p(t) = p(0[p(/i) - 1] -» 0 as h -> 0.
Therefore p(t) is continuous at every real number (see Appendix II). Then the
unique solution of the Cauchy equation (1)
p(t) = e~c\ c > 0 and t > 0.
Since p(t) = p (- f), p{t) = e~c^\ c > 0.
Exercise 8.5
X{t) = Z cos (2nt + 0) = Z cos 0 cos 2nt - Z sin 0 sin 2nt
= A cos (2nt) - B sin (2nt)
where A = Z cos 0 and Z? = Z sin 0.
The joint density of A and B is given by
cos 0 - Z sin 0 1
f a .b(Zcos 0, Zsin 6 = 1/2 7rZ<rl/2(z2 >
sin 6 Z cos 6
= 1/2 ;rze-'(1/2):2 l / z = l/2 ^ e z2/2
Therefore / ^ ( a , £) = 1/2/r £ -1/2(fl2+£2). So ,4 and 5 are independent normal

variables.
Therefore every linear combination of X(tjYs
n f n \ f n \
2 CjX(tj= Z cj cos {2ntj) A - Z cj sin ( 2 ntj) B
7 = 1
V;=1 yj=]
is a Gaussian r.v. By definition (X(f), - 00 < / < 00} is a Gaussian Process.
Exercise 8.6 (a) Consider the characteristic function
E< exp i Z ukX(tk + r)

I
r pin
l/2n exp iA Z cos (27rf* + 2 /rr +0 )
I Jo *=1
p 2n{ r+1)
= Ej 1 1 exp
1 /A Z uk cos (2 /rf* + y/) dyf\ (\\f- 2nt + 6)
[ 2 n \' 2nx *=1
r-"
=E exp iA 2 uk cos (2ntk + y/) dW\
& Jo k=\
( n ^
= E i2 uk (v X(0 = A cos (2nt + yr)
*=' y
So by uniqueness theorem of characteristic function (X(r)} is stationary.
pin
(b) E(X(t)) = E\A( \l2 n
/ .2*
rJ
Jo
cos (2m + 0 ) d 0 | = E(A).
1/21 cos (2nt+G) dG = 0 for all f > 0.

V Jo
1/2 T J XT(co) dt = A((o) — J cos [2tt/+ 0(<w)] r fM O
a.s. as 7 —> oo.

Therefore MT(a)} —> E(X(0) a.s. as T oo.
(c) Let T = A cos 0 and Z = A sin G, then
X(r) = T cos 27rr - Z sin 2/rr. So {X(t), - «> < t < «>}
is Gaussian iff (K, Z) are Gaussian. Since
pin
E(YZ) = E(A2)l/2n sin Gcos GdQ = 0
Jo
E(K2) = EZ2 = 1/2 E(A2) = a 2.
Therefore (T, Z) is Gaussian iff
1 exp 1
fr ,z (y ,z ) = ()>2 + z2)
2no2 2(j2
Hence, by the transformation rule for r.v.s (c.f. exercise 8.5) we have,
cos 0 a sin G
\ ! 2 n f A(a) = f)\ z(a cos sin 0)
sin G a cos G
= ——
—zr~exp (- a 2! 2 o 2), a > 0,
2n o 1
which is the Raleigh density.
{X(r), - oo < t < oo} is not a Markov Process because for t > 1/4
E(X(t) | X(0) = y, X(l/4) = - z) = y cos (2 m) - z sin (2m)
which depends on y, contrary to the Markov property.
Exercise 8.7
E(U(t)U(s)) = e~'e~sE(X(e2')X(e2s)},t > j
= e~‘e~s min (e2‘, e2s) = e~‘e~'e2s = e~,+s = />5
Therefore E(t/(r) { /( j )) = e - 1 ' - ' 1.
Exercise 8.8
EX2(t) = E
f
Jo
W(s)ds =E
Jo' W(v)dv\
= f f E[W(u)W(v)] dudv (by Fubini’s Theorem)

Jo Jo
E[W(u)W(v)]du + | J
■III
=2 r r )W(v)du] dv = 2J du
| J > dv
u
Jo Jo
= 2 Jof y2/2dv = r3/3.

Since X{t) is the limit of Riemann sums of normal random variables, X(t) is
normal. Also W and X(t) have jointly normal distribution. Given W(t) = w, X(t)
is normally distributed with mean
E[X(t) | Wit) = w]=\I2tw and variance E[(X(t) - U2tw)2 \ W(t) = w
= r3/ 12.
**Exercise 8.9 Since Brownian motion is separable
P( sup I W(t) I > e) = lim P(max I W(t) I > £), e > 0

0 < ,< , a i- ^ o o tE Sn
where Sn = {k/2n, k = 0, 1,. . 2n - 1}. For fixed W, let rb t2, .. . tN be the points
of Sn with tx < t2 < . . . < tN. Let T((d) be the first i (if any) such that
| W(th (d) | > £, and write
N
EW2(1) = 2 P(T= k)E(W2{1) 1 7
k=I tesn
N
E(W2( 1) I max I X(t) \ < e) > I P(T= k)E(W2( 1) I T= k)
te s n k=\
E (W 2(1 ) | T=k)= £[ ( W ( 1 ) - W{tk))2 \T = k } + E(W2(tk) |
+ 2E[W(tk)(W(l) - W(tk))2 1
Because the event T = kdepends only on W(t\), W(t2), ■■ W(tk),
W[W(tk) ( W ( l ) - W ( t k))T=k]
= E[W(tk)V
[(\ , - W(tk) | W(t,),. .
E
(by martingale property of Brownian motion, see Appendix I).
It follows that E { W \ 1) |T=k)> e2

and
N
EW2( 1) > e 2 E P(T = k) = e 2P(max I W(f) I > e)
*=1
—» e 2P( sup | W(0 | > £) as n —> °o
0<f<l
proving Kolmogorov’s inequality for Brownian motion.
Exercise 8.10 Let Yk = W(k) - W(k - 1). Then Yk are independent /V(0, 1)
variables, X(n) = Yl + . . . + Yn is the sum of i.i.d. r.vs. Since EYk = 0, by strong
law of large numbers for i.i.d. r.v.s, X(n)/n —> 0 a.s. a sn -)< » .
Define Zk = max | W(k + t) - W(k) \

0<t<\
For t e [ k , k + 1], | (t)/ -

X (k)/<|
X l+ | X(*)| + Z*.
The first term —> 0 a.s. Since Zk has the same distribution as
max I W(t) I and P( sup I W(t) I > £) = 2P(\ W(t) | > e)

0 < t <\ 0<t<\
(or by Kolmogorov’s inequality for martingle; see exercise 8.9),

EZk < oo. Now Zk s are i.i.d. r.v.s with EZk < oo implies Zk!k —> 0 a.s. as k —> «>.
Therefore lim = 0 a s-
Exercise 8.11 Fix x > 0. The characteristic function of T(x) is given by
0(m) = £[exp (iu Y(x))] = f

Jo
£[exp (iu W2(t))] dP[T(x) < t\.
since T(x) is determined by the Wx process, T(x) and W2 are independent.

Since W2(t) is N(0, t), £[exp (iu W2(t)] = exp (-1/2 u2t).
Also Laplace transform of T(x) is given by
E[e~m x ) ] = f
Jo
e’ 0' dP[T(x) <t] = exp [- jcJ2Q]
Therefore 0(m) = f
Jo
exp (-1/2 u2t) dP[T(x) < t]
= £[exp (- 1/2 u2T(x))] = exp (- | u \ x), - oo < u < oo.
Therefore Y(x) is a Cauchy process.

The characteristic function of T(x{ + x2) is given by
<PXi +x2(6) =E(e,eT(x'+X2)) = exp [-(x , +
= exp [- x, *j2iG ] exp [- x2 = <&, (0) • <Px2(0 )-

This implies that the first passage time r.v. T(x\ + x2) for reaching x } + x2 > 0
when W(0) = 0 a.s. is the same as the distribution of the sum of T(xx) and T(jc2),
where T(xx) and T(x2) are independent, X\ > 0, x2 > 0, i.e. {T(jc), ;t > 0} has
stationary independent increaments and it is nondecreasing. From this it follows
that
Y(x) = W2(T( x ))
also has stationary independent increments.
Exercise 8.12 Since Y(t) = K(0)^X(°"X(0), using the characteristic function of

normal distribution, we get
E[Y(t)I Y(0) = y]= yE[em ~m ) ]= y exp [tQi + (1
and E[Y\t) \ Y(0) = y] = yÊ[e2U(,)~x m \
= y2 exp (1/2) 4a1]},
so that the variance of Y(t) is
Var [Y(t) | Y(0) = y] = E[Y2(t) \ Y(0) = y] - {E[Y(t) \ Y(0) = y]}2
= y2{exp [2t(ju + a 2)] - exp [2t(ju + l/2<72)]}
= y2 exp [2t(ii + 1/2 cr2)][exp (tcr2) - 1].
T(r + h) - K(r) = K(0[^X(r+/l)”X(/)-1 -1], again using characteristic function, we get

E[Y(t + h) - Y(t) | K(f) = y] = yE[eX(t+h)-X(t) - 1]
= y(exp {h{jx + (l/2)cr2)} - 1). Therefore b{y) = y(ju + l/2cr2).
Similarly
E [r(t + * ) - r < > ) ) i |l-(< )= ;).) = _ 1]2/ft
Hence lim E[{ K(f + h)- K(/)}2 1

hiO
Exercise 8.13 By Reflection principle
P[M(t) > m, W(t) <x] = P(W(t) > 2 m - x ]
f2m -
= i -<t> , m > 0, m > x.
l~ v rj
where <p(x) is the c.d.f. of N(0, 1). Differentiating with respect to x and then with
respect to m, changing the sign and using the relation dldx(j>(x) = -0(jc), we get
the joint p.d.f. of M(t) and W{t)
d__d j _ f 2m - x d f 2 m - jc ^
dm dx V > dm
2m - x 2 2m - x
* V7 ~ 7 T
where Q(x) is the p.d.f. of /V(0, 1). Joint density of M and W is given by
/(ra, x) = — -r- (2m - x) exp [- (2m - x)2/21] if 0 < m and x < m.

^ /rr
Hence the Joint p.d.f. of M(t) and Y(t) = M(t) - W(t) is given by
g(m, y) =/(m, m - y )
= I——■ (m + y) exp [- (m + y)2/2t], m < 0 and y > 0.

^ /rr
**Exercise 8.14 Let Tab = inf {/> 0 : VT(0 = « or W(/) = b }

and T a n = min (T, /?).
Now K(/) = W2(t) - t is a continuous parameter martingale,
W a «)] = £ [ K ( 0 ) ] = 0 .
Therefore
E [T a /i] = E[W2(T a w)] < (| a |2 + Z>)2
Thus E(T) = lim f P[T > r] * = lim £[7 a n] < (I a I + 6 ) 2 < oo

n—>°° Jq n —>oo
Let p = /W (7-fli) = *).

Now {W(f)} is a martingale, T= Tab<°° and (X(T is bounded. By optional
stopping theorem (see Appendix IV)
0 = E(W(T)) a P [ W ( T ) bP[W(T) = b]
= a(1- p ) + bp.
Therefore
P ~\a\ +b
E(Y(T)) = 0 implies E(Tab] = E= a \ l - p ) +
= a2■ +b = \a\b.
a|+b |a | + b
By changing the origin and scale we get the result for Wiener process with
variance cr2 and starting position W(0) = x.
Exercise 8.15 Let Z(t) = exp {X(W(t)) - (A/i + 1/2 A2cr2)r} is a martingale,
where W(t) is a wiener process with drift ji * 0 and variance parameter a2. If
Ao = - 2/i/o2. Then Z$(t) = exp (A2(W(/))} is a martingale. Let Tab be the first
time the process reaches a < 0 or b < 0. Then by optional stopping theorem (see
Appendix IV) 1 = E(Z0(Tab)) = P(W(Tab) = a) exp ( V ) + P(W(Tab) = b) exp (A^),
and
1 - exp (A0 a)
P(W(Tab) = b) = , where A0 = - 2 f il e
exp (A0 6 ) - exp (A0 a)
changing the origin to the case W(0) = jc. We get the result
« W ( 7 ^ ) = b | IV(0) = x) = < = * p ( - ^ ; > - « p < - ^ ' - f >

exp (—jucilo>~) - exp {-2jja/a2)
Since ju < 0, letting a —> - ©o, exp (- /i ala1) —> 0 and

lim P(W(Tab) = b | W(0) = x) = exp {2/i(6 - jt)/<r}.
Now left hand side becomes the probability that the process ever reaches b,
i.e. the probability that the maximum of the process ever exceeds b. Thus, for
m -b-x,
P(A/ < m) - P( max W(0 > m | W(0) = x) = exp 2

,m > 0.
o</ < °o
° 2)
**Exercise 8.16 Now Y(t)= exp {AW<t) - \l7?}t) is a martingale for a standard
Brownian motion W(t). Also W(T) = a + bT, a > 0, b > 0. By optional stopping
theorem £[exp {\W{T) - 1/2 X2T}] = 1. Therefore
£[exp {Ma + bT) - 1/2 A2 T}] = 1.
Putting 6 = A- 1/2 A2, yields E[exp {6T + Aa}] = 1 .
Hence £[exp (GT)] = e~
Now to get the value of A, we have A2 + 20 - 2Xb = 0 and
X = b± -fi - 20
We require A> 0, which implies
A= 26> if d = b 2/2.
Therefore £[exp(07)] = exp { -a(6 + -^0 < d
Chapter 9
Exercise 9.1
The autocovariance generating function for ARM A (1,1) process is given by
C ( Z) = a 2 a + f c -1
j B z ) ( l + ) = c t 2 (1 + + f c + f c - 1 )
(1 - a z)(l - az"1) [1 - ( a 2 + az + az"1)
= a 2 (1 + )32 + j8z + jSz"‘)[l + (a2 az"1)

+ (a2 +az+az~')2 + . .. + (a2+ az + az"1)* + .. .]
= a2^ 1 + 2 a ( 5 + j82)(l + a 2 + a4 +.)
+ (1 + a)(l + jS )(1 + or+ a 4 + .. .)
+ a( 1+ a + (3+ )ap(1 +
+ a*"‘( 1 + a + (5 +a /3 )(1 +
Therefore coefficients of give
1 + 2a/3 + (1+ « ) ( ! + /?)

7(0) %7(1) = a 2
1 - a- 1- a 2
y(k)= ay(k - 1),
The auto-correlations are given by
p ( 0 ) = 1, p ( l ) = 7 ( l ) / 7 ( 0 ) = (1 + a ) (1 + + 2
and p(k) = y(k)/0 ) = a *-1p ( l ) , k >

Exercise 9.2 The autocovariance of Zt is given by
Yz( T ) = E(Zt Zt_T) = E(Xt + Yt) (X,_T Yt_r)
+
= Yx(T) + 0 + 0 + Yy(t ) = Yx(T) + Yy(t )

Since Xt is a MA(qx), yx(r) Ofor T> qx and Yt is &MA(q2) ==> Yy (t ) = 0 for t > q2.
Therefore yz(r) = yx(r) + yK(r) = 0 for r > Max (qh qi).
From a result of Wold (1954) a necessary and sufficient condition for a MA(q)
process {X,} is that it has spectral density/(A) > 0, - n< X< n given by
f(A) = C ( z ) = X
J=~<J
Therefore the process Zt = Xt + Yt is a MA(q), where q < max (q}, q2)-
Exercise 9.3 Let B be the Backshift operator. Then <plpi(B)Xt = 6i(n(B)eu and
<kn (B)Y, = 02n(B)e2„ where <pi<n(z), <P\q](z) and <t>2 q2(z) are
polynomials in z of degree P\,P2^ \ y ch respectively and eu and £%are independent,
Zero means white noise processes.
Since Zt = Xt + Yt, it follows that
0lp](B),4>2p2(B) Zt = 02p2(fl) 4>]pi( B ) = X t Hh 4>lpi(B) 02p2(B)Yt
= fopjiB) B\ql(B) eu + 0]p](B) <l>2q2(B)82t.

The first term on the right hand side is MA(p2 + qx) and the second term is
MA(p{ + q2) so the whole of the expression on the right-hand side is MA{q)
where q < max (p{ + q2, p2 + qx) (by exercise 9.2). Now the polynomials 0ip](z)
and 02/j2(^) rnay contain common root and so the part of the operator need not
be applied twice and hence the order p of the polynomial 0 02P2(B) is less
than equal to p x + p2.
Exercise 9.4 For ARM A (1,1) model equate the coefficients of the right and
left sides of (1 - 0 - B) \i/(B) = 1 + QB, where y/(B) is a function of back shift
operator B given by y/(B) = iff jB j . From the constant term \f/0 = 1, the
coefficient of B gives - 0 4- y/l = 0, so y/] = 6 + 0; equating the coefficients of
B2 gives -0iff\ + iff2 = 0, so ¥2 = $V\ = 0(0 + 0). and so on, to give
Xt = £t - (0 + 6)et_x + 0(0 + 0) e(_2 + 0"(0 + 0)£t_3+ . . .

(-0 )<>l(0 + 0)e,_t + • ■•
Therefore Xt_k = + Vi£t-k-2 + . . . , so is a linear function of
. . . etc. but not of £t. As the £,’s are independent by assumption, £t
must therefore be independent of £,_* £,_*_!, . . . and so of Xt_k. This is true for
any k > 0 and any stationary ARMA process.
(b) In the ARMA (1,1) case the ARMA equation gives
£t - Xf - 0XM - 0Et_i - Xt - 0 Xt_\ - 6 (Xt_i - 0(X,_2 - 0£t_2) - • • •
= X, - (0 + 0)X,_, + 0(0 + 0)X,_2 - 0 2(0 + 0)X(_3 + . . .
+ (-0)*"1 (0 + 9)X,_k+ . . .
Exercise 9.5 (a) If {Xt} is zero mean second order stationary non-deterministic
time series then by Wold’s Theorem Xt has a representation of the form
( 1)
Since we have the knowledge of the entire past Xh Xt_u . . . of the series at
time r, the linear predictor of Xt+k k > 0 Xt+k\t is a linear combination of the
known past values, viz.
(2)
So we can rewrite (1) in the form
Xt+k\t = 2 [Si€t-i for some sequence {#,}.
Therefore Xt+k - X l+k[l = . 2 0 ,e (+,_, - .2
k- 1
= .2 V , £ , +k-i + 2
/= 0 z = A: 7=0
M *-l
= I y/jE,+k-i+ 2 - *,•
/=0 z=A:
E[(Xl+k - Xl+k]l)2] = a 2 ,2 wf + ct2 .2 (i//,+t - gi)2

which is minimized by choosing
8 i ~ V i+ b
so the optimal mean square predictor is
— V i+ k £f -i
and the minimum value of the mean square error is
E[(Xl+
k- X , +*„)2 ] = (X 2 2 V?.
(b) For the AR(1) process X, = a XM + £1? | a | < 1
Xt = Z a j e . i so that w = a j
1 i=0 1J TJ
We find the &-step predictor has
gj = a j+
k,given X,+t|, = jS a 2+*e,-, = a* 2q a*e H = a*X,
For A/A(l) process X, = £, + /fcM, | j8 | < 1, the optimum one step ahead
predictor is
*r+i|, = P*, (3)

However, we do not observe that the {£,} series, but just the {X,} series. To
get a useful predictor we express et interms of Xt, XM X,_2, . . . .
MA(1) can be written as
£, = X, - 0eM ,~8(X,_,
X
= j - =..
= X, - PX,_1 + P2 -
X Xj_3 + . . .
Substituting this in (3) we get one-step predictor as
X,+1|, = PX, - j32X„, + - 3+ • • •
Exercise 9.6 Let Xt = 6 £M + £, be a MA{\) process where {£,} are i.i.d. r.v.s
with zero mean and finite variance <r2.
(1 + 0 2) if /t = 0
Then Cov (X/+/,, X,) = 0cr2 if h = ± 1
0 if | h | > 1
If | p | > 1/2 then k{h) is the autocovariance function of the above MA( 1)
process with a2 = (1 + 0 2) ' 1 and 0 = (2p)"1(1 ± ^(1 - 4 p 2)).
If p > 1/2, X = [*(i -y )]"y-=1 and a = (1, -1, 1, -1, . . .)

with ^-component, then
a K a = n - 2(n - l)p < 0 for n > 2p/(2p - 1),
which shows that k(h) is not non-negative definite and therefore by Bochner’s
Theorem it is not an autocovariance function. If p < - (1/2), the same argument
using n-component vector a = (1, 1, 1, . . .)' again shows that k(h) is not non
negative definite.
Exercise 9.7
S\ = [i + S$ + X \ = /a + X\
E(S\) = juy E(St) = p + £(Sm ) = tju
Cov(S,, 5,+/,) = E(St - tju)(St+h - t + h ju) = Var (5, - tju) = fcr2
Thus, S, is not second order stationary,
Let Yt = VS,. Then VS, = (1 - - SM = p + Xt
E(VSt) = ju, E(Yt+h) = ju for all h = . . . , -1, 0, 1, 2, . . .
f or2 if h=0
Cov(K„ T,+„) = £(Y„ X,+/>) =
[0 if | A | * 0
Obviously Yt - ju + Xt is a (i.i.d.) stationary process and also a covariance
stationary process.
Exercise 9.8 For one-sided infinite moving average representation of AR{2),

the roots of (p{B) = 1 foB2 = 0 must lie outside the unit circle, from the
properties of quadratic equation this implies that the AR(2) parameters and 02
must lie in the triangular region
02 + 0i < 1
02 - 01 < 1 ( 1)
-1 < 02 < 1
From Yule-Walker equations in AR(2)

Pi 1) = 0i + 02 p O ) ar,d pi2) = 0i p ( l ) + 02-
Solving we get
P(D(1 - P ( 2 ) ) p ( 2 ) - p 2 (l)
0i = S 02 =
1 - P 2 (D 1 - p 2 (l)
which imply
_ 0 f_
P(D = r ^ . n d p ( 2 ) - f c + ( 2)
1 -0 2
From (1) and (2), we get the admissible values of p ( l) and p (2) given by
p 2(l) < (l/2)(p(2) + 1), | pi 1) | < 1, | p(2) | < 1.
Exercise 9.9 Now since | 0 | < 1, Xt = can be written as a linear

process as
X,I = i=
£Q 0*8'^
l l and Xr+*
[+K = /=0
I 0' et+k*
I.
Hence y(Jt) = £ [ £ 0 ‘£,_,

^ t=0 7=0
Therefore yik) = ———- 0l*L
1 - 02
Since spectral density of {£,} is o 2!2n, by calculating covariance generating

function we easily get
f U ) = ^ \ l - 6 e - a \2.
Since {eb e2, . . . , £n) is distributed as multivariate normal with zero mean
vector and covariance matrix
"(1 - 0 2)- 1 a2 0 . . .0
0 a2 0.. .0
E=
0 0 a 2. . .0
0 0.. . . . . a 2
iii = G 2n
1 - G2
n
and x'lr' x a 2 (1 - 0 2) X2 +
Therefore
Ln = -1/2 [n log(27K72) - log( 1 - 6)2
+ or “ (i - 9 2) x,2 + I -
7= 2
P <7
**Exercise 9.10 Let {X,} be a ARMA (/?, q) process E a rXt_r = r=0
E f3s et_s,
r-0
Oq, /30 * 0 and {£,} is a white noise with £(£,) = 0 and {£,} are i.i.d. r.v.s with
2
s=0
£ £ 2 = cr2. Then the spectral density of X{ is given by •—
p
2
r=0
a rzr
s = e~lX. Let = 38 (Xt, t < n) denote the closed linear space (closed in the sense
of mean square convergence) generated by linear forms in the Xt's, t < n. Now
£; e J i t. Xt has a linear process (one sided) representation Xt = Eq yj e t_j,

p
since the zeros of a(z) = E^ a rzr have modulus greater than one. Therefore £, is
orthogonal to (independent of) It follows that
p «
E a* lot0Xt_k + &E
X, = -
£=l =1 7
/?, f a 0£t_j is in
7
and Xt - Xt - /30/ a 0et is
orthogonal (independent of) to J ( t_\. Thus Xt = E{Xt | ^ M), i.e. X, is the projection
of X, on and consequently the best linear predictor of Xt in terms of the past
Xpj < t - 1. The variance of the prediction error is E | Xt - x t |2 = Po a 2.

a0
p
If Zu s are the zeros of a(z) = rZ
=0
a rz r
I a{e-iX ) |2 = n (e a,
M=1
= n, 12 , 12 n | i - e- ‘V |2 | « p|2
Now log I lA) \2 = log I (Xq |:2 + n log I I - e lAz J |2.
Since for | | > 1, the log(l - z^1) expansion is valid and hence
f log(l - e -a z~l )dA = - Z z"1/ j\ f e~ajdA = 0.

J-n 7~1 J-7T
By assumption all the zeros of a(z) and b(z) are outside the unit disc. Therefore
1/27T j* logf(A)dA = l o g j ^ - + log | /30 |2 - log | a 0 |2
and hence Po
ao
= 2 êxp J log/(A)<M.J.
Exercise 9.14 (Explosive A/?( 1) process)

Using backshift operator Xt = aXM + et can be written as (1 - aB)Xt_{ = et or as
aB(a~lB~l - 1) Xt = et and hence that
Xt = - o r xB~\\ + o f1#"1 + cC2BT2 + . . . + orJ B~J + . . . ]et (since | o f11< 1)

oo
= - Z a ” 7 et+j, where B~j et = et+J.

Exercise 9.15
n
Xn = a X n_i + en, n > 2 gives Xn = Z et a n~\
i=i
n J
and =n 1Z Z aH =n 1 Z £.-(1 - a n‘7+1 )/(l - a).

l ;=i ;=i 7
Hence
= n - '( A i/( l- a ) ) S (1 - a 2)
7=1
= n~'(n/(l - a))(n - a (1 - a")/(l - a))
= /i/(l - a) - (a^/n)((l - a")/(l - a 2)2) -» - a),
Var ( „ =) n~2E a 2( l - a""y+1) 2/ ( l - a ) 2 < cx2/n —> 0,

X
7= 1
and(E(Xn) - ///(l - a) )2= Var(X / /(l
+ 2E(X„)(EX„ - / i / ( 1 - a )) -> 0.
Exercise 9.16 Let | a | < 1. Since X0 = 0 and Xj = + ep we have

n n -1 n
;=i 7 j=i 7 ;=i 7

n n
This implies that (1 - a) 2 X;- = 2 Ej - a X n
or Vn((l - a ) X n - ju) = - AO - a X n/ J n .
By C.L.T for i.i.d. r.v.s, yfn{£n - ju) ~^>N{0, a 2).

n
By Exercise 9.15, Xn = 2 Ej a n~j and hence
n
E( X„ /J n) E
n a n-j 14n ->
=
7=1
and Var (Xnl j n ) = a 2 2 a 2{n' j) In —» 0.

7=1
This implies Xnl 4n - ---» 0 and hence Xn / J n -A- 0.

By Slutsky’s Theorem Vfl((l - cx)Xn - /u) - A N(0, cr2)
or
For a = 1, X„ = h"1(£2 + £a + • • • + £*) for even n and X„ = /T1(£1+ £3+ • • • + £n)

for n odd. In either case,
Vn(X„ - ,u /2 )-A W(0, a 212).

n n
For a = 1, we have Xn = 2 £y and X„ = n ”1 2 (n + 1 - j)Ej, If /i > 0
Xn - A <» (by Weak law of large members).
__ n
Since £,’s are i.i.d., X„ has same distribution as rc^Z,,, where Z„ = £ /£., so
/=i J
n
we consider Zn - EZ„ = 2 j(£j - jjl) . Define for 1 < j < n
/=i
n
Xni = j(£j - n) and B2 = E(SX„ )2 = 2 j 2cr2= cPnin + 1) {In + l)/6,
J J i=i
since we have E(Xnj) = 0, Var (Xnj) = j 2a 2. Then,
B f i E { X 2n
jl {\Xnj\> e Bn))
= b; 2 2 j 2{(£j - h )2
/ [(e,
i= l J
 0, because B2 In2 -> <*>.

Hence by Lindberg’s CLT (Zn - EZn)/Bn -=^> N(0, 1) which implies
n m ( Xn - npJ2) N(0, o2^ ).
— n __ _ n
** Exercise9.17 Let Z„ = n"1 2 X, XM since aXn + bZn = n~' 2 X,(a + bXM).
i= l i=1
We let T, = Xj(a + bXi+l). Then Y2, . . . is a stationary 1-dependent sequence
with
E(Y,) = an + bn2, T = Var (K,) = EX2 E(a + bX2
= (a2 + n2) (b2a 2 +
= O2(an + bo2 + bn2) (

= /r(a + bn)ba2.
Therefore,
V^(aX„ + Z>Z„ - /r(a + fyr)) A(0, r\2 + 2£)
= A(0, cr2 (a2 + Aabn + b2( a2 + 4/i2))).
Now aV«(X - n) + b ^ ( Z „ n )2 aX +
^ (72 2a 2n
(X, Z) ~ N2(0, 2), 2 =
\2 a 2n <J4 +4o2n 2j
which implies 4n(X - Z
n, rt - n 2) W2 (0, 2) •
Now y (l) = Z„ - X2. Apply Cramer’s (delta method) Theorem using the
function g(x, z) = z - x 2. We have =

3jc’ dz
= (-2x, i),g(n,n2) = Q
and g(n, n 2) = (-2 n,!)• Therefore
Vny(l) = V^(Z„ - X ; ) ^ N ( 0 , ( T 4).
(b) (i) Let (/, = X,2 and V, = X,X,+I. Then W, = aX, + bUj + cV, is a 1-dependent
sequence with mean EW, = b o 2. ThenE= era2 + 2abf.li, + b2fiA +
EWjWi+i = /ro"\ so that Var (VV,) = a2o 2 + + + c2r f -
aXn + bUn + cVn - 2bo ) N(0, 202 + 2aZ>p3
b2(pA- o4
) + c2o 4).
and V^(X„ , U n- o 2,Vn) ^ > /V3(0, £),
^ o2 ^3 0 '
where 1 = Hi p A - O4 0 , p 3 = EX2 and p 4 = £X,4
v 0 0 *4 J
Therefore U
„_-o_2,Vn)-^> N}(0, £).
(ii) p ( 1) = g(X„, Un, V„), where £(x, m, v) = (v - x)2/(u - x2). We have
g(0, o 2, 0) = 0, and hence g ( x , u , v ) = ((2jc(u - w), -( - at2),
we have g(0, cr2,0) = (0, 0, 1/ct2). Hence by Cramer’s theorem (delta method),
V n(p( 1) - 0) o 4t o 4)=
Chapter 10
Exercise 10.1 This is a Queueing system with mean arrival and service rates
A and ju, respectively, where no more than k units are allowed to enter the system
at any time, we set
A„ = A(0 < n < k)

= 0(a1 > k)
n„ = p(n> 1)
Then 7T01 in the generalized Birth and Death model in the text is given by
\ _ pk+l
s = 7Tq] = 1 + p + p 2 + .. . + p* = —— — where p = A/p is the traffic
intensity. Since s is finite, we conclude that the steady state exists for all values
of p. Then nn= (1 - p )p ”/(l - p*+1), p * 1, = 0, 1, . . . ,When p = 1
s = £ + 1 and therefore n„ = Y +~T’ ôr^~ * an<* n = 1, • • -

Then the expected number in the system is
k
L= 2 Mt„ =pi 1- (k +\ ) p k + 1 - p )(l - p k+]) if p * 1
n- 0
= ik!2f p = 1.
The expected number in the Queue is

k
Lq= Z (n - l)7ln = p 2[l - kpk~' + (k - l ) p k]/(l - p)(l - p k+]) ifp ^ 1
n= \
= k ( k - l)/{2(k+ 1)} if p = L
Exercise 10.2
Here Xn = A if 0 < n < k
= 0 ifn> k
fin = n/u if n < c
= cfi if c < n < k
Writing Xlcp = p, we have
nn = /Zo cnff/n\ \i 0 < n < c
= 7l§c(pPIn! if c < n < k
where 7r0 will be found from the boundary condition Z nn = 1 i.e.,

«=o
i i m n ^0 = 1
n= 0 n \ \ jU ° «=r cn~Cc\ I P
AV
and hence 5= = Z (cp)n/n\ + Z —-— f —
« = r C n~ r c ! IP,
r-1
= Z (cp)n/n! + — - Z p"
n=0 Cl n=0
= S (cpr/n! + ^ i ^ l ifp3tl
n=0 cl 1-p
r-1
z
= Z ^ +^ a - c + l )
M=o ni cl
ifp = 1
Note that the steady state exists for all values of p. Now, the expected number
in the queue (waiting) is
Lq= ^ - C) 7T„
n=r+l
= {ô(cp)‘7c!} Z (
n=c+\
k-c
= {7T0(cp)<;p / c !} Z np n - \
n=\
= { T t o i c p Y pic'.}^ [ ( 1 - p * - f+ ,) /l - p ] if p ^ 1
- Q;-;C P / P [1 - P*~'+l - (1 - p ) ( * - c + 1 if p ^1
c!(l - p ) “
The average number being served or the mean number of busy channels is
<-] k c-1 k
Ls = Z nn„ +Z cn„= Zn Zc n 0ccp n/c\
n=0 n-c n=0 n=r+l
r r k
= Z { (cp )'"'/(H - 1)!} + —
p=l : M
2C: p=r+1
" -'
—c p 710 Z (cp)i/j\ + ( c p y / c ' . k £ ' ,

7=0 A2=0
= c p TTottfo1 - (c p ) c /c\ p
= c p ( l - ^ ) = — (1 - A*)
Therefore expected number in the system is
L - Lq + Lv = Lq + —■(1 - 7T*)
The effective arrival rate at the system is A' = A(1 - /fy).

Therefore from Little’s formula E(W) = LIX
= L„lh' + — = W„ +—, where /A'.

q JLl q JU q q
Exercise 10.3 By Bayes theorem = P {a is system | arrival about to occur}

_ P {arrival about to occur | n in system} n n
X P {arrival about to occur | n in system} n n
U n&t+ o(At)]nn
= lim k- 1
A f-> 0 X [AwAf + o(At)]nn + 0 .nk
n-0
[A„ + 0 (Aa)/(Aa)]tt„
lim k-1
Af —>0 X [A„Ar + o(A/)/Af]7Tn
a?=0
Aw7Tw A7T„ 7T „
k-1
(n < k - 1) ( 1)
1 - 7Ti
X An7Tn A X 7Tn
p=0 p=0
The waiting time c.d.f.

k-\
Wq (t) = Z P[n - c + 1 completions in < 1 1arrival founds n in system}
n- c
<7n + WJO)
f ' ycincx)'
qn
ii
n-c Jo (n ■ - c) 1
k-j JUC |3
1
= 2 Qn + WAO)
n-c
i J, ( n - c)\
k- 1 n-c (pct)-i e ~ ^
7^
Qn - z 2 :I + WJO)
II
n-c n=( Qn 7=o J-

k k- 1 n-c
(/ic /)7 e
= 2 Qn
n-c
- n=zc Qn 7=0
2
j
, v V (net)i e
i - 2 <7 ,, 2 -----------n----------
n=c j-c J1
An7Tn
In MIMIC/oo,qn = by leting k —> °° in (1)
Z An7Tn
n=0
?l7l„
= 7in(n> 1)
A Z un
n=0
since A„ = A for all n > 1 in Af/M/cA»

Exercise 10.4 Taking A as the mean arrival rate and fi as the mean service rate
for each channel, we have
Xn = A for n < c
= 0 for n > c
jxn - nju for n < c
Therefore i1= n q1 = 1 + h i p + (A//i)2/ 2! + . . . + jc\
Thus, the steady state exists for all values of A//x and
7in = {XlfX)kln\7^ (0 < n < c).
The probability that all the channels are busy in the steady state is given by
(Xlju)c/c\
71r
1+ {X! fi) + (A//i)2/ 2! + . .. + ( X / j u ) c lc\
nc is called Erlang’s first loss formula. An application of this model is a

telephone exchange with a trunk lines with no facility for holding subcribers
who fail to get a free line.
Exercise 10.5 Using M IM IC system’s result
N \/( N - n )\( AV
— ^— [ ttJ *<> (0- w<c)
n„ =
N \/(N - n)\ ( V \ n
n 0 (c < n < N )
cn~cc\
(A.) 7t0 (0 < n < c)

or equivalently nn =
UJ
'AP n
— I n o (c < n < N)
Kn s cn
where /r0 is found in the usual way from Z n n = 1 as

n- 0
C —1 i y n n!
Ur} = 2 - 1 + 2
n=0
V “ V
n=c
\ n J cn Cc\ I I1
This is a finite source queue. A typical application o f this model is that of
servicing of N machines, where arrivals corresponds to breakdown of machines
and the repairmen are the servers. The service times are assumed to be exponentially
distributed with mean IIfi.
Exercise 10.6
c- 1 'AP a ' i " 'AP
* i n\
E—
n=0
n n —7Tq Z Al
n=0 y , UJ + Z n\
n -c \y j c n~cc\ UJ J
c~l fAT| A Y \ 1 $ N\ n\ X Y
- n 0 2j n — + -r Z
n=0 c! n=c
n) cn~cc\
"c-l n N 'AP
where n 0x = z ( N \ fA] +— z n\
fiVl
, n . UJ c\ n=0 y , UJ _
n=c c n~cc\
The average number of operational machines is N-L.
N N c -1
N
Lq = Z (n - c) n n = E n„- 1 - 2 nn c
n -0 n =0 n- 0 n=Q n=0
c-l
= L - c + X (c - n)7tn
n- 0
AA
Hence Lq = L - c + /r0 Z (c - n)
n=0
If the system has n units in it, there are N - n possible arrival units outside
the system, each with a mean arrival rate A, so the mean arrival to the system is
(N - n) A.
Hence the effective mean arrival rate,
N N N
A' = Z (N - n )A n n = AM Z n n - A Z az7T„
n=0 n=0 n=0
= A M - A L = A(Af - L).
Hence by Little’s formula we have
E ( W) = 1—- L ) a"d
Hence the expected downtime is L/A(M - L). The average percent idle time of
each server is
TTq + ^ (tfi + 712 + . • • + n v- 1).
So the expected percent idle time of each repairman is
\ rj } ( N )( AX '
100 7l0 1 + 1/c
n-0
V " J ’ J)
Exercise 10.7 From Exercise 10.1
p - p k+] - p 2 + p k+2 p d - p ) - p t+id - p )

j a ( L - Lq) = n
(i - p x i - p t+i) (i - p ) o - P k+x)
p d - p *)
=p
i - p k+[
A,sc Ail ~ Ki) = t t = 4 "r.

I -p ” ' F P l - p ‘*'
Therefore /z(L - Lq) = A(1 - nk).
Exercise 10.8 Let Xn be the number of arrivals during the service time of the
nth customer. Then in the M/G/l system we have Qn+] = Qn - H{Qn) + Xn+h
where H(x) is the heavyside’s function, given by
H(jc)= 1 if jc > 0
= 0 if jc< 0.
Equating expectations Qn+l and Qn gives E(H(Qn)) = E{Qn+1).

i* o o oo
Now E ( X n+l) = l j [ A v y e - ^ / j ' ] d G ( v ) = P:

Jo J=1
Exercise 10.9 (Takacs integral equation) Regard the busy period as made up of
the service time of the first customer plus the busy period initfated by customers
in the system.
Now G(x) = P(T < x), where T is the busy period with
G*(j)= f e~sxdG(x).
Jo
Let V be the service time with d.f. F(u) = P(V < v),
F*(s) = f e~svdF(v).
Jo
Let G{n) be the Ai-fold convolution of G with itself and A(t) denotes the number
of arrivals in [0, t]. By conditioning on the duration of the service time V of the
initial customer initiating the busy period and on the number A of arrivals during
his service tipie, we get
G(x) =e~**V(x) 4 £
"=' Jo
f P [ T < x \ V = v , A ( v ) = n}dV(v)P{A(v) = n}.
The first term corresponds to the case that none arrives during the service
time of the initial customer (and then v = x), let TLbe the total service time of the
ith “descendent” and all its other customer initied by him. Then T,’s are i.i.d. r.v.s
with same distribution as T. We have
( T < t | V = v, A(v) = n) = P(TX+ . . . Tn < t - v) = G(n\ t - v), n > 1.
Therefore G(x) = S f ------j 1— G(n} (x - v)dV(v).

"=o J 0 n\
Pluging the above expression o f G(x) on the r.h.s. of
G * ( s ) = f esxdG(x)
Jo
and changing the order of integration (on the r.h.s.) and after simplification, we
get
G*(s) =P* (s +A - AG*(j)).
References
1. Anderson, T.W. The Statistical Analysis of Time Series, John Wiley, New York
(1971).
2. Andre, D. (1887).
3. Arthey, K.B. and Ney, P. Branching Processes, Springer-Verlag, Berlin (1972).
4. Bachliar, L. Theorie de la speculation. Ann. Sci. Ecole Norm Sup (3), 21-86
(1900).
5a. Bailey, N.T.J. The Elements of Stochastic Proceses. John Wiley, New York (1964).
5b. Bailey, N.T.J Mathematical Theory of Epidemics, Giriffin and Co., London.
6. Barlow, R.E. and Proschan, F. Statistical Theory of Relability and Life Testing,
Holt, Rinehart, New York (1975).
7. Bartlett, M.S. An introduction to stochastic process, Camb. Univ. Press, London
(1954).
8. Basu, A.K. An introduction to measure theoretic probability, Prentice Hall, New
Delhi (1999).
9. Bharucha-Ried, A.T. Elements of the theory of Markov processes and their
applications, McGraw-Hill Book Co, Inc, New York (1960).
10. Bhatt, U.N. Elements of Applied Stochastic Processes. John Wiley, New York
(1984).
11. Bhattacharyya, R.N. and Waymire, E. Stochastic Processes with Application,
John Wiley, New York (1990).
12. Billingsley, P. Convergence of Probability measure, 2nd ed, John Wiley New
York (1999).
13. Blackwell, D. Extension of renewal theorem, Pacific J. Math, 3, 315-320 (1953).
14. Blumenthal, R.M. and Gatoor, R.K. Markov Processes and Potential Theory,
Academic Press, New York (1960).
15. Bloomfield, P. Fourier Analysis of Time Series: An introduction, John Wiley,
New York (1976).
16. Bochner, S. Stochastic Processes, Ann. Math, 48, 1014-1061 (1947).
17. Box, G.E.P. and Jenkins, G.M. time series Analysis: Forecasting and control,
Holdey Dey, Sanfrancisco (1996).
18. Blanc - Lapierre, A. and Fortet, R. Theorie des Fonetions Aleatories, Masson et.
Cie, Paris (1953).
19. Brieman, L. Probability, Addison-Wesley, Reading. Massachusetts (1998).
20. Champemowne, D.C. An elementary method of solution of the Queueing problem
with a single server and a constant parameter. J.R.S.S. B, 18, 125-128(1956).
21. Chatfield, C. The analysis of Time Series: An Introduction, 5th ed. Chapman and
Hall, London (1989).
22. Christer, A.H. Refined Asymptotic costs for renewal reward processes. J. Opns.
Res. Soc., 29, 577-583 (1978).
23. Chow, Y.S. and Tiecher, H. Probability Theory, Springer Verlag, 3rd ed. (1998).
24. Chung, K.L. Markov Chains with Stationary Tranisition Probabilities, Springer
Verlag, 2nd ed. Berlin (1967).
412 Introduction to stochastic process
25. Chung, K.L. and Ornstein, D. On the Recurrence of Sums of Random Variables,
Bull. Am. Math. Soc., 5, 332-334 (1954).
26. Cox, D.R. Renewal Theory. Methuen, London (1962).
27. Cox, D.R. and Miller, H.D. The Theory of Stochastic Processes, Methuen, London
(1977).
28. Cramer, H. and Leadbetter, M.R. Stationary and Related Stochastic Processes,
29. Doeblin, W. Sur les proprietis asymptotiques de movement rigis par certains
types des chaines simples. Bull. Math. Soc. Roum. Sci, 39, No. 1, 57-115, No.
2, 3-61 (1936).
30. Doob, J.L. Stochastic processes, John Wiley, New York (1953).
31. Doob, J.L. Renewal theory from the point of view of the theory of probability.
Trans. Amer. Math. Soc. 63, 422-438 (1948).
32. Dvoretzky, A, Erdos, P. and Katutani, S. Nonincrease everywhere of the Brownion
motion process. Proc. 4th Berkley. Symp. On Math. Stat. and Prob. vol. II, 103—
116 (1961).
33. Erdos, P, Feller, W., and Pollard, H. A Property of power series with positive
coefficients, Bull. Amer. Math. Soc. Vol-55, 201-204 (1949).
34. Fabens, A.T. The solution of Queueing and Inventory models by Semi-Markov
processes, J.R.S.S. B, 23, 113-127 (1961).
35. Feller, W. On the integral equation of renewal theory, Ann. Math. Statistics, 12,
243-267 (1941).
36. Feller, W. An introduction to probability theory and its applications, vol. I & II,
John Wiley, New York (1968, 71).
37. Fisher, R.A. Tests of significance in harmonic Analysis. Proc. Roy. Soc, Ser. A,
125, 54-59.
38. Folkes, J.L. and Chikara, R.S. The Inverse Gaussian Distribution and its Statistical
Application—a review, J.R.S.S.B. 40, 263-275(1978).
39. Foster, F.G. On Stochastic Matrices Associated with Certin Queueing Processes.
Ann. Math. Statist., 24, 355-360 (1953).
40. Fuchs, W.H.J. and Chung, K.L. On the distribution of values of sums of random
variables. Mem. Am. Math. soc. No. 6, 1-12 (1951).
41. Fuller, W.A. Introduction to Statistical Time Series, John Wiley, 2nd ed, New
York (1996).
42. Galton, F. and Watson, H.W. On the Probability of extinction of families. J. of
Anthr. Inst., 6, 138-144 (1874).
43. Gikhman, 1.1, and Skorohod, A.V. Introduction to the theory of random functions,
Saunders, Philadelphia (1969).
44. Granger, C.W.J. and Newbold, P. Forecasting Economic Time Series. Academic
Press, New York (1977).
45. Grenander, V. and Rosenblatt, M. Statistical Analysis of Stotionary Time Series.
46. Grimmet, G. and Stirzakar, D. Probability and Random Processes, Oxford University
Press, 3rd. ed. (2001).
47. Good, I.J. The Number of Individuals in a cascade process, proc. Camb. Phil.
Soc., 45, 360-363 (1948).
48. Gross, D. and Harris, C.M. Fundamentals of Queueing theory. 2nd ed. John
Wiley, New York (1985).
49. Harris, T.E. The theory of Branching Processes, Springer-Verlag, Berlin (1963).
50. Hannan, E.J. Time Series Analysis. Methuen, London (1960).
51. Hannan, E.J. Multiple Time Series. John Wiley, New York (1970).
References 413
52. Isifescu, M. and Tautu, P. Stochastic Processes and Applications in Biology and
Medicine, I, Springer-Verlag, Berlin (1973).
53. Ito. K. Stochastic integral. Proc. Imp. Acad. Tokyo 20, 519-524 (1944).
54. Jagers, P. Branching Process with Biological Applications, John Wiley, New York
(1975).
55. Kalkutani, S. Ergodic theorems and the Markov Process with a stable distribution,
proc. Acad. Tokyo, 16, 49-54 (1940).
56. Karakowski, M. Conservation methods in queueing Theory. Rev.Fran d’auto.
Infor. Rech. Oper (R.A.l.R.O) 7e annee, vol. I, 3-20 (1973).
57. Karhunen, K. Uber Lineare Method Methoden in der Wahrschein lich Keit
srechnung. Ann. Acad. Sci. Fenn. A. 37 (1947).
58. Karlin, S. and Taylor H.M. A first course in stochastic processes, Academic
Press, New York, 2nd ed. (1975).
59. Keilson, J. and Kooharian, A. On time dependent Queueing processes. Ann.
Math. Statist, 31, 104-112 (1960).
60. Kemeny, J.G. and Snell, J.L. Finite Markov Chains, Academic Press, New York
(1966).
61. Kendall, D.G. Some Problems in the Theory of Queues, J.R.S.S. B, 13, 151-185
(1951).
62. Kendall, D.G. On the generalized birth and death processes, Ann Math, Statist.,
19, 1-15.
63. Khintchine, A.Y. Correlation theory of stationary random processes. Usp. Math.
Nauk, 5,42-51 (1938).
64. Kingman, J.F.C. The single server queue in heavy traffic, proc. Camb. Phil. Soc.,
57, 902-904 (1961).
65. Kleinrock, L. Queueing Systems, Vol. 1 & II, John Wiley, New York (1975).
66. Kolmogorov, A.N. Interpolation and extrapolation of Stationary random sequences,
IZV. Akad. Nauk. SSSR Ser. Math. 5, 3-14 (1941).
67. Kolmogorov, A.N. On the problem of differencetiability of transition probabilities
of time-homogenous Markov Processes with Countable number of states. Uchenye
Zapiski Markovskovo Gos. 148 (Mathematika 4), 53-59 (in Russian).
68. Loeve, M, Probability Theory. Vol I & II, Springer verlag New York (1982).
69. Levy, P Theorie de l ’addition des variables aleatories, Paris (1937).
70. Levy, P. processus stochastiques et mouvement Brownien, Paris (1948).
71. Lotka, J.A. Stochastic convergence. Academic Press (1937).
72. Markov, A.A. Calclus of Probability (Russian) 4th ed. Moscow (1924).
73. Medhi, J. Stochastic Processes, New Age International Pub. New Delhi, 2nd ed
(1994).
74. Miller, H.D. A Generalization of Wald’s identity with Applications to Random
Walks. Ann. Math. Statist., 32, 549-560 (1961).
75. Morse, PM. Queues, Inventories and Maintenance. John Wiley, New York (1958).
76. Neuts, M.F. A general class of bulk queues with Poisson input. Ann. Math.
Statist., 38, 759-770 ( 1967).
77. Omstein, L.S. and Uhlenbeck, G.E. on the theory of Brownian motion. Physical
Review, 36.93-112 (1930). [Reprinted in selected papers on Noise and Stochastic
Processes, (Ed. Wase, N.) Dover, New York (1954)].
78. Palm, C. (1943) Intensitatschwarkungen in Femsprechovorkehr Eriesson technics,
44, 1-189, Stocholm: Tele fonaktiebolaget I.M. Ericeson. I.M. Ericeson.
79. Parzen, E. Stochastic Processes, SIAM, Philadelphia (1999).
80. Prabhu, N.U. Queues and Inventories, John Wiley, New York (1965).
81. Prabhu, N.U. Stochastic Processes, Macmillan, New York (1965).
82. Priestley, M.B. Spectral analysis and time Series, Vol. I & II, Academic Press,
New York (1981).
83. Quenouille, M.H. Approximate tests for correlation in time Series. J. Roy. Stat.
Soc. B, 11, 68-84 (1957).
84. Resnik, S.I. Adventures in stochastic Processes. Birkhairser, Boston (1999).
85. Rolski, T., schmidli, H., Schmidt, V., Tengels, J. Stochastic Processes for Insurance
and Finance, Wiley, New York (1998).
86. Rosenblatt, M. Some regression problems in time series Analysis. Proc. 3rd.
Berkeley. Symp. On Math. Statist. Prob., Univ. of California Press, Berkeley
165-186 (1956).
87. Ross, S. Stochastic Processes. 2nd ed. John Wiley, New York (2000).
88. Slutsky, E. Sur Les Fonctions Eventuelles Continues, Integrates et Drivables
Dans le sens stochastique. C.R. Acad. Sci. Paris, 187, 878-880 (1928).
89. Smith, W.L. Renewal Theory and its Ramifications, J. Roy. Statist. Soc., B 20,
243-302 (1958).
90. Smoluchowlski, M. V. Molekularthearetische studen uber Umkehn thermodynamisch
irreversilder vorgange and uber wideskehn abnormale zustance, S.B. Akad. Wiss
Wien (2a): 124-339 (1915).
91. Spitzer, F. Principles of random walk. D. Van Nonstrand, Co. Princeton (1964).
92. Stidham, S. A last work on L = W, Opns. Res. 22, 417-421 (1974).
93. Stratonovich, R.L. A new representation for stochastic Integrals and equation,
Bull. Moscow Univ. Math, and Mech. Ser. No: 1, 3-12 (in Russian) (1964)
Translation in J. SIAM CONTROL, 4, 362-371 (1965).
94. Takacs, L. Investigation of wating time problems by reduction to Markov processes,
Act. Math. Hung., 6, 101-129 (1955).
95. Tichmmarsh, E.C. Theory of Functions, Griffin, London (1936).
96. Wald, A. Sequential Analysis, John wiley, New York (1945).
97. Wiener, N. Extrapolation, interpolation and smoothing of stationary time series;
with Engineering Applications. Cambridge - New York (1949).
98. Wold, H. A study in the Analysis of stationary Time series, 2nd ed. Almqvist &
Wiksells, Stockholm (1954).
99. Wong, E. Stochastic Processes in Information and Dynamical systems. McGraw-
Hill, New York (1971).
100. Yule, G.D. A Mathematical Theory of Evolution, Phil. Trans. Roy. Soc., London
B, 213, 21-87.
Subject Index
Abel’s Lemma, 16 with drift, 158, 161

Absorbing state, 13 geometric, 174
Accessible state, 12 martingales related to, 260
Amplitudes, 185, 186, 187 multidimensional Brownian motion,
Age replacement, 82 170
Aperiodic Markov Chains, 23 radial, see Bessel process, 170
Arley Process (Feller), 128, 129 reflected at the origin, 173
(Lattice) Arithmatic distribution, 70, 85 quadratic variation of, 262
Autocorrelation functions, 178, 187 (bounded)
Autocovariance function, 178 total variation of, 263
Autocorrelation Generating function
(AGF), 201 Caley-Hamilton Theorem, 42
Autoregressive process, 180, 188, 190 Cauchy - Schwartz inequality,
Autoregressive moving averages, 181 Central limit theorem
ARMA (p, q), 181,203 in a renewal process, 74
ARIMA (p, d, q), 181, 193 Chapman-Kolmogorov equation,
Average Power of random process, 185 connected with branching processes,
99
Backward Kolmogorov Closed (Transient) class, 13
differential equation, 155 Communicating class of states, 12
Bessel process, 170 Conservative (nonexplosive) Markov
Birth and Death Processes, 117 process, 112
with linear growth, 125 Continuity of sample paths, 151, 261
linear growth with immigration, 126 Correlation function, 178
nonhomogeneous, 121 Correlogram analysis, 184, 187
probability of absorption, 126 Covariance function, 178
pure birth processes, 134 Covariance matrix, 4, 167, 177
Block replacement, 82 Covariance stationary process, 4, 177
Bochner-Khintchine’s Theorem, 195, 197 Current life,
Borel Zero-one law, 267 limiting distribution, 80
Branching processess, 85 in a Poisson process, 137
in continuous time, 98 in a renewal process, 78
extinction probability, 91
generating functions, 86 Deterministic process, 194
with linear birth and death process, Differential difference equation, 112
102 of conservative process, 112
factorial moments, 103 of birth and death processes, 117
Kolmogorv equations for branching Diffusion equation, 155
process, 100, 129 Diffusion process, 151
with multiple types, Dominated Convergence theorem, 17
Brownian motion, 156 Doob’s renewal theorem, 71
Elementary renewal theorem, 70, 71 Instantaneous state, 112

Erlang’s models, 230, 247 Inventory (Storage) models,
Erlang loss system, 247 Irreducible (ergodic) Markov chains, 13
Entry time, 14
Ergodicity, 3, 182 Joint distribution, 4, 154
Ergodic states in a Mrkov Chain, 28
Ergodic Theorem, 182 Kemperman and Miller’s indentity, 60
Ehrenfest chain, 34, 35 Karhunen-Loe’ve expansion, 274, 282
embedded markov chain, 243 Kolmogorov’s formula, 261
Essential state, 12 Kronecker’s lemma, 183, 198
Inessential state, 12
Excess life, 78 Lengevian equation, 163
Exponential Smoothing, 212 Laplace transform
Extinction in branching processes of a renewal equation, 67, 76
in continuous time, 104 Limit theorems
in discrete time, 91 for renewal process, 70-71
for branching processes,
Fair game, 2 for Markov chains, 23, 26, 28, 30
FCFS, 227 Limiting distribution
First come, first serve discipline (FCFS), of current life in a renewal process, 80
227 of excess life in a renewal process, 79
First entrance time, 14 in a Markov renewal process.
First Fourier Transform, 210 Linear fractional transformation, 98
Fisher’s Information, 216 Theory of Linear predictor,
Fisher’s test for Hidden periodicties, 210 Little’s formula, 97
Forward Kolmogorov differential Lotka model, 97
equations, 155
Fourier Analysis, 196 Markov branching process, 98
Frequency, 185, 195 Markov chain,
Birth and Death, 32
Gambler’s ruin, 52 Markov process
Gaussian process, 4, 167 basic limit theorems, 23, 26
Generating functions Classifications of states, 14
Generating functions relations for Periodicity, 23
branching processes, 88, 96 recurrence of , 16
Fokker - Planck equation, 155 Markov renewal process,
Foster type theorems, 136 Markov time, 301
Genetic models, Martingales
General input, 228 Continuous parameter, 260, 301
Geometric Brownian motion, 174 Maxwell-Boltzmn distribution, 164, 165
Mean recurrence time, 16
Hamming, 216 Mean square convergence, 182, 271
Hanning, 210, 212 Mean square distance, 275
Heat equation, 155 Mean square error, 182, 270
Hilbert space, 275 Measurable progressively, 260
Mercer’s theorem, 282
Independent increments, 2, 133, 156 Moving average processes, 179
Index parameter, 1 Multivariate normal distribution, 4, 167
Inequalities for Brownian motion, 174
Infinitesimal generator, 114 NBU distribution, 78
Infinitesimal matrix, 114 NBUE, 78
Subject Index 417
Negative binomial distribution, 6 Queueing sheory, 237

Nonanticipating, 289 bulk service rule, 257
Nonhomogeneous Poisson process, 143 Queueing time, 232
null recurrent states, 16 Steady state distribution, 228
non-Markovian, 243
Offspring distribution, 87-88 utilization (load) factor, 231
Optional sampling theorem, 300, 302 Queues
Optional stopping theorem, 300 bulk, 256
Order statistics, 146 group arrival, 256
Ornstain-Uhlenbeck process, 162 multichannel, 240
non-Markovian, 243
Periodic chain, 23 Zero avoiding process, 238
Periodicity of Markov Chains, 23
Periodogram, 184, 209 Random function (process), 1
Point processes, 2 Random walk.
Poisson distribution, 133 Correlated, 49
Poisson process, 133 Realization of aprocess,
Poisson increment process, 145 Recurrence time (forward/backward), 78
Characterization of, 140 Recurrent events, 1
Compound (cumulative), 144 Recurrent states, 16
Interrival time of, 139 Reducible chain, 12
Nonhomogeneous (time dependent), closed class, 12
143 Reflection principle, 159
Postulates, for, 134 Regeneration point (epoch), 243
Properties of, 138 Regenerative stochastic process, 243
Positive recurrent, state, 16 Regular (standard) Markov process,
Prediction theorem, 195 Reliability theory, 77
Pollaczek - Khintchine formula, 229 Renewal argument, 74
- mean-value formula, 249 Renewal counting process, 65
- transform formula, 250 Renewal function, 65
Polya’s urn model, 45 Renewal density, 74
Power spectrum, 185 Renewal equation, 66
Probability generating function, 16 Renewal process, 65
Probability vector, 9 Delayed, 75
Process-Kendall, 126 equilibrium (stationary), 77, 75
Process with independent increments, 2 in continuous time, 65
Separable, 259 in discrete time, 65
Pure birth process, 118 Renewal reward process, 80
Purely random process, 179 Renewal theorems,
Blackwell’s, 70
Queueing models Doob’s, 71
M/M/C, 239 elementary, 70
MIDI 1, 249 Key (Smith’s), 70
Ed Ml 1, 251 Renewal theory,
GI/G/1, 246 Replacement policies,
MIE/JX, 249 age, 82
MIGl 1, 246 block, 82
Ml MIX, 230 Residual and excess (or spent life) life
Mix)/MI 1, 256 times, 79, 80
Queing chain, 23 asymptotic distributions of - 79, 80
Queing processes, 238 Rouche’s theorem, 234
Sample function, 1 Supermartingales, 299

Sample correlogram, 191 Supplementary variable technique, 246
SARIMA (p, d, q) x (P, D.Q), 213
Seasonal components, 209 Tackcs integral equation, 258
Second order stationary process, 276 Tauberian theorems, 16
Semi-markov process, 246 Telegraph signals, 7
embedded markov chain, 246 random - , 7
Slutsky's theorem, 217 Time domian, 184
Sojurn time, 232 Time series
Spectral analysis, 184 autoregressive process, 180
Spectral decomposition, 195 first order Markov process, 188
Spectral density function, 184 moving average process, 181
Spectral distribution function, 186, 196 Total life
Spectral representation, 186, 196 in a renewal process, 78
Spectral theorem, 195 Transient state, 16
Spectrum, 185 Traffic intensity, 231
Spectrogram, 207 Transient behaviour
Steady state, of M/M/l queue, 233
Standrd Brownian motion, 156 Transition density
State space, 1 of continuous time Markov chain,
Stationary increments, 133 152, 157
Stationary distribution for a M.C., 27 Transition function, 151
Stationary process, 2, 177 Transition distribution of waiting time,
Statistical equilibrium, 28 233
covariance function of, Transition (Markov) matrix, 8
convariance stationary (weakly or Two state process, 30
wide sense stationary), 4
evaluationary process, 4 Uniformly integrable random variables,
Gaussian process, 4 72
multi-dimensional, 4 Urn models, 45, 47
Stictly stationary, Ehrenfest model, 34
with independent increments, 177 modified Ehrenfest model, 35
Stationary transition functions
(probabilities), 8 Waiting times
Stochastic matrix, 8 in a Poisson process, 138
Stochastic process, 1 Waiting time distribution
regular, 5 - in M/M/l queue, 231
orthogonal, 5 - M/M/C queue, 242
Stratonovich integral, 289 Wald’s equation, 59
Stochastic sequence, 1 Wald’s indentity, 60
independent, 2 Wald’s martingale, 393
Stochastic Integral, 273, 287 Weakly stationary process, 2, 117
Stopping time (Markov time), 59, 299, White noise process (Purely random
301 process), 117
Submatringales, 280 Wiener process (Wiener Einstein), 156
martingales, 2, 95, 260 Wiener-Khinchine’s theorem, 195
Subordinator, 174 backward diffusion equation, 155
Success runs, 21 definition of, 155
Sums of independent random variables differential equation of, 155
associated with renewal processes, 64 distribution of maximum of, 160
Sum of a random number of r.vs., 59 drift of, 161
Subject Index 419
first passage time distribution of, 161 - Tukey, 212

forward diffusion equation of, 155, Wold Decomposition Theorem, 195, 284
165
Kolmogorov equation of, 155 Yule (Yule-Furry) process, 118
sample path of Brownian motion, 260 Yule-Walker equations, 190
Windows
Parzen, 212 Zero-avoiding state probability, 238
Introduction to
Stochastic Process
Professor A. K. Basu
This book, suitable for advanced undergraduate,

graduate and research courses in such subjects as
statistics, applied mathematics, operation research,
computer science, different branches o f engineering,
business and management, economics and life sciences
etc., is aimed between elementary probability texts and
advanced works on stochastic processes. What
distinguishes the text is the illustration o f the theorems
by examples and applications. The key features of the
book are:
• Well integrated elementary probability and
advanced stochastic processes
• Exercises at the end of each chapter with complete
solutions (provided separately)
• Appendix on second order process, sample path
properties of Brownian motion along with
stochastic integrals.
ISBN cl?fl-l-fl4EtjS-lDS-b
©
Introduction to
Stochastic Process i Introduction to
Stochastic Process
Professor A.K. Basu
This book, suitable for advanced undergraduate,

graduate and research courses in such subjects as
statistics, applied mathematics, operation research,
computer science, different branches of engineering,
business and management, economics and life sciences
etc., is aimed between elementary probability texts and
advanced works on stochastic processes. What
distinguishes the text is the illustration of the theorems
by examples and applications. The key features of the
book are:
• Well integrated elementary probability and
advanced stochastic processes
• Exercises at the end of each chapter with complete
solutions (provided separately)
• Appendix on second order process, sample path
properties o f Brownian motion along with
stochastic integrals.
ISBN t»?fi-l-642tS-lDS-b
©
Alpha Science In

A. K. Basu - Introduction To Stochastic Process

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

A. K. Basu - Introduction To Stochastic Process

Uploaded by

Copyright:

Available Formats

Introduction to

Copyright © 2003 Narosa Publishing House Pvt. Ltd.

Alpha Science International Ltd.

From my thirty years experience of teaching the subject of stochastic process to

P{A) Probability of the event A

Sn Sum of N i.i.d. r.v.s Xh where /V is a r.v.

2. Discrete Time Markov Chains 8

6. Continuous Time Discrete State Markov Processes 107

7. Poisson Process 133

8. Continuous Time and Continuous State Markov Process 150

9. Time Series Analysis 175

9.7 Forecasting 2/2

Solution to Exercises 303

1.1 Notion of Stochastic Processes

1.2 Different Types of Stochastic Processes

t0 < r, < . . . < tn, tj e T, i = I, 2, . . . , n (1.1)

if X, , X, - XtQ, Xt2 - Xt] , . . . , Xt - Xtn_] are independent for all possible

/o < r, < t2 < ■• • < C+1, tj G 7 • / = 0, 1, 2, . . . , n + 1 (1.2)

If E[Xlfj+] \Xtn —an , Xtn ^ —an_\, . . . , XtQ—Oq] —an

t\ < t2 < . . . < tn, tj e T, ti + h E T

1.3 An Introduction to Stationary Processes

Examples of stationary processes

with intensity parameter A (to be defined in Chapter 7) and L is a positive

constant. Its mean function m(t) - E(Xt) = X{t + L) - X{t) + cos

Cm,„ = E( XmX„)= £ £ a * k £(£„_*

(72 £ a 2i+»-m if in - « < 0

(72 £ a 2i+"-m if m - n >0

Therefore, the process is covariance stationary.

Exercises and Complements

(Au)a'-' e - x\ a > 0, A > 0 .

Assume that the distribution of T is F(t) = P[T < /].

(a) Show that EPx (/(x)) < Epi (/(x ) if p x < p 2

2.1 Definition and Transition Probabilities

Definition 2.2 A square matrix P = (ptj) is called a stochastic matrix if each of

= Z P[Xn =j I X„_, = k}P[Xn_, (by Markov property)

P<m'n) = 2 p l r u)Ptt'n) for m < u < n

and for all values of /, j in S, where p (tjn'n)= P(Xn = j \ Xm = i ).

where P<m'n) = ( p ^ n)).

We also have P{m'n) = p ^ n^ p ^ ~

Proof P ^ ’n) =ZP(Xn = j | Xn= k , X m = i)P(Xn_t

2.2 A Few More Examples

(b) Success runs

This follows either directly or from Chapman-Kolmogorov’s equation. It can

be shown that Pn converges to a matrix whose all elements in the column j

(c) Two state M.C.

i -p p } , 0 < p < 1 and 0 < a < 1.

The system is said to be in state Ex if a particle moves in the positive direction

(d) Random walk with absorbing barriers

(e) Random walk with reflecting barriers

This chain may be interpreted in Gambling language by considering two

2.3 Classification of States

2.3.1 Decomposition of state space

0 < p - p p ) kl) < Z p l p p ^ = p (ik+m) (Chapman-Kolmogorov)

Lemma 2.2 If C is closed then

p?i = S c PikPki = S c PikPkj + = 0 fory 2 C

Corollary 2.1 A Markov chain is irreducible if the only closed set in S is S

P T =ZPJ ' PJ ? ^ pT '=^ p T pV h«'d

In this example p55 = 1, i.e. the state 5 is absorbing.

2.3.2 Classification of states

Define /^ 0) = 0. Note that f - p = P[Tjj = n], where

Tij = min {n: X„= j \ X0 = i]

Note = p jj gives the diagonal of the transition matrix.

Theorem 2.4 p ' f 1= 2^ f ' f ' ' for all = 1, 2, . . . n.

= S F[X„ =y | Xm =y, Xm_, y, . . . . X, y, X0 =

___________ n ! _ n ! ___