MARKOV ANALYSIS

Session 11

 The

term Markov Analysis refers to a quantitative technique

that involves the analysis of the current behavior of some
variable in order to predict the future behavior of that same
variable.
Markov analysis is applicable to systems in which we can
determine probability of movement from one state to another
over time.
A Markov process is a special type of stochastic process in
which the current state of a system depends only upon the
immediately preceding state of the system.
An example of the use of Markov Analysis is the
determination of the probability that a customer will change
brands of substitute product from one month to the next
month, also it has been used to describe the probability that a
machine is functioning in one period will continue to function
in the next period.

Introduction

 A wholesale

distributor is attempting to determine a more

effective set of credit-control policies. The company would
classify all of its accounts receivable into several
categories.
First, at any point in time in which an accounts receivable
balance is paid in full, it is immediately placed in full
category, and second if the payment exceeds 90 days, that
portion is placed in the defaulted category.
Third, the payment category according to the oldest unpaid
bill for each customer. For example, the total accounts
receivable balance is $6000. A payment on November 30,
then it would be classified as the delinquent (31-90 days)
category since the delivery placed on October 10 (41 days
old). And on December 5, the customer paid the October
10 bill of $1000. The remaining total balance of $5000,
corresponding to the November 20 purchase, is classified
as the delinquent (0-30 days) category since it less than 31
days old.

An Illustration : The Accounts

Receivable Control Problem

 Use

of this method of accounts receivable aging means

that an accounts receivable classified in the delinquent
(31-90 days) category at one point in time may appear in
the delinquent (0-30 days) category at a later point.
The various accounts receivable categories are the states
of the system and describe the status of the system at a
point in time.
Markov analysis assumes that those states are
collectively exhaustive (list all the possible states of the
system) and mutually exclusive (a property that the
system can only be in one state at any point in time).
In this discussion, it is assumed that Markov analysis has
a finite number of states for the system being analyzed.
Once the states have been identified, the next step in
Markov analysis process is to determine the probabilities
associated with the system being in a particular state.

An Illustration : The Accounts

For

this case, assume that there was $100,000

accounts receivable position and it is distributed
(probability) as follows.
State 1 Paid in full
State 2 Defaulted
State 3 Delinquent (0-30 days)
State 4 Delinquent (31-90 days)

$45,000/$100,000 = .45
$15,000/$100,000 = .15
$25,000/$100,000 = .25
$15,000/$100,000 = .15

Written

in V(1) = ( .45 .15 .25 .15 )

General form V(i) = (v1, v2, v3, ,vn)
V sounds to the vector of states
probabilities.

An Illustration : The Accounts

...
The

next step is to specify the matrix of transition

probabilities. A transition probability (pij) is defined
as conditional probability of the system being in
state j one period, or step, in the future, given that
system is in state i currently.
S1 S2 S3 S4
P=

S1

S2

P11 P12 P13 P14

=

P21 P22 P23 P24

S3 .5 .2 .1 .2

P31 P32 P33 P34

S4 .4 .4 .1 .1

P41 P42 P43 P44

 Property

1 : the transition probabilities for a given

beginning state of the system must sum to 1.0
Property 2 : the transition probabilities encompass all
possibilities for the system.
Property 3 : the transition probabilities are constant
over time.
Property 4 : the states of the transition probability matrix
are independent over time.

An Illustration : The Accounts

...
Now,

if there is any period n, the state

probabilities for period n+1 can be
computed as follows
S1 1 0 0 0
S2 0 1 0 0
V(n+1) = V(n) P. P
S 3 .5 .2 .1 .2
S 4 .4 .4 .1 .1

 For

the case
V(2) = V(1) . P
V(2) = (.45 .15 .25 .15)

.5

.2

.1

.2

.4

.4

.1

.1

= [ (.45)(1) + (.15)(0) + (.25)(.5) + (.15)(.4) ]

= [ (.45)(0) + (.15)(1) + (.25)(.2) + (.15)(.4) ]
= [ (.45)(0) + (.15)(0) + (.25)(.1) + (.15)(.1) ]
= [ (.45)(0) + (.15)(0) + (.25)(.2) + (.15)(.1) ]
= [ .635
.26
.04
.065]
S1
S2
S3
S4
Paid in full Defaulted Delinquent Delinquent
(0-30 days) (31-90 days)

An Illustration : The Accounts

 Continuing

the computation process of the state probabilities:

Period 3 =V(3) = V(2) . P
1
0
0
0
V(3) = (.681 .26 .04 .065)

.5

.2

.1

.2

.4

.4

.1

.1

V(3) = [ .681 .26 .04 .065 ]

Period 4 =V(4) = V(3) . P
V(4) = [ .6921 .3019 .0025 .0036 ]
Period 5 = V(5) = V(4) . P
V(5) = [ .6947 .3038 .0006 .0009 ]
Period 6 = V(6) = V(5) . P
V(6) = [ .6954 .3043 .0001 .0002 ]
Computing Steady State Probabilities

 Period

7 =V(7) = V(6) . P
V(7) = [ .69552 .30439 .00004 .000051 ]
Period 8 = V(8) = V(7) . P
V(8) = [ .6956 .3044 .0000 .0000 ]
Note that as it proceeded from period 2 to period 8,
the changes in the state probabilities become
smaller and smaller. The virtually constant state
probabilities that occur after a certain number of
periods are referred to as steady-state
probabilities.
Therefore, for accounts receivable problem, the
steady state are the same transition probabilities at
period 8, means that each probability being in its
category after a number of periods in the future is
independent toward its disposition in period 1.
Computing Steady State Probabilities

Category Of
Accounts
Receivable

Steady
-State
Probab
ility

Paid in full
Defaulted
Delinquent, 0-30 days
Delinquent, 31-90 days

.6956
.3044
.0000
.0000

Number of
Accounts
Receivable
1000
1000
1000
1000

x
x
x
x

.6956
.3044
.0000
.0000

=
=
=
=

695.6
304.4
0
0

Dollar Amount of
Accounts Receivable
$1,000,000
$1,000,000
$1,000,000
$1,000,000

x
x
x
x

.6956
.3044
.0000
.6956

=
=
=
=

$695,600
$304.400
$0
$0

For other variation state probabilities

S1

S2

S3

S1

.5

.3

.2

S2

.4

.3

.3

S3

.3

.3

.4

Beginning Condition V(1) = (.05

.05

.90)

State

Period
1

Period
2

Period
3

Period
4

Period
5

Period 6
(steady-state prob.)

S1
S2
S3

.3150
.3000
.3850

.3930
.3000
.3070

.4086
.3000
.2914

.4117
.3000
.2883

.4123
.3000
.2877

.4125
.3000
.2875

Computing Steady State Probabilities

If the steady-state probabilities is designated as 1, 2 , and 3 ,

for both period n and period n+1:
V(n) = V(n+1) = [1 2 3 ]

Consequently, for the previous example, the computation

given by
.5 .3 .2
[1 2 3 ]
.4 .3 .3
.3 .3 .4

The results is
1 = .51 + .42 + .33
2 = .31 + .32 + .33
3 = .21 + .32 + .43

Since the steady-state probabilities must sum to 1.0, then

1 = 1 + 2 + 3

An Algebraic approach to compute steady-state

probabilities

The above equations can be rewritten as follows:

0
0
0
1

= -.51 + .42 + .33

eq. 1
= .31 - .72 + .33
= .21 + .32 - .63
eq. 3
= 1 + 2 + 3
eq. 4

eq. 2

Arbitrarily it can be solved that:

Step 1 equate eq. 1 and eq. 2
-.51 + .42 + .33 = .31 - .72 + .33
-.51 + .42 = .31 - .72
- 81 = -1.1 2
1 = 1.1/.8 2
Step 2 add 3 times eq.1 to 5 times eq. 2
0 = -1.51 + 1.22 + .93
0 = 1.51 3.52 + 1.53
0 =

- 2.3 2 + 2.43

3 = 2.3/2.4 2

An Algebraic approach to compute steady-state

probabilities

3 solve for 1 2 3
1 = 1 + 2 + 3
1 = 1.1/.8 2 + 2 + 2.4/2.4 2
1 = 3.333 2
2 = 1/3.333 = .3000
1 = 1.1/.8 2 = .4125
3 = 1 (.3000 + .4125) = .2875

Step

It

is necessary to emphasize that this method for

solving the steady-state probabilities will work only for
non-absorbing Markov processes.

An Algebraic approach to compute steady-state

probabilities

A special case of a Markov process involves an absorbing or

trapping state.
An absorbing state is a state that has a zero probability of
being left once it is entered. Once the absorbing state is
entered, the process either stops completely or stops then is
reinitiated from some other state.
A Markov process can be shown to be an absorbing Markov
process if
1. it has at least one absorbing state.
2. It is possible to move from every non-absorbing state in a
finite number of steps.
For the case accounts receivable, it has two absorbing states
1. S1 : paid in full
2. S1 : defaulted
The probability of being in both categories above, in a future
time period given for the current period is 1.0 or 100 percent.

Analysis of Absorbing Markov Processes

Observe also that both of other states, S3 and S4, are nonabsorbing but it is possible to move from any of these nonabsorbing states to the absorbing states, S1 and S2. Or as
analyzed, after period 6, the steady-state probabilities of both
states had been reduced to zero.
In particular, the probability of absorption by any absorbing state;
and the expected number of steps before the process is absorbed
can be determined.
To begin this process, initially the matrix of transition possibilities
is transform into the following general form.

P=

where
I = an r-by-r identity matrix defining the probabilities of staying within
an
absorbing state once it reached
O = an r-by-s null matrix indicating the probabilities of going from an
absorbing
state to a non-absorbing state.

Analysis of Absorbing Markov Processes

P=

where
A = an s-by-r matrix containing the probabilities of going from a
non-absorbing
state to an absorbing state.
N = an s-by-s matrix showing the probabilities of going from one
non-absorbing
state to another non-absorbing state.
For accounts receivable case :

P=

1
0

0
1

0
0

0
0

.5 .2 .1 .2
.4 .4 .1 .1

The fundamental matrix F, can be calculated as

F = (1 N)-1

Analysis of Absorbing Markov Processes

First, determining (1 - N)

.9

-.2

-.1

.9

.1

.2

.1

.1

9/10

-2/10

-1/10

9/10

Next, the above matrix is inverted to obtain F = (1-N)-1

=

90/79

20/79

10/79

90/79

1.14

.25

.13

1.14

To interpret the results given by the fundamental matrix (F)

matrix, recall that the non-absorbing states are 3 and 4. The
expected number of steps before absorption is the sum of the
times the process in each absorbing state.

Analysis of Absorbing Markov Processes

Beginning State

Expected steps before absorption

S3
S4

90/79 + 20/79 = 110/79 =1.39

10/79 + 90/79 = 100/79 = .27

To compute the probability of absorption = FA = (1-N)-1 . A

=

1.14

.25

.5

.2

.13

1.14

.4

.4

.67

.33

.52

.48

The interpretation is that (first row) there is .67 probability that a delinquent
account in state 3 will end up in state 1 (paid in full), and .33 probability a
delinquent account beginning in state 3 will end up in state 2 (bad debt).
The second row indicates that a delinquent account in state 4 will end up in
state 1 at .52 probability and state 2 at .48 probability.

Analysis of Absorbing Markov Processes

 Finally,

assume that the company has found that it has

$25,000 accounts receivable in state 3 (delinquent,030 days) and $15,000 in state 4 (delinquent, 31-90
days), the absorption probability matrix that how many
of these dollars will eventually end up in state 1 or 2 is
determined (denoted by T) by

T` = T . Probability of absorption
T=

S3

S4

($25,000)
S1

($15,000)
S2

S3

S4

S3

.67

.33

($25,000)
S3

($15,000)

S4

.52

.48

($24,550)

S4

($15,450)

Therefore, if this company currently has $25,000 (state S3) +

$15,000 (state S4) = $40,000 of delinquent accounts receivable,
it can expect that eventually $24,450 will be collected in, while
$15,450 will become bad debts.

Analysis of Absorbing Markov Processes