Is

a mathematical system that undergoes transitions from one state to another, between a finite or countable number of possible states. It is a random process characterized as memorylessness.
the next state depends only on the current state and not on the sequence of events that preceded it. This specific kind of "memorylessness" is called the Markov property. Markov chains have many applications as statistical models of real-world processes.

MARKOV CHAINS

INTRODUCTION

Markov chain is the so-called "drunkard's walk", a random walk on the number line where, at each step, the position may change by +1 or 1 with equal probability. From any position there are two possible transitions, to the next or previous integer. The transition probabilities depend only on the current position, not on the manner in which the position was reached. For example, the transition probabilities from 5 to 4 and 5 to 6 are both 0.5, and all other transition probabilities from 5 are 0. These probabilities are independent of whether the system was previously in 4 or 6.

ANOTHER EXAMPLE IS THE DIETARY HABITS OF A CREATURE WHO EATS ONLY GRAPES, CHEESE OR LETTUCE, AND WHOSE DIETARY HABITS CONFORM TO THE FOLLOWING RULES:

It eats exactly once a day. If it ate cheese today, tomorrow it will eat lettuce or grapes with equal probability. If it ate grapes today, tomorrow it will eat grapes with probability 1/10, cheese with probability 4/10 and lettuce with probability 5/10. If it ate lettuce today, it will not eat lettuce again tomorrow but will eat grapes with probability 4/10 or cheese with probability 6/10.

FORMULA DEFINITION

A Markov chain is a sequence of random variables X1, X2, X3, ... with the Markov property, namely that, given the present state, the future and past states are independent. Formally,

The possible values of Xi form a countable set S called the state space of the chain. Markov chains are often described by a directed graph, where the edges are labeled by the probabilities of going from one state to the other states.

VARIATIONS

Continuous-time Markov processes have a continuous index. Time-homogeneous Markov chains (or stationary Markov chains) are processes where

REDUCIBILITY

A state j is said to be accessible from a state i (written i j) if a system started in state i has a non-zero probability of transitioning into state j at some point. Formally, state j is accessible from state i if there exists an integer n 0 such that

Allowing n to be zero means that every state is defined to be accessible from itself.

PERIODICITY

A state i has period k if any return to state i must occur in multiples of k time steps. Formally, the period of a state is defined as

Note that even though a state has period k, it may not be possible to reach the state in k steps. For example, suppose it is possible to return to the state in {6, 8, 10, 12, ...} time steps; k would be 2, even though 2 does not appear in this list. If k = 1, then the state is said to be a periodic: returns to state i can occur at irregular times. Otherwise (k > 1), the state is said to be periodic with period k. It can be shown that every state in a communicating class must have overlapping periods with all equivalent-or-larger occurring samples.

RECURRENCE

A state i is said to be transient if, given that we start in state i, there is a non-zero probability that we will never return to i. Formally, let the random variable Ti be the first return time to state i (the "hitting time"):

The number is the probability that we return to state i for the first time after n steps. Therefore, state i is transient if

State i is recurrent (or persistent) if it is not transient. Recurrent states have finite hitting time with probability 1.

ERGODICITY

A state i is said to be ergodic if it is aperiodic and positive recurrent. If all states in an irreducible Markov chain are ergodic, then the chain is said to be ergodic.It can be shown that a finite state irreducible Markov chain is ergodic if it has an aperiodic state.

FINITE STATE SPACE

If the state space is finite, the transition probability distribution can be represented by a matrix, called the transition matrix, with the (i, j)th element of P equal to

Since each row of P sums to one and all elements are non-negative, P is a right stochastic matrix. Time-homogeneous Markov chain with a finite state space If the Markov chain is time-homogeneous, then the transition matrix P is the same after each step, so the kstep transition probability can be computed as the k-th power of the transition matrix, Pk.

REVERSIBLE MARKOV CHAIN

A Markov chain is said to be reversible if there is a probability distribution over states, , such that

for all times n and all states i and j. This condition is also known as the detailed balance condition.

BERNOULLI SCHEME

A Bernoulli scheme is a special case of a Markov chain where the transition probability matrix has identical rows, which means that the next state is even independent of the current state (in addition to being independent of the past states). A Bernoulli scheme with only two possible states is known as a Bernoulli process.

APPLICATIONS

Markov chains are applied in a number of ways to many different fields. Often they are used as a mathematical model from some random physical process; if the parameters of the chain are known, quantitative predictions can be made. In other cases, they are used to model a more abstract process, and are the theoretical underpinning of an algorithm.
Physics Chemistry Testing Information sciences Queueing theory Internet applications Statistics

APPLY IN:

Economics and finance Social sciences Mathematical biology Games Music

STATE MACHINE

In general, a state machine is any device that stores the status of something at a given time and can operate on input to change the status and/or cause an action or output to take place for any given change.

To summarize it, a state machine can be described as:

An initial state or record of something stored someplace A set of possible input events A set of new states that may result from the input A set of possible actions or output events that result from a new state

IN THEIR BOOK REAL-TIME OBJECT-ORIENTED MODELING, BRAN SELIC & GARTH GULLEKSON VIEW A STATE MACHINE AS:
A set of input events A set of output events A set of states A function that maps states and input to output A function that maps states and inputs to states (which is called a state transition function) A description of the initial state

FINITE STATE MACHINE

A finite state machine is a mathematical abstraction used to design algorithms. In simple terms, a state machine will read a series of inputs. When it reads an input it will switch to a different state. Each state specifies which state to switch for a given input. This sounds complicated but it is really quite simple. Imagine a device that reads a long piece of paper. For every inch of paper there is a single letter printed on iteither the letter a or the letter b.

As the state machine reads each letter it changes state. Here is a very simple state machine.

The circles are states that the machine can be in. The arrows are the transitions. So if you are in state sand read an a youll transition to state q. If you read a b, youll stay in state s.

DETERMINISTIC FINITE STATE MACHINE

The state machines weve looked at so far are all deterministic state machines. From any state there is only one transition for any allowed input. In other words there arent two paths leading out of a state when you read the letter a. At first this sounds silly to even make this distinction.

NON-DETERMINISTIC FINITE STATE MACHINE

Non-deterministic finite state machines are finite state machines where a given input from a particular state can lead to more than one different state. For example, lets say we want to build a finite state machine that can recognize strings of letters that start with a and are then followed by zero or more occurrences of the letter b or zero or more occurrences of the letter c terminated by the next letter of the alphabet. Valid strings would be: abbbbbbbbbc abbbc acccd acccccd ac (zero occurrences of b) ad (zero occurrences of c)

So it will recognize the letter a followed by zero or more of the same letter of b or c followed by the next letter of the alphabet. A very simple way to represent this is with a state machine that looks like the one below, where a final state of t means that the string was accepted and matches the pattern.

The machine below is a deterministic version of the nondeterministic machine above. In the machine below, a final state of t or v is reached by any string that is accepted by the machine.

The non-deterministic model has four states and six transitions. The deterministic model has 6 states, 10 transitions and two possible final states. That isnt that much more, but the complexity usually grows exponentially and a moderately sized non-deterministic machine can produce an absolutely huge deterministic machine.

REGULAR EXPRESSION

If you have done any type of programming, youve probably encountered regular expressions. Regular expressions and finite state machines are functionally equivalent. Anything you can accept or match with a regular expression can be accepted or matched with a state machine. For example the pattern above could be matched with: a(b*c|c*d)

Regular expressions and finite state machines also have the same limitations. In particular they both can only match or accept patterns that can be handled with finite memory. So what type of patterns cant they match? Lets say you want to only match strings of a and b, where there are a number of as followed by an equal number of bs. Or n as followed by n bs where n is some number. Examples would be:

ab aabb aaaaaabbbbbb Aaaaaaaaaaaaaaaaaaaabbbbbbbbbbbbbbbbbbbb

At first this looks like an easy job for a finite state machine. The problem is that youll quickly run out of states or youll have to assume an infinite number of statesat which point it is no longer a finite state machine.

TURING MACHINES

There is a theoretical device that is similar to a state machine called a Turing Machine. It is similar to a finite state machine that it has a paper strip that it reads, but a Turing Machine can erase and write on the paper tape. Explaining a Turing Machine will take more space that we have here, but there are a few important points relevant to our discussion of finite state machines and regular expressions.