MUTUAL INFORMATION The mutual information is defined as the amount of information transferred when xi is transmitted and yj is received.

It is represented by I(xi,yj) and given as, I(xi,yj)= log

bits

Here I(xi,yj) is the mutual information P(xi/yj) is the conditional probability that xi was transmitted and yj is received. P(xi) is the probability of symbol xi for transmission. The average mutual information is represented by I(X;Y). It is calculated in bits/symbol. The average mutual information is defined as the amount of source information gained per received symbol. It is given as,

Thus I(xi,yj is weighted by joint probabilities P(xi,yj) over all possible joint events. Putting for I(xi,yj) from equation we get

( Properties of Mutual Information

i) The mutual information of the channel is symmetric i.e.., I(X;Y)=I(Y;X) Soln. Let us consider some standard relationships from probability theory. These are as follows. P(xi yj)=P(xi/yj)p(yj) P(xiyj)=P(yj/xi)p(xi) Here P(xi yj) is the joint probability that xi is transmitted and yj is received. P(xi/yj) is the conditional probability of that xi is transmitted and yj is received.

P(yj/xi) is the conditional probability that yj is received and xi is transmitted. P(xi) is the probability of symbol xi for transmission. P(yj) is the probability of symbol yj is received. P(xi/yj)P(yj)=P(yj/xi)P(xi)

The average mutual information is given by equation ( Hence we can write I(Y;X) as follows. I(Y;X)= )

I(X;Y)=I(Y;X) ii) Prove the following relationships. I(X;Y)=H(X)-H(X/Y) I(X;Y)=H(Y)-H(Y/X) Sol. Here H(X,Y) is the conditional entropy and it is given as,

( )

H(X/Y) is the information or uncertainty in X after Y is received. In other words H(X/Y) is the information lost in the noisy channel. It is the average conditional self information. Consider the equation ( ) ( )

( )

Above equation can be written as ( )

Here let us use the standard probability relation which is given as follows: ( )

First term of the above equation represents entropy I(X;Y)=H(X)-H(X/Y)

Here note that I(X;Y) is the average information transferred per symbol across the channel. It is equal to source entropy minus information lost in the noisy channel is given by above equation. Similarly consider the average mutual information given by equation ( ) ( )

( )

( )

The conditional entropy H(Y/X) is given as,

( )

Here H(Y/X) is the uncertainty in Y when X was transmitted. With this result, the equation becomes,

 ( Here let us use the standard probability equation, ( Hence equation becomes, ( ) The entropy is given by equation

( )

( ) Hence we can write the above equation as

( )

I(Y;X)=H(Y)-H(Y/X) iii) Prove that the mutual information is always positive

Mutual information is given by equation as ( )

Putting above value of ( ) in equation

We know that

can be written as,

. Hence above equation becomes,

 ( This equation can be written as,

( Earlier we have derived one result for entropy (

This result can be applied to the above equation. We can consider pk be P(xi,yj) and qk be p(xi) p(yj). Both pk and qk are two probability distributions on some alphabet. Then equation becomes

iv) Prove the following I(X;Y)=H(X)+H(Y)-H(X,Y) We have derived following relation: H(X,Y)=H(X/Y)+H(Y) H(X/Y)=H(X,Y)-H(Y) Mutual information is given by equation also I(X;Y)=H(X)-H(X/Y) Putting for H(X/Y) in above equation I(X;Y)=H(X)+H(Y)-H(X,Y) CONDITIONAL ENTROPY The conditional entropy H(X/Y) is called equivocation. It is defined as, ( )

And the joint entropy H(X,Y) is given as, ( )

The conditional entropy H(X/Y) represents uncertainty of X , on average, when Y is known. Similarly the conditional entropy H(Y/X) represents uncertainty of Y, on average, When X is transmitted. H(Y/X) can be given as, ( )

The conditional entropy H(X/Y) is an average measure of uncertainty in X after Y is received. In other words H(X/Y) represents the information lost in the noisy channel.

PROVE THAT H(X,Y)=H(X/Y)+H(Y) =H(Y/X)+H(X) Consider equation ( )

( From probability theory we know that,

( ) ( ) ) ( ) [ ( ) ]

( [ ( ) ]=

We know that

( )

Hence above equation becomes,

( )

( )

{ (

)}

( )

The first term in above equation is H(X/Y). From the standard probability theory, ( Hence H(X,Y) will be written as, ( ) ( ) )

As per the definition of entropy, the second term in the above equation is H(Y). Hence, ( ) ( )

( )

( )

As per the definition of entropy, the second term in the above equation is H(Y). Hence, Thus the first given equation is proved. From the probability theory we know that P(AB)=P(B/A)P(A) ( Putting this result in the )

term of equation we get,

([

])

( )

As per equation the first term of above equation is H(Y/X).And from standard probability theory,

Hence H(X,Y) will be written as,

As per the definition of entropy, the second term in the above equation is H(X). Hence Thus the second part of the given equation is proved.