Chapter 1

What is Information?
Form 1995 we have been witnessing a transformation or revolution in the way we communicate
and the process is still under way. It includes ever growing use of internet, explosion of mobile
networks and video conferencing. Information i.e. text, audio, pictures and video is transmitted
on networks. Data compression is one enabling technologies for each of aspects of multimedia
revolution.
Information treated in various disciplines
physics basic property as energy, matter
biology senses
neurophysiology brain processes
psychology behavior, perception
cognitive science cognition
telecommunication, computer science signals, bits
philosophy knowledge
The discussion centers on information as an energy and signals/bits.
Define the scope of information theory and coding.
Information theory is a branch of applied mathematics, electrical engineering, and computer
science involving the quantification of information.
Information theory studies the transmission, processing, utilization, and extraction of
information. In the case of communication of information over a noisy channel, this abstract
concept was made concrete in 1948 by Claude Shannon in A Mathematical Theory of
Communication, in which "information" is thought of as a set of possible messages, where the
goal is to send these messages over a noisy channel, and then to have the receiver reconstruct the
message with low probability of error, in spite of the channel noise. Shannon's main result, the
Noisy-channel coding theorem showed that, in the limit of many channel uses, the rate of
information that is asymptotically achievable is equal to the Channel capacity, a quantity
dependent merely on the statistics of the channel over which the messages are sent.
Coding theory is concerned with finding explicit methods, called codes, for increasing the
efficiency and reducing the error rate of data communication over noisy channels to near the
Channel capacity. These codes can be roughly subdivided into data compression (source coding)
and error-correction (channel coding) techniques. A third class of information theory codes are
cryptographic algorithms (both codes and ciphers). Concepts, methods and results from coding
theory and information theory are widely used in cryptography and cryptanalysis

2 Describe Shannons model for digital communications

It consists of essentially five parts:

1. An information source which produces a message or sequence of messages to be
communicated to thereceiving terminal. The message may be of various types: (a) A sequence of
letters as in a telegraphof teletype system; (b) A single function of time f (t) as in radio or
telephony; (c) A function oftime and other variables as in black and white television here the
message may be thought of as afunction f (x;y; t) of two space coordinates and time, the light
intensity at point (x;y) and time t on apickup tube plate; (d) Two or more functions of time, say f
(t), g(t), h(t)this is the case in threedimensionalsound transmission or if the system is
intended to service several individual channels inmultiplex; (e) Several functions of
severalvariablesin color television themessage consists of threefunctions f (x;y; t), g(x;y; t),
h(x;y; t) defined in a three-dimensional continuumwe may also thinkof these three functions as
components of a vector field defined in the region similarly, severalblack and whitetelevision
sources would produce messages consisting of a number of functions
of three variables; (f) Various combinations also occur, for example in television with an
associatedaudio channel.
2. A transmitter which operates on the message in some way to produce a signal suitable for
transmissionover the channel. In telephony this operation consists merely of changing sound
pressureinto a proportional electrical current.
3. The channel is merely the medium used to transmit the signal from transmitter to receiver. It
may bea pair of wires, a coaxial cable, a band of radio frequencies, a beam of light, etc.
4. The receiver ordinarily performs the inverse operation of that done by the transmitter,
reconstructingthe message from the signal.
5. The destination is the person (or thing) for whom the message is intended.

3 Define following terms discussed in Shannons Theory

The meaning of probability

Probabilities can describe frequencies of outcomes in random experiments, but giving

noncircular definitions of the terms `frequency' and `random' is achallenge { what does it mean
to say that the frequency of a tossed coin'scoming up heads is 1/2? If we say that this frequency
is the average fraction ofheads in long sequences, we have to define`average'; and it is hard to
define`average' without using a word synonymous to probability!
Information entropy and Uncertainty
Shannon had introduced quantitative measures of information, defined self information. A key
measure of information is entropy, which is usually expressed by the average number of bits
needed to store or communicate one symbol in a message. Entropy quantifies the uncertainty
involved in predicting the value of a random variable.Probability associated with event can be
used to calculate entropy
Suppose we have an event A, which is set of outcomes. If p (A) probability that Event A occurs
the self information associated with A is given by
i(A)= - log b P(A)
If event A has outcomes a1 and a2 information content is given by
I(A)=i(a1)+i(a2).
Then Shannon entropy of event can be calculated by

H ( A ) = pilog pi bits
i=0
Entropy is typically measured in bits, nats, or bans.

The capacity C of a discrete channel is given by

WhereN(T) is the number of allowed signals of duration T.

4 Explain Nyquist Shannon theorem with a suitable example
Sampling Theorem:
A bandlimited signal can be reconstructed exactly if it issampled at a rate atleast twice
themaximum frequencycomponent in it."
The frequency 2fmax is called the Nyquist sampling rate. Half of this value, fmax, is sometimes
called the Nyquist frequency

For example, the human ear can detect sound across the frequency range of 20 Hz to 20 kHz.
According to the sampling theorem, one should sample sound signals at least at 40 kHz in order
for the reconstructed sound signal to be acceptable to the human ear. Components higher than
20 kHz cannot be detected, but they can still pollute the sampled signal through aliasing.
Therefore, frequency components above 20 kHz are removed from the sound signal before
sampling by a band-pass or low-pass analog filter. Practically speaking, the sampling rate is
typically set at 44 kHz (rather than 40 kHz) in order to avoid signal contamination from the filter
rolloff.
Oversampling
In practice signal are oversampled, where fsis significantlyhigher than Nyquist rate to avoid
aliasing.
5. Entropy encoding
Entropy Coding

The process of entropy coding (EC) can be split in two parts: modeling and coding. Modeling
assigns probabilities to the symbols, and coding produces a bit sequence from these probabilities.
As established in Shannon's source coding theorem, there is a relationship between the symbol
probability and it's corresponding bit sequence. A symbol with probabilitiy p gets a bit sequence
of length -log(p).
In order to achieve a good compression rate, an exact propability estimation is needed. Since the
model is responsible for the probability of each symbol, modeling is one the most important
tasks in data compression.
Entropy coding can be done with a coding scheme, which uses a discrete number of bits for each
symbol, for example Huffman coding, or with a coding scheme, which uses a discrete number of
bits for several symbols. In the last case we get arithmetoc coding, if the number of symbols is
equal to the total number of symbols to encode.
For example, suppose we want to transmit messages composed of the four letters a, b, c, and d. A
straightforward scheme for coding these messages in bits would be to represent a by 00, b by
01, c by 10 and d by 11. However, suppose we know that for any letter of the message
(independent of all other letters), a occurs with probability .5, b occurs with probability .25, and
c or d occur with probability .125 each. Then we might choose a shorter representation for a, at
the necessary cost of accepting longer representations for the other letters. We could represent a
by 0, b by 10, c by 110, and d by 111. This representation is more compact on average
than the first one; indeed, it is the most compact representation possible (though not uniquely
so). In this simple example, the modeling part of the problem is determining the probabilities for
each possible symbol value; the entropy-coding part of the problem is determining the
representations in bits from those probabilities; let us emphasize that the probabilities associated
with the symbol values play a fundamental role in entropy coding.
6 problems on entropy calculation
1)aaabbbccdbbddc
2)11213444543323
3) B D C B C E C C C A D C B D D A A E C E E A

4) A B B D A E E C A C E E B A E E C B C E A D
5) Calculate the entropy for following events
a) From a pack of 52 cards, a card is drawn at random the independent outcome set
={ queen of club, a face card i.e. Jack, Queen and King , 8 of red color }
b) A fair coin tossed ten times outcome set T T H T T H H H T H T

Web references for further reading

Online entropy calculator is available on http://www.shannonentropy.netmark.pl/
web.mit.edu/6.933/www/Fall2001/Shannon1.pdf
http://mlg.eng.cam.ac.uk/zoubin/papers/ECS-infotheory02.pdf