National University of Sciences & Technology (NUST) School of Electrical Engineering and Computer Science (SEECS) Department of Electrical

Engineering

Information and Coding Theory

Course Code: Credit Hours: Instructor: Office: Lecture Days: Class Room: Knowledge Group: EE-831 3+0 Dr. Adnan K. Kiani Top Floor IAEC Building Mon, Thurs Semester: Spring 2013 Prerequisite Codes: Discipline: MS-EE (T&N) Telephone: +92 (0)51 9085 2104 E-mail: adnan.khalid@seecs.edu. pk Consulting Hours: Updates on Every Week LMS:

Telecomm & Networks

Course Description:
This course will provide the students an introduction to classical Information Theory. The main course objective is to introduce the students to well-known information theoretic tools that can be used to solve engineering problems. The course will begin with information measurement and characterization leading to the Asymptotic Equipartition Property. After that, the students will study markov chains, Fundamentals of noiseless source coding, information capacity of some well known channels, optimal source coding techniques, channel limitations and channel coding techniques will also be taught. Finally, gambling and its relation to compression will be discussed.

Course Outcomes/Objectives:
On the completion of the course, the student will be able to: Use information theory principles for modeling communication systems. Understand the basic compression limits for a source and their relations to entropy. Understand markov processes and their importance in information theory. To relate coding with information theory. To understand optimum source coding techniques. To evolve capacities of different types of communication channels. To understand various channel coding techniques. To understand the relation between gambling and data compression.

Books:
Text Elements of Information Theory by Thomas M. Cover and Joy A. Thomas. 2nd Book: ed. Wiley-Interscience, July 18, 2006. ISBN-10:0471241954, ISBN-13: 9780471241959 Referenc e Books:
A First Course in Information Theory by Raymond W. Yeung, 1st ed., Springer, 2006. Coding and Information Theory by Richard W. Hamming. 2nd ed. Prentice Hall.

Page 1 of 3

National University of Sciences & Technology (NUST) School of Electrical Engineering and Computer Science (SEECS) Department of Electrical Engineering

Main Topics to be Covered:

The course spans over a number of different topics as under: The basics of Probability theory and R Shannons Channel Coding Theorem Random Processes Entropy of a Source Linear Block/Hamming Codes Joint and Conditional Entropy Error detection and correction Mutual Information and Conditional Singleton Bound Mutual Information Properties inequalities and chain rules of Convolutional Codes information measures Turbo Codes Source Coding Theorem Optimum Source Codes AEP of an iid Source Huffman Codes AEP for stationary and ergodic sources Arithmetic Codes Gambling and Data Compression Capacity of Noiseless Channel Capacity of DMC Capacity of Erasure Channel

Lecture Breakdown:
Chapter Topics 1 Introduction to probability theory and random process 2 Entropy of a Source 3 Joint and Conditional Entropy, and Mutual Information 4 Chain rules 5 Source Coding Theorem 6 Optimum Source Codes 7 Channel Capacity 8 Channel Coding Theorem 9 Block Codes 10 Error Correction/detection, Singleton Bound 11 Convolutional Codes 12 AEP Total: Lectures 3 3 6 4 3 5 5 3 5 3 3 2 45

Weightages:
Quizzes: Assignments: OHT 1 & 2: Final Exam: 15% 15% 25% 45% Page 2 of 3

National University of Sciences & Technology (NUST) School of Electrical Engineering and Computer Science (SEECS) Department of Electrical Engineering
Grading Policy:
Quiz Policy: The quizzes will be unannounced and normally last for ten minutes. The question framed is to test the concepts involved in last few lectures. Number of quizzes that will be used for evaluation is at the instructors discretion. Grading for quizzes will be on a fixed scale of 0 to 10. A score of 10 indicates an exceptional attempt towards the answer and a score of 1 indicates your answer is entirely wrong but you made a reasonable effort towards the solution. Scores in between indicate very good (8-9), good (67), satisfactory (4-5), and poor (2-3) attempt. Failure to make a reasonable effort to answer a question scores a 0. Assignment In order to develop comprehensive understanding of the subject, Policy: assignments will be given. Late assignments will not be accepted / graded. All assignments will count towards the total (No best-of policy). The students are advised to do the assignment themselves. Copying of assignments is highly discouraged and violations will be dealt with severely by referring any occurrences to the disciplinary committee. The questions in the assignment are meant to be challenging to give students confidence and extensive knowledge about the subject matter and enable them to prepare for the exams. Plagiarism: SEECS maintains a zero tolerance policy towards plagiarism. While collaboration in this course is highly encouraged, you must ensure that you do not claim other peoples work/ ideas as your own. Plagiarism occurs when the words, ideas, assertions, theories, figures, images, programming codes of others are presented as your own work. You must cite and acknowledge all sources of information in your assignments. Failing to comply with the SEECS plagiarism policy will lead to strict penalties including zero marks in assignments and referral to the academic coordination office for disciplinary action.

Page 3 of 3