Communication Systems by Simon HaykinChapter 9 : Fundamental Limits in Information Theory INTRODUCTION The of a communication system is to carry information bearing baseband signals from one place to another over a communication channel. INFORMATION THEORY ● It deals with mathematical modeling and analysis of a communication system rather than with physical sources and physical channel. ● It is a highly theoretical study of the efficient use of bandwidth to propagate information through electronic communications systems. INFORMATION THEORY ● It provides answers to the two fundamental questions: ○ What is the irreducible complexity below which a signal cannot be compressed? ○ What is the ultimate transmission rate for reliable communication over a noisy channel? The answer to these questions lie in the ENTROPY of a source and the CAPACITY of a channel, respectively. INFORMATION THEORY Entropy ● It is defined in terms of the probabilistic behavior of a source information. ● It is named in deference to the parallel use of this concept in thermodynamics. Capacity ● The intrinsic ability of a channel to convey information. ● It is naturally related to the noise characteristic of the channel. INFORMATION THEORY A remarkable result that emerges from information theory is that if the entropy of the source is less than the capacity of the channel, then error free communication over channel can be achieved. UNCERTAINTY, INFORMATION, AND ENTROPY DISCRETE RANDOM VARIABLE, Suppose that a probabilistic experiment involves the observation of the output emitted by a discrete source during every unit of time (signaling interval). The source output is modeled as a discrete random variable, S , which takes on symbols from a fixed finite alphabet: (9.1) DISCRETE RANDOM VARIABLE, with probabilities: (9.2) that must satisfy the condition: (9.3) Assuming that the symbols emitted by the source during successive signaling intervals are statistically independent. A source having such properties are called DISCRETE MEMORYLESS SOURCE, a memoryless in the sense that the symbol emitted at any time is independent of previous choices. Can we find a measure of how much information is produced by DISCRETE MEMORYLESS SOURCE? Note: idea of information is closely related to that of uncertainty or surprise EVENT K Consider the event S = s k [describing the emission of symbol s k by the source with a probability p k ] Before the event occurs: >there is an amount of uncertainty. During the event: >there is an amount of surprise. After the event: > there is a gain in the amount of information, which is the resolution of uncertainty. The amount of information is related to the inverse of the probability of occurrence. The amount of information gained after observing the event S = s k , which occurs with probability p k , is the logarithmic function (9.4) **base of logarithmic is arbitrary This definition exhibits the following important properties that are intuitively satisfying: 1. (9.5) If we are absolutely certain of the outcome of an event, even before it occurs, there is no information gained. 2. (9.6) The occurrence of an event S= s k either provides some or no information, but never brings about a loss of information. 3. (9.7) The less the probable an event is, the more information we gain when it occurs. 4. if s k and s l are statistically independent. Using Equation 9.4 in logarithmic base 2. The resulting unit of information is called the bit (a contraction of binary digit). (9.8) When k , we have k . Hence, one bit is the amount of information that we gain when one of two possible and equally likely events occurs. K The amount of information I(s k ) produced by the source during an arbitrary signaling interval depends on the symbol s k emitted by the source at the time. Indeed I (s k ) is a discrete random variable that takes on the values with probabilities , respectively. K The mean of I(s k ) over the source alphabet is given by (9.9) ENTROPY OF A DISCRETE MEMORYLESS SOURCE The important quantity H (S ) is called the entropy of a discrete memory less source with source alphabet. It is a measure of the average information content per source symbol. It depends only on the probabilities of the symbols in the alphabet S of the source. A discrete memory less source whose mathematical model is defined by equations 9.1 & 9.2. The entropy H(l) of such source is bounded as follows: (9.10) where K is the radix of the alphabet of the source. Furthermore, we may make two statements: 1. H(S )= 0, if and only if the probability p k = 1 for some k, and the remaining probabilities in the set are all zero; this lower bound on entropy corresponds to no uncertainty. 2. H(S )= log K, if and only if p k =1/K for all k; this upper bond on entropy corresponds to maximum uncertainty. ENTROPY OF BINARY MEMORY LESS SOURCE Consider a binary memory less source for which symbol 0 occurs with probability p 0 and symbol 1 with probability p 1 = 1 - p 0 , with entropy of: (9.15) SOLUTION For which we observe the following: 1. When p 0 = 0, the entropy H(S ) =0; this follows from the fact that x log x→0 as x→0. 2. When p 0 = 1, the entropy H (S ) = 0. 3. The entropy H (S ) attains its maximum value, H max =1 bit, when p 1 = p 0 =1/2, that is, symbol 1 and 0 are equally probable. SOLUTION The function p 0 is frequently encountered in information theoretic problems, and defined as: (9.16) This function is called as the entropy function. This is a function of prior probability p 0 defined on the interval [0,1].Plotting the entropy function H(p 0 ) versus p 0 defined on the interval [0,1] as in Figure 9.2. FIGURE 9.2 ENTROPY FUNCTION The curve highlights the observations made under points 1,2, and 3. EXTENSION OF DISCRETE MEMORYLESS SOURCE -Consider blocks rather than individual symbols -Each block consisting of n successive source symbols. (9.17) the probability of a source symbol S is equal to the product of the probabilities of the n source symbols in S constituting the particular symbol in S . ENTROPY OF EXTENDED SOURCE Consider a discrete source with source alphabet S = {s0, s1, s2} with respective probabilities: p 0 = 1/4 p 1 = 1/4 p 2 = 1/2. Find the entropy of the extended source. : SOLUTION The entropy of the source is: : SOLUTION Consider next the second order extension of the source. With the source alphabet S consisting of three symbols, it follows that the source has nine symbols. Table 9.1 present the nine symbols, its corresponding sequences, and its probabilities. Table 9.1 Alphabet particulars of second-order extension of a discrete memoryless source Symbols of S 2 0 1 2 3 4 5 6 7 8 Correspondin g sequences of symbols of S s 0 s 0 s 0 s 1 s 0 s 2 s 1 s 0 s 1 s 1 s 1 s 2 s 2 s 0 s 2 s 1 s 2 s 2 Probability p ( i ), i = 0, 1, ... , 8 1/16 1/16 1/8 1/16 1/16 1/8 1/8 1/8 1/4 : SOLUTION The entropy of the extended source is: : SOLUTION The entropy of the extended source is: Which proves: Presented by Roy Sencil and Janyl Jane Nicart END OF PRESENTATION