Information Theory

The entropy function H( X) is a measure of the uncertainty of X, in formula

$$ H(X)=-\sum \limits_{a:{p}_X(a)>0}{p}_X(a)\cdot {\log}_2\, {p}_X(a), $$

where p _X( a) =  Pr [ x =  A] denotes the probability that random variable X takes on value a. The interpretation is that with probability p _X ( a), X can be described by log ₂   p _X ( a) bits of information.

The conditional entropy or equivocation (Shannon 1948) H( X ∕ Y) denotes the uncertainty of X provided Y is known:

$$\begin{aligned} &H\left(\left.X\right|Y\right)\\ &\ =-\sum \limits_{a,b:{p}_{\left.X\right|Y}\left(\left.a\right|b\right)>0}\!{p}_{X,Y}\left(a,b\right){\cdot} \log_2\, {p}_{\left.X\right|Y}\left(\left.a\right|b\right)\end{aligned} $$

where p _{X, Y} ( a,  b)= _def Pr [( X =  a) ∧ ( Y =  b)] and p _{X| Y} ( a|  b) obeys Bayes’ rule for conditional probabilities:

$$ {\displaystyle \begin{aligned}{p}_{X,Y}\left(a,b\right)&={p}_Y(b)\cdot {p}_{X/Y}\left(a|b\right),\textrm{thus}\\ &\quad-{\log}_2\, {p}_{X,Y}\left(a,b\right)\\...