Log sum inequality

__NOTOC__
The log sum inequality is an inequality, which is useful for proving several theorems in information theory.

Statement
Let  and  be nonnegative numbers.  Denote the sum of all s by  and the sum of all s by .  The log sum inequality states that

with equality if and only if  are equal for all , in other words  for all .

Proof

Notice that after setting  we have

where the inequality follows from Jensen's inequality since , , and  is convex.

Applications

The log sum inequality can be used to prove several inequalities in information theory such as Gibbs' inequality or the convexity of Kullback-Leibler divergence.

For example, to prove Gibbs' inequality it is enough to substitute s for s, and s for s to get

Generalizations

The inequality remains valid for  provided that  and .
The proof above holds for any function  such that  is convex, such as all continuous non-decreasing functions. Generalizations to convex functions other than the logarithm is given in Csiszár, 2004.

References
 T.S. Han, K. Kobayashi, Mathematics of information and coding. American Mathematical Society, 2001. .
 Information Theory course materials, Utah State University . Retrieved on 2009-06-14.
 

Category:Inequalities
Category:Information theory
Category:Articles containing proofs