<story><title>Kullback–Leibler divergence</title><url>https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence</url></story><parent_chain></parent_chain><comment><author>jwarden</author><text>Here&amp;#x27;s how I describe KL Divergence, building up from simple to complex concepts.&lt;p&gt;surprisal: how surprised I am when I learn the value of X&lt;p&gt;&lt;pre&gt;&lt;code&gt;  Suprisal(x) = -log p(X=x)
&lt;/code&gt;&lt;/pre&gt;
entropy: how surprised I expect to be&lt;p&gt;&lt;pre&gt;&lt;code&gt;  H(p)        = 𝔼_X -log p(X) 
              = ∑_x p(X=x) * -log p(X=x)
&lt;/code&gt;&lt;/pre&gt;
cross-entropy: how surprised I expect Bob to be (if Bob&amp;#x27;s beliefs are q instead of p)&lt;p&gt;&lt;pre&gt;&lt;code&gt;  H(p,q)      = 𝔼_X -log q(X) 
              = ∑_x p(X=x) * -log q(X=x)
&lt;/code&gt;&lt;/pre&gt;
KL divergence: how much *more* surprised I expect Bob to be than me&lt;p&gt;&lt;pre&gt;&lt;code&gt;  Dkl(p || q) = H(p,q) - H(p,p)
              = ∑_x p(X=x) * log p(X=x)&amp;#x2F;q(X=x)
&lt;/code&gt;&lt;/pre&gt;
information gain: how much less surprised I expect Bob to be if he knew that Y=y&lt;p&gt;&lt;pre&gt;&lt;code&gt;  IG(q|Y=y)   = Dkl(q(X|Y=y) || q(X))
&lt;/code&gt;&lt;/pre&gt;
mutual information: how much information I expect to gain about X from learning the value of Y&lt;p&gt;&lt;pre&gt;&lt;code&gt;  I(X;Y)      = 𝔼_Y IG(q|Y=y)
                𝔼_Y Dkl(q(X|Y=y) || q(X))&lt;/code&gt;&lt;/pre&gt;</text></comment>
<story><title>Kullback–Leibler divergence</title><url>https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence</url></story><parent_chain></parent_chain><comment><author>golwengaud</author><text>I found &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.lesswrong.com&amp;#x2F;posts&amp;#x2F;no5jDTut5Byjqb4j5&amp;#x2F;six-and-a-half-intuitions-for-kl-divergence&quot; rel=&quot;nofollow noreferrer&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.lesswrong.com&amp;#x2F;posts&amp;#x2F;no5jDTut5Byjqb4j5&amp;#x2F;six-and-a-...&lt;/a&gt; very helpful for getting intuition for what the K-L divergence is and why it&amp;#x27;s useful. The six intuitions:&lt;p&gt;&lt;pre&gt;&lt;code&gt;    1. Expected surprise
    2. Hypothesis testing
    3. MLEs
    4. Suboptimal coding
    5a. Gambling games -- beating the house
    5b. Gambling games -- gaming the lottery
    6. Bregman divergence&lt;/code&gt;&lt;/pre&gt;</text></comment>