<story><title>Gaussian Distributions Are Soap Bubbles</title><url>http://www.inference.vc/high-dimensional-gaussian-distributions-are-soap-bubble/</url></story><parent_chain></parent_chain><comment><author>cs702</author><text>Of course. As others here point out, the hypervolume inside an n-dimensional hypersphere grows as the nth power of a linear increase in radius. In high dimensions, tiny increases in radius cause hypervolume to grow by more than 100%. The concentration of hypervolume is always highest at the edge.[0]&lt;p&gt;The theoretical tools (and intuitions) we have today for making sense of the distribution of data, developed over the past three centuries, &lt;i&gt;break down&lt;/i&gt; in high dimensions. The fact that in high dimensions Gaussian distributions are not &amp;quot;clouds&amp;quot; but actually &amp;quot;soap bubbles&amp;quot; is a perfect example of this breakdown. Can you imagine trying to model a cloud of high-dimensional points lying on or near a lower-dimensional manifold with soap bubbles?&lt;p&gt;If the data is not only high-dimensional but also non-linearly entangled, we don&amp;#x27;t yet have &amp;quot;mental tools&amp;quot; for reasoning about it:&lt;p&gt;* &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;medium.com&amp;#x2F;intuitionmachine&amp;#x2F;why-probability-theory-should-be-thrown-under-the-bus-36e5d69a34c9&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;medium.com&amp;#x2F;intuitionmachine&amp;#x2F;why-probability-theory-s...&lt;/a&gt;&lt;p&gt;* &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;news.ycombinator.com&amp;#x2F;item?id=15620794&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;news.ycombinator.com&amp;#x2F;item?id=15620794&lt;/a&gt;&lt;p&gt;[0] See kgwgk&amp;#x27;s comment below.</text></comment>
<story><title>Gaussian Distributions Are Soap Bubbles</title><url>http://www.inference.vc/high-dimensional-gaussian-distributions-are-soap-bubble/</url></story><parent_chain></parent_chain><comment><author>smallnamespace</author><text>Isn&amp;#x27;t the unsuitability of the high-dimensional Gaussian intimately related to the fact that for most realistic problem spaces, we actually believe there are really far fewer than the N &amp;gt;&amp;gt; 1 measured dimensions?&lt;p&gt;A uniform Gaussian presupposes that the variates are either linearly orthogonal, or all have the same linear interaction with each other (in the case of fixed positive correlation).&lt;p&gt;If your actual problem has dimension 20, but you&amp;#x27;ve measured it with N dimensions, then that means there are strong interactions between your measured variates, and moreover the intervariate interactions do not have a single &lt;i&gt;fixed&lt;/i&gt; interaction strength (like a single Gaussian correlation), but probably vary like a random matrix.&lt;p&gt;This might be related to the Tracy-Widom[1] distribution somehow. Perhaps the the distribution you use to replace the Gaussian should really be something like: first generate a random positive semi-definite matrix as C, then generate random data based on different random choices of C.&lt;p&gt;[1] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;en.wikipedia.org&amp;#x2F;wiki&amp;#x2F;Tracy%E2%80%93Widom_distribution&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;en.wikipedia.org&amp;#x2F;wiki&amp;#x2F;Tracy%E2%80%93Widom_distributi...&lt;/a&gt;</text></comment>