<story><title>The Neural Network Zoo</title><url>http://www.asimovinstitute.org/neural-network-zoo/#</url></story><parent_chain><item><author>Houshalter</author><text>Look at computer science. They have invented so many different algorithms that work! This is a sign that computer science lacks a decent unifying theoretical framework.&lt;p&gt;Having different algorithms for different purposes is fine. For instance, autoencoders can do unsupervised learning, GANs can learn generative models that sample from the entire distribution, recurrent neural networks can handle time series, etc. Also while there are many different types of networks, research has shown which ones work and which ones don&amp;#x27;t. Few people pretrain autoencoders or bother with RBMs anymore, for instance. And I think we have good theoretical reasons why they aren&amp;#x27;t as good.&lt;p&gt;But to continue the analogy to computer science, imagine all the different kinds of sorting algorithms. They will each work better or worse based on how the data you are sorting is arranged. If it&amp;#x27;s already sorted in reverse, that&amp;#x27;s a lot different than if it&amp;#x27;s sorted completely randomly, or if it was sorted and then big chunks were randomly rearranged.&lt;p&gt;There&amp;#x27;s no way to prove that one sorting algorithm will always do better than another, because there&amp;#x27;s always special cases where they do do better. The same is true of neural networks, it&amp;#x27;s impossible to formally prove they will work, because it depends on how real world problems are distributed.</text></item><item><author>cs702</author><text>Good job.&lt;p&gt;Alas, the proliferation of different kinds of neural net architectures &lt;i&gt;that work&lt;/i&gt;, over the past few years, is a sign that we lack a decent unifying theoretical framework that can explain, from fundamental principles, what works, what doesn&amp;#x27;t, and why.  We&amp;#x27;re not there yet.</text></item></parent_chain><comment><author>danharaj</author><text>Type theory is a unifying theory of program semantics. It is one among many, each of which is a facet of one whole. In particular, the theory of algebraic and coalgebraic data types is a unifying theory for computation that allows one to &lt;i&gt;derive&lt;/i&gt; efficient algorithms automatically. It does not encompass all algorithms design by a long shot, it is actually quite niche, but as a coherent mathematical theory that gets to the essence of computation, it is very promising.&lt;p&gt;I highly recommend this book to anyone who has access to a library and is interested in seeing algorithms from the thousand-mile-high point of view: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.amazon.com&amp;#x2F;Algebra-Programming-Prentice-Hall-International-Computer&amp;#x2F;dp&amp;#x2F;013507245X&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.amazon.com&amp;#x2F;Algebra-Programming-Prentice-Hall-Int...&lt;/a&gt;&lt;p&gt;I suppose the point of my comment is that theoretical computer science is actually a field with a lot of unifying theories that approach computation in coherent ways. Applied computer science is much, much messier because it is interested in the particularities and flaws of real world computational models and getting practical results now, leaving explanations to come later.&lt;p&gt;There are unifying theories of inference for AI, but they don&amp;#x27;t really cover deep neural networks. There are a few tantalizing hints that deep learning is intimately related to profound concepts in physics (renormalization) and functional programming.</text></comment>
<story><title>The Neural Network Zoo</title><url>http://www.asimovinstitute.org/neural-network-zoo/#</url></story><parent_chain><item><author>Houshalter</author><text>Look at computer science. They have invented so many different algorithms that work! This is a sign that computer science lacks a decent unifying theoretical framework.&lt;p&gt;Having different algorithms for different purposes is fine. For instance, autoencoders can do unsupervised learning, GANs can learn generative models that sample from the entire distribution, recurrent neural networks can handle time series, etc. Also while there are many different types of networks, research has shown which ones work and which ones don&amp;#x27;t. Few people pretrain autoencoders or bother with RBMs anymore, for instance. And I think we have good theoretical reasons why they aren&amp;#x27;t as good.&lt;p&gt;But to continue the analogy to computer science, imagine all the different kinds of sorting algorithms. They will each work better or worse based on how the data you are sorting is arranged. If it&amp;#x27;s already sorted in reverse, that&amp;#x27;s a lot different than if it&amp;#x27;s sorted completely randomly, or if it was sorted and then big chunks were randomly rearranged.&lt;p&gt;There&amp;#x27;s no way to prove that one sorting algorithm will always do better than another, because there&amp;#x27;s always special cases where they do do better. The same is true of neural networks, it&amp;#x27;s impossible to formally prove they will work, because it depends on how real world problems are distributed.</text></item><item><author>cs702</author><text>Good job.&lt;p&gt;Alas, the proliferation of different kinds of neural net architectures &lt;i&gt;that work&lt;/i&gt;, over the past few years, is a sign that we lack a decent unifying theoretical framework that can explain, from fundamental principles, what works, what doesn&amp;#x27;t, and why.  We&amp;#x27;re not there yet.</text></item></parent_chain><comment><author>LeanderK</author><text>I don&amp;#x27;t think that this is a sign that computer science lacks a decent unifying theoretical framework.&lt;p&gt;We can even proof that there is no optimal sorting algorithm. &amp;quot;For every sorting algorithm x exists an input i for which an sorting algorithm y exists, so that y sorts i faster than x&amp;quot; is provable</text></comment>