<story><title>Do simpler machine learning models exist and how can we find them?</title><url>https://statmodeling.stat.columbia.edu/2022/12/22/do-simpler-machine-learning-models-exist-and-how-can-we-find-them/</url></story><parent_chain></parent_chain><comment><author>tbalsam</author><text>I recently released a codebase in beta that modernizes a tiny model that gets really good performance on CIFAR-10 in about 18.1 or so seconds on the right single GPU -- a number of years ago the world record was 10 minutes, down from several days a few years previously.&lt;p&gt;While most of my work was porting and cleaning up certain parts of the code for a different purpose (just-clone-and-hack experimentation workbench), I&amp;#x27;ve spent years optimizing neural networks at a very fine grained level, and many of the lessons learned here in debugging reflected that.&lt;p&gt;I believe that there are fundamentally a few big NP-hard layers (at least two that I can define, and likely several other smaller ones) unfortunately but they are not hard blockers to progress. The model I mentioned above is extremely simple and has little &amp;quot;extra fat&amp;quot; where it is not needed. It also importantly seems to have good gradient and such flow throughout, something that&amp;#x27;s important for a model to be able to learn quickly. There are a few reasonable priors, like initializing and freezing the first convolution to whiten the inputs based upon some statistics from the training data. That does a shocking amount of work in stabilizing and speeding up training.&lt;p&gt;Ultimately, the network is simple, and there are a number of other methods to help it reach near-SOTA, but they are as simple as can be. I think as this project evolves and we get nearer to the goal (&amp;lt;2 seconds in a year or two), we&amp;#x27;ll keep uncovering good puzzle pieces showing exactly what it is that&amp;#x27;s allowing such a tiny network to perform so well. There&amp;#x27;s a kind of exponential value to having ultra-short training times -- you can somewhat open-endedly barrage-test your algorithm, something that&amp;#x27;s already led to a few interesting discoveries that I&amp;#x27;d like to refine before publishing to the repo.&lt;p&gt;If you&amp;#x27;re interested, the code is here. The running code is a single .py with the upsides and downsides that come with that. If you&amp;#x27;re interested or have any questions, let me know! :D :))))&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;tysam-code&amp;#x2F;hlb-CIFAR10&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;tysam-code&amp;#x2F;hlb-CIFAR10&lt;/a&gt;</text></comment>
<story><title>Do simpler machine learning models exist and how can we find them?</title><url>https://statmodeling.stat.columbia.edu/2022/12/22/do-simpler-machine-learning-models-exist-and-how-can-we-find-them/</url></story><parent_chain></parent_chain><comment><author>donkeyboy</author><text>Yes, they exist, and they are called Linear Regression and Decision Tree. Not everything needs to be a neural network.&lt;p&gt;Anyway, residual connections in NNs as well as distillation being only a 1% hit to performance imply our models are way too big.</text></comment>