<story><title>Some thoughts on machine learning with small data</title><url>https://niklasriewald.com/2022/06/02/some-thoughts-on-machine-learning-with-small-data/</url></story><parent_chain></parent_chain><comment><author>ChrisRackauckas</author><text>There&amp;#x27;s also data in another source: domain knowledge and physical laws. This is the core tenant of scientific machine learning techniques (SciML). I have a talk that walks through how you can go from something without domain knowledge (like a neural ODE) and how as you add more prior model knowledge to the system the extrapolation accuracy improves even for small amounts of data (&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.youtube.com&amp;#x2F;watch?v=FihLyzdjN_8&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.youtube.com&amp;#x2F;watch?v=FihLyzdjN_8&lt;/a&gt;). People have used these techniques in all kinds of places, like extrapolating black hole trajectories and building earthquake-safe buildings, off of like 20-40 data points (&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.youtube.com&amp;#x2F;watch?v=eSeY4K4bITI&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.youtube.com&amp;#x2F;watch?v=eSeY4K4bITI&lt;/a&gt;). While there is a lot of work to be done in the domain still, there&amp;#x27;s a lot of empirical evidence that this approach does indeed decrease the data requirements (but is then no longer &amp;quot;pure&amp;quot; machine learning in some sense)</text></comment>
<story><title>Some thoughts on machine learning with small data</title><url>https://niklasriewald.com/2022/06/02/some-thoughts-on-machine-learning-with-small-data/</url></story><parent_chain></parent_chain><comment><author>evrydayhustling</author><text>A powerful concept for thinking about small data is bias&amp;#x2F;variance tradeoff.  The ELI5 is: data is how a learner selects confidently between one model and another.  So if you have little data, you must choose between high bias (fewer models to choose from) or low confidence (can&amp;#x27;t be sure you picked the best one).&lt;p&gt;Bias in a lot of contexts is Bad, and large data techniques like DNN are basically about how you can have biases that are so vague the learner has plenty of room to surprise you.  But when you add bias that truly aligns with the structure your learner is going to be exposed to, you enable it to reach confidence sooner.  Many important techniques for small data are about allowing you to express specific biases.  For example:&lt;p&gt;- PGMs and Probabilistic Programming are about giving you a specific, interpretable structure for how your data are related to each other.&lt;p&gt;- Picking informed bayesian priors&lt;p&gt;- Data augmentation lets you add bias by expressing what kinds of variation &lt;i&gt;don&amp;#x27;t&lt;/i&gt; matter (I.e. permitting and fuzzing your dataset)&lt;p&gt;- feature engineering is about selectively adding and removing biases in terms of what kinds of data transformations are informative&lt;p&gt;With all that said, the large data ML community has produced an incredible tool for practical work with small data: transfer learning.  If your dataset is related to a larger one (e.g. by including natural language), you can borrow informed biases from models that were trained on a much larger corpus.</text></comment>