<story><title>Machine learning for the impatient: algorithms tuning algorithms</title><url>http://www.aelag.com/147952673</url></story><parent_chain><item><author>brador</author><text>Question:  Why is the data for machine learning split into a training dataset and a test dataset?  Wouldn&apos;t using the entire dataset to build the model result in greater accuracy of the predictions?</text></item><item><author>scottfr</author><text>Such aggressive usage of the test data set in determining the tuning parameters in effect makes your test data set part of your training data set.&lt;p&gt;The more times you go back to your test data set to evaluate the effectiveness of a model, the more optimistic your error predictions will be and the greater your chance of overfitting. Several iterations of his loop will probably improve the model, but if you keep repeating it eventually the true model performance will start to degrade.</text></item></parent_chain><comment><author>reginaldo</author><text>When you develop a model, for instance, when implementing a classifier, you supposedly want to apply the developed model to &lt;i&gt;other&lt;/i&gt; data, i.e., data you don&apos;t have available during development.&lt;p&gt;In many situations, it doesn&apos;t make sense to test your model &lt;i&gt;only&lt;/i&gt; when it&apos;s put to make or influence decisions in the real world (although you have to test in the real world too). You&apos;ll want to test the predictions of your model on data you already have the &lt;i&gt;actual&lt;/i&gt; results for. To test your model you&apos;ll split your data into data you know &lt;i&gt;and will let the model know about&lt;/i&gt; (training dataset), and data you know &lt;i&gt;but the model can&apos;t know about&lt;/i&gt; (test dataset). That way you can use the data the model doesn&apos;t know about to make controlled experiments and compare models (and, if your data is really representative of the real world, your mofrl comparison and the performance of your chosen model will hold).&lt;p&gt;The moral of the story is: if you don&apos;t split your data, you won&apos;t have any idea of how it performs in the real world, you&apos;ll only know how it performs with data it already knew about.</text></comment>
<story><title>Machine learning for the impatient: algorithms tuning algorithms</title><url>http://www.aelag.com/147952673</url></story><parent_chain><item><author>brador</author><text>Question:  Why is the data for machine learning split into a training dataset and a test dataset?  Wouldn&apos;t using the entire dataset to build the model result in greater accuracy of the predictions?</text></item><item><author>scottfr</author><text>Such aggressive usage of the test data set in determining the tuning parameters in effect makes your test data set part of your training data set.&lt;p&gt;The more times you go back to your test data set to evaluate the effectiveness of a model, the more optimistic your error predictions will be and the greater your chance of overfitting. Several iterations of his loop will probably improve the model, but if you keep repeating it eventually the true model performance will start to degrade.</text></item></parent_chain><comment><author>numlocked</author><text>To prevent overfitting. A common technique is to use cross-validation, where you train some percentage of the data (90%) and test on the remaining (10%), and cycle through which 10% you use as test (each train/test split is called a &quot;fold&quot;). Once you&apos;ve identified a model that resists overfitting, you can train on all of the data.&lt;p&gt;edit: As noted in the other reply, having a truly blind validation set is still ideal.</text></comment>