<story><title>Population-based training of neural networks</title><url>https://deepmind.com/blog/population-based-training-neural-networks/</url></story><parent_chain></parent_chain><comment><author>knexer</author><text>Two things stick out to me after a first read:&lt;p&gt;First, this actually learns a schedule for each hyperparameter, not just a good set of fixed values, automatically discovering learning rate annealing and related techniques.  This seems incredibly powerful.  It is also learning hyperparameter schedules specific to a single training run - which seems interesting but not obviously helpful, especially since many of the learned schedules fairly closely match the baseline hand-tuned ones.&lt;p&gt;Second, it seems like they&amp;#x27;re optimizing against their validation metric directly; isn&amp;#x27;t that basically &amp;#x27;cheating&amp;#x27; (i.e. defeats much of the point of having a separate validation metric in the first place)?  It also seems completely orthogonal to their technique - could they not have optimized for the same loss function as the network itself?  Is this an improvement over state of the art, or is it just overfitting to the validation metric?</text></comment>
<story><title>Population-based training of neural networks</title><url>https://deepmind.com/blog/population-based-training-neural-networks/</url></story><parent_chain></parent_chain><comment><author>0x63_Problems</author><text>This is really cool!  I haven&amp;#x27;t read through the real paper yet but it&amp;#x27;s very impressive that this method does not incur a significant performance cost.  I had assumed that using a genetic-style algorithm would be costly since you would need to train a large number of networks individually, but treating all variations equally in terms of training time now seems naive.  Distributing the training time using intelligent exploration and exploitation is an awesome idea to fix this.</text></comment>