<story><title>Population-based training of neural networks</title><url>https://deepmind.com/blog/population-based-training-neural-networks/</url></story><parent_chain></parent_chain><comment><author>knexer</author><text>Two things stick out to me after a first read:&lt;p&gt;First, this actually learns a schedule for each hyperparameter, not just a good set of fixed values, automatically discovering learning rate annealing and related techniques.  This seems incredibly powerful.  It is also learning hyperparameter schedules specific to a single training run - which seems interesting but not obviously helpful, especially since many of the learned schedules fairly closely match the baseline hand-tuned ones.&lt;p&gt;Second, it seems like they&amp;#x27;re optimizing against their validation metric directly; isn&amp;#x27;t that basically &amp;#x27;cheating&amp;#x27; (i.e. defeats much of the point of having a separate validation metric in the first place)?  It also seems completely orthogonal to their technique - could they not have optimized for the same loss function as the network itself?  Is this an improvement over state of the art, or is it just overfitting to the validation metric?</text></comment>
<story><title>Population-based training of neural networks</title><url>https://deepmind.com/blog/population-based-training-neural-networks/</url></story><parent_chain></parent_chain><comment><author>margorczynski</author><text>I think this is what Google and the others aim for - no hand-tuning. You simply specify the problem (some function to optimize) and the data. Everything in a nice concise package running on Google Cloud using their custom software and hardware.</text></comment>