<story><title>Antifragility in complex dynamical systems</title><url>https://www.nature.com/articles/s44260-024-00014-y</url></story><parent_chain><item><author>bob1029</author><text>I&amp;#x27;ve found that intentionally causing abrupt (but reasonable) changes to hyperparameters in evolutionary spiking neural network simulations (I.e between each generation) results in far more robust simulations that will meet fitness criteria with less likelihood of getting stuck somewhere. The tradeoff being that simulating will take longer, but this may be worth things like reducing the chances of your resource requirements going asymptotic half way through.&lt;p&gt;My current method of perturbation is to cap the total # of activations per candidate globally. When at least 3 improvements in the best global fitness score have been achieved, I cut the global activation limit in &lt;i&gt;half&lt;/i&gt;. Obviously, this is fairly catastrophic for the current population. Each generation can increase the limit by 1% if no improvements made. This provides aggressive selection pressure for more efficient networks and forces a different kind of partial restart of the training process (I.e. do what you just did with 50% resources now). Very often this does result in optimization of the network. It also seems to make the networks rebound faster after restarts. At first, a restart can take 10-20 generations to recover. By generation 1000, this is closer to 1-2.</text></item></parent_chain><comment><author>seanhunter</author><text>&amp;gt; I&amp;#x27;ve found that intentionally causing abrupt (but reasonable) changes to hyperparameters in evolutionary spiking neural network simulations (I.e between each generation) results in far more robust simulations that will meet fitness criteria with less likelihood of getting stuck somewhere.&lt;p&gt;If I understand you correctly this is similar to the philosophy behind simulated annealing isn&amp;#x27;t it? The idea being that adding some peturbation can be enough to jog the evolution out of local minima and you probably want to reduce the amount of those jogs over time so it ends up converging.&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;en.wikipedia.org&amp;#x2F;wiki&amp;#x2F;Simulated_annealing&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;en.wikipedia.org&amp;#x2F;wiki&amp;#x2F;Simulated_annealing&lt;/a&gt;</text></comment>
<story><title>Antifragility in complex dynamical systems</title><url>https://www.nature.com/articles/s44260-024-00014-y</url></story><parent_chain><item><author>bob1029</author><text>I&amp;#x27;ve found that intentionally causing abrupt (but reasonable) changes to hyperparameters in evolutionary spiking neural network simulations (I.e between each generation) results in far more robust simulations that will meet fitness criteria with less likelihood of getting stuck somewhere. The tradeoff being that simulating will take longer, but this may be worth things like reducing the chances of your resource requirements going asymptotic half way through.&lt;p&gt;My current method of perturbation is to cap the total # of activations per candidate globally. When at least 3 improvements in the best global fitness score have been achieved, I cut the global activation limit in &lt;i&gt;half&lt;/i&gt;. Obviously, this is fairly catastrophic for the current population. Each generation can increase the limit by 1% if no improvements made. This provides aggressive selection pressure for more efficient networks and forces a different kind of partial restart of the training process (I.e. do what you just did with 50% resources now). Very often this does result in optimization of the network. It also seems to make the networks rebound faster after restarts. At first, a restart can take 10-20 generations to recover. By generation 1000, this is closer to 1-2.</text></item></parent_chain><comment><author>UniverseHacker</author><text>Very cool, but I&amp;#x27;m surprised you&amp;#x27;re sharing this on here and not in a job interview with a deep learning startup and&amp;#x2F;or an arXiv paper.</text></comment>