<story><title>MuGo: A minimalist Go engine modeled after AlphaGo</title><url>https://github.com/brilee/MuGo</url></story><parent_chain><item><author>cr0sh</author><text>Something I&amp;#x27;m finding interesting is how so many of these deep-learning neural nets use RELU for their activation function; RELU is known as a &amp;quot;lazy engineer&amp;#x27;s activation&amp;quot; function - very simple to implement, and despite looking like a hack, seems to work very well for many tasks.&lt;p&gt;I tend to wonder - beyond &amp;quot;ease of implementation and good &amp;#x27;nuff&amp;quot; reasons - if there are other reasons to use RELU, over other activation functions like TANH or Sigmoid?&lt;p&gt;I&amp;#x27;m beginning to suspect that we may be seeing the &amp;quot;engineering side&amp;quot; of neural networks coming into play; that instead of using the more &amp;quot;biologically accurate&amp;quot; activation of the sigmoid function, we instead use RELU (and other ELU derivatives) because it works well, and is easier to understand?&lt;p&gt;Much like how things progressed better in heavier-than-air flight once engineers realized that flapping wings weren&amp;#x27;t absolutely needed, and low-weight engines turning propellers, with fixed wings, worked better for flying than what nature uses...?</text></item></parent_chain><comment><author>jimfleming</author><text>The reasons to use ReLUs are sparsity and improved gradient flow. ReLUs encourage sparsity because when the input to the ReLU is less than 0 as the activation becomes 0. This means some fraction of activations in a given layer will be omitted which can encourage better representations. They also have improved gradient flow because the gradients are zero or constant and thus don&amp;#x27;t suffer from vanishing&amp;#x2F;exploding gradients.&lt;p&gt;In deep learning, I would _generally_ not look towards biology for the reasons behind why things are done as this is usually an after-the-fact explanation. When in doubt, blame the gradients.</text></comment>
<story><title>MuGo: A minimalist Go engine modeled after AlphaGo</title><url>https://github.com/brilee/MuGo</url></story><parent_chain><item><author>cr0sh</author><text>Something I&amp;#x27;m finding interesting is how so many of these deep-learning neural nets use RELU for their activation function; RELU is known as a &amp;quot;lazy engineer&amp;#x27;s activation&amp;quot; function - very simple to implement, and despite looking like a hack, seems to work very well for many tasks.&lt;p&gt;I tend to wonder - beyond &amp;quot;ease of implementation and good &amp;#x27;nuff&amp;quot; reasons - if there are other reasons to use RELU, over other activation functions like TANH or Sigmoid?&lt;p&gt;I&amp;#x27;m beginning to suspect that we may be seeing the &amp;quot;engineering side&amp;quot; of neural networks coming into play; that instead of using the more &amp;quot;biologically accurate&amp;quot; activation of the sigmoid function, we instead use RELU (and other ELU derivatives) because it works well, and is easier to understand?&lt;p&gt;Much like how things progressed better in heavier-than-air flight once engineers realized that flapping wings weren&amp;#x27;t absolutely needed, and low-weight engines turning propellers, with fixed wings, worked better for flying than what nature uses...?</text></item></parent_chain><comment><author>xiphias</author><text>What&amp;#x27;s interesting is that Jeff Dean noted that ReLU was one of the big things why deep learning took off. Wikipedia summarizes tha advantages, but my guess is getting rid of the vanishing gradients problem:&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;en.m.wikipedia.org&amp;#x2F;wiki&amp;#x2F;Rectifier_%28neural_networks%29?wprov=sfla1&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;en.m.wikipedia.org&amp;#x2F;wiki&amp;#x2F;Rectifier_%28neural_networks...&lt;/a&gt;</text></comment>