<story><title>Neural Networks, Types, and Functional Programming</title><url>http://colah.github.io/posts/2015-09-NN-Types-FP/</url></story><parent_chain></parent_chain><comment><author>ericjang</author><text>I really liked the mention of the &amp;quot;three narratives&amp;quot; of neural networks: 1) human brain 2) nonlinear &amp;quot;squashing and folding&amp;quot; of the input space 3) probabilistic &amp;#x2F; latent variables. Seeing different ways to look at something is always refreshing and opens up deep learning to rich analytical techniques developed in optimization, type theory, etc.&lt;p&gt;HN: how do YOU think about NNs? As a matter of pure preference, most of my daydreaming of NNs comes from my inductive biases of (1) and (2), with a few ideas from MCMC methods and optimization thrown in the mix.&lt;p&gt;&amp;gt; Representation theory&lt;p&gt;One question I&amp;#x27;ve always wondered is whether it is better to get a network to (1) learn representation transformations A-&amp;gt;B-&amp;gt;C-&amp;gt;D via a series of layers, or (2) try to learn A-&amp;gt;D via a single (highly nonlinear) transformation.&lt;p&gt;Obviously the former lends itself better to analytical understanding, and this seems to be what emerges when you train ImageNet. However, do researchers have any control over whether the learned net does (1) or (2)? I&amp;#x27;m thinking that scattering &amp;quot;skip arcs&amp;quot; that pass gradients through layers (or an LSTM) would cause representations to mix more between layers.&lt;p&gt;&amp;gt; Each layer is a function, acting on the output of a previous layer. As a whole, the network is a chain of composed functions. This chain of composed functions is optimized to perform a task.&lt;p&gt;This is certainly true for the majority of models in use today, but I wonder if this will remain true for future architectures. In practice, most layer types store some state, and implement &amp;quot;forward&amp;quot; and &amp;quot;backward&amp;quot; functions that may utilize that state.&lt;p&gt;One could argue that state is not necessary if we only consider the network to be defined by its feedforward passes (de-coupled from the optimization problem, which requires the &amp;quot;backward&amp;quot; functions). But the neuroscience narrative argues that function and learning cannot be decoupled.&lt;p&gt;In the case of a network that is supposed to update it&amp;#x27;s own weights during normal operation (i.e. topic discovery), the functional narrative is less clear to me. Thoughts?</text></comment>
<story><title>Neural Networks, Types, and Functional Programming</title><url>http://colah.github.io/posts/2015-09-NN-Types-FP/</url></story><parent_chain></parent_chain><comment><author>kailuowang</author><text>Interesting thoughts there.&lt;p&gt;I am writing a deep neural network lib in scala as part of my DQN implementation. It&amp;#x27;s developed in the FP paradigm with some type safety. If anyone is interested, the code is here. 
&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;A-Noctua&amp;#x2F;glaux&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;A-Noctua&amp;#x2F;glaux&lt;/a&gt;
Right now only feedforward and convolution with Relu are implemented, no documentation either. But would love to collaborate with anyone also interested in the area.</text></comment>