Deep neural networks are computational systems composed of neuron-like units connected through synapse-like contacts (Figure REF b). Each unit transmits a scalar value, analogous to a spike rate, which is computed based on the sum of its inputs, that is, the activities of ‘upstream’ units multiplied by the strength of the transmitting synapse or connection [1]}. Critically, unit activity is a non-linear function of these inputs, allowing networks with layers of units interposed between the ‘input’ and ‘output’ sides of the system – i.e., ‘deep’ neural networks – to approximate any function mapping activation inputs to activation outputs [2]}. Furthermore, when the connectivity pattern includes loops, as in ‘recurrent’ neural networks, the network's activations can preserve information about past events, allowing the network to compute functions based on sequences of inputs.
<FIGURE>