Many of the mechanisms found in modern
Convolutional Neural Networks have their origins in
the Neocognitron [1]}.
This model consisted of multiple layers of
units inspired by the simple and complex cells
described by [2]}.
Local features were learned by S-cells
which, crucially, share their parameters
across the input array.
This creates a field of replicated feature detectors,
which are then aggregated locally by C-cells,
creating wider detectors with limited shift invariance.
In this way, the receptive fields of these units grow
as processing proceeds through the layers of the network,
building more complex features and expanding the
tolerance to spatial translation.
