Image-based machine learning can be generally described in the framework of artificial neural networks, where technological advancements typically involve reducing the complexity of neural network training the large parameter space and increasing the ability to generalise from smaller training data sets. Artificial neural networks were developed to model and simulate biological neural network processing, i.e. the perceptron model of Rosenblatt where the output of a neuron is a weighted linear combination of the inputs [1]}. 'Deep' networks emerged shortly thereafter with the multi-layer perceptron (MLP) [2]}, where perceptrons were organized into multiple, feed-forward layers evaluated in a sequential fashion. A persistent theme has been the incorporation of mathematical invariance, allowing networks to train from fewer data points, or equivalently to generalize to unseen data. LeCun developed the translation-invariant convolutional neural network (CNN) architecture in the 1980s [3]}, which greatly reduced the memory requirements of the weights from \(O(N^2)\)  in the number \(N\)  of pixels for fully-connected MLP network layers to \(O(N)\)  for sets of weights computed locally and shared across the image in a manner invariant to image translation, i.e. convolution filters. While highly effective, CNN training via the backpropagation algorithm remained too computationally complex for practical applications from sequential CPU-based computers until the development of highly parallel GPU-based convolution allowed training of deep CNNs from 100000s of images [4]}, e.g. the ImageNet dataset consisting of 1000 objects with 1000 training examples per category).
