Deep learning has steadily gained popularity throughout the past several years due to a variety of complementary reasons. First and foremost, deep learning has shown incredible performance in almost all domains it has been applied to from speech processing [1]}, translation [2]}, image classification [3]}, and probabilistic modeling [4]}. This stellar performance has arose in part to significant advances in network optimization, primarily arising from advances in gradient based methods and computations. Networks often take the form of differentiable functions with latent parameters, optimized using empirical risk minimization. This gradient acts as rich source of counter-factual information, effectively allowing the optimizer to explore and understand the local neighborhood of a network's parameter space with only a single function and gradient computation. In recent deep learning frameworks such as the Cognitive Toolkit, Tensorflow, and Caffe, the calculation of the gradient has been abstracted away. This significantly simplifies code and simultaneously allows for automatic compilation of gradient computations to GPU architectures. It is now easier than ever to develop performant and novel network architectures using these automatic differentiation libraries.
