BACKGROUND

This specification relates to training neural networks.

Neural networks are machine learning models that employ one or more layers of transformations to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current values of a respective set of parameters.

Some neural networks are convolutional neural networks. A convolutional neural network is a neural network that includes one or more convolutional layers. Generally, an output generated by a convolutional layer can be computed as a linear transformation of an input to the layer.

SUMMARY

This specification describes a system implemented as computer programs on one or more computers in one or more locations that trains a convolutional neural network to generate one or more outputs based on the received one or more inputs.

The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages.

In many convolutional neural network training scenarios, at each iteration of the training process, current values of parameters of the network are updated using a determined gradient, i.e., the gradient of an objective function that measures the quality of the training outputs. Typically, the size of the determined gradient directly affects the quality of the training of the network. In some cases where the convolutional neural network is a deep convolutional neural network, i.e., having a plurality of convolutional layers, in order to complete a training iteration, the determined gradient has to be propagated back from a last convolutional layer all the way to a first convolutional layer in the convolutional neural network.

Because of this, in circumstances where the size of the determined gradient is large, then it may become even larger during the back-propagation process, causing numerical overflow or irregular oscillations in the objective function that is used to calculate the gradient. This scenario is referred to as an exploding gradient. Exploding gradients inhibit the convolutional neural network from further training. That is, existing techniques for training the network in such circumstances consume a large amount of computational resources and wall clock time due to exploding gradients inhibiting successful learning.

The described techniques, on the other hand, augment the training process by bounding the size of the gradient during the back-propagation process. In so doing, the described system is capable of maintaining an effective training process of the convolutional neural network by mitigating any impact caused by exploding gradients. In other words, the system can effectively train the network to generate high quality outputs for a particular task in a much more computationally efficient manner than other techniques.

The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

