BACKGROUND

This specification relates to processing data using neural networks.

Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current values of a respective set of parameters.

SUMMARY

This specification describes a capsule neural network implemented as computer programs on one or more computers in one or more locations.

According to a first aspect, there is provided a neural network that is configured to receive a network input and to generate a network output for the network input, wherein: the neural network comprises a plurality of layers arranged in a sequence, including a plurality of capsule layers; each capsule layer comprises a respective plurality of capsules; and each particular capsule in a particular capsule layer is configured to: receive respective inputs including: (i) outputs generated by capsules of a previous capsule layer that is before the particular capsule layer in the sequence, and (ii) final routing factors between capsules of the previous capsule layer and the particular capsule, wherein the final routing factors are generated by a routing subsystem; and determine a particular capsule output based on the received inputs, wherein the particular capsule output is of dimension greater than one; and a routing subsystem that is configured to perform operations including: receiving outputs generated by capsules in a first capsule layer; determining final routing factors between capsules of the first capsule layer and capsules of a second capsule layer that is after the first capsule layer in the sequence; and providing the final routing factors to the capsules in the second capsule layer.

In some implementations, determining final routing factors between capsules of the first capsule layer and capsules of a second capsule layer that is after the first capsule layer in the sequence comprises: performing one or more iterations of an expectation-maximization routing algorithm, wherein each iteration of the expectation-maximization routing algorithm includes an M-step and an E-step, wherein for a particular iteration: the M-step includes, for each second capsule in the second capsule layer: determining weighted votes for the second capsule from one or more first capsules of the first capsule layer for the particular iteration, comprising: determining a vote for the second capsule from a particular first capsule of the first capsule layer by multiplying an output of the particular first capsule by a weight matrix of the second capsule that is associated with the particular first capsule; weighting the vote for the second capsule from the particular first capsule of the first capsule layer by a routing factor between the particular first capsule and the second capsule for the particular iteration; determining values of one or more parameters characterizing a distribution of weighted votes for the second capsule from the one or more first capsules of the first capsule layer for the particular iteration; determining a cost associated with the second capsule for the particular iteration based at least in part on the values of the parameters characterizing the distribution of weighted votes for the second capsule for the particular iteration; determining an activation of the second capsule for the particular iteration based at least in part on the cost associated with the second capsule for the particular iteration; and the E-step includes determining routing factors for a next iteration based on at least the activations of the capsules of the second capsule layer for the particular iteration and the values of the parameters characterizing the distributions of weighted votes associated with the capsules of the second capsule layer for the particular iteration; determining final routing factors to be the routing factors determined in a last iteration of the expectation-maximization routing algorithm.

In some implementations, weighting the vote for the second capsule from the particular first capsule of the first capsule layer further comprises: weighting the vote for the second capsule from the particular first capsule by an activation of the particular first capsule.

In some implementations, the cost associated with the second capsule for the particular iteration is further determined based at least in part on the routing factors between the first capsules of the first capsule layer and the second capsule for the particular iteration.

In some implementations, the cost associated with the second capsule for the particular iteration is further determined based at least in part on an inverse temperature parameter.

In some implementations, the parameters characterizing the distribution of weighted votes for the second capsule for the particular iteration include: a mean of the weighted votes for the second capsule from the one or more first capsules of the first capsule layer for the particular iteration; and a variance of the weighted votes for the second capsule from the one or more first capsules of the first capsule layer for the particular iteration.

In some implementations, determining final routing factors between capsules of the first capsule layer and capsules of a second capsule layer that is after the first capsule layer in the sequence comprises: performing one or more iterations of a dynamic routing algorithm, wherein a particular iteration of the dynamic routing algorithm includes: for each second capsule of the second capsule layer: determining weighted votes for the second capsule from one or more first capsules of the first capsule layer for the particular iteration, comprising: determining a vote for the second capsule from a particular first capsule of the first capsule layer by multiplying the output of the particular first capsule by a weight matrix of the second capsule that is associated with the particular first capsule; weighting the vote for the second capsule from the particular first capsule of the first capsule layer by a normalized routing factor between the particular first capsule and the second capsule for the particular iteration; determining routing factors for a next iteration, wherein determining a routing factor between a first capsule of the first capsule layer and a second capsule of the second capsule layer for the next iteration comprises: determining a sum of the weighted votes for the second capsule from the one or more first capsules of the first capsule layer for the particular iteration; determining a similarity measure between the sum of the weighted votes for the second capsule for the particular iteration and the vote for the second capsule from the first capsule; determining the routing factor between the first capsule and the second capsule for the next iteration by combining the determined similarity measure with the routing factor between the first capsule and the second capsule for the particular iteration; determining final routing factors to be the routing factors determined in a last iteration of the dynamic routing algorithm.

In some implementations, determining the sum of the weighted votes for the second capsule further comprises normalizing the sum of the weighted votes for the second capsule.

In some implementations, the similarity measure between the sum of the weighted votes for the second capsule and the vote for the second capsule from the first capsule comprises: the inner product of the sum of the weighted votes for the second capsule and the vote for the second capsule from the first capsule.

In some implementations, determining a particular capsule output based on the received inputs comprises: determining weighted votes for the particular capsule from the capsules of the previous capsule layer, comprising: determining a vote for the particular capsule from a given capsule in the previous capsule layer by multiplying the output of the given capsule by a weight matrix of the particular capsule that is associated with the given capsule; weighting the vote for the particular capsule from the given capsule in the previous capsule layer by a final routing factor between the given capsule and the particular capsule; determining the particular capsule output by combining the weighted votes for the particular capsule from the capsules of the previous capsule layer.

In some implementations, the final routing factors are normalized.

In some implementations, weighting the vote for the particular capsule from the given capsule in the previous capsule layer further comprises: weighting the vote for the particular capsule from the given capsule by an activation of the given first capsule.

In some implementations, combining the weighted votes for the particular capsule from the capsules of the previous capsule layer comprises: determining a mean of the weighted votes for the particular capsule from the capsules of the previous capsule layer.

In some implementations, determining a particular capsule output based on the received inputs further comprises: determining values of one or more parameters characterizing a distribution of the weighted votes for the particular capsule from the capsules of the previous capsule layer; determining a cost associated with the particular capsule based at least in part on the values of the parameters characterizing the distribution of the weighted votes for the particular capsule; determining an activation of the particular capsule based at least in part on the cost associated with the particular capsule.

In some implementations, the output of a capsule comprises an activity vector.

In some implementations, the output of a capsule comprises an activity matrix.

In some implementations, the neural network comprises an initial convolutional layer followed by a primary capsule layer.

In some implementations, the primary capsule layer is followed by a plurality of convolutional capsule layers.

In some implementations, a last convolutional capsule layer of the plurality of convolutional capsule layers is followed by a class capsule layer.

In some implementations, for each capsule of the class capsule layer, weight matrices of the capsule that are associated with capsules of the last convolutional capsule layer of a same type share weight values, wherein: a first capsule of a convolutional capsule layer is of a same type as a second capsule of the convolutional capsule layer if weight matrices of the first capsule share weight values with weight matrices of the second capsule; values based on the coordinates of the center of the receptive fields of capsules of the last convolutional capsule layer are added to votes of capsules of the last convolutional capsule layer.

In some implementations, the neural network is configured to receive a network input and to classify the network input as belonging to one or more of a predetermined set of classes, and wherein the class capsule layer includes a respective capsule for each class in the predetermined set.

In some implementations, an activation of a respective capsule of the class capsule layer is greater than activations of other capsules of the class capsule layer; the network input is classified as belonging to a particular class corresponding to the respective capsule of the class capsule layer.

In some implementations, the network input is an image.

The above aspects can be implemented in any convenient form. For example, aspects and implementations may be implemented by appropriate computer programs which may be carried on appropriate carrier media which may be tangible carrier media (e.g. disks) or intangible carrier media (e.g. communications signals). Aspects may also be implemented using suitable apparatus which may take the form of programmable computers running computer programs.

Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages. Capsule neural networks can include a dynamic routing system to route the outputs of lower layers of the capsule network to higher layers of the capsule network, where the routing of outputs between capsule network layers is dynamically determined based on the outputs themselves. The dynamic routing system described in this specification facilitates superior performance (e.g., superior neural network prediction accuracy) relative to conventional routing methods. In some conventional routing methods, the routing of outputs between neural network layers is pre-determined, and in some other conventional routing methods such as max-pooling methods, the routing is dynamically determined by routing only the largest outputs and discarding the others. The dynamic routing system as described in this specification facilitates superior performance relative to such conventional routing methods because it enables a more flexible routing of outputs, thereby allowing a neural network including the dynamic routing system as described in this specification to route relevant information more effectively and generate more accurate predictions as a result.

Capsule networks as described in this specification may consume fewer computational resources than conventional neural networks. For example, in the setting of image processing and computer vision, a conventional neural network that is configured to process input images is typically trained on a large number of input images depicting objects and scenes from different viewpoints. Training a conventional neural network in this manner may endow the conventional neural network with some capacity to make accurate predictions about objects and scenes depicted from a variety of viewpoints. Capsule networks as described in this specification can naturally generalize to make accurate predictions about objects and scenes depicted from a variety of viewpoints, even when trained on less training data than conventional neural networks. Since training a neural network based on a larger training set consumes more computational resources, training capsule networks may consume fewer computational resources (e.g., memory and processing power) than training conventional neural networks. Furthermore, since capsule networks as described in this specification can naturally generalize to make accurate predictions about objects and scenes depicted from a variety of viewpoints, a capsule network that has similar prediction accuracy as a conventional neural network can (in some situations) be implemented with fewer parameters than the conventional neural network, thereby reducing the amount of memory required to maintain the neural network, the amount of data. required to train the neural network, and the amount of processing power required to process inputs using the neural network.

The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

