FIELD

The present disclosure relates generally to performing clustering using machine-learning. More particularly, the present disclosure relates to systems and methods for training a clustering and/or embedding model.

BACKGROUND

Clustering is a common operation in machine learning and data analysis. Traditional clustering algorithms, of which there are many, are applied outside deep learning systems in a predefined representation space. Most clustering algorithms rest on the premise that natural datasets consists of a collection of discernible distributional modes (i.e., the clusters) in some high dimensional space. Conventional clustering algorithms also typically employ a hard encoding of input datapoints to clusters. Such clustering algorithms suffer various drawbacks. For example, they are not always satisfactory for complex data sets and/or can require substantial manual adjustment or tuning to be adapted for different situations. Further, prior art neural network-based clustering methods generally depend on external clustering in a pre-existing embedding space to seed a clustering objective, which is then reinforced.

SUMMARY

Aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or can be learned from the description, or can be learned through practice of the embodiments.

One example aspect of the present disclosure is directed to a computing system including one or more processors and an embedding model configured to receive each of a plurality of inputs and to respectively process each input to produce a respective embedding. The computing system may include a clustering model configured to receive the respective embedding for each input and to respectively process the respective embedding for each input to produce a respective cluster assignment for each input. The respective cluster assignment for each input can include a respective probability distribution for the respective embedding with respect to a plurality of clusters. The computing system can include one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations. The operations may include inputting each of the plurality of inputs into the embedding model; receiving the respective embeddings for the plurality of inputs as outputs of the embedding model; inputting the respective embeddings for the plurality of inputs into the clustering model; receiving the respective cluster assignments for the plurality of inputs as outputs of the clustering model; evaluating a clustering loss function; and modifying one or more parameters of one or both of the clustering model and the embedding model based on the clustering loss function. The clustering loss function can evaluate a first average, across the plurality of inputs, of a respective first entropy of each respective probability distribution and a second entropy of a second average of the probability distributions for the plurality of inputs.

Another example aspect of the present disclosure is directed to a computing system including one or more processors and an embedding model configured to receive each of a plurality of inputs and to respectively process each input to produce a respective embedding. The computing system can include a clustering model configured to receive the respective embedding for each input and to respectively process the respective embedding for each input to produce a respective cluster assignment for each input. The respective cluster assignment for each input can include a respective probability distribution for the respective embedding with respect to a plurality of clusters. At least one of the embedding model or the clustering model can have been trained based on a clustering loss function. The clustering loss function can include a first average, across the plurality of inputs, of a respective first entropy of each respective probability distribution and a second entropy of a second average of the probability distributions for the plurality of inputs. The computing system can include one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations. The operations can include inputting an additional input into the embedding model and receiving an additional embedding as an output of the embedding model. The additional embedding can be generated by the embedding model by processing the additional input. The operations can include inputting the additional embedding into the clustering model and receiving an additional cluster assignment as an output of the clustering model. The additional cluster assignment can be generated by the clustering model by processing the additional embedding.

Another example aspect of the present disclosure is directed to a method for training one or more machine learned models. The method may include inputting, by one or more computing devices, each of a plurality of inputs into an embedding model that is configured to respectively process each input to produce a respective embedding; receiving, by the one or more computing devices, the respective embeddings for the plurality of inputs as outputs of the embedding model; and inputting, by the one or more computing devices, the embeddings of the plurality of inputs into a clustering model that is configured to receive the respective embedding for each input and to respectively process the respective embedding for each input to produce a respective cluster assignment for each input. The respective cluster assignment for each input can include a respective probability distribution for the respective embedding with respect to a plurality of clusters. The operations can include receiving, by the one or more computing devices, the cluster assignment as an output of the clustering model and evaluating, by the one or more computing devices, a clustering loss function. The clustering loss function can include a mean per-example entropy that includes a first average, across the plurality of inputs, of a respective entropy of each respective probability distribution and a batch average distribution entropy of a second average of the probability distributions for the plurality of inputs. The method can include modifying, by the one or more computing devices, parameters of at least one of the clustering model or the embedding model based on the clustering loss function.

Other aspects of the present disclosure are directed to various systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices.

These and other features, aspects, and advantages of various embodiments of the present disclosure will become better understood with reference to the following description and appended claims. The accompanying drawings, which are incorporated in and constitute a pan of this specification, illustrate example embodiments of the present disclosure and, together with the description, serve to explain the related principles.

