We initialize the clustering of all instances in the training set \(S\)  using k-means [1]}. We obtain a representation for clustering which is not only aware of the visual features but also the semantic embedding. For this, we create a visual-semantic representation as follows.
Given video \(i\) , we compute visual features \(x_{i}\)  and a semantic embedding of their class \(y_i\)  as \(a(y_i)\)  (see Sec.  for details how we obtain these representations). We want to map both to an equal dimension and similar magnitude so they have a similar weight during clustering.
Additionally, to reduce overfitting to the training classes and improve generalization, we learn a multi-layer perceptron (MLP) [2]} which maps \(a(y_i)\)  in the space of \(x_{i}\)  with an MLP, which consists of two fully connected (FC) layers and a ReLU. This MLP is trained with a least-square embedding loss to minimize the distance between \(x_{i}\)  and the output from the MLP, which we call \(a^{\prime }(y)\) . \(x_{i}\)  is kept fixed.
Given these two representations, we concatenate \(x_{i}\)  and \(a^{\prime }(y)\)  to obtain the visual-semantic representations that we cluster with k-means. Each resulting cluster \(j\)  has a centroid \(c_{j}\) , that is the average of all visual-semantic samples in that particular cluster.
Note that we keep the MLP fixed after this initial training.
