While the multimodal alignment aligns the visual and corresponding semantic embeddings in the latent space, it does not learn a domain-agnostic latent space, which is partitioned according to the semantic concepts that relate to the different classes.
In order to achieve a structured and domain-invariant latent space, we propose to cluster the latent embeddings based on class-level (domain-invariant) semantic concepts across different domains. The latent space is then conceptually structured, since the visual latent embeddings \(\mathbf {z}_v\)  and semantic latent embeddings \(\mathbf {z}_a\)  of a class are clustered together.
To this end, we adopt the center loss  [1]} in a multimodal setting. Formally, we first randomly initialise \(S\)  centers, i.e., \(\lbrace \mathbf {c}_{j}|j=1,\ldots , S\rbrace \)  for each of the seen classes in the training set and compute the loss, \(\mathcal {L}_{center}\)  due to each class \(y\)  present in a mini-batch. Then, for every class \(y\)  that is present in a mini-batch, the center update \(\Delta \mathbf {c}_y\)  is computed for incrementing the corresponding center \(\mathbf {c}_y\) . The loss \(\mathcal {L}_{center}\)  and update \(\Delta \mathbf {c}_y\)  are given by:
\(\mathcal {L}_{center}=\delta [\mathbb {E}(||\mathbf {z}_{v}-\mathbf {c}_y||_{2}^2)+\mathbb {E}(||\mathbf {z}_{a}-\mathbf {c}_y||_{2}^2)]; \qquad & \Delta \mathbf {c}_y = \mathbb {E} [\mathbf {c}_y-\mathbf {z}_v] + \mathbb {E} [\mathbf {c}_y-\mathbf {z}_a].\) 
