To better exploit the paired features of our data, we apply cross-modality alignment loss to force the soft assignments of every image-caption pair to be similar. The idea is that the text and image from the same pair should be assigned to the same cluster (and, more generally, should have similar soft assignment distributions). We use Jensen–Shannon divergence (JSD) [1]} to capture the similarity between the cluster assignment distributions:
\(L_{align} &= JSD(\mathbf {q}||\mathbf {r}) \\&= \frac{1}{2}KL(\mathbf {r}||\mathbf {s}) + \frac{1}{2}KL(\mathbf {q}||\mathbf {s})\) 
