Step #3: Learning the student model.
We simultaneously optimize the two branches of the student model using the contextualized semantic similarities as synthetic supervision. Since its value is soft, however, the contextualized semantic similarity cannot be incorporated with the ordinary metric learning losses that take discrete labels as supervision.
We thus employ the relaxed contrastive loss [1]}, which is designed to utilize soft relational labels for metric learning.
Let \(w_{ij}\)  be the contextualized semantic similarity between \(x_i\)  and \(x_j\) , and \(z_i^{f_s}\)  denote the student embedding vector of \(x_i\)  generated by \(f^s\) . Defining \(d_{ij}^{f_s}:={||z_i^{f_s} - z_j^{f_s}||_2}/({\frac{1}{n}\sum _{k=1}^{n}||z_i^{f_s} - z_k^{f_s}||_2})\)  as the relative distance between \(f_i^s\)  and \(f_j^s\) , the loss is then given by
\(\begin{split}\mathcal {L}_\textrm {RC}(\mathcal {Z}^{f_s}) &= \frac{1}{n}\sum _{i=1}^n\sum _{j\ne i}^n{{w_{ij} \big (d_{ij}^{f_s}}\big )^2} \\&+ \frac{1}{n}\sum _{i=1}^n\sum _{j\ne i}^n{{(1-w_{ij})\big [\delta - d_{ij}^{f_s}}\big ]_+^2},\end{split}\) 
