The success of STML depends heavily on the quality of predicted semantic similarities, and our second contribution lies in the way of estimating semantic similarities using contexts.
Specifically, given a pair of data, we compute their semantic similarity considering the overlap of their contexts (i.e., neighborhoods in the embedding space) as well as their pairwise distance.
We found that the contextualized semantic similarity approximates class-equivalence precisely.
Further, since it has a real value indicating the degree of semantic similarity, it provides rich information beyond binary class-equivalence.
Also, since it requires no external module nor memory banks [1]}, [2]}, it makes STML efficient and concise.
To further enhance the quality of predicted semantic similarity, we design the teacher network to learn a higher dimensional embedding space than the student counterpart.
This asymmetric design of the two networks allows the teacher to provide more effective supervision thanks to its improved expression power while the student, i.e., our final model, remains compact.
