Possible Vector Collaps Issue

#10
by Banso - opened

Hey, amazing work!

In some of my experiments, the model seems to predict vectors in certain areas, which causes the relatively high (> 0.7) cosine similarity.

I think this might be due to vector collapse. Maybe it's worth investigating in this direction.

You are right, the produced embeddings concentrate in a certain area of the vector space.

I suspect this is a result of using contrastive loss with low temperature (0.01), but it is surely a good research question.

Sign up or log in to comment