One relatively early approach try to find orthogonal or close to orthogonal transformations [1]}, [2]}, [3]}, [3]} to improve interpretability.
These models' advantage is that they do not lose information due to
the orthogonality. Other works [5]}, [6]}, [7]} transform the original embedding to a sparse embedding to improve interpretability. Unlike our method, the above models provide a limited explainability and they need embedding matrix which limit them especially in unbounded embedding space (like sentence embedding that we use here).
