At present, there is no common corpus and evaluation metric to evaluate the intrinsic quality of SSE models. For instance, [1]}, [2]}, [3]}
use pre-segmented word tokens to perform a query-by-example task measured by MAP [4]}. Others rely on an ABX test on word triplets [5]}.
All these research papers compute their metrics on different corpora featuring different languages. This plurality of metrics and corpora makes it hard to monitor improvement in the field. We propose to create a common benchmark using the most popular metric so far: the MAP, on a widely known corpus: the LibriSpeech dataset [6]}.
