There are two main methods to evaluate word representations: Intrinsic and Extrinsic.
Intrinsic evaluations measure syntactic or semantic relationships between words directly, e.g., word similarity in word clusters.
Extrinsic evaluations measure the performance of word embeddings as input features to a downstream task, e.g., named entity recognition (NER) and text classification.
Several studies have shown that there is no consistent correlation between intrinsic and extrinsic evaluation results [1]}, [2]}, [3]}.
Hence, we evaluate our representation by both intrinsic and extrinsic metrics.
Specifically, we use 8 intrinsic datasets (6 word similarity and 2 word cluster tasks):
RareWord [4]},
SimLex [5]},
MTurk [6]},
MEN [7]},
WordSim [8]},
Simverb [8]},
AP [10]} and
BLESS [11]}.
We use four extrinsic datasets (2 text classification and 2 NER tasks):
SST2 [12]}, MR [13]},
CoNLL-03 [14]} and BC2GM [15]}.
It is worth noting that the RareWord dataset contains many long-tail words and the BC2GM is a domain-specific NER dataset.
All data augmentations and typo simulations are implemented by NLPAUGhttps://github.com/makcedward/nlpaug.
Appendix  provides more details on our datasets and experimental settings.
