Some datasets, such as HMDB51 [1]}, do not have semantic manual annotations. Thus, semantic representations are often computed using a word2vec model [2]}. In most action recognition datasets, labels are phrases (e.g. “playing guitar") and thus the semantic embeddings are computed by averaging the word2vec embeddings of each word.
This works in some cases, however, simple averaging does not always capture the inter-dependency of action classes.
For example, “horse riding" and “horse racing" lie far apart in the word2vec space.
