size_categories:n<1K license:apache-2.0 region:us
---
license: apache-2.0
size_categories:
- n<1K
---

HEROS is a dataset used to compare the sentence cosine similarity among sentences with high lexical overlapping but differ in their semantics.
Please refer to the paper, "Revealing the Blind Spot of Sentence Encoder Evaluation by HEROS" for more details of how the dataset is constructed and the comparison of different sentence encoders.

The dataset `heros.tsv` consists of 6 columns: `Original`, `Synonym`, `Antonym`, `Negation`, `Random`, `Typo`, `Negation`. 
The first column, `Original` are the sentences from GoEmotion dataset, and sentences in the other columns are constructed by replacing some words in the original sentences based on different rules, making up different subsets in HEROS.
Different subsets in HEROS capture various aspects of semantics.
Comparing the average cosine similarity between minimal pairs in Synonym and Antonym allows one to understand whether replacing a word with an antonym is more dissimilar to the original semantics than replacing a word with a synonym.
The average cosine similarity between minimal pairs in Negation can tell us how negation affects sentence embedding similarity.
Typos are realistic and happen every day.
While humans can infer the original word from a typo and get the original meaning of the sentence, it will be interesting to see how the typos affect the sentences' similarity with the original sentences.
The Random MLM subset can tell us how similar the sentence embedding can be when two sentences are semantically different but with high lexical overlaps.
By comparing the performance of different SEs on different subsets in HEROS, we can further understand the trait of different SEs.
			
