language:ind language:eng concept-alignment-classification region:us
In this work, we present AM2iCo, a wide-coverage and carefully designed cross-lingual and multilingual evaluation set;
it aims to assess the ability of state-of-the-art representation models to reason over cross-lingual 
lexical-level concept alignment in context for 14 language pairs. 

This dataset only contain Indonesian - English language pair.
@inproceedings{liu-etal-2021-am2ico,
    title = "{AM}2i{C}o: Evaluating Word Meaning in Context across Low-Resource Languages with Adversarial Examples",
    author = "Liu, Qianchu  and
      Ponti, Edoardo Maria  and
      McCarthy, Diana  and
      Vuli{\'c}, Ivan  and
      Korhonen, Anna",
    booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2021",
    address = "Online and Punta Cana, Dominican Republic",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.emnlp-main.571",
    doi = "10.18653/v1/2021.emnlp-main.571",
    pages = "7151--7162",
    abstract = "Capturing word meaning in context and distinguishing between correspondences and variations across languages is key to building successful multilingual and cross-lingual text representation models. However, existing multilingual evaluation datasets that evaluate lexical semantics {``}in-context{''} have various limitations. In particular, 1) their language coverage is restricted to high-resource languages and skewed in favor of only a few language families and areas, 2) a design that makes the task solvable via superficial cues, which results in artificially inflated (and sometimes super-human) performances of pretrained encoders, and 3) no support for cross-lingual evaluation. In order to address these gaps, we present AM2iCo (Adversarial and Multilingual Meaning in Context), a wide-coverage cross-lingual and multilingual evaluation set; it aims to faithfully assess the ability of state-of-the-art (SotA) representation models to understand the identity of word meaning in cross-lingual contexts for 14 language pairs. We conduct a series of experiments in a wide range of setups and demonstrate the challenging nature of AM2iCo. The results reveal that current SotA pretrained encoders substantially lag behind human performance, and the largest gaps are observed for low-resource languages and languages dissimilar to English.",
}
---
tags:
- concept-alignment-classification
language:
- ind
- eng
---

# id_am2ico

In this work, we present AM2iCo, a wide-coverage and carefully designed cross-lingual and multilingual evaluation set;

it aims to assess the ability of state-of-the-art representation models to reason over cross-lingual 

lexical-level concept alignment in context for 14 language pairs. 



This dataset only contain Indonesian - English language pair.

## Dataset Usage

Run `pip install nusacrowd` before loading the dataset through HuggingFace's `load_dataset`.

## Citation

```
@inproceedings{liu-etal-2021-am2ico,
    title = "{AM}2i{C}o: Evaluating Word Meaning in Context across Low-Resource Languages with Adversarial Examples",
    author = "Liu, Qianchu  and
      Ponti, Edoardo Maria  and
      McCarthy, Diana  and
      Vuli{'c}, Ivan  and
      Korhonen, Anna",
    booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2021",
    address = "Online and Punta Cana, Dominican Republic",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.emnlp-main.571",
    doi = "10.18653/v1/2021.emnlp-main.571",
    pages = "7151--7162",
    abstract = "Capturing word meaning in context and distinguishing between correspondences and variations across languages is key to building successful multilingual and cross-lingual text representation models. However, existing multilingual evaluation datasets that evaluate lexical semantics {``}in-context{''} have various limitations. In particular, 1) their language coverage is restricted to high-resource languages and skewed in favor of only a few language families and areas, 2) a design that makes the task solvable via superficial cues, which results in artificially inflated (and sometimes super-human) performances of pretrained encoders, and 3) no support for cross-lingual evaluation. In order to address these gaps, we present AM2iCo (Adversarial and Multilingual Meaning in Context), a wide-coverage cross-lingual and multilingual evaluation set; it aims to faithfully assess the ability of state-of-the-art (SotA) representation models to understand the identity of word meaning in cross-lingual contexts for 14 language pairs. We conduct a series of experiments in a wide range of setups and demonstrate the challenging nature of AM2iCo. The results reveal that current SotA pretrained encoders substantially lag behind human performance, and the largest gaps are observed for low-resource languages and languages dissimilar to English.",
}
```

## License

CC-BY 4.0

## Homepage

[https://github.com/cambridgeltl/AM2iCo](https://github.com/cambridgeltl/AM2iCo)

### NusaCatalogue

For easy indexing and metadata: [https://indonlp.github.io/nusa-catalogue](https://indonlp.github.io/nusa-catalogue)