size_categories:1K<n<10K language:en license:cc-by-4.0 figurative-language multimodal-figurative-language  commonsense-reasoning visual-reasoning arxiv:2303.15445 region:us
---
license: cc-by-4.0
language:
- en
tags:
- figurative-language
- multimodal-figurative-language
- ' commonsense-reasoning'
- visual-reasoning
size_categories:
- 1K<n<10K
---

# Dataset Card for IRFL

- [Dataset Description](#dataset-description)
  - [Leaderboards](#leaderboards)
  - [Colab notebook code for IRFL evaluation](#colab-notebook-code-for-irfl-evaluation)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Fields](#data-fields)
- [Dataset Creation](#dataset-creation)
- [Considerations for Using the Data](#considerations-for-using-the-data)
  - [Licensing Information](#licensing-information)
  - [Citation Information](#citation-information)

## Dataset Description

The IRFL dataset consists of idioms, similes, and metaphors with matching figurative and literal images, as well as two novel tasks of multimodal figurative understanding and preference.

We collected figurative and literal images for textual idioms, metaphors, and similes using an automatic pipeline we created (idioms) and manually (metaphors + similes). We annotated the relations between these images and the figurative phrase they originated from. Using these images we created two novel tasks of figurative understanding and preference.

The figurative understanding task evaluates Vision and Language Pre-Trained Models’ (VL-PTMs) ability to understand the relation between an image and a figurative phrase. The task is to choose the image that best visualizes the figurative phrase out of X candidates. The preference task examines VL-PTMs' preference for figurative images. In this task, the model needs to classify phrase images of different categories correctly based on their ranking by the model matching score.


The figurative understanding task evaluates Vision and Language Pre-Trained Models’ (VL-PTMs) ability to understand the relation between an image and a figurative phrase. The task is to choose the image that best visualizes the figurative phrase out of X candidates. The preference task examines VL-PTMs' preference for figurative images. In this task, the model needs to classify phrase images of different categories correctly based on their ranking by the model matching score.

We evaluated state-of-the-art VL models and found that the best models achieved 22%, 30%, and 66% accuracy vs. humans 97%, 99.7%, and 100% on our understanding task for idioms, metaphors, and similes respectively. The best model achieved an F1 score of 61 on the preference task.



- **Homepage:** 
https://irfl-dataset.github.io/
- **Repository:**
https://github.com/irfl-dataset/IRFL
- **Paper:**
https://arxiv.org/abs/2303.15445
- **Leaderboard:**
https://irfl-dataset.github.io/leaderboard
- **Point of Contact:**
irfl.dataset@gmail.com; ron.yosef@mail.huji.ac.il

### Leaderboards

https://irfl-dataset.github.io/leaderboard 

### Colab notebook code for IRFL evaluation
https://colab.research.google.com/drive/1zbW7R8Cn9sXICV3x_FGKjKIKu8GCrCme?usp=sharing

### Languages

English. 

## Dataset Structure

### Data Fields
★ - refers to idiom-only fields

Understanding task
- query (★): the idiom definition the answer image originated from.
- distractors: the distractor images
- answer: the correct image 
- figurative_type: idiom | metaphor | simile
- images_metadata: the metadata of the distractors and asnwer images. 
- type: the correct image type (Figurative or Figurative Literal).
- definition (★): list of all the definitions of the idiom
- phrase: the figurative phrase.

Preference task
- type: the rival categories FvsPO (Figurative images vs. Partial Objects) or FLvsPO (Figurative Literal images vs. Partial Objects)
- figurative_type: idiom | metaphor | simile
- first_category: the first category images (Figurative images if FvsPO, Figurative Literal images if FLvsPO)
- second_category: the second category images (Partial Objects)
- definition (★): list of all the definitions of the idiom
- phrase: the figurative phrase.

The idioms, metaphor, and similes datasets contain all the figurative phrases, annotated images, and corresponding metadata. <br/>


## Dataset Collection
We collected figurative and literal images for textual idioms, metaphors, and similes using an automatic pipeline we created (idioms) and manually (metaphors + similes). We annotated the relations between these images and the figurative phrase they originated from.

#### Annotation process

We paid Amazon Mechanical Turk Workers to annotate the relations between each image and phrase (Figurative vs. Literal).  

## Considerations for Using the Data
5 annotators annotated all of the data releases.


### Licensing Information

CC-By 4.0 

### Citation Information

@misc{yosef2023irfl,
      title={IRFL: Image Recognition of Figurative Language}, 
      author={Ron Yosef and Yonatan Bitton and Dafna Shahaf},
      year={2023},
      eprint={2303.15445},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}