task_categories:visual-question-answering task_ids:visual-question-answering annotations_creators:machine-generated language_creators:machine-generated multilinguality:monolingual source_datasets:clevr language:en license:cc-by-4.0 reasoning neuro-symbolic multimodal arxiv:2208.05358 region:us
CLEVR-Math is a dataset for compositional language, visual and mathematical reasoning. CLEVR-Math poses questions about mathematical operations on visual scenes using subtraction and addition, such as "Remove all large red cylinders. How many objects are left?". There are also adversarial (e.g. "Remove all blue cubes. How many cylinders are left?") and multihop questions (e.g. "Remove all blue cubes. Remove all small purple spheres. How many objects are left?").
@misc{https://doi.org/10.48550/arxiv.2208.05358,
  doi = {10.48550/ARXIV.2208.05358},
  url = {https://arxiv.org/abs/2208.05358},
  author = {Lindström, Adam Dahlgren and Abraham, Savitha Sam},
  keywords = {Machine Learning (cs.LG), Computation and Language (cs.CL), Computer Vision and Pattern Recognition (cs.CV), FOS: Computer and information sciences, FOS: Computer and information sciences, I.2.7; I.2.10; I.2.6; I.4.8; I.1.4},
  title = {CLEVR-Math: A Dataset for Compositional Language, Visual, and Mathematical Reasoning},
  publisher = {arXiv},
  year = {2022},
  copyright = {Creative Commons Attribution Share Alike 4.0 International}
}
---
annotations_creators:
- machine-generated
language:
- en
language_creators:
- machine-generated
license:
- cc-by-4.0
multilinguality:
- monolingual
pretty_name: CLEVR-Math - Compositional language, visual, and mathematical reasoning
size_categories:
#- 100K<n<1M
source_datasets: [clevr]
tags:
- reasoning
- neuro-symbolic
- multimodal
task_categories:
- visual-question-answering
task_ids:
- visual-question-answering
---

# Dataset Card for CLEVR-Math

## Table of Contents
- [Table of Contents](#table-of-contents)
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
- [Considerations for Using the Data](#considerations-for-using-the-data)
  - [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
  - [Dataset Curators](#dataset-curators)
  - [Licensing Information](#licensing-information)
  - [Citation Information](#citation-information)
  - [Contributions](#contributions)

## Dataset Description

- **Homepage:**
- **Repository:*https://github.com/dali-does/clevr-math*
- **Paper:*https://arxiv.org/abs/2208.05358*
- **Leaderboard:**
- **Point of Contact:*dali@cs.umu.se*

### Dataset Summary

Dataset for compositional multimodal mathematical reasoning based on CLEVR.

#### Loading the data, preprocessing text with CLIP

```
from transformers import CLIPPreprocessor
from datasets import load_dataset, DownloadConfig


dl_config = DownloadConfig(resume_download=True,
        num_proc=8,
        force_download=True)

# Load 'general' instance of dataset
dataset = load_dataset('dali-does/clevr-math', download_config=dl_config)

# Load version with only multihop in test data
dataset_multihop = load_dataset('dali-does/clevr-math', 'multihop',
                                download_config=dl_config)


model_path = "openai/clip-vit-base-patch32"
extractor = CLIPProcessor.from_pretrained(model_path)
def transform_tokenize(e):
  e['image'] = [image.convert('RGB') for image in e['image']]
  return extractor(text=e['question'],
                           images=e['image'],
                           padding=True)

dataset = dataset.map(transform_tokenize,
                      batched=True,
                      num_proc=8,
                      padding='max_length')

dataset_subtraction = dataset.filter(lambda e:
        e['template'].startswith('subtraction'), num_proc=4)
```


### Supported Tasks and Leaderboards

Leaderboard will be announced at a later date.

### Languages

The dataset is currently only available in English. To extend the dataset to other languages, the CLEVR templates must be rewritten in the target language.

## Dataset Structure

### Data Instances

* `general` containing the default version with multihop questions in train and test
* `multihop` containing multihop questions only in test data to test generalisation of reasoning


### Data Fields


```
features = datasets.Features(
  {
    "template": datasets.Value("string"),
    "id": datasets.Value("string"),
    "question": datasets.Value("string"),
    "image": datasets.Image(),
    "label": datasets.Value("int64")
  }
)
```

### Data Splits

train/val/test

## Dataset Creation

Data is generated using code provided with the CLEVR-dataset, using blender and templates constructed by the dataset curators.


## Considerations for Using the Data

### Other Known Limitations

[More Information Needed]

## Additional Information

### Dataset Curators

Adam Dahlgren Lindström - dali@cs.umu.se

### Licensing Information

Licensed under Creative Commons Attribution Share Alike 4.0 International (CC-by 4.0).
### Citation Information

[More Information Needed]

```
@misc{https://doi.org/10.48550/arxiv.2208.05358,
  doi = {10.48550/ARXIV.2208.05358},
  url = {https://arxiv.org/abs/2208.05358},
  author = {Lindström, Adam Dahlgren and Abraham, Savitha Sam},
  keywords = {Machine Learning (cs.LG), Computation and Language (cs.CL), Computer Vision and Pattern Recognition (cs.CV), FOS: Computer and information sciences, FOS: Computer and information sciences, I.2.7; I.2.10; I.2.6; I.4.8; I.1.4},
  title = {CLEVR-Math: A Dataset for Compositional Language, Visual, and Mathematical Reasoning},
  publisher = {arXiv},
  year = {2022},
  copyright = {Creative Commons Attribution Share Alike 4.0 International}
}
```

### Contributions

Thanks to [@dali-does](https://github.com/dali-does) for adding this dataset.

