arxiv:2307.06281 region:us
---
dataset_info:
  features:
  - name: question
    dtype: string
  - name: hint
    dtype: string
  - name: A
    dtype: string
  - name: B
    dtype: string
  - name: C
    dtype: string
  - name: D
    dtype: string
  - name: label
    dtype:
      class_label:
        names:
          '0': A
          '1': B
          '2': C
          '3': D
  - name: image
    dtype: image
  splits:
  - name: train
    num_bytes: 102942038.498
    num_examples: 4377
  download_size: 99866501
  dataset_size: 102942038.498
---
# Dataset Card for "MMBench_dev"

## Dataset Description

* **Homepage**: https://opencompass.org.cn/mmbench
* **Repository**: https://github.com/internLM/OpenCompass/
* **Paper**: https://arxiv.org/abs/2307.06281
* **Leaderboard**: https://opencompass.org.cn/leaderboard-multimodal
* **Point of Contact**: opencompass@pjlab.org.cn

### Dataset Summary

In recent years, the field has seen a surge in the development of numerous vision-language (VL) models, such as MiniGPT-4 and LLaVA. These models showcase promising performance in tackling previously challenging tasks. However, effectively evaluating these models' performance has become a primary challenge hindering further advancement in large VL models. Traditional benchmarks like VQAv2 and COCO Caption are widely used to provide quantitative evaluations for VL models but suffer from several shortcomings:

Dataset Construction: Dataset Construction: Traditional benchmarks tend to evaluate models based on their performance in various tasks, such as image captioning and visual question answering. Unfortunately, these tasks do not fully capture the fine-grained abilities that a model possesses, potentially impeding future optimization efforts.

Evaluation Metrics: Existing evaluation metrics lack robustness. For example, VQAv2 targets a single word or phrase, while many current VL models generate sentences as outputs. Although these sentences may correctly answer the corresponding questions, the existing evaluation metric would assign a Fail score due to an inability to exactly match the given answer. Moreover, recently proposed subjective evaluation metrics, such as that used in mPLUG-Owl, offer comprehensive evaluation of VL models. However, these metrics struggle to scale smoothly due to the significant amount of human labor required for evaluation. Additionally, these evaluations are highly biased and difficult to reproduce.

To address these limitations, we propose a novel approach by defining a set of fine-grained abilities and collecting relevant questions for each ability. We also introduce innovative evaluation strategies to ensure more robust assessment of model predictions. This new benchmark, called MMBench, boasts the following features:

Data Collection: To date, we have gathered approximately 3000 questions spanning 20 ability dimensions. Each question is a multiple-choice format with a single correct answer.

Evaluation: For a more reliable evaluation, we employ ChatGPT to match a model's prediction with the choices of a question, and then output the corresponding label (A, B, C, D) as the final prediction.

### Languages

All of our questions are presented in single-choice question format, with the number of options ranging from 2 to 4. In addition, all these questions, options, and answers are in English.

## Dataset Structure

### Data Instances

We provide a overview of an instance in MMBench as follows:

```text
{
  'index': 241,
  'question': 'Identify the question that Madelyn and Tucker's experiment can best answer.',
  'hint': 'The passage below describes an experiment. Read the passage and then follow the 
          instructions below.\n\nMadelyn applied a thin layer of wax to the underside of her  
          snowboard and rode the board straight down a hill. Then, she removed the wax and rode 
          the snowboard straight down the hill again. She repeated the rides four more times, 
          alternating whether she rode with a thin layer of wax on the board or not. Her friend 
          Tucker timed each ride. Madelyn and Tucker calculated the average time it took to slide 
          straight down the hill on the snowboard with wax compared to the average time on the 
          snowboard without wax.\nFigure: snowboarding down a hill.'
  'A': 'Does Madelyn's snowboard slide down a hill in less time when it has a thin layer of wax or 
        a thick layer of wax?'
  'B': 'Does Madelyn's snowboard slide down a hill in less time when it has a layer of wax or     
        when it does not have a layer of wax?'
  'image': xxxxxx,
  'category': 'identity_reasoning',
  'l2-category': 'attribute_reasoning',
  'split': 'dev',
  'source': 'scienceqa',
}
```

### Data Fields

* `index`: the index of the instance in the dataset.
* `question`: the question of the instance.
* `hint (optional)`: the hint of the instance.
* `A`: the first option of the instance.
* `B`: the second option of the instance.
* `C (optional)`: the third option of the instance.
* `D (optional)`: the fourth option of the instance.
* `image`: the raw image of the instance.
* `category`: the leaf category of the instance. 
* `l2-category`: the L-2 category of the instance.
* `split`: the split of the instance.
* `source`: the source of the instance comes from.


### Data Splits

Currently, MMBench contains 2974 instances in total, and is splitted into **dev** and **test** splits according to a 4:6 ratio.

## Additional Information

### Citation Information

```
@article{MMBench,
    author  = {Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhnag, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, Dahua Lin},
    journal = {arXiv:2307.06281},
    title   = {MMBench: Is Your Multi-modal Model an All-around Player?},
    year    = {2023},
}
```