size_categories:1M<n<10M language:en language:zh language:es language:ja language:fr language:ko language:ar license:mit arxiv:2306.05425 region:us
MIMIC-IT offers a diverse and extensive dataset of 2.8M multimodal instruction-response pairs, designed to enhance the performance of Vision-Language Models (VLMs) in real-life scenarios, enabling VLMs to excel in perception, reasoning, and planning while also catering to a multilingual audience.
@article{li2023mimicit,
    title={MIMIC-IT: Multi-Modal In-Context Instruction Tuning},
    author={Bo Li and Yuanhan Zhang and Liangyu Chen and Jinghao Wang and Fanyi Pu and Jingkang Yang and Chunyuan Li and Ziwei Liu},
    year={2023},
    eprint={2306.05425},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}
---
license: mit
language:
- en
- zh
- es
- ja
- fr
- ko
- ar
arxiv: 2306.05425
extra_gated_prompt: |
  <h1>MIMIC-IT Dataset Download
  Agreement</h1>
  <p>S-Lab, Nanyang Technological University (S-Lab) provides access to
  the MIMIC-IT Dataset (referred to as the Dataset) under the following
  conditions.</p>
  <p>By signing, the researcher agrees to the following terms of use:</p>
  <ol type="1">
  <li>S-Lab makes no warranties regarding the Dataset, including but not
  limited to being up-to-date, correct or complete. S-Lab cannot be held
  liable for providing access to the Dataset or usage of the Dataset.</li>
  <li>The Dataset should only be used for scientific or research purposes.
  Any other use is explicitly prohibited.</li>
  <li>The researcher agrees to the following terms and conditions of data
  sources of the Dataset:
  <ul>
    <li>TVC: <a href="https://tvqa.cs.unc.edu/">https://tvqa.cs.unc.edu/</a></li>
    <li>LLaVA: <a href="https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K">https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K</a>; <a href="https://cocodataset.org/#termsofuse">https://cocodataset.org/#termsofuse</a></li>
    <li>DC: <a href="http://activity-net.org/index.html">http://activity-net.org/index.html</a></li>
    <li>VIST: <a href="https://visionandlanguage.net/VIST/index.html">https://visionandlanguage.net/VIST/index.html</a></li>
    <li>SD: <a href="https://goo.gl/forms/HJiFJSllupqeCbax1">https://goo.gl/forms/HJiFJSllupqeCbax1</a></li>
    <li>Ego4D: <a href="https://ego4ddataset.com/">https://ego4ddataset.com/</a></li>
  </ul></li>
  <li>The researcher takes full responsibility for usage of the Dataset at
  any time.</li>
  <li>S-Lab reserves the right to terminate the researcher's access to the
  Dataset at any time.</li>
  <li>The place of jurisdiction is Singapore.</li>
  <li>If any part of this agreement is legally invalid, this shall not
  affect the remaining agreement.</li>
  </ol>
extra_gated_fields:
  Verifiable Name: text
  Institution Email: text
  Institutional Affiliation: text
  I agree with the agreement: checkbox
pretty_name: 'MIMIC-IT: Multi-Modal In-Context Instruction Tuning'
size_categories:
- 1M<n<10M
---

<p align="center" width="100%">
<img src="https://i.postimg.cc/sxy8v9PS/mimicit-logo.png"  width="80%" height="80%">
</p>


<div>
<div align="center">
    <a href='https://brianboli.com/' target='_blank'>Bo Li<sup>*,♠,1</sup></a>&emsp;
    <a href='https://zhangyuanhan-ai.github.io/' target='_blank'>Yuanhan Zhang<sup>*,♠,1</sup></a>&emsp;
    <a href='https://cliangyu.com/' target='_blank'>Liangyu Chen<sup>*,1</sup></a>&emsp;
    <a href='https://king159.github.io/' target='_blank'>Jinghao Wang<sup>*,1</sup></a>&emsp;
    <a href='https://pufanyi.github.io/' target='_blank'>Fanyi Pu<sup>*,1</sup></a>&emsp;
    </br>
    <a href='https://jingkang50.github.io/' target='_blank'>Jingkang Yang<sup>1</sup></a>&emsp;
    <a href='https://chunyuan.li/' target='_blank'>Chunyuan Li<sup>2</sup></a>&emsp;
    <a href='https://liuziwei7.github.io/' target='_blank'>Ziwei Liu<sup>&#x2709,1</sup></a>
</div>
<div>
<div align="center">
    <sup>1</sup>S-Lab, Nanyang Technological University&emsp;
    <sup>2</sup>Microsoft Research, Redmond
    </br>
    <sup>♠</sup> Co-Project Lead&emsp;
    <sup>*</sup> Equal Contribution&emsp;
    <sup>&#x2709</sup> Corresponding Author
    
</div>

## Dataset Description

- **Homepage: https://otter-ntu.github.io**
- **Repository: https://github.com/Luodian/Otter**
- **Paper: https://arxiv.org/abs/2306.05425**

### Dataset Summary

MIMIC-IT offers a diverse and extensive dataset of 2.8M multimodal instruction-response pairs, designed to enhance the performance of Vision-Language Models (VLMs) in real-life scenarios, enabling VLMs to excel in perception, reasoning, and planning while also catering to a multilingual audience. 

MIMIC-IT enables the application of egocentric visual assistant model that can serve that can answer your questions like **Hey, Do you think I left my keys on the table?**. Harness the power of MIMIC-IT to unlock the full potential of your AI-driven visual assistant and elevate your interactive vision-language tasks to new heights.

MIMIC-IT provides multilingual instructions, supporting English, Chinese, Korean, Japanese, German, French, Spanish, and Arabic, thereby allowing a larger global audience to altogether enjoy from the convenience brought about by advancements in artificial intelligence.

<p align="center" width="100%">
<img src="https://i.postimg.cc/4x66gHhw/mimic-it.jpg"  width="100%" height="100%">
</p>

## Using MIMIC-IT

You can following the steps to obtain the MIMIC-IT dataset. Each task (e.g. `DC`, `LA`) in MIMIC-IT is composed of three parts, including:
1. `xx.json` file: the images in base64 format.
2. `xx_instructions.json` file: the instruction-response pairs (also includes image ids and related instructions ids for each instruction-response pair) for each task.
3. `xx_train.json` file: the customized related instruction-response pairs for each instruction.

You can directly download the contents in the `data` folder. The distribution of the `data` folder is as follows:

```plain
data/
	CGD/
		CGD.json
		CGD_images_preview.csv
		CGD_instructions.json
	...
```

For each `dataset_name`, there are three main files **except for `DC` and `E4D`**:

1. `{dataset_name}.json`: Stores the image numbers and their corresponding base64 codes in lossless compressed PNG format.
   ```json
   {
     "image_id_1": "base64_code_1",
     "image_id_2": "base64_code_2",
     ...
   }
   ```
2. `{dataset_name}_images_preview.csv`: Stores the image numbers and their corresponding base64 codes in lossy compressed JPG format, mainly used for display in the Dataset Card.
   ```csv
   id, image
   "image_id_1", "base64_code_1"
   "image_id_2", "base64_code_2"
   ...
   ```
3. `{dataset_name}_instructions.json`: Stores each instruction and its associated answer.
   ```json
   {
     "meta": {
       "version": current_version,
       "time": update_time,
       "author": "ntu"
     },
     "data": {
       "instruction_id_1": {
         "instruction": "instruction_1",
         "answer": "answer_of_instruction_1",
         "image_ids": [
           "image_id_1",
           "image_id_2",
           ...
         ],
         "rel_ins_ids": [
           "related_instruction_id_1",
           "related_instruction_id_2",
           ...
         ]
       },
       ...
     }
   }
   ```

Of course, you can also use `wget` or `curl` for direct downloads. Below is an example.

Before proceeding with the downloads, you need to set your Hugging Face token. For that, please refer to [this page](https://huggingface.co/docs/hub/security-tokens).


```shell
$ # Set Hugging Face Token
$ HF_TOKEN="YOUR_HUGGING_FACE_TOKEN"

$ # Set the dataset you want to download
$ DATASET_NAME="DATASET_YOU_WANT_TO_DOWNLOAD" # e.g. CGD

$ # Download {DATASET_NAME}.json
$ wget --header="Authorization: Bearer $HF_TOKEN" "https://huggingface.co/datasets/pufanyi/MIMICIT/resolve/main/data/${DATASET_NAME}/${DATASET_NAME}.json"

$ # Download {DATASET_NAME}_instructions.json
$ wget --header="Authorization: Bearer $HF_TOKEN" "https://huggingface.co/datasets/pufanyi/MIMICIT/resolve/main/data/${DATASET_NAME}/${DATASET_NAME}_instructions.json"

$ # Download {DATASET_NAME}_images_preview.csv (usually not necessary)
$ wget --header="Authorization: Bearer $HF_TOKEN" "https://huggingface.co/datasets/pufanyi/MIMICIT/resolve/main/data/${DATASET_NAME}/${DATASET_NAME}_images_preview.csv"
```

Or

```shell
$ # Set Hugging Face Token
$ HF_TOKEN="YOUR_HUGGING_FACE_TOKEN"

$ # Set the dataset you want to download
$ DATASET_NAME="DATASET_YOU_WANT_TO_DOWNLOAD" # e.g. CGD

$ # Download {DATASET_NAME}.json
$ curl -LJO -H "Authorization: Bearer $HF_TOKEN" "https://huggingface.co/datasets/pufanyi/MIMICIT/resolve/main/data/${DATASET_NAME}/${DATASET_NAME}.json"

$ # Download {DATASET_NAME}_instructions.json
$ curl -LJO -H "Authorization: Bearer $HF_TOKEN" "https://huggingface.co/datasets/pufanyi/MIMICIT/resolve/main/data/${DATASET_NAME}/${DATASET_NAME}_instructions.json"

$ # Download {DATASET_NAME}_images_preview.csv (usually not necessary)
$ curl -LJO -H "Authorization: Bearer $HF_TOKEN" "https://huggingface.co/datasets/pufanyi/MIMICIT/resolve/main/data/${DATASET_NAME}/${DATASET_NAME}_images_preview.csv"
```

Alternatively, you can use `dataset.load_dataset` for downloading. However, due to Hugging Face's size limitations, all images can only be loaded in JPG format. Below is an example using `CGD` dataset:


### CGD_Images

Download the JPG format images and their corresponding identifiers:

```python
from datasets import load_dataset
data = load_dataset("pufanyi/MIMICIT", "CGD_Images")
```

The format will be like:

```json
{
	"id": "CGD_IMG_000000426149",
	"image": <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=224x224 at 0x7F84601D62F0>
}
```

It should be noted that, due to size limitations, for `DC` (Dense Captions), this command will only extract a portion of the images from the `DC` collection for downloading.

### CGD_Instructions

Download all instructions:

```python
from datasets import load_dataset
data = load_dataset("pufanyi/MIMICIT", "CGD_Instructions")
```

The format will be like:

```json
{
    "id": "CGD_INS_000000",
    "instruction": "What is the difference between the two pizzas in these images?",
    "answer": "The pizza in the first image is on a red plate and being held by an old lady, while the pizza in the second image is on a metal counter being prepared by a woman in a blue shirt.",
    "images": [
        "CGD_IMG_000000069568",
        "CGD_IMG_000000328270"
    ],
    "related instructions": [
        "CGD_INS_000001"
    ]
}
```

### CGD_Preview

Download all instructions along with their corresponding JPG images:

```python
from datasets import load_dataset
data = load_dataset("pufanyi/MIMICIT", "CGD_Preview")
```

The format will be like:

```json
{
    "id": "CGD_INS_000000",
    "instruction": "What is the difference between the two pizzas in these images?",
    "answer": "The pizza in the first image is on a red plate and being held by an old lady, while the pizza in the second image is on a metal counter being prepared by a woman in a blue shirt.",
    "images": [
        <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=224x224 at 0x7F8460267DF0>,
        <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=224x224 at 0x7F8460267700>
    ],
    "related instructions": [
        "CGD_INS_000001"
    ]
}
```

It should be noted that, due to size limitations, for `DC` (Dense Captions), this command will only extract a portion of the images from the `DC` collection for downloading.