language:en language:multilingual license:other region:us
---
license: other
language:
- en
- multilingual
pretty_name: mBLIP instructions
---


# mBLIP Instruct Mix Dataset Card

## Important!
This dataset currently does not work directly with `datasets.load_dataset(Gregor/mblip-train)`! 
Please download the data files you need and load them with `datasets.load_dataset("json", data_files="filename")`.

## Dataset details

**Dataset type:**
This is the instruction mix used to train [mBLIP](https://github.com/gregor-ge/mBLIP).

See https://github.com/gregor-ge/mBLIP/data/README.md for more information on how to reproduce the data.


**Dataset date:**
The dataset was created in May 2023.

**Dataset languages:**
The original English examples were machine translated to the following 95 languages:
`
af, am, ar, az, be, bg, bn, ca, ceb, cs, cy, da, de, el, en, eo, es, et, eu, fa, fi, fil, fr, ga, gd, gl, gu, ha, hi, ht, hu, hy, id, ig, is, it, iw, ja, jv, ka, kk, km, kn, ko, ku, ky, lb, lo, lt, lv, mg, mi, mk, ml, mn, mr, ms, mt, my, ne, nl, no, ny, pa, pl, ps, pt, ro, ru, sd, si, sk, sl, sm, sn, so, sq, sr, st, su, sv, sw, ta, te, tg, th, tr, uk, ur, uz, vi, xh, yi, yo, zh, zu
`

Languages are translated proportional to their size in [mC4](https://www.tensorflow.org/datasets/catalog/c4#c4multilingual), i.e., as 6% of examples in mC4 are German, we translate 6% of the data to German.




**Dataset structure:**
- `task_mix_mt.json`: The instruction mix data in the processed, translated, and combined form.
- Folders: The folders contain 1) the separate tasks used to generate the mix 
and 2) the files of the tasks used to evaluate the model.

**Images:** 
We do not include any images with this dataset. 
Images from the public datasets (MSCOCO for instruction training, and others for evaluation) can be downloaded
from the respective websites.
For the BLIP captions, we provide the URLs and filenames as used by us [here](blip_captions/ccs_synthetic_filtered_large_2273005_raw.json).
To download them, [our code](https://github.com/gregor-ge/mBLIP/tree/main/data#blip-web-capfilt) can be adapted, for example. 


**License:**
Must comply with license of the original datasets used to create this mix. See https://github.com/gregor-ge/mBLIP/data/README.md for more.

Translations were produced with [NLLB](https://huggingface.co/facebook/nllb-200-distilled-1.3B) so use has to comply with
their license.


**Where to send questions or comments about the model:**
https://github.com/gregor-ge/mBLIP/issues

## Intended use
**Primary intended uses:**
The primary  is research on large multilingual multimodal models and chatbots.

**Primary intended users:**
The primary intended users of the model are researchers and hobbyists in computer vision, natural language processing, machine learning, and artificial intelligence.