license:apache-2.0 region:us
---
license: apache-2.0
---
## Anh multilingual chat dataset

This is about 24M multilingual synthetic instructions intended to perform continued pretraining and finetuning a chatbot. 

- cross_lingual.jsonl (~800000)
  This dataset contains both the multi-lingual and cross-lingual version of the Anh data in the form of `Human: instruction\nAssistant: response` described here:
  https://github.com/LAION-AI/Anh/tree/main/data .
  The data is translated from a portion of the OIG dataset, which includes synthic_qa, prosocial and anthropic data. Read more about the data in LAION's OIG hf repo.
  Covers these langs: zh, vi, ru, ms, pt, ja, id, hi, fr, es, de.
  
- xp3_sample.jsonl (~650000)
  This dataset also contains a portion of the xp3 dataset converted into the standard Human/Assistant format.
  See https://huggingface.co/datasets/bigscience/xP3 for the 43 languages covered by xp3.
  
- sungai_ul2_instructions.jsonl (~23000000)
  This dataset also contains a UL2 like instruction set based on 140 languages from a subset of cc100, OSCAR and mc4.
  You can find the individual datasets from which this UL2 version was created here: https://github.com/ontocord/sungai 

## Disclaimer
- Translations may be inaccurate. The web text found in the UL2 file may contain inappropriate content as it is based on web scrapped data.
- Translations were generated by M2M 12B and the output generations were limited at 512 tokens due to VRAM limit (40G).

## License
The Anh dataset that is authored by LAION volunteers is released under an Apache 2.0 license. However, the data also includes content licensed under other permissive licenses, or web-crawled data which is used under fair use principles.

## Acknowledgement
- Thanks to LAION's Anh multilingual chat team: @yp_yurilee, @cahya, @kevin ko, @lasse, @mattdf, @theblackcat102, @yongzx, @acul3, @logus2, @paulovn, and many others. 
- Thanks to @rallio67 for the original English version of the cross_lingual dataset.
- Thanks to @theblackcat102 for his translations at https://huggingface.co/datasets/theblackcat102/instruction_translations, from which the cross-lingual data is based.
- Thanks to the authors of all the underlying datasets from which Anh is based, including the xp3 authors, OSCAR, cc100 and mc4 authors. 