bigscience
/

mt0-xxl-mt

@@ -1,7 +1,9 @@
 ---
 license: apache-2.0
-language:
-- multilingual
 - af
 - am
 - ar
@@ -103,315 +105,798 @@ language:
 - yo
 - zh
 - zu
-datasets:
-- mc4
-- bigscience/xP3
 ---
-Multilingual Text-to-Text Transfer Transformer Zero (MT0)
-Version 1. / 28 October 2022
----
-# Models
-mT5 is pretrained on the [mC4](https://www.tensorflow.org/datasets/catalog/c4#c4multilingual) corpus, covering 101 languages:
-Afrikaans, Albanian, Amharic, Arabic, Armenian, Azerbaijani, Basque, Belarusian, Bengali, Bulgarian, Burmese, Catalan, Cebuano, Chichewa, Chinese, Corsican, Czech, Danish, Dutch, English, Esperanto, Estonian, Filipino, Finnish, French, Galician, Georgian, German, Greek, Gujarati, Haitian Creole, Hausa, Hawaiian, Hebrew, Hindi, Hmong, Hungarian, Icelandic, Igbo, Indonesian, Irish, Italian, Japanese, Javanese, Kannada, Kazakh, Khmer, Korean, Kurdish, Kyrgyz, Lao, Latin, Latvian, Lithuanian, Luxembourgish, Macedonian, Malagasy, Malay, Malayalam, Maltese, Maori, Marathi, Mongolian, Nepali, Norwegian, Pashto, Persian, Polish, Portuguese, Punjabi, Romanian, Russian, Samoan, Scottish Gaelic, Serbian, Shona, Sindhi, Sinhala, Slovak, Slovenian, Somali, Sotho, Spanish, Sundanese, Swahili, Swedish, Tajik, Tamil, Telugu, Thai, Turkish, Ukrainian, Urdu, Uzbek, Vietnamese, Welsh, West Frisian, Xhosa, Yiddish, Yoruba, Zulu.
-mt5 was then finetuned on:
- - [xP3](https://huggingface.co/bigscience/xP3) to obtain [mt0-small](https://huggingface.co/bigscience/mt0-small)/[mt0-base](https://huggingface.co/bigscience/mt0-base)/[mt0-large](https://huggingface.co/bigscience/mt0-large)/[mt0-xl](https://huggingface.co/bigscience/mt0-xl)/[mt0-xxl](https://huggingface.co/bigscience/mt0-xxl)
- - [P3](https://huggingface.co/bigscience/P3) to obtain [mt0-p3-xxl](https://huggingface.co/bigscience/mt0-p3-xxl)
- - [xP3mt](https://huggingface.co/bigscience/xP3mt) to obtain [mt0-mt-xxl](https://huggingface.co/bigscience/mt5-mt-xxl)
-## Model Flavors
-Multilingual model capable of following user instructions in a variety of languages. Together with our paper [TODO: LINK], we release the following models:
-----
-- [mt0-small](https://huggingface.co/bigscience/mt0-small): 300M parameters multitask finetuned version of [mt5-small](https://huggingface.co/google/mt5-small) on [xP3](https://huggingface.co/bigscience/xP3)
-- [mt0-base](https://huggingface.co/bigscience/mt0-base): 580M parameters multitask finetuned version of [mt5-base](https://huggingface.co/google/mt5-base) on [xP3](https://huggingface.co/bigscience/xP3)
-- [mt0-large](https://huggingface.co/bigscience/mt0-large): 1.2B parameters multitask finetuned version of [mt5-large](https://huggingface.co/google/mt5-large) on [xP3](https://huggingface.co/bigscience/xP3)
-- [mt0-xl](https://huggingface.co/bigscience/mt0-xl): 3.7B parameters multitask finetuned version of [mt5-xl](https://huggingface.co/google/mt5-xl) on [xP3](https://huggingface.co/bigscience/xP3)
-- [mt0-xxl](https://huggingface.co/bigscience/mt0-xxl): 13B parameters multitask finetuned version of [mt5-xxl](https://huggingface.co/google/mt5-xxl) on [xP3](https://huggingface.co/bigscience/xP3)
-----
-- [mt0-p3-xxl](https://huggingface.co/bigscience/mt0-p3-xxl): 13B parameters multitask finetuned version of [mt5-xxl](https://huggingface.co/google/mt5-xxl) on [P3](https://huggingface.co/bigscience/P3)
-- [mt0-mt-xxl](https://huggingface.co/bigscience/mt5-mt-xxl): 13B parameters multitask finetuned version of [mt5-xxl](https://huggingface.co/google/mt5-xxl) on [xP3mt](https://huggingface.co/bigscience/xP3mt)
-## Basics
-*This section provides information about the model type, version, license, funders, release date, developers, and contact information.*
-*It is useful for anyone who wants to reference the model.*
-<details>
-<summary>Click to expand</summary>
-*All collaborators are either volunteers or have an agreement with their employer. (Further breakdown of participants forthcoming.)*
-**Model Type:** Transformer-based Language Model
-**Checkpoints format:** `transformers`
-**Version:** 1.0.0
-**Languages:** Multiple; see [training data](#training-data)
-**License:** Apache 2.0
-**Release Date Estimate:** Friday, 28.October.2022
-**Send Questions to:** niklas@huggingface.co
-**Funded by:**
-* The French government.
-* Hugging Face ([website](https://huggingface.co)).
-</details>
-## Technical Specifications
-*This section includes details about the model objective and architecture, and the compute infrastructure.*
-*It is useful for people interested in model development.*
 <details>
-<summary>Click to expand</summary>
-### Model Architecture and Objective
-* Same architecture as [mt5](https://arxiv.org/abs/2010.11934)
-* Encoder-decoder architecture
-**Objective Function:** Cross Entropy with mean reduction on target tokens (see [API documentation](https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html#torch.nn.CrossEntropyLoss)).
-### Compute infrastructure
-Models were finetuned on [TPUv4](https://cloud.google.com/tpu/docs/system-architecture-tpu-vm#tpu_v4):
- - `mt0-small` was finetuned on TPUv4-64
- - `mt0-base` was finetuned on TPUv4-64
- - `mt0-large` was finetuned on TPUv4-64
- - `mt0-xl` was finetuned on TPUv4-128
- - `mt0-xxl` was finetuned on TPUv4-256
- - `mt0-mt-xxl` was finetuned on TPUv4-256
- - `mt0-p3-xxl` was finetuned on TPUv4-256
-#### Software
-* T5X([Github link](https://github.com/google-research/t5x), [paper](https://arxiv.org/abs/2203.17189))
 </details>
----
-# Training
-*This section provides information about the training data, the speed and size of training elements, and the environmental impact of training.*
-*It is useful for people who want to learn more about the model inputs and training footprint.*
 <details>
-<summary>Click to expand</summary>
-## Training Data
-*This section provides a high-level overview of the training data. It is relevant for anyone who wants to know the basics of what the model is learning.*
-It was pretrained on mC4 and then finetuned on xP3, P3 or xP3mt.
-### Languages
-// TODO @thomasw21: Copy list from mt5
-## Speeds, Sizes, Times
-// TODO @adarob: Maybe we can push tensorboard on this repo as well
-- Training logs:
-- Checkpoint size: 51.7GB (Bf16 weights)
-- Number of epochs: 1
-- Precision: bfloat16
-## Environmental Impact
-The evaluation supercomputer, [Jean Zay](http://www.idris.fr/eng/jean-zay/), uses mostly nuclear energy. The heat generated by it is reused for heating campus housing.
 </details>
----
-# Uses
-*This section addresses questions around how the model is intended to be used, discusses the foreseeable users of the model (including those affected by the model), and describes uses that are considered out of scope or misuse of the model.*
-*It is useful for anyone considering using the model or who is affected by the model.*
 <details>
-<summary>Click to expand</summary>
-## How to use
-This model can be easily used and deployed using HuggingFace's ecosystem. This needs `transformers` and `accelerate` installed. The model can be downloaded as follows:
 ```python
 from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
-checkpoint = "..." # "checkpoint_1006000" for example
-model_name = "bigscience/mt0-xxl"
-model = AutoModelForSeq2SeqLM.from_pretrained(model_name, revision=checkpoint, torch_dtype="auto", device_map="auto")
-tokenizer = AutoTokenizer.from_pretrained(model_name, revision=checkpoint)
-inputs = tokenizer.encode("Commentaire: C'est la meilleure crêpière que j'ai jamais eu. Je l'adore.\nCe commentaire est-il positif ou négatif?", return_tensors="pt")
 outputs = model.generate(inputs)
 print(tokenizer.decode(outputs[0]))
 ```
-## Intended Use
-This model is being created in order to enable public research on large language models (LLMs). LLMs are intended to be used for language generation or as a pretrained base model that can be further finetuned for specific tasks. Use cases below are not exhaustive.
-### Direct Use
--   Text generation
--   Exploring characteristics of language generated by a language model
-    -   Examples: Cloze tests, counterfactuals, generations with reframings
-### Downstream Use
--   Tasks that leverage language models include: Information Extraction, Question Answering, Summarization
 </details>
----
-# Risks and Limitations
-*This section identifies foreseeable harms and misunderstandings.*
-<details>
-<summary>Click to expand</summary>
-Model may:
--   Overrepresent some viewpoints and underrepresent others
--   Contain stereotypes
--   Contain [personal information](#personal-data-and-information)
--   Generate:
-    -   Hateful, abusive, or violent language
-    -   Discriminatory or prejudicial language
-    -   Content that may not be appropriate for all settings, including sexual content
--   Make errors, including producing incorrect information as if it were factual
--   Generate irrelevant or repetitive outputs
--   Induce users into attributing human traits to it, such as sentience or consciousness
-</details>
----
 # Evaluation
-*This section describes the evaluation protocols and provides the results.*
-<details>
-<summary>Click to expand</summary>
-##  Results
-*Results are based on the [Metrics](#metrics).*
-**Train-time Evaluation:**
-// TODO @adarob: Pending if we can get access to tensorboard
-**Zero-shot evaluations:**
-// TODO @niklas
-</details>
----
-# Recommendations
-*This section provides information on warnings and potential mitigations.*
-<details>
-<summary>Click to expand</summary>
--   Indirect users should be made aware when the content they're working with is created by the LLM.
--   Users should be aware of [Risks and Limitations](#risks-and-limitations), and include an appropriate age disclaimer or blocking interface as necessary.
--   Models trained or finetuned downstream of MT0 should include an updated Model Card.
--   Users of the model should provide mechanisms for those affected to provide feedback, such as an email address for comments.
-</details>
----
-# Glossary and Calculations
-*This section defines common terms and how metrics are calculated.*
-<details>
-<summary>Click to expand</summary>
--   <a name="loss">**Loss:**</a> A calculation of the difference between what the model has learned and what the data shows ("groundtruth"). The lower the loss, the better. The training process aims to minimize the loss.
--   <a name="perplexity">**Perplexity:**</a> This is based on what the model estimates the probability of new data is. The lower the perplexity, the better.  If the model is 100% correct at predicting the next token it will see, then the perplexity is 1. Mathematically this is calculated using entropy.
--   <a name="high-stakes">**High-stakes settings:**</a> Such as those identified as "high-risk AI systems" and "unacceptable risk AI systems" in the European Union's proposed [Artificial Intelligence (AI) Act](https://artificialintelligenceact.eu/annexes/).
--   <a name="critical-decisions">**Critical decisions:**</a> Such as those defined in [the United States' proposed Algorithmic Accountability Act](https://www.congress.gov/117/bills/s3572/BILLS-117s3572is.pdf).
--   <a name="human-rights">**Human rights:**</a> Includes those rights defined in the [Universal Declaration of Human Rights](https://www.un.org/sites/un2.un.org/files/2021/03/udhr.pdf).
--  <a name="personal-data-and-information">**Personal Data and Personal Information:**</a> Personal data and information is defined in multiple data protection regulations, such as "[personal data](https://gdpr-info.eu/issues/personal-data/)" in the [European Union's General Data Protection Regulation](https://gdpr-info.eu); and "personal information" in the Republic of South Africa's [Protection of Personal Information Act](https://www.gov.za/sites/default/files/gcis_document/201409/3706726-11act4of2013popi.pdf), The People's Republic of China's [Personal information protection law](http://en.npc.gov.cn.cdurl.cn/2021-12/29/c_694559.htm).
-- <a name="sensitive-characteristics">**Sensitive characteristics:**</a> This includes specifically protected categories in human rights (see [UHDR, Article 2](https://www.un.org/sites/un2.un.org/files/2021/03/udhr.pdf)) and personal information regulation (see GDPR, [Article 9; Protection of Personal Information Act, Chapter 1](https://www.gov.za/sites/default/files/gcis_document/201409/3706726-11act4of2013popi.pdf))
-- <a name="deception">**Deception:**</a> Doing something to intentionally mislead individuals to believe something that is false, such as by creating deadbots or chatbots on social media posing as real people, or generating text documents without making consumers aware that the text is machine generated.
-</details>
----
-# More Information
-*This section provides links to writing on dataset creation, technical specifications, lessons learned, and initial results.*
-<details>
-<summary>Click to expand</summary>
-## Intermediate checkpoints
-For academic (or any) usage, we published the intermediate checkpoints, corresponding to the model state at each 1000 steps. There are available as branches in this repository. You can use them using `transformers`:
-```python
-from transformers import AutoModel
-checkpoint = "..." # "checkpoint_1006000" for example
-model = AutoModel.from_pretrained("bigscience/mt0-xxl", revision=checkpoint, torch_dtype="auto", device_map="auto")
-```
-## Dataset Creation
-// TODO @niklas: Point to the arxiv paper
-## Original checkpoints
-The checkpoints in this repo correspond to the HuggingFace Transformers format. We'll provide T5X checkpoints as well.
-# Citing MT0
-Please use the following bibtex entry to cite T0:
 ```bibtex
-TODO @niklas
-```

 ---
+datasets:
+- bigscience/xP3mt
+- mc4
 license: apache-2.0
+language:
 - af
 - am
 - ar
 - yo
 - zh
 - zu
+pipeline_tag: text-generation
+widget:
+- text: "一个传奇的开端，一个不灭的神话，这不仅仅是一部电影，而是作为一个走进新时代的标签，永远彪炳史册。Would you rate the previous review as positive, neutral or negative?"
+  example_title: "zh-en sentiment"
+- text: "一个传奇的开端，一个不灭的神话，这不仅仅是一部电影，而是作为一个走进新时代的标签，永远彪炳史册。你认为这句话的立场是赞扬、中立还是批评？"
+  example_title: "zh-zh sentiment"
+- text: "Suggest at least five related search terms to \"Mạng neural nhân tạo\"."
+  example_title: "vi-en query"
+- text: "Proposez au moins cinq mots clés concernant «Réseau de neurones artificiels»."
+  example_title: "fr-fr query"
+- text: "Explain in a sentence in Telugu what is backpropagation in neural networks."
+  example_title: "te-en qa"
+- text: "Why is the sky blue?"
+  example_title: "en-en qa"
+- text: "Write a fairy tale about a troll saving a princess from a dangerous dragon. The fairy tale is a masterpiece that has achieved praise worldwide and its moral is \"Heroes Come in All Shapes and Sizes\". Story (in Spanish):"
+  example_title: "es-en fable"
+- text: "Write a fable about wood elves living in a forest that is suddenly invaded by ogres. The fable is a masterpiece that has achieved praise worldwide and its moral is \"Violence is the last refuge of the incompetent\". Fable (in Hindi):"
+  example_title: "hi-en fable"
+model-index:
+- name: mt0-xxl-mt
+  results:
+  - task:
+      type: Coreference resolution
+    dataset:
+      type: winogrande
+      name: Winogrande XL (xl)
+      config: xl
+      split: validation
+      revision: a80f460359d1e9a67c006011c94de42a8759430c
+    metrics:
+    - type: Accuracy
+      value: 62.67
+  - task:
+      type: Coreference resolution
+    dataset:
+      type: Muennighoff/xwinograd
+      name: XWinograd (en)
+      config: en
+      split: test
+      revision: 9dd5ea5505fad86b7bedad667955577815300cee
+    metrics:
+    - type: Accuracy
+      value: 83.31
+  - task:
+      type: Coreference resolution
+    dataset:
+      type: Muennighoff/xwinograd
+      name: XWinograd (fr)
+      config: fr
+      split: test
+      revision: 9dd5ea5505fad86b7bedad667955577815300cee
+    metrics:
+    - type: Accuracy
+      value: 78.31
+  - task:
+      type: Coreference resolution
+    dataset:
+      type: Muennighoff/xwinograd
+      name: XWinograd (jp)
+      config: jp
+      split: test
+      revision: 9dd5ea5505fad86b7bedad667955577815300cee
+    metrics:
+    - type: Accuracy
+      value: 80.19
+  - task:
+      type: Coreference resolution
+    dataset:
+      type: Muennighoff/xwinograd
+      name: XWinograd (pt)
+      config: pt
+      split: test
+      revision: 9dd5ea5505fad86b7bedad667955577815300cee
+    metrics:
+    - type: Accuracy
+      value: 80.99
+  - task:
+      type: Coreference resolution
+    dataset:
+      type: Muennighoff/xwinograd
+      name: XWinograd (ru)
+      config: ru
+      split: test
+      revision: 9dd5ea5505fad86b7bedad667955577815300cee
+    metrics:
+    - type: Accuracy
+      value: 79.05
+  - task:
+      type: Coreference resolution
+    dataset:
+      type: Muennighoff/xwinograd
+      name: XWinograd (zh)
+      config: zh
+      split: test
+      revision: 9dd5ea5505fad86b7bedad667955577815300cee
+    metrics:
+    - type: Accuracy
+      value: 82.34
+  - task:
+      type: Natural language inference
+    dataset:
+      type: anli
+      name: ANLI (r1)
+      config: r1
+      split: validation
+      revision: 9dbd830a06fea8b1c49d6e5ef2004a08d9f45094
+    metrics:
+    - type: Accuracy
+      value: 49.5
+  - task:
+      type: Natural language inference
+    dataset:
+      type: anli
+      name: ANLI (r2)
+      config: r2
+      split: validation
+      revision: 9dbd830a06fea8b1c49d6e5ef2004a08d9f45094
+    metrics:
+    - type: Accuracy
+      value: 42.0
+  - task:
+      type: Natural language inference
+    dataset:
+      type: anli
+      name: ANLI (r3)
+      config: r3
+      split: validation
+      revision: 9dbd830a06fea8b1c49d6e5ef2004a08d9f45094
+    metrics:
+    - type: Accuracy
+      value: 48.17
+  - task:
+      type: Natural language inference
+    dataset:
+      type: super_glue
+      name: SuperGLUE (cb)
+      config: cb
+      split: validation
+      revision: 9e12063561e7e6c79099feb6d5a493142584e9e2
+    metrics:
+    - type: Accuracy
+      value: 87.5
+  - task:
+      type: Natural language inference
+    dataset:
+      type: super_glue
+      name: SuperGLUE (rte)
+      config: rte
+      split: validation
+      revision: 9e12063561e7e6c79099feb6d5a493142584e9e2
+    metrics:
+    - type: Accuracy
+      value: 84.84
+  - task:
+      type: Natural language inference
+    dataset:
+      type: xnli
+      name: XNLI (ar)
+      config: ar
+      split: validation
+      revision: a5a45e4ff92d5d3f34de70aaf4b72c3bdf9f7f16
+    metrics:
+    - type: Accuracy
+      value: 58.03
+  - task:
+      type: Natural language inference
+    dataset:
+      type: xnli
+      name: XNLI (bg)
+      config: bg
+      split: validation
+      revision: a5a45e4ff92d5d3f34de70aaf4b72c3bdf9f7f16
+    metrics:
+    - type: Accuracy
+      value: 59.92
+  - task:
+      type: Natural language inference
+    dataset:
+      type: xnli
+      name: XNLI (de)
+      config: de
+      split: validation
+      revision: a5a45e4ff92d5d3f34de70aaf4b72c3bdf9f7f16
+    metrics:
+    - type: Accuracy
+      value: 60.16
+  - task:
+      type: Natural language inference
+    dataset:
+      type: xnli
+      name: XNLI (el)
+      config: el
+      split: validation
+      revision: a5a45e4ff92d5d3f34de70aaf4b72c3bdf9f7f16
+    metrics:
+    - type: Accuracy
+      value: 59.2
+  - task:
+      type: Natural language inference
+    dataset:
+      type: xnli
+      name: XNLI (en)
+      config: en
+      split: validation
+      revision: a5a45e4ff92d5d3f34de70aaf4b72c3bdf9f7f16
+    metrics:
+    - type: Accuracy
+      value: 62.25
+  - task:
+      type: Natural language inference
+    dataset:
+      type: xnli
+      name: XNLI (es)
+      config: es
+      split: validation
+      revision: a5a45e4ff92d5d3f34de70aaf4b72c3bdf9f7f16
+    metrics:
+    - type: Accuracy
+      value: 60.92
+  - task:
+      type: Natural language inference
+    dataset:
+      type: xnli
+      name: XNLI (fr)
+      config: fr
+      split: validation
+      revision: a5a45e4ff92d5d3f34de70aaf4b72c3bdf9f7f16
+    metrics:
+    - type: Accuracy
+      value: 59.88
+  - task:
+      type: Natural language inference
+    dataset:
+      type: xnli
+      name: XNLI (hi)
+      config: hi
+      split: validation
+      revision: a5a45e4ff92d5d3f34de70aaf4b72c3bdf9f7f16
+    metrics:
+    - type: Accuracy
+      value: 57.47
+  - task:
+      type: Natural language inference
+    dataset:
+      type: xnli
+      name: XNLI (ru)
+      config: ru
+      split: validation
+      revision: a5a45e4ff92d5d3f34de70aaf4b72c3bdf9f7f16
+    metrics:
+    - type: Accuracy
+      value: 58.67
+  - task:
+      type: Natural language inference
+    dataset:
+      type: xnli
+      name: XNLI (sw)
+      config: sw
+      split: validation
+      revision: a5a45e4ff92d5d3f34de70aaf4b72c3bdf9f7f16
+    metrics:
+    - type: Accuracy
+      value: 56.79
+  - task:
+      type: Natural language inference
+    dataset:
+      type: xnli
+      name: XNLI (th)
+      config: th
+      split: validation
+      revision: a5a45e4ff92d5d3f34de70aaf4b72c3bdf9f7f16
+    metrics:
+    - type: Accuracy
+      value: 58.03
+  - task:
+      type: Natural language inference
+    dataset:
+      type: xnli
+      name: XNLI (tr)
+      config: tr
+      split: validation
+      revision: a5a45e4ff92d5d3f34de70aaf4b72c3bdf9f7f16
+    metrics:
+    - type: Accuracy
+      value: 57.67
+  - task:
+      type: Natural language inference
+    dataset:
+      type: xnli
+      name: XNLI (ur)
+      config: ur
+      split: validation
+      revision: a5a45e4ff92d5d3f34de70aaf4b72c3bdf9f7f16
+    metrics:
+    - type: Accuracy
+      value: 55.98
+  - task:
+      type: Natural language inference
+    dataset:
+      type: xnli
+      name: XNLI (vi)
+      config: vi
+      split: validation
+      revision: a5a45e4ff92d5d3f34de70aaf4b72c3bdf9f7f16
+    metrics:
+    - type: Accuracy
+      value: 58.92
+  - task:
+      type: Natural language inference
+    dataset:
+      type: xnli
+      name: XNLI (zh)
+      config: zh
+      split: validation
+      revision: a5a45e4ff92d5d3f34de70aaf4b72c3bdf9f7f16
+    metrics:
+    - type: Accuracy
+      value: 58.71
+  - task:
+      type: Sentence completion
+    dataset:
+      type: story_cloze
+      name: StoryCloze (2016)
+      config: "2016"
+      split: validation
+      revision: e724c6f8cdf7c7a2fb229d862226e15b023ee4db
+    metrics:
+    - type: Accuracy
+      value: 94.66
+  - task:
+      type: Sentence completion
+    dataset:
+      type: super_glue
+      name: SuperGLUE (copa)
+      config: copa
+      split: validation
+      revision: 9e12063561e7e6c79099feb6d5a493142584e9e2
+    metrics:
+    - type: Accuracy
+      value: 88.0
+  - task:
+      type: Sentence completion
+    dataset:
+      type: xcopa
+      name: XCOPA (et)
+      config: et
+      split: validation
+      revision: 37f73c60fb123111fa5af5f9b705d0b3747fd187
+    metrics:
+    - type: Accuracy
+      value: 81.0
+  - task:
+      type: Sentence completion
+    dataset:
+      type: xcopa
+      name: XCOPA (ht)
+      config: ht
+      split: validation
+      revision: 37f73c60fb123111fa5af5f9b705d0b3747fd187
+    metrics:
+    - type: Accuracy
+      value: 79.0
+  - task:
+      type: Sentence completion
+    dataset:
+      type: xcopa
+      name: XCOPA (id)
+      config: id
+      split: validation
+      revision: 37f73c60fb123111fa5af5f9b705d0b3747fd187
+    metrics:
+    - type: Accuracy
+      value: 90.0
+  - task:
+      type: Sentence completion
+    dataset:
+      type: xcopa
+      name: XCOPA (it)
+      config: it
+      split: validation
+      revision: 37f73c60fb123111fa5af5f9b705d0b3747fd187
+    metrics:
+    - type: Accuracy
+      value: 88.0
+  - task:
+      type: Sentence completion
+    dataset:
+      type: xcopa
+      name: XCOPA (qu)
+      config: qu
+      split: validation
+      revision: 37f73c60fb123111fa5af5f9b705d0b3747fd187
+    metrics:
+    - type: Accuracy
+      value: 56.0
+  - task:
+      type: Sentence completion
+    dataset:
+      type: xcopa
+      name: XCOPA (sw)
+      config: sw
+      split: validation
+      revision: 37f73c60fb123111fa5af5f9b705d0b3747fd187
+    metrics:
+    - type: Accuracy
+      value: 81.0
+  - task:
+      type: Sentence completion
+    dataset:
+      type: xcopa
+      name: XCOPA (ta)
+      config: ta
+      split: validation
+      revision: 37f73c60fb123111fa5af5f9b705d0b3747fd187
+    metrics:
+    - type: Accuracy
+      value: 81.0
+  - task:
+      type: Sentence completion
+    dataset:
+      type: xcopa
+      name: XCOPA (th)
+      config: th
+      split: validation
+      revision: 37f73c60fb123111fa5af5f9b705d0b3747fd187
+    metrics:
+    - type: Accuracy
+      value: 76.0
+  - task:
+      type: Sentence completion
+    dataset:
+      type: xcopa
+      name: XCOPA (tr)
+      config: tr
+      split: validation
+      revision: 37f73c60fb123111fa5af5f9b705d0b3747fd187
+    metrics:
+    - type: Accuracy
+      value: 76.0
+  - task:
+      type: Sentence completion
+    dataset:
+      type: xcopa
+      name: XCOPA (vi)
+      config: vi
+      split: validation
+      revision: 37f73c60fb123111fa5af5f9b705d0b3747fd187
+    metrics:
+    - type: Accuracy
+      value: 85.0
+  - task:
+      type: Sentence completion
+    dataset:
+      type: xcopa
+      name: XCOPA (zh)
+      config: zh
+      split: validation
+      revision: 37f73c60fb123111fa5af5f9b705d0b3747fd187
+    metrics:
+    - type: Accuracy
+      value: 87.0
+  - task:
+      type: Sentence completion
+    dataset:
+      type: Muennighoff/xstory_cloze
+      name: XStoryCloze (ar)
+      config: ar
+      split: validation
+      revision: 8bb76e594b68147f1a430e86829d07189622b90d
+    metrics:
+    - type: Accuracy
+      value: 91.0
+  - task:
+      type: Sentence completion
+    dataset:
+      type: Muennighoff/xstory_cloze
+      name: XStoryCloze (es)
+      config: es
+      split: validation
+      revision: 8bb76e594b68147f1a430e86829d07189622b90d
+    metrics:
+    - type: Accuracy
+      value: 93.38
+  - task:
+      type: Sentence completion
+    dataset:
+      type: Muennighoff/xstory_cloze
+      name: XStoryCloze (eu)
+      config: eu
+      split: validation
+      revision: 8bb76e594b68147f1a430e86829d07189622b90d
+    metrics:
+    - type: Accuracy
+      value: 91.13
+  - task:
+      type: Sentence completion
+    dataset:
+      type: Muennighoff/xstory_cloze
+      name: XStoryCloze (hi)
+      config: hi
+      split: validation
+      revision: 8bb76e594b68147f1a430e86829d07189622b90d
+    metrics:
+    - type: Accuracy
+      value: 90.73
+  - task:
+      type: Sentence completion
+    dataset:
+      type: Muennighoff/xstory_cloze
+      name: XStoryCloze (id)
+      config: id
+      split: validation
+      revision: 8bb76e594b68147f1a430e86829d07189622b90d
+    metrics:
+    - type: Accuracy
+      value: 93.05
+  - task:
+      type: Sentence completion
+    dataset:
+      type: Muennighoff/xstory_cloze
+      name: XStoryCloze (my)
+      config: my
+      split: validation
+      revision: 8bb76e594b68147f1a430e86829d07189622b90d
+    metrics:
+    - type: Accuracy
+      value: 86.7
+  - task:
+      type: Sentence completion
+    dataset:
+      type: Muennighoff/xstory_cloze
+      name: XStoryCloze (ru)
+      config: ru
+      split: validation
+      revision: 8bb76e594b68147f1a430e86829d07189622b90d
+    metrics:
+    - type: Accuracy
+      value: 91.66
+  - task:
+      type: Sentence completion
+    dataset:
+      type: Muennighoff/xstory_cloze
+      name: XStoryCloze (sw)
+      config: sw
+      split: validation
+      revision: 8bb76e594b68147f1a430e86829d07189622b90d
+    metrics:
+    - type: Accuracy
+      value: 89.61
+  - task:
+      type: Sentence completion
+    dataset:
+      type: Muennighoff/xstory_cloze
+      name: XStoryCloze (te)
+      config: te
+      split: validation
+      revision: 8bb76e594b68147f1a430e86829d07189622b90d
+    metrics:
+    - type: Accuracy
+      value: 90.4
+  - task:
+      type: Sentence completion
+    dataset:
+      type: Muennighoff/xstory_cloze
+      name: XStoryCloze (zh)
+      config: zh
+      split: validation
+      revision: 8bb76e594b68147f1a430e86829d07189622b90d
+    metrics:
+    - type: Accuracy
+      value: 93.05
 ---
+![xmtf](https://github.com/bigscience-workshop/xmtf/blob/master/xmtf_banner.png?raw=true)
+#  Table of Contents
+1. [Model Summary](#model-summary)
+2. [Use](#use)
+3. [Limitations](#limitations)
+4. [Training](#training)
+5. [Evaluation](#evaluation)
+7. [Citation](#citation)
+# Model Summary
+> We present BLOOMZ & mT0, a family of models capable of following human instructions in dozens of languages zero-shot. We finetune BLOOM & mT5 pretrained multilingual language models on our crosslingual task mixture (xP3) and find our resulting models capable of crosslingual generalization to unseen tasks & languages.
+- **Repository:** [bigscience-workshop/xmtf](https://github.com/bigscience-workshop/xmtf)
+- **Paper:** [Crosslingual Generalization through Multitask Finetuning](https://arxiv.org/abs/2211.01786)
+- **Point of Contact:** [Niklas Muennighoff](mailto:[email protected])
+- **Languages:** Refer to [mc4](https://huggingface.co/datasets/mc4) for pretraining & [xP3](https://huggingface.co/bigscience/xP3) for finetuning language proportions. It understands both pretraining & finetuning languages.
+- **BLOOMZ & mT0 Model Family:**
+<table>
+  <tr>
+<th colspan="12">Multitask finetuned on <a style="font-weight:bold" href=https://huggingface.co/datasets/bigscience/xP3>xP3</a>. Recommended for prompting in English.
+</tr>
+<tr>
+<td>Parameters</td>
+<td>300M</td>
+<td>580M</td>
+<td>1.2B</td>
+<td>3.7B</td>
+<td>13B</td>
+<td>560M</td>
+<td>1.1B</td>
+<td>1.7B</td>
+<td>3B</td>
+<td>7.1B</td>
+<td>176B</td>
+</tr>
+<tr>
+<td>Finetuned Model</td>
+<td><a href=https://huggingface.co/bigscience/mt0-base>mt0-base</a></td>
+<td><a href=https://huggingface.co/bigscience/mt0-small>mt0-small</a></td>
+<td><a href=https://huggingface.co/bigscience/mt0-large>mt0-large</a></td>
+<td><a href=https://huggingface.co/bigscience/mt0-xl>mt0-xl</a></td>
+<td><a href=https://huggingface.co/bigscience/mt0-xxl>mt0-xxl</a></td>
+<td><a href=https://huggingface.co/bigscience/bloomz-560m>bloomz-560m</a></td>
+<td><a href=https://huggingface.co/bigscience/bloomz-1b1>bloomz-1b1</a></td>
+<td><a href=https://huggingface.co/bigscience/bloomz-1b7>bloomz-1b7</a></td>
+<td><a href=https://huggingface.co/bigscience/bloomz-3b>bloomz-3b</a></td>
+<td><a href=https://huggingface.co/bigscience/bloomz-7b1>bloomz-7b1</a></td>
+<td><a href=https://huggingface.co/bigscience/bloomz>bloomz</a></td>
+</tr>
+</tr>
+  <tr>
+<th colspan="12">Multitask finetuned on <a style="font-weight:bold" href=https://huggingface.co/datasets/bigscience/xP3mt>xP3mt</a>. Recommended for prompting in non-English.</th>
+</tr>
+<tr>
+<td>Finetuned Model</td>
+<td></td>
+<td></td>
+<td></td>
+<td></td>
+<td><a href=https://huggingface.co/bigscience/mt0-xxl-mt>mt0-xxl-mt</a></td>
+<td></td>
+<td></td>
+<td></td>
+<td></td>
+<td><a href=https://huggingface.co/bigscience/bloomz-7b1-mt>bloomz-7b1-mt</a></td>
+<td><a href=https://huggingface.co/bigscience/bloomz-mt>bloomz-mt</a></td>
+</tr>
+<th colspan="12">Multitask finetuned on <a style="font-weight:bold" href=https://huggingface.co/datasets/Muennighoff/P3>P3</a>. Released for research purposes only. Strictly inferior to above models!</th>
+</tr>
+<tr>
+<td>Finetuned Model</td>
+<td></td>
+<td></td>
+<td></td>
+<td></td>
+<td><a href=https://huggingface.co/bigscience/mt0-xxl-p3>mt0-xxl-p3</a></td>
+<td></td>
+<td></td>
+<td></td>
+<td></td>
+<td><a href=https://huggingface.co/bigscience/bloomz-7b1-p3>bloomz-7b1-p3</a></td>
+<td><a href=https://huggingface.co/bigscience/bloomz-p3>bloomz-p3</a></td>
+</tr>
+<th colspan="12">Original pretrained checkpoints. Not recommended.</th>
+<tr>
+<td>Pretrained Model</td>
+<td><a href=https://huggingface.co/google/mt5-base>mt5-base</a></td>
+<td><a href=https://huggingface.co/google/mt5-small>mt5-small</a></td>
+<td><a href=https://huggingface.co/google/mt5-large>mt5-large</a></td>
+<td><a href=https://huggingface.co/google/mt5-xl>mt5-xl</a></td>
+<td><a href=https://huggingface.co/google/mt5-xxl>mt5-xxl</a></td>
+<td><a href=https://huggingface.co/bigscience/bloom-560m>bloom-560m</a></td>
+<td><a href=https://huggingface.co/bigscience/bloom-1b1>bloom-1b1</a></td>
+<td><a href=https://huggingface.co/bigscience/bloom-1b7>bloom-1b7</a></td>
+<td><a href=https://huggingface.co/bigscience/bloom-3b>bloom-3b</a></td>
+<td><a href=https://huggingface.co/bigscience/bloom-7b1>bloom-7b1</a></td>
+<td><a href=https://huggingface.co/bigscience/bloom>bloom</a></td>
+</tr>
+</table>
+# Use
+## Intended use
+We recommend using the model to perform tasks expressed in natural language. For example, given the prompt "*Translate to English: Je t’aime.*", the model will most likely answer "*I love you.*". Some prompt ideas from our paper:
+- 一个传奇的开端，一个不灭的神话，这不仅仅是一部电影，而是作为一个走进新时代的标签，永远彪炳史册。你认为这句话的立场是赞扬、中立还是批评?
+- Suggest at least five related search terms to "Mạng neural nhân tạo".
+- Write a fairy tale about a troll saving a princess from a dangerous dragon. The fairy tale is a masterpiece that has achieved praise worldwide and its moral is "Heroes Come in All Shapes and Sizes". Story (in Spanish):
+- Explain in a sentence in Telugu what is backpropagation in neural networks.
+**Feel free to share your generations in the Community tab!**
+## How to use
+### CPU
 <details>
+<summary> Click to expand </summary>
+```python
+# pip install -q transformers
+from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
+checkpoint = "bigscience/mt0-xxl-mt"
+tokenizer = AutoTokenizer.from_pretrained(checkpoint)
+model = AutoModelForSeq2SeqLM.from_pretrained(checkpoint)
+inputs = tokenizer.encode("Translate to English: Je t’aime.", return_tensors="pt")
+outputs = model.generate(inputs)
+print(tokenizer.decode(outputs[0]))
+```
 </details>
+### GPU
 <details>
+<summary> Click to expand </summary>
+```python
+# pip install -q transformers accelerate
+from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
+checkpoint = "bigscience/mt0-xxl-mt"
+tokenizer = AutoTokenizer.from_pretrained(checkpoint)
+model = AutoModelForSeq2SeqLM.from_pretrained(checkpoint, torch_dtype="auto", device_map="auto")
+inputs = tokenizer.encode("Translate to English: Je t’aime.", return_tensors="pt").to("cuda")
+outputs = model.generate(inputs)
+print(tokenizer.decode(outputs[0]))
+```
 </details>
+### GPU in 8bit
 <details>
+<summary> Click to expand </summary>
 ```python
+# pip install -q transformers accelerate bitsandbytes
 from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
+checkpoint = "bigscience/mt0-xxl-mt"
+tokenizer = AutoTokenizer.from_pretrained(checkpoint)
+model = AutoModelForSeq2SeqLM.from_pretrained(checkpoint, device_map="auto", load_in_8bit=True)
+inputs = tokenizer.encode("Translate to English: Je t’aime.", return_tensors="pt").to("cuda")
 outputs = model.generate(inputs)
 print(tokenizer.decode(outputs[0]))
 ```
 </details>
+<!-- Necessary for whitespace -->
+###
+# Limitations
+**Prompt Engineering:** The performance may vary depending on the prompt. For BLOOMZ models, we recommend making it very clear when the input stops to avoid the model trying to continue it. For example, the prompt "*Translate to English: Je t'aime*" without the full stop (.) at the end, may result in the model trying to continue the French sentence. Better prompts are e.g. "*Translate to English: Je t'aime.*", "*Translate to English: Je t'aime. Translation:*" "*What is "Je t'aime." in English?*", where it is clear for the model when it should answer. Further, we recommend providing the model as much context as possible. For example, if you want it to answer in Telugu, then tell the model, e.g. "*Explain in a sentence in Telugu what is backpropagation in neural networks.*".
+# Training
+## Model
+- **Architecture:** Same as [mt5-xxl](https://huggingface.co/google/mt5-xxl), also refer to the `config.json` file
+- **Finetuning steps:** 7000
+- **Finetuning tokens:** 1.29 billion
+- **Precision:** bfloat16
+## Hardware
+- **TPUs:** TPUv4-256
+## Software
+- **Orchestration:** [T5X](https://github.com/google-research/t5x)
+- **Neural networks:** [Jax](https://github.com/google/jax)
 # Evaluation
+We refer to Table 7 from our [paper](https://arxiv.org/abs/2211.01786) & [bigscience/evaluation-results](https://huggingface.co/datasets/bigscience/evaluation-results) for zero-shot results on unseen tasks. The sidebar reports zero-shot performance of the best prompt per dataset config.
+# Citation
 ```bibtex
+@misc{muennighoff2022crosslingual,
+      title={Crosslingual Generalization through Multitask Finetuning},
+      author={Niklas Muennighoff and Thomas Wang and Lintang Sutawika and Adam Roberts and Stella Biderman and Teven Le Scao and M Saiful Bari and Sheng Shen and Zheng-Xin Yong and Hailey Schoelkopf and Xiangru Tang and Dragomir Radev and Alham Fikri Aji and Khalid Almubarak and Samuel Albanie and Zaid Alyafeai and Albert Webson and Edward Raff and Colin Raffel},
+      year={2022},
+      eprint={2211.01786},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL}
+}
+```