
# MT-LLaMA Model Card

## Model details

**Model type:**
MT-LLaMA is an open-source multi-task model trained by fine-tuning LLaMA on the massive tasks in [P3](https://huggingface.co/datasets/bigscience/P3) (i.e., T0 Train). Concretely, the used datasets during training and task taxonomy are listed below:
* Multi-choice QA: CommonsenseQA, Cosmos QA, DREAM, QuAIL, QuaRTz, QASC, QuaRel, SciQ, Social IQA, Wiki Hop, WiQA  
* Extractive QA: Adversarial QA, DuoRC, Quoref, ROPES  
* Close-Book QA: Hotpot QA, Wiki QA  
* Sentiment Classification: Amazon, App Reviews, IMDB, Rotten Tomatoes, Yelp  
* Topic Classification: AG News, DBPedia, TREC  
* Structure-to-Text Generation: Common Gen, Wiki Bio  
* Text Summarization: CNN Daily Mail, Gigaword, MultiNews, SamSum, XSum  
* Paraphrase Identification: MRPC, PAWS, QQP  

**Organizations developing the model:**
The MT-LLaMA team with members from Alibaba Damo Academy and the Chinese University of Hong Kong.

## Intended use

You can try the codes from our [github repo](https://github.com/DAMO-NLP-SG/MT-LLaMA).

                                          
## Zero-shot Evaluation

We primarily follow the protocols of [Bigscience T0](https://openreview.net/forum?id=9Vrb9D0WI4) to assess the generalization capability of our Multi-task LLaMA to: (1) _**Unseen Datasets**_ (i.e., datasets from seen tasks); (2) _**Unseen Tasks**_.
                                                     
#### Prompt Format

Extractive QA:

1. XQuAD, TyDiQA, MLQA, SQuAD
   ```angular2html
    Input: Answer the question according to the context. Question: ${question}. Context: ${context}. Answer:
    Output: ${Answer}
   ```

Sentiment:

1. SST-2
   ```angular2html
   Input: ${sentence} Based on this review, would the user recommend this product? No or Yes?
   Output: Yes / No
   ```
Multiple-Choice QA:

1. OpenbookQA
   ```angular2html
   Input: ${question} Which is the correct answer? - (A) ${choiceA} - (B) ${choiceB} - (C) ${choiceC} - (D) ${choiceD}
   Output: ${choiceA} / ${choiceB} / ${choiceC} / ${choiceD}
   ```
Sentence Completion:

1. COPA
   ```angular2html
   Input: ${premise} {% if question == "cause" %} This happened because... {% else %} As a consequence... Help me pick the more plausible option: - ${text1} - ${text2}
   Output: ${text1} / ${text2}
   ```
Coreference Resolution:
1. Winogrande:
   ```angular2html    
   Input: ${sentence} In the previous sentence, does _ refer to ${option1} or ${option2}?
   Output: ${option1} / ${option2}
   ```
Word Sense Disambiguation:
1. WiC
   ```angular2html
   Input: Does the word "${word}" have the same meaning in these two sentences? Yes, No? ${sentence1} ${sentence2}
   Output: ${sentence1} / ${sentence2}
   ```
Natural Language Inference:

1. MNLI:
   ```angular2html
   Input: ${premise} Question: Does this imply that ${hypothesis}? Please response with 'Yes', 'No', or 'Maybe'.
   Output: Yes / No / Maybe
   ```
2. RTE
   ```angular2html  
   Input: Given ${premise} Is it guaranteed true that "${hypothesis}"? Yes or no?
   Output: Yes / no
   ```
#### Results on _Unseen Datasets_

| Model       | XQuAD-en (F1/EM) | TyDiQA-en (F1/EM) | MLQA-en (F1/EM) | SQuAD (F1/EM) | SST-2 (Acc.) | OpenbookQA (Acc.) |
|:------------|------------------|-------------------|-----------------|---------------|--------------|-------------------|
| LLaMA-7b    | 9.5 / 2.0        | 14.3 / 2.6        | 13.4 / 3.3      | 29.4 / 11.5   | 50.5         | 32.4              |
| MT-LLaMA-7b | 42.3 / 31.1      | 38.9 / 26.9       | 45.4 / 31.5     | 85.9 / 77.6   | 92.6         | 38.2              |
#### Results on _Unseen Tasks_                                                     
| Model       | COPA (Acc.) | Winogrande (Acc.)  | WiC (Acc.) | MNLI (Acc.) | RTE (Acc.) |
|:------------|-------------|--------------------|------------|-------------|------------|
| LLaMA-7b    | 56.0        | 49.3               | 51.7       | 30.2        | 52.7       |
| MT-LLaMA-7b | 88.0        | 54.9               | 52.2       | 49.6        | 79.1       |

## Acknowledgement

* Our training codes are largely borrowed from [FastChat](https://github.com/lm-sys/FastChat)
* We are also grateful for the efforts of [LLaMA](https://github.com/facebookresearch/llama) (from FAIR) and [T0](https://github.com/bigscience-workshop/t-zero) (from BigScience), which serve as the foundation of our work 

If you find this resource useful, please cite the repo as follows:
```
@software{damonlpsg2023mtllama,
  author = {Xu, Weiwen and Li, Xin and Bing, Lidong},
  title = {Multi-task Instruction-tuned LLaMA},
  year = 2023,
  url = {https://github.com/DAMO-NLP-SG/MT-LLaMA}
}
```

{"license": "mit"}
transformers pytorch llama text-generation license:mit autotrain_compatible endpoints_compatible text-generation-inference region:us
TAGS
#transformers #pytorch #llama #text-generation #license-mit #autotrain_compatible #endpoints_compatible #text-generation-inference #region-us 

MT-LLaMA Model Card
===================


Model details
-------------


Model type:
MT-LLaMA is an open-source multi-task model trained by fine-tuning LLaMA on the massive tasks in P3 (i.e., T0 Train). Concretely, the used datasets during training and task taxonomy are listed below:


* Multi-choice QA: CommonsenseQA, Cosmos QA, DREAM, QuAIL, QuaRTz, QASC, QuaRel, SciQ, Social IQA, Wiki Hop, WiQA
* Extractive QA: Adversarial QA, DuoRC, Quoref, ROPES
* Close-Book QA: Hotpot QA, Wiki QA
* Sentiment Classification: Amazon, App Reviews, IMDB, Rotten Tomatoes, Yelp
* Topic Classification: AG News, DBPedia, TREC
* Structure-to-Text Generation: Common Gen, Wiki Bio
* Text Summarization: CNN Daily Mail, Gigaword, MultiNews, SamSum, XSum
* Paraphrase Identification: MRPC, PAWS, QQP


Organizations developing the model:
The MT-LLaMA team with members from Alibaba Damo Academy and the Chinese University of Hong Kong.


Intended use
------------


You can try the codes from our github repo.


Zero-shot Evaluation
--------------------


We primarily follow the protocols of Bigscience T0 to assess the generalization capability of our Multi-task LLaMA to: (1) *Unseen Datasets* (i.e., datasets from seen tasks); (2) *Unseen Tasks*.


#### Prompt Format


Extractive QA:


1. XQuAD, TyDiQA, MLQA, SQuAD


Sentiment:


1. SST-2


Multiple-Choice QA:


1. OpenbookQA


Sentence Completion:


1. COPA


Coreference Resolution:


1. Winogrande:


Word Sense Disambiguation:


1. WiC


Natural Language Inference:


1. MNLI:
2. RTE


#### Results on *Unseen Datasets*



#### Results on *Unseen Tasks*



Acknowledgement
---------------


* Our training codes are largely borrowed from FastChat
* We are also grateful for the efforts of LLaMA (from FAIR) and T0 (from BigScience), which serve as the foundation of our work


If you find this resource useful, please cite the repo as follows:



#### Prompt Format


Extractive QA:


1. XQuAD, TyDiQA, MLQA, SQuAD


Sentiment:


1. SST-2


Multiple-Choice QA:


1. OpenbookQA


Sentence Completion:


1. COPA


Coreference Resolution:


1. Winogrande:


Word Sense Disambiguation:


1. WiC


Natural Language Inference:


1. MNLI:
2. RTE #### Results on *Unseen Datasets* #### Results on *Unseen Tasks*



Acknowledgement
---------------


* Our training codes are largely borrowed from FastChat
* We are also grateful for the efforts of LLaMA (from FAIR) and T0 (from BigScience), which serve as the foundation of our work


If you find this resource useful, please cite the repo as follows:
TAGS
#transformers #pytorch #llama #text-generation #license-mit #autotrain_compatible #endpoints_compatible #text-generation-inference #region-us 
 #### Prompt Format


Extractive QA:


1. XQuAD, TyDiQA, MLQA, SQuAD


Sentiment:


1. SST-2


Multiple-Choice QA:


1. OpenbookQA


Sentence Completion:


1. COPA


Coreference Resolution:


1. Winogrande:


Word Sense Disambiguation:


1. WiC


Natural Language Inference:


1. MNLI:
2. RTE #### Results on *Unseen Datasets* #### Results on *Unseen Tasks*



Acknowledgement
---------------


* Our training codes are largely borrowed from FastChat
* We are also grateful for the efforts of LLaMA (from FAIR) and T0 (from BigScience), which serve as the foundation of our work


If you find this resource useful, please cite the repo as follows: