File size: 5,033 Bytes
f5cb629 692dbf7 f5cb629 692dbf7 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 |
---
license: apache-2.0
datasets:
- shareAI/ShareGPT-Chinese-English-90k
language:
- zh
- en
pipeline_tag: text-generation
---
<div align="center">
<h2>
Aurora: Activating chinese chat capability for Mistral-8x7B sparse Mixture-of-Experts through Instruction-Tuning
</h2>
</div>
1. <h1>Please follow our Github: <a href="https://github.com/WangRongsheng/Aurora">https://github.com/WangRongsheng/Aurora</a></h1>
2. <h1>Please follow our Paper: <a href="https://arxiv.org/abs/2312.14557">https://arxiv.org/abs/2312.14557</a></h1>
## Overview
Existing research has demonstrated that refining large language models (LLMs) through the utilization of machine-generated instruction-following data empowers these models to exhibit impressive zero-shot capabilities for novel tasks, without requiring human-authored instructions. In this paper, we systematically investigate, preprocess, and integrate three Chinese instruction-following datasets with the aim of enhancing the Chinese conversational capabilities of Mixtral-8x7B sparse Mixture-of-Experts model. Through instruction fine-tuning on this carefully processed dataset, we successfully construct the Mixtral-8x7B sparse Mixture-of-Experts model named "Aurora." To assess the performance of Aurora, we utilize three widely recognized benchmark tests: C-Eval, MMLU, and CMMLU. Empirical studies validate the effectiveness of instruction fine-tuning applied to Mixtral-8x7B sparse Mixture-of-Experts model. This work is pioneering in the execution of instruction fine-tuning on a sparse expert-mixed model, marking a significant breakthrough in enhancing the capabilities of this model architecture.
## Usage
```python
import gradio as gr
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, StoppingCriteria, StoppingCriteriaList, TextIteratorStreamer
from threading import Thread
from peft import PeftModel
import time
model_name_or_path = "mistralai/Mixtral-8x7B-Instruct-v0.1" # download weights from https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1
lora_weights = "wangrongsheng/Aurora" # download weights from https://huggingface.co/wangrongsheng/Aurora
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)
model0 = AutoModelForCausalLM.from_pretrained(model_name_or_path, load_in_4bit=True, device_map="auto", torch_dtype=torch.bfloat16)
model = PeftModel.from_pretrained(
model0,
lora_weights,
)
class StopOnTokens(StoppingCriteria):
def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor, **kwargs) -> bool:
stop_ids = [0,]
for stop_id in stop_ids:
if input_ids[0][-1] == stop_id:
return True
return False
def convert_history_to_text(history):
text = ""
if len(history) > 1:
text = "<s> " + "".join(
[
"".join(
[
f"[INST]{item[0]}[/INST] {item[1]} ",
]
)
for item in history[:-1]
]
) + "</s> "
text += "".join(
[
"".join(
[
f"[INST]{history[-1][0]}[/INST]",
]
)
]
)
return text
def predict(message, history):
history_transformer_format = history + [[message, ""]]
stop = StopOnTokens()
messages = convert_history_to_text(history_transformer_format)
model_inputs = tokenizer([messages], return_tensors="pt").to("cuda")
streamer = TextIteratorStreamer(tokenizer, timeout=10., skip_prompt=True, skip_special_tokens=True)
generate_kwargs = dict(
model_inputs,
streamer=streamer,
max_new_tokens=4096,
do_sample=True,
top_p=0.95,
top_k=1000,
temperature=1.0,
num_beams=1,
pad_token_id=tokenizer.eos_token_id,
stopping_criteria=StoppingCriteriaList([stop])
)
t = Thread(target=model.generate, kwargs=generate_kwargs)
t.start()
partial_message = ""
t1 = time.time()
count = 0
for new_token in streamer:
if new_token != '<':
partial_message += new_token
count += 1
yield partial_message
t2 = time.time()
speed = count/(t2-t1)
print("inference speed: %f tok/s" % speed)
gr.ChatInterface(predict,chatbot=gr.Chatbot(height=600,),title="MoE").queue().launch()
```
## Citation
If you find our work helpful, feel free to give us a cite.
```latex
@misc{wang2023auroraactivating,
title={Aurora:Activating Chinese chat capability for Mixtral-8x7B sparse Mixture-of-Experts through Instruction-Tuning},
author={Rongsheng Wang and Haoming Chen and Ruizhe Zhou and Yaofei Duan and Kunyan Cai and Han Ma and Jiaxi Cui and Jian Li and Patrick Cheong-Iao Pang and Yapeng Wang and Tao Tan},
year={2023},
eprint={2312.14557},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
``` |