|
--- |
|
library_name: peft |
|
pipeline_tag: conversational |
|
datasets: |
|
- fnlp/moss-003-sft-data |
|
--- |
|
|
|
<div align="center"> |
|
<img src="https://github.com/InternLM/lmdeploy/assets/36994684/0cf8d00f-e86b-40ba-9b54-dc8f1bc6c8d8" width="600"/> |
|
|
|
|
|
[](https://github.com/InternLM/xtuner) |
|
|
|
|
|
</div> |
|
|
|
## Model |
|
|
|
Qwen-7B-qlora-moss-003-sft is fine-tuned from [Qwen-7B](https://huggingface.co/Qwen/Qwen-7B) with [moss-003-sft](https://huggingface.co/datasets/fnlp/moss-003-sft-data) dataset by [XTuner](https://github.com/InternLM/xtuner). |
|
|
|
|
|
## Quickstart |
|
|
|
### Usage with HuggingFace libraries |
|
|
|
```python |
|
import torch |
|
from peft import PeftModel |
|
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, StoppingCriteria |
|
from transformers.generation import GenerationConfig |
|
|
|
class StopWordStoppingCriteria(StoppingCriteria): |
|
def __init__(self, tokenizer, stop_word): |
|
self.tokenizer = tokenizer |
|
self.stop_word = stop_word |
|
self.length = len(self.stop_word) |
|
def __call__(self, input_ids, *args, **kwargs) -> bool: |
|
cur_text = self.tokenizer.decode(input_ids[0]) |
|
cur_text = cur_text.replace('\r', '').replace('\n', '') |
|
return cur_text[-self.length:] == self.stop_word |
|
|
|
tokenizer = AutoTokenizer.from_pretrained('Qwen/Qwen-7B', trust_remote_code=True) |
|
quantization_config = BitsAndBytesConfig(load_in_4bit=True, load_in_8bit=False, llm_int8_threshold=6.0, llm_int8_has_fp16_weight=False, bnb_4bit_compute_dtype=torch.float16, bnb_4bit_use_double_quant=True, bnb_4bit_quant_type='nf4') |
|
model = AutoModelForCausalLM.from_pretrained('Qwen/Qwen-7B', quantization_config=quantization_config, device_map='auto', trust_remote_code=True).eval() |
|
model = PeftModel.from_pretrained(model, 'xtuner/Qwen-7B-qlora-moss-003-sft') |
|
gen_config = GenerationConfig(max_new_tokens=512, do_sample=True, temperature=0.1, top_p=0.75, top_k=40) |
|
|
|
# Note: In this example, we disable the use of plugins because the API depends on additional implementations. |
|
# If you want to experience plugins, please refer to XTuner CLI! |
|
prompt_template = ( |
|
'You are an AI assistant whose name is Qwen.\n' |
|
'Capabilities and tools that Qwen can possess.\n' |
|
'- Inner thoughts: disabled.\n' |
|
'- Web search: disabled.\n' |
|
'- Calculator: disabled.\n' |
|
'- Equation solver: disabled.\n' |
|
'- Text-to-image: disabled.\n' |
|
'- Image edition: disabled.\n' |
|
'- Text-to-speech: disabled.\n' |
|
'<|Human|>: {input}<eoh>\n' |
|
'<|Inner Thoughts|>: None<eot>\n' |
|
'<|Commands|>: None<eoc>\n' |
|
'<|Results|>: None<eor>\n') |
|
|
|
text = '请给我介绍五个上海的景点' |
|
inputs = tokenizer(prompt_template.format(input=text), return_tensors='pt') |
|
inputs = inputs.to(model.device) |
|
pred = model.generate(**inputs, generation_config=gen_config, stopping_criteria=[StopWordStoppingCriteria(tokenizer, '<eom>')]) |
|
print(tokenizer.decode(pred.cpu()[0], skip_special_tokens=True)) |
|
""" |
|
好的，以下是五个上海的景点介绍： |
|
1. 上海博物馆：上海博物馆是中国最大的综合性博物馆之一，收藏了大量的历史文物和艺术品，包括青铜器、陶瓷、书画、玉器等。 |
|
2. 上海城隍庙：上海城隍庙是上海最古老的庙宇之一，建于明朝，是上海的标志性建筑之一。庙内有各种神像和文物，是了解上海历史文化的好去处。 |
|
3. 上海科技馆：上海科技馆是一座集科技、文化、教育为一体的综合性博物馆，展示了各种科技展品和互动体验项目，适合全家人一起参观。 |
|
4. 上海东方明珠塔：上海东方明珠塔是上海的标志性建筑之一，高468米。游客可以乘坐高速电梯到达观景台，欣赏上海的美景。 |
|
5. 上海迪士尼乐园：上海迪士尼乐园是中国第一个迪士尼主题公园，拥有各种游乐设施和表演节目，适合全家人一起游玩。 |
|
""" |
|
``` |
|
|
|
### Usage with XTuner CLI |
|
|
|
#### Installation |
|
|
|
```shell |
|
pip install -U xtuner |
|
``` |
|
|
|
#### Chat |
|
|
|
```shell |
|
export SERPER_API_KEY="xxx" # Please get the key from https://serper.dev to support google search! |
|
xtuner chat Qwen/Qwen-7B --adapter xtuner/Qwen-7B-qlora-moss-003-sft --bot-name Qwen --prompt-template moss_sft --system-prompt moss_sft --with-plugins calculate solve search |
|
``` |
|
|
|
#### Fine-tune |
|
|
|
Use the following command to quickly reproduce the fine-tuning results. |
|
|
|
```shell |
|
NPROC_PER_NODE=8 xtuner train qwen_7b_qlora_moss_sft_all_e2_gpu8 |
|
``` |
