
This is an attempt to replicate the RLHF pipeline

### Base Model

  We used [bloomz-7b1-mt](https://huggingface.co/bigscience/bloomz-7b1-mt) because of its less-restricted license and multilingual ability.

### Supervised Fintune

  For SFT we used a combination of multiple datasets including:
  - [RyokoAI/ShareGPT52K](https://huggingface.co/datasets/RyokoAI/ShareGPT52K)
  - [GPTeacher](https://github.com/teknium1/GPTeacher)
  - [Alpaca-GPT4](https://github.com/Instruction-Tuning-with-GPT-4/GPT-4-LLM) en & zh
  - Filtered subset of machine-translated ShareGPT dataset into Chinese

### Reward Model

  For RM we used the code of [reward-modeling](https://github.com/Dahoas/reward-modeling) repo and datasets from
  - [oasst1](https://huggingface.co/datasets/OpenAssistant/oasst1)
  - [Dahoas/full-hh-rlhf](https://huggingface.co/datasets/Dahoas/full-hh-rlhf)
  - [liswei/rm-static-m2m100-zh](https://huggingface.co/datasets/liswei/rm-static-m2m100-zh)

### Reinforcement Learning

  For RL we used the code of [trlx](https://github.com/CarperAI/trlx) with slight modification.

  Instead of building value network upon the policy network with a single linear layer, we add another hydra head upon the reference network's frozen bottom layers as value network.

### Example

  We used Vicuna v1.1 template for model training

  ```
  from transformers import AutoModelForCausalLM, AutoTokenizer

  checkpoint = "keyfan/bloomz-rlhf"

  tokenizer = AutoTokenizer.from_pretrained(checkpoint)
  model = AutoModelForCausalLM.from_pretrained(checkpoint).cuda()

  template = ("A chat between a curious human and an artificial intelligence assistant. "
              "The assistant gives helpful, detailed, and polite answers to the human's questions. "
              "USER: {}\nASSISTANT:")
  question = template.format("Who was the president of the United States in 1955?")
  inputs = tokenizer.encode(question, return_tensors="pt").cuda()
  outputs = model.generate(inputs, do_sample=True, top_p=0.8, max_new_tokens=512)
  print(tokenizer.decode(outputs[0]))
  ```

### Evalutions

Result on the Chinese [BELLE eval set](https://github.com/LianjiaTech/BELLE/tree/main/eval)

| others | rewrite | classification | generation | summarization | extract | open qa | brainstorming | closed qa | macro ave | macro ave w/o others |
| ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |
| 0.619 | 0.873 | 0.706 | 0.934 | 0.755 | 0.619 | 0.527 | 0.908 | 0.615 | 0.728 | 0.742 |

* We found in GPT-4 evaluation the order in which the responses were presented has unneglectable affect on the final score even with the very-well designed Vicuna prompt. So we removed the score on the Vicuna eval set.

{"language": ["zh", "en"], "license": "bigscience-bloom-rail-1.0", "datasets": ["OpenAssistant/oasst1", "RyokoAI/ShareGPT52K", "Dahoas/full-hh-rlhf", "liswei/rm-static-m2m100-zh", "fnlp/moss-002-sft-data"]}
transformers pytorch bloom text-generation zh en dataset:OpenAssistant/oasst1 dataset:RyokoAI/ShareGPT52K dataset:Dahoas/full-hh-rlhf dataset:liswei/rm-static-m2m100-zh dataset:fnlp/moss-002-sft-data license:bigscience-bloom-rail-1.0 autotrain_compatible endpoints_compatible text-generation-inference region:us
TAGS
#transformers #pytorch #bloom #text-generation #zh #en #dataset-OpenAssistant/oasst1 #dataset-RyokoAI/ShareGPT52K #dataset-Dahoas/full-hh-rlhf #dataset-liswei/rm-static-m2m100-zh #dataset-fnlp/moss-002-sft-data #license-bigscience-bloom-rail-1.0 #autotrain_compatible #endpoints_compatible #text-generation-inference #region-us 

This is an attempt to replicate the RLHF pipeline


### Base Model


We used bloomz-7b1-mt because of its less-restricted license and multilingual ability.


### Supervised Fintune


For SFT we used a combination of multiple datasets including:


* RyokoAI/ShareGPT52K
* GPTeacher
* Alpaca-GPT4 en & zh
* Filtered subset of machine-translated ShareGPT dataset into Chinese


### Reward Model


For RM we used the code of reward-modeling repo and datasets from


* oasst1
* Dahoas/full-hh-rlhf
* liswei/rm-static-m2m100-zh


### Reinforcement Learning


For RL we used the code of trlx with slight modification.


Instead of building value network upon the policy network with a single linear layer, we add another hydra head upon the reference network's frozen bottom layers as value network.


### Example


We used Vicuna v1.1 template for model training


### Evalutions


Result on the Chinese BELLE eval set



* We found in GPT-4 evaluation the order in which the responses were presented has unneglectable affect on the final score even with the very-well designed Vicuna prompt. So we removed the score on the Vicuna eval set.



### Base Model


We used bloomz-7b1-mt because of its less-restricted license and multilingual ability. ### Supervised Fintune


For SFT we used a combination of multiple datasets including:


* RyokoAI/ShareGPT52K
* GPTeacher
* Alpaca-GPT4 en & zh
* Filtered subset of machine-translated ShareGPT dataset into Chinese ### Reward Model


For RM we used the code of reward-modeling repo and datasets from


* oasst1
* Dahoas/full-hh-rlhf
* liswei/rm-static-m2m100-zh ### Reinforcement Learning


For RL we used the code of trlx with slight modification.


Instead of building value network upon the policy network with a single linear layer, we add another hydra head upon the reference network's frozen bottom layers as value network. ### Example


We used Vicuna v1.1 template for model training ### Evalutions


Result on the Chinese BELLE eval set



* We found in GPT-4 evaluation the order in which the responses were presented has unneglectable affect on the final score even with the very-well designed Vicuna prompt. So we removed the score on the Vicuna eval set.
TAGS
#transformers #pytorch #bloom #text-generation #zh #en #dataset-OpenAssistant/oasst1 #dataset-RyokoAI/ShareGPT52K #dataset-Dahoas/full-hh-rlhf #dataset-liswei/rm-static-m2m100-zh #dataset-fnlp/moss-002-sft-data #license-bigscience-bloom-rail-1.0 #autotrain_compatible #endpoints_compatible #text-generation-inference #region-us 
 ### Base Model


We used bloomz-7b1-mt because of its less-restricted license and multilingual ability. ### Supervised Fintune


For SFT we used a combination of multiple datasets including:


* RyokoAI/ShareGPT52K
* GPTeacher
* Alpaca-GPT4 en & zh
* Filtered subset of machine-translated ShareGPT dataset into Chinese ### Reward Model


For RM we used the code of reward-modeling repo and datasets from


* oasst1
* Dahoas/full-hh-rlhf
* liswei/rm-static-m2m100-zh ### Reinforcement Learning


For RL we used the code of trlx with slight modification.


Instead of building value network upon the policy network with a single linear layer, we add another hydra head upon the reference network's frozen bottom layers as value network. ### Example


We used Vicuna v1.1 template for model training ### Evalutions


Result on the Chinese BELLE eval set



* We found in GPT-4 evaluation the order in which the responses were presented has unneglectable affect on the final score even with the very-well designed Vicuna prompt. So we removed the score on the Vicuna eval set.