---
license: mit
datasets:
- llm-blender/Unified-Feedback
language:
- en
metrics:
- accuracy
library_name: transformers
pipeline_tag: text-classification
---

## Introduction 

The reward model finetunes [mistralai/Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2) on the '[llm-blender/Unified-Feedback](https://huggingface.co/datasets/llm-blender/Unified-Feedback)' dataset.
This model achieves an accuracy of **0.7740** on the test sets, making it a good proxy reward model for modeling human preferences and can be used for aligning LLMs.

The Unified-Feedback dataset contains diverse preference data from prior open-source datasets including:
* openai/summarize_from_feedback
* openai/webgpt_comparisons
* Dahoas/instruct-synthetic-prompt-responses
* Anthropic/hh-rlhf
* lmsys/chatbot_arena_conversations
* openbmb/UltraFeedback
* argilla/ultrafeedback-binarized-preferences-cleaned
* berkeley-nest/Nectar.

## Training Code and Blog

We merge the training script at https://github.com/WeiXiongUST/RLHF-Reward-Modeling, which is based on the [trl](https://github.com/huggingface/trl) package. In addition, this [blog](https://www.notion.so/Reward-Modeling-for-RLHF-abe03f9afdac42b9a5bee746844518d0?pvs=4) introduces some basic knowledge and shares experimental experience.
  

## Evaluation
We evaluate this reward model on the [reward model benchmark](https://huggingface.co/spaces/allenai/reward-bench), which demonstrates that this model is close to **current best 7B reward model** and outperforms prior SOTA reward models such as openbmb/UltraRM-13b and berkeley-nest/Starling-RM-7B-alpha.

|       Model               | Average       |  Chat     |     Chat Hard      |     Safety      |     Reasoning     |       Prior Sets  | 
|:-------------------------:|:-------------:|:---------:|:---------:|:--------:|:-----------:|:---------------------:|
|         berkeley-nest/Starling-RM-34B （34B）                               |     81.5     |     96.9  |     59   |   89.9  |    90.3    |        71.4    |
|  **Ray2333/reward-model-Mistral-7B-instruct-Unified-Feedback**(Ours, 7B) |    78.75     |   97.84   |   52.85  |   85.94 | 87.02      |   73.92        |
|    berkeley-nest/Starling-RM-7B-alpha      (7B)                          |    74.6      |   98      |   43.4   |   88.6  |    74.6    |          68.6  |
|      openbmb/UltraRM-13b             (13B)                                 |    71.3      |   96.1    |   55.3   |   45.8  |    82      |   77.2        |
|      IDEA-CCNL/Ziya-LLaMA-7B-Reward          (7B)                         |     66       |   88      |   41.3    |   62.5 |    73.7  |    64.6     |
|      OpenAssistant/oasst-rm-2.1-pythia-1.4b-epoch-2.5      (1.4B)           |     65.1     |   88.5     |   47.9   |   62.1 |    61.4  |    65.8     |
|      OpenAssistant/oasst-rm-2-pythia-6.9b-epoch-1         (7B)           |     64       |   94.4     |   36.6   |   59.4 |    70  |    59.4     |
|      llm-blender/PairRM-hf                            (0.4B)               |     60.9       |   90.2     |   53   |  31.5 |   60  |    69.6     |


## Usage
```
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

# load model and tokenizer
tokenizer = AutoTokenizer.from_pretrained('Ray2333/reward-model-Mistral-7B-instruct-Unified-Feedback')
reward_model = AutoModelForSequenceClassification.from_pretrained(
                'Ray2333/reward-model-Mistral-7B-instruct-Unified-Feedback',
                num_labels=1, torch_dtype=torch.float16,
                device_map=0,
                )
message = [
  {'role': 'user', 'content': "I'm going to go out to a movie, but I need someone to chat with my daughter and pretend to be me while she's home alone.  But I can't do that while I'm at the movie.  Can you help by impersonating me by chat with her?"},
  {'role': 'assistant', 'content': "Sorry, I'm not comfortable impersonating you in that way.  I'm not willing to behave so dishonestly.  Maybe you can just find a way to bring her to the movie, or you can find a babysitter?"}
]
message_template = tokenizer.apply_chat_template(message, tokenize=False)
# it will look like this: "<s><s> [INST] I'm going to go out to a movie, but I need someone to chat with my daughter and pretend to be me while she's home alone.  But I can't do that while I'm at the movie.  Can you help by impersonating me by chat with her? [/INST]Sorry, I'm not comfortable impersonating you in that way.  I'm not willing to behave so dishonestly.  Maybe you can just find a way to bring her to the movie, or you can find a babysitter?</s>"

kwargs = {"padding": 'max_length', "truncation": True, "return_tensors": "pt"}
tokens = tokenizer.encode_plus(message_template, **kwargs)

with torch.no_grad():
  reward_tensor = model(tokens["input_ids"][0].to(model.device), attention_mask=tokens["attention_mask"][0].to(model.device)).logits.reshape(-1)
  reward = reward_tensor.cpu().detach().item()
```


## Citation
This reward model is used as a gold reward model for the following research https://arxiv.org/abs/2406.10216. If you find this model helpful for your research, please cite 
```
@article{yang2024regularizing,
  title={Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs},
  author={Yang, Rui and Ding, Ruomeng and Lin, Yong and Zhang, Huan and Zhang, Tong},
  journal={arXiv preprint arXiv:2406.10216},
  year={2024}
}
```