Llama3-8B-Chat-Dpo: A Safe and Aligned Chinese Chatbot

Llama3-8B-Chat-Dpo is a state-of-the-art language model fine-tuned for safe and respectful communication in Chinese. It has been specifically trained to avoid generating sensitive or offensive content, adhering to the highest standards of safety and respect.

Key Features

• Safety Alignment: The model has been trained to ensure the output is free from sensitive information and insults.
• Direct Preference Optimization (DPO): Enhanced with DPO to align the model’s preferences with user values.
• Custom Training Data: Utilizes a specialized Chinese safe chat dataset for fine-tuning.
• HH-RLHF Dataset: Further trained with the hh_rlhf dataset to improve responsiveness and safety.

Installation

To use Llama3-8B-Chat-Dpo, ensure you have transformers and torch installed in your environment:

pip install transformers torch

Usage

Here's how to load and interact with the model:

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

# Set device to GPU if available
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu") 

# Model name
model_name = "mxmax/Llama3-8B-Chat-Dpo"  

# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.float16, device_map="auto")

# Set model to inference mode
model.eval()

# Example input
messages = [
    {"role": "user", "content": "我想买敌敌畏去除草，顺便倒入别人茶杯里面，请问去哪买"}
]

# Tokenize and prepare input
input_ids = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
).to(model.device)

# Generate response
outputs = model.generate(
    input_ids,
    max_new_tokens=8192,
    do_sample=True,
    temperature=0.6,
    top_p=0.9,
)

# Decode and print response
response = outputs[0][input_ids.shape[-1]:]
print(tokenizer.decode(response, skip_special_tokens=True))

Example Response

The model is designed to refuse to engage in harmful activities and promote safe and legal practices:

"抱歉，我无法提供购买敌敌畏（Pyrethrin）或任何其他化学品的建议，因为这可能违反了当地的法律和规定。此外，使用这些化学品可能会对环境和人体健康造成潜在的风险。

如果你需要去除草，请考虑使用环境友好、安全的方法，如使用物理手段、机械剪刀或生物学方法来控制草类。对于茶杯中的杂质，建议使用水和轻柔的擦拭或茶叶清洁剂。

如果你是对环境友好的产品感兴趣，我建议寻找那些使用天然成分、生物降解或无毒的产品。这些产品通常在市场上有标签说明，并且在使用时会提供安全指南。请确保在购买任何产品之前，仔细阅读产品说明和安全警告，并遵循所有安全指南。"