Edit model card

DiscoPOP-zephyr-7b-gemma

This model is a fine-tuned version of HuggingFaceH4/zephyr-7b-gemma-sft-v0.1 on the argilla/dpo-mix-7k dataset.

This model is from the paper "Discovering Preference Optimization Algorithms with and for Large Language Models"

Read the blog post on it here!

See the codebase to generate it here: https://github.com/SakanaAI/DiscoPOP

Model description

This model is identical in training to HuggingFaceH4/zephyr-7b-gemma-v0.1, except instead of using Direct Preference Optimization (DPO), it uses DiscoPOP.

DiscoPOP is our Discovered Preference Optimization algorithm, which is defined as follows:

def log_ratio_modulated_loss(
    self,
    policy_chosen_logps: torch.FloatTensor,
    policy_rejected_logps: torch.FloatTensor,
    reference_chosen_logps: torch.FloatTensor,
    reference_rejected_logps: torch.FloatTensor,
) -> torch.FloatTensor:
    pi_logratios = policy_chosen_logps - policy_rejected_logps
    ref_logratios = reference_chosen_logps - reference_rejected_logps
    logits = pi_logratios - ref_logratios
    # Modulate the mixing coefficient based on the log ratio magnitudes
    log_ratio_modulation = torch.sigmoid(logits)
    logistic_component = -F.logsigmoid(self.beta * logits)
    exp_component = torch.exp(-self.beta * logits)
    # Blend between logistic and exponential component based on log ratio modulation
    losses = logistic_component * (1 - log_ratio_modulation) + exp_component * log_ratio_modulation
    return losses

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 5e-07
  • train_batch_size: 2
  • eval_batch_size: 4
  • seed: 42
  • distributed_type: multi-GPU
  • num_devices: 8
  • gradient_accumulation_steps: 8
  • total_train_batch_size: 128
  • total_eval_batch_size: 32
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_ratio: 0.1
  • num_epochs: 2

Framework versions

  • Transformers 4.40.1
  • Pytorch 2.1.2+cu121
  • Datasets 2.19.0
  • Tokenizers 0.19.1
Downloads last month
4,316
Safetensors
Model size
8.54B params
Tensor type
BF16
·
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Model tree for SakanaAI/DiscoPOP-zephyr-7b-gemma

Base model

google/gemma-7b
Finetuned
(32)
this model
Finetunes
7 models
Quantizations
4 models

Dataset used to train SakanaAI/DiscoPOP-zephyr-7b-gemma

Space using SakanaAI/DiscoPOP-zephyr-7b-gemma 1