dmariko's picture
Update README.md
cf11bd8 verified
metadata
license: cc-by-nc-4.0
tags:
  - trl
  - dpo
  - generated_from_trainer
base_model: HuggingFaceTB/SmolLM-360M-Instruct
model-index:
  - name: SmolLM-360M-Instruct-dpo-16k
    results: []
language:
  - en

SmolLM-360M-Instruct-dpo-16k

This model is a fine-tuned version of HuggingFaceTB/SmolLM-360M-Instruct on an unknown dataset. It achieves the following results on the evaluation set:

  • Loss: 0.8873
  • Rewards/chosen: 0.0047
  • Rewards/rejected: 0.3539
  • Rewards/accuracies: 0.0326
  • Rewards/margins: -0.3493
  • Logps/rejected: -470.7575
  • Logps/chosen: -546.0133
  • Logits/rejected: 0.3014
  • Logits/chosen: 0.6045

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 5e-06
  • train_batch_size: 2
  • eval_batch_size: 2
  • seed: 42
  • gradient_accumulation_steps: 2
  • total_train_batch_size: 4
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_steps: 2
  • num_epochs: 6

Training results

Training Loss Epoch Step Validation Loss Rewards/chosen Rewards/rejected Rewards/accuracies Rewards/margins Logps/rejected Logps/chosen Logits/rejected Logits/chosen
0.5225 0.9999 3368 0.8679 0.0092 0.3258 0.0337 -0.3166 -471.0385 -545.9679 0.3212 0.6250
0.4511 2.0 6737 0.8863 0.0171 0.3649 0.0283 -0.3477 -470.6477 -545.8885 0.2889 0.5939
0.4453 2.9999 10105 0.8880 0.0006 0.3516 0.0304 -0.3510 -470.7807 -546.0537 0.3259 0.6291
0.4439 4.0 13474 0.8894 0.0067 0.3598 0.0228 -0.3531 -470.6990 -545.9932 0.2699 0.5815
0.4441 4.9999 16842 0.8881 0.0058 0.3569 0.0293 -0.3511 -470.7278 -546.0020 0.2999 0.6028
0.4442 5.9991 20208 0.8873 0.0047 0.3539 0.0326 -0.3493 -470.7575 -546.0133 0.3014 0.6045

Framework versions

  • Transformers 4.41.0
  • Pytorch 2.2.0
  • Datasets 2.19.1
  • Tokenizers 0.19.1