Edit model card

sexed-mistral-7b-sft-lora

This model is a fine-tuned version of mistralai/Mistral-7B-v0.1 on the generator dataset. It achieves the following results on the evaluation set:

  • Loss: 0.2681

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0001
  • train_batch_size: 4
  • eval_batch_size: 4
  • seed: 42
  • gradient_accumulation_steps: 128
  • total_train_batch_size: 512
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: cosine
  • num_epochs: 20

Training results

Training Loss Epoch Step Validation Loss
0.5069 0.8380 4 0.4851
0.4188 1.8854 9 0.4025
0.3461 2.9329 14 0.3485
0.3204 3.9804 19 0.3211
0.3053 4.8183 23 0.3053
0.2914 5.8658 28 0.2923
0.2802 6.9133 33 0.2851
0.2733 7.9607 38 0.2795
0.273 8.7987 42 0.2762
0.2746 9.8462 47 0.2732
0.2673 10.8936 52 0.2709
0.2591 11.9411 57 0.2695
0.2602 12.9885 62 0.2686
0.26 13.8265 66 0.2683
0.2668 14.8740 71 0.2682
0.2598 15.9214 76 0.2681
0.26 16.7594 80 0.2681

Framework versions

  • PEFT 0.10.0
  • Transformers 4.40.2
  • Pytorch 2.1.1+cu121
  • Datasets 2.19.1
  • Tokenizers 0.19.1
Downloads last month
2
Inference API
Unable to determine this model’s pipeline type. Check the docs .

Model tree for ben-wycliff/sexed-mistral-7b-sft-lora

Adapter
(1171)
this model