Edit model card

fine-tuning-Phi2-with-webglm-qa-with-lora_7

This model is a fine-tuned version of microsoft/phi-2 on an unknown dataset. It achieves the following results on the evaluation set:

  • Loss: 0.0950

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 5e-05
  • train_batch_size: 2
  • eval_batch_size: 2
  • seed: 42
  • gradient_accumulation_steps: 5
  • total_train_batch_size: 10
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • lr_scheduler_warmup_steps: 60
  • training_steps: 1000
  • mixed_precision_training: Native AMP

Training results

Training Loss Epoch Step Validation Loss
7.3505 0.31 20 6.2863
4.0914 0.63 40 0.9255
0.6517 0.94 60 0.5762
0.4621 1.26 80 0.4062
0.3128 1.57 100 0.3056
0.2536 1.89 120 0.2604
0.2227 2.2 140 0.2247
0.1901 2.52 160 0.2041
0.176 2.83 180 0.1812
0.1453 3.14 200 0.1683
0.1557 3.46 220 0.1592
0.1441 3.77 240 0.1488
0.1282 4.09 260 0.1430
0.1215 4.4 280 0.1348
0.1217 4.72 300 0.1323
0.117 5.03 320 0.1271
0.109 5.35 340 0.1255
0.1094 5.66 360 0.1210
0.1057 5.97 380 0.1175
0.0937 6.29 400 0.1158
0.0942 6.6 420 0.1159
0.1007 6.92 440 0.1125
0.0876 7.23 460 0.1119
0.0894 7.55 480 0.1099
0.0827 7.86 500 0.1072
0.0894 8.18 520 0.1069
0.0805 8.49 540 0.1075
0.0782 8.81 560 0.1043
0.0881 9.12 580 0.1034
0.0839 9.43 600 0.1015
0.0694 9.75 620 0.1000
0.068 10.06 640 0.1007
0.072 10.38 660 0.0994
0.0709 10.69 680 0.0985
0.0712 11.01 700 0.0986
0.0673 11.32 720 0.0999
0.0669 11.64 740 0.0974
0.0706 11.95 760 0.0981
0.0641 12.26 780 0.0969
0.0652 12.58 800 0.0964
0.0668 12.89 820 0.0962
0.0617 13.21 840 0.0972
0.0628 13.52 860 0.0960
0.0637 13.84 880 0.0949
0.0633 14.15 900 0.0951
0.0577 14.47 920 0.0953
0.0646 14.78 940 0.0947
0.06 15.09 960 0.0946
0.0584 15.41 980 0.0949
0.0638 15.72 1000 0.0950

Framework versions

  • PEFT 0.7.1
  • Transformers 4.36.2
  • Pytorch 2.0.0
  • Datasets 2.15.0
  • Tokenizers 0.15.0
Downloads last month
0
Inference API
Unable to determine this model’s pipeline type. Check the docs .

Model tree for Gunslinger3D/fine-tuning-Phi2-with-webglm-qa-with-lora_7

Base model

microsoft/phi-2
Adapter
(636)
this model