Hugging Face's logo

KrithikV
/
phi-3-mini-LoRA

PEFT
Safetensors
trl
sft
Generated from Trainer
Model card Files Community
phi-3-mini-LoRA / README.md
KrithikV's picture
KrithikV
HuggingFaceUser/adapter-medmobile
dbc2bad verified
preview code
|
raw
history blame contribute delete
6.14 kB
metadata
base_model: microsoft/Phi-3-mini-4k-instruct
library_name: peft
license: mit
tags:
  - trl
  - sft
  - generated_from_trainer
model-index:
  - name: phi-3-mini-LoRA
    results: []

phi-3-mini-LoRA

This model is a fine-tuned version of microsoft/Phi-3-mini-4k-instruct on an unknown dataset. It achieves the following results on the evaluation set:

  • Loss: 0.8445

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0001
  • train_batch_size: 32
  • eval_batch_size: 32
  • seed: 42
  • gradient_accumulation_steps: 4
  • total_train_batch_size: 128
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • lr_scheduler_warmup_ratio: 0.1
  • num_epochs: 3

Training results

Training Loss Epoch Step Validation Loss
1.0374 0.0329 100 1.0194
0.9715 0.0658 200 0.9314
0.9106 0.0987 300 0.8969
0.888 0.1316 400 0.8869
0.8902 0.1645 500 0.8813
0.8826 0.1974 600 0.8777
0.8763 0.2303 700 0.8745
0.8728 0.2632 800 0.8723
0.8707 0.2961 900 0.8701
0.8702 0.3289 1000 0.8684
0.8631 0.3618 1100 0.8664
0.8623 0.3947 1200 0.8647
0.8655 0.4276 1300 0.8624
0.863 0.4605 1400 0.8602
0.858 0.4934 1500 0.8586
0.859 0.5263 1600 0.8578
0.8527 0.5592 1700 0.8569
0.8587 0.5921 1800 0.8563
0.8551 0.625 1900 0.8557
0.8548 0.6579 2000 0.8550
0.8515 0.6908 2100 0.8546
0.8531 0.7237 2200 0.8542
0.8567 0.7566 2300 0.8535
0.8589 0.7895 2400 0.8532
0.8547 0.8224 2500 0.8529
0.8537 0.8553 2600 0.8525
0.85 0.8882 2700 0.8521
0.8518 0.9211 2800 0.8519
0.8456 0.9539 2900 0.8515
0.8585 0.9868 3000 0.8512
0.849 1.0197 3100 0.8509
0.8549 1.0526 3200 0.8507
0.8502 1.0855 3300 0.8504
0.8504 1.1184 3400 0.8502
0.8488 1.1513 3500 0.8500
0.8504 1.1842 3600 0.8497
0.8465 1.2171 3700 0.8495
0.8471 1.25 3800 0.8494
0.8467 1.2829 3900 0.8491
0.8439 1.3158 4000 0.8489
0.8467 1.3487 4100 0.8487
0.8461 1.3816 4200 0.8485
0.8525 1.4145 4300 0.8483
0.8426 1.4474 4400 0.8481
0.8479 1.4803 4500 0.8480
0.853 1.5132 4600 0.8478
0.8432 1.5461 4700 0.8477
0.8416 1.5789 4800 0.8475
0.8527 1.6118 4900 0.8474
0.849 1.6447 5000 0.8472
0.8446 1.6776 5100 0.8471
0.8427 1.7105 5200 0.8469
0.8464 1.7434 5300 0.8468
0.8444 1.7763 5400 0.8466
0.8479 1.8092 5500 0.8465
0.8452 1.8421 5600 0.8465
0.8387 1.875 5700 0.8466
0.845 1.9079 5800 0.8463
0.8402 1.9408 5900 0.8461
0.8459 1.9737 6000 0.8460
0.8431 2.0066 6100 0.8460
0.8395 2.0395 6200 0.8459
0.8395 2.0724 6300 0.8458
0.8457 2.1053 6400 0.8457
0.8438 2.1382 6500 0.8457
0.8411 2.1711 6600 0.8456
0.8386 2.2039 6700 0.8456
0.8393 2.2368 6800 0.8454
0.8406 2.2697 6900 0.8454
0.8386 2.3026 7000 0.8453
0.8456 2.3355 7100 0.8453
0.8424 2.3684 7200 0.8452
0.8437 2.4013 7300 0.8451
0.8426 2.4342 7400 0.8451
0.8393 2.4671 7500 0.8450
0.8398 2.5 7600 0.8450
0.8434 2.5329 7700 0.8449
0.8456 2.5658 7800 0.8449
0.8393 2.5987 7900 0.8449
0.8401 2.6316 8000 0.8448
0.838 2.6645 8100 0.8448
0.8432 2.6974 8200 0.8447
0.8471 2.7303 8300 0.8447
0.8435 2.7632 8400 0.8446
0.8441 2.7961 8500 0.8446
0.8399 2.8289 8600 0.8446
0.8391 2.8618 8700 0.8446
0.8432 2.8947 8800 0.8446
0.8459 2.9276 8900 0.8446
0.8446 2.9605 9000 0.8445
0.8412 2.9934 9100 0.8445

Framework versions

  • PEFT 0.12.0
  • Transformers 4.43.3
  • Pytorch 2.3.1+cu121
  • Datasets 2.20.0
  • Tokenizers 0.19.1