---
base_model: microsoft/Phi-3-mini-4k-instruct
library_name: peft
license: mit
tags:
- trl
- sft
- generated_from_trainer
model-index:
- name: phi-3-mini-QLoRA
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# phi-3-mini-QLoRA

This model is a fine-tuned version of [microsoft/Phi-3-mini-4k-instruct](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 0.6126

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 6e-05
- train_batch_size: 4
- eval_batch_size: 4
- seed: 0
- gradient_accumulation_steps: 4
- total_train_batch_size: 16
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 5
- training_steps: 250

### Training results

| Training Loss | Epoch   | Step | Validation Loss |
|:-------------:|:-------:|:----:|:---------------:|
| 1.3693        | 0.6667  | 5    | 1.3378          |
| 1.1643        | 1.3333  | 10   | 1.1047          |
| 0.8388        | 2.0     | 15   | 0.8767          |
| 0.6894        | 2.6667  | 20   | 0.6828          |
| 0.5636        | 3.3333  | 25   | 0.5688          |
| 0.4496        | 4.0     | 30   | 0.5110          |
| 0.3487        | 4.6667  | 35   | 0.4549          |
| 0.3169        | 5.3333  | 40   | 0.4148          |
| 0.2595        | 6.0     | 45   | 0.3893          |
| 0.2002        | 6.6667  | 50   | 0.3733          |
| 0.2437        | 7.3333  | 55   | 0.3597          |
| 0.1669        | 8.0     | 60   | 0.3456          |
| 0.1873        | 8.6667  | 65   | 0.3491          |
| 0.1831        | 9.3333  | 70   | 0.3422          |
| 0.1581        | 10.0    | 75   | 0.3664          |
| 0.0831        | 10.6667 | 80   | 0.3644          |
| 0.1277        | 11.3333 | 85   | 0.3822          |
| 0.0539        | 12.0    | 90   | 0.3868          |
| 0.0799        | 12.6667 | 95   | 0.4190          |
| 0.066         | 13.3333 | 100  | 0.4375          |
| 0.0564        | 14.0    | 105  | 0.4581          |
| 0.0356        | 14.6667 | 110  | 0.4715          |
| 0.0493        | 15.3333 | 115  | 0.4896          |
| 0.0399        | 16.0    | 120  | 0.5066          |
| 0.0452        | 16.6667 | 125  | 0.5022          |
| 0.0305        | 17.3333 | 130  | 0.5246          |
| 0.036         | 18.0    | 135  | 0.5492          |
| 0.0282        | 18.6667 | 140  | 0.5537          |
| 0.0327        | 19.3333 | 145  | 0.5703          |
| 0.0341        | 20.0    | 150  | 0.5699          |
| 0.0315        | 20.6667 | 155  | 0.5761          |
| 0.0284        | 21.3333 | 160  | 0.5781          |
| 0.027         | 22.0    | 165  | 0.5818          |
| 0.0258        | 22.6667 | 170  | 0.5858          |
| 0.0224        | 23.3333 | 175  | 0.5884          |
| 0.0253        | 24.0    | 180  | 0.5960          |
| 0.0232        | 24.6667 | 185  | 0.6015          |
| 0.0256        | 25.3333 | 190  | 0.6088          |
| 0.0226        | 26.0    | 195  | 0.6106          |
| 0.0226        | 26.6667 | 200  | 0.6096          |
| 0.0259        | 27.3333 | 205  | 0.6102          |
| 0.0217        | 28.0    | 210  | 0.6100          |
| 0.022         | 28.6667 | 215  | 0.6115          |
| 0.0219        | 29.3333 | 220  | 0.6115          |
| 0.0239        | 30.0    | 225  | 0.6109          |
| 0.0226        | 30.6667 | 230  | 0.6123          |
| 0.0219        | 31.3333 | 235  | 0.6140          |
| 0.0201        | 32.0    | 240  | 0.6128          |
| 0.0198        | 32.6667 | 245  | 0.6130          |
| 0.0234        | 33.3333 | 250  | 0.6126          |


### Framework versions

- PEFT 0.13.2
- Transformers 4.45.2
- Pytorch 2.4.1+cu121
- Datasets 3.0.1
- Tokenizers 0.20.0