phi-3-mini-QLoRA

This model is a fine-tuned version of microsoft/Phi-3-mini-4k-instruct on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.5739

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss
1.0193	0.0181	10	1.0582
1.0408	0.0362	20	1.0129
0.9957	0.0543	30	0.9095
0.8047	0.0724	40	0.7711
0.699	0.0905	50	0.6689
0.6143	0.1085	60	0.6385
0.6472	0.1266	70	0.6175
0.6077	0.1447	80	0.6073
0.6028	0.1628	90	0.6012
0.5929	0.1809	100	0.5978
0.6117	0.1990	110	0.5948
0.5904	0.2171	120	0.5925
0.5852	0.2352	130	0.5909
0.5662	0.2533	140	0.5895
0.6183	0.2714	150	0.5880
0.5872	0.2895	160	0.5873
0.5807	0.3076	170	0.5863
0.6169	0.3256	180	0.5853
0.5705	0.3437	190	0.5841
0.6143	0.3618	200	0.5835
0.5705	0.3799	210	0.5828
0.5683	0.3980	220	0.5821
0.6077	0.4161	230	0.5818
0.586	0.4342	240	0.5811
0.5724	0.4523	250	0.5804
0.5941	0.4704	260	0.5799
0.5989	0.4885	270	0.5798
0.5582	0.5066	280	0.5793
0.5798	0.5246	290	0.5792
0.5545	0.5427	300	0.5785
0.597	0.5608	310	0.5783
0.6093	0.5789	320	0.5779
0.5736	0.5970	330	0.5778
0.5698	0.6151	340	0.5772
0.5659	0.6332	350	0.5769
0.5877	0.6513	360	0.5764
0.5837	0.6694	370	0.5763
0.5858	0.6875	380	0.5761
0.5877	0.7056	390	0.5760
0.5802	0.7237	400	0.5756
0.6009	0.7417	410	0.5754
0.5713	0.7598	420	0.5751
0.5509	0.7779	430	0.5751
0.5646	0.7960	440	0.5750
0.5458	0.8141	450	0.5748
0.5694	0.8322	460	0.5746
0.576	0.8503	470	0.5744
0.5864	0.8684	480	0.5742
0.5645	0.8865	490	0.5741
0.5531	0.9046	500	0.5742
0.6176	0.9227	510	0.5742
0.5987	0.9408	520	0.5742
0.5703	0.9588	530	0.5740
0.6023	0.9769	540	0.5740
0.5637	0.9950	550	0.5739

Framework versions

• PEFT 0.13.0
• Transformers 4.45.1
• Pytorch 2.4.1+cu121
• Datasets 3.0.1
• Tokenizers 0.20.0