zephyr-dpo-qlora-uf-ours-5e-7

This model is a fine-tuned version of alignment-handbook/zephyr-7b-sft-full on the generation/UF dataset. It achieves the following results on the evaluation set:

• Loss: 0.6585
• Rewards/chosen: -0.1318
• Rewards/rejected: -0.2668
• Rewards/accuracies: 0.6360
• Rewards/margins: 0.1350
• Rewards/margins Max: 0.7123
• Rewards/margins Min: -0.4480
• Rewards/margins Std: 0.4011
• Logps/rejected: -285.2624
• Logps/chosen: -297.7779
• Logits/rejected: -2.6359
• Logits/chosen: -2.6686

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-07
• train_batch_size: 4
• eval_batch_size: 8
• seed: 42
• distributed_type: multi-GPU
• num_devices: 2
• gradient_accumulation_steps: 2
• total_train_batch_size: 16
• total_eval_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 3

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Rewards/margins Max	Rewards/margins Min	Rewards/margins Std	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.6872	0.28	100	0.6920	0.0070	0.0040	0.5890	0.0030	0.0201	-0.0117	0.0106	-258.1791	-283.8921	-2.7682	-2.8069
0.6585	0.56	200	0.6870	0.0392	0.0222	0.5990	0.0171	0.1011	-0.0563	0.0523	-256.3612	-280.6689	-2.7497	-2.7877
0.6099	0.85	300	0.6805	0.0140	-0.0224	0.5980	0.0364	0.2144	-0.1227	0.1127	-260.8226	-283.1934	-2.7177	-2.7552
0.5685	1.13	400	0.6739	-0.0306	-0.0901	0.6020	0.0596	0.3565	-0.2113	0.1904	-267.5928	-287.6518	-2.6871	-2.7235
0.5275	1.41	500	0.6691	-0.0498	-0.1325	0.6110	0.0827	0.4946	-0.3139	0.2729	-271.8259	-289.5688	-2.6519	-2.6881
0.4992	1.69	600	0.6633	-0.0820	-0.1909	0.6240	0.1089	0.6097	-0.3758	0.3364	-277.6682	-292.7885	-2.6447	-2.6793
0.4721	1.97	700	0.6607	-0.0596	-0.1836	0.6320	0.1240	0.6617	-0.3928	0.3590	-276.9363	-290.5549	-2.6452	-2.6783
0.4401	2.25	800	0.6595	-0.1036	-0.2352	0.6340	0.1316	0.6923	-0.4292	0.3871	-282.0996	-294.9506	-2.6407	-2.6735
0.4402	2.54	900	0.6586	-0.1091	-0.2452	0.6450	0.1361	0.7080	-0.4378	0.3951	-283.1014	-295.5010	-2.6372	-2.6699
0.4835	2.82	1000	0.6584	-0.1313	-0.2662	0.6380	0.1349	0.7122	-0.4482	0.4013	-285.1998	-297.7245	-2.6391	-2.6714

Framework versions

• PEFT 0.7.1
• Transformers 4.39.0.dev0
• Pytorch 2.1.2+cu121
• Datasets 2.14.6
• Tokenizers 0.15.2