fresh-2-layer-medmcqa100000-distill-of-fresh-2-layer-mmlu_EVAL_mmlu

This model is a fine-tuned version of on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 207.8812
• Accuracy: 0.4391

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0005
• train_batch_size: 32
• eval_batch_size: 32
• seed: 321
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 500
• training_steps: 5000

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
No log	0.03	100	200.5408	0.246
No log	0.06	200	200.4924	0.318
No log	0.1	300	189.5536	0.362
No log	0.13	400	213.1945	0.408
142.2534	0.16	500	200.2095	0.41
142.2534	0.19	600	183.4482	0.434
142.2534	0.22	700	181.7445	0.446
142.2534	0.26	800	174.5725	0.446
142.2534	0.29	900	172.2695	0.456
95.7189	0.32	1000	189.9845	0.446
95.7189	0.35	1100	200.3398	0.446
95.7189	0.38	1200	176.7680	0.422
95.7189	0.42	1300	184.6660	0.424
95.7189	0.45	1400	206.7043	0.466
83.1508	0.48	1500	188.5695	0.454
83.1508	0.51	1600	206.9309	0.452
83.1508	0.54	1700	186.1902	0.454
83.1508	0.58	1800	191.8201	0.45
83.1508	0.61	1900	185.2374	0.466

Framework versions

• Transformers 4.34.0.dev0
• Pytorch 2.0.1+cu117
• Datasets 2.14.5
• Tokenizers 0.14.0