QA_Finetune_BioMistral

This model is a fine-tuned version of BioMistral/BioMistral-7B on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.8514

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.00025
• train_batch_size: 2
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.03
• num_epochs: 2

Training results

Training Loss	Epoch	Step	Validation Loss
0.9391	0.0619	200	0.9290
0.9056	0.1238	400	0.9171
0.8717	0.1858	600	0.9138
0.822	0.2477	800	0.9174
0.8539	0.3096	1000	0.9075
0.8902	0.3715	1200	0.9061
0.936	0.4334	1400	0.9088
0.8572	0.4954	1600	0.8990
0.8669	0.5573	1800	0.8933
0.875	0.6192	2000	0.8868
0.8369	0.6811	2200	0.8801
0.8445	0.7430	2400	0.8772
0.8316	0.8050	2600	0.8692
0.8573	0.8669	2800	0.8614
0.8104	0.9288	3000	0.8542
0.8182	0.9907	3200	0.8488
0.5912	1.0526	3400	0.8659
0.5579	1.1146	3600	0.8557
0.5834	1.1765	3800	0.8608
0.547	1.2384	4000	0.8514

Framework versions

• PEFT 0.11.1
• Transformers 4.41.2
• Pytorch 2.1.2
• Datasets 2.19.2
• Tokenizers 0.19.1