llama_8b_lima_40

This model is a fine-tuned version of meta-llama/Llama-3.1-8B on the open_webui_dataset dataset. It achieves the following results on the evaluation set:

• Loss: 0.9288

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 6e-06
• train_batch_size: 2
• eval_batch_size: 2
• seed: 66
• distributed_type: multi-GPU
• num_devices: 2
• gradient_accumulation_steps: 7
• total_train_batch_size: 28
• total_eval_batch_size: 4
• optimizer: Use OptimizerNames.ADAMW_8BIT with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: polynomial
• lr_scheduler_warmup_ratio: 0.04
• num_epochs: 1.0

Training results

Training Loss	Epoch	Step	Validation Loss
1.0252	0.0769	80	1.0117
0.8185	0.1538	160	0.9820
0.9686	0.2308	240	0.9702
0.7962	0.3077	320	0.9604
0.8669	0.3846	400	0.9520
0.8883	0.4615	480	0.9506
0.7636	0.5385	560	0.9441
0.8221	0.6154	640	0.9404
0.9417	0.6923	720	0.9345
0.9623	0.7692	800	0.9317
0.9654	0.8462	880	0.9302
0.9169	0.9231	960	0.9286
0.9005	1.0	1040	0.9288

Framework versions

• Transformers 4.46.1
• Pytorch 2.4.0
• Datasets 3.1.0
• Tokenizers 0.20.3