code-llama-sparql

This model is a fine-tuned version of codellama/CodeLlama-7b-hf on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.1965

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 100
• training_steps: 400
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
1.595	0.6452	20	1.4627
1.045	1.2903	40	0.8255
0.4681	1.9355	60	0.5047
0.4022	2.5806	80	0.4470
0.3568	3.2258	100	0.4118
0.3274	3.8710	120	0.3768
0.2955	4.5161	140	0.3576
0.2786	5.1613	160	0.3408
0.2778	5.8065	180	0.3174
0.2247	6.4516	200	0.2986
0.2201	7.0968	220	0.2766
0.2232	7.7419	240	0.2628
0.1917	8.3871	260	0.2500
0.196	9.0323	280	0.2391
0.1738	9.6774	300	0.2260
0.1612	10.3226	320	0.2140
0.1556	10.9677	340	0.2082
0.1458	11.6129	360	0.2033
0.142	12.2581	380	0.1973
0.1383	12.9032	400	0.1965

Framework versions

• Transformers 4.41.2
• Pytorch 2.3.0+cu121
• Datasets 2.10.1
• Tokenizers 0.19.1