G0514HMA16H

This model is a fine-tuned version of google/gemma-2b on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: -17.8971

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 16
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine_with_restarts
• lr_scheduler_warmup_steps: 100
• num_epochs: 3
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
0.9162	0.09	10	0.0830
-0.7664	0.18	20	-2.0466
-3.27	0.27	30	-4.9484
-6.242	0.36	40	-7.9963
-9.1799	0.45	50	-10.7742
-11.7646	0.54	60	-13.2319
-14.1063	0.63	70	-15.1473
-15.7143	0.73	80	-16.3945
-16.7127	0.82	90	-17.0741
-17.2299	0.91	100	-17.4041
-17.4683	1.0	110	-17.5479
-17.5857	1.09	120	-17.6235
-17.6418	1.18	130	-17.6631
-17.6771	1.27	140	-17.6957
-17.703	1.36	150	-17.7160
-17.7218	1.45	160	-17.7272
-17.7369	1.54	170	-17.7463
-17.7561	1.63	180	-17.7646
-17.7704	1.72	190	-17.7808
-17.7897	1.81	200	-17.7972
-17.8056	1.9	210	-17.8223
-17.8326	1.99	220	-17.8447
-17.8508	2.08	230	-17.8658
-17.8699	2.18	240	-17.8773
-17.8777	2.27	250	-17.8862
-17.8827	2.36	260	-17.8912
-17.889	2.45	270	-17.8936
-17.8917	2.54	280	-17.8948
-17.8936	2.63	290	-17.8942
-17.8949	2.72	300	-17.8967
-17.8934	2.81	310	-17.8970
-17.8964	2.9	320	-17.8971
-17.8956	2.99	330	-17.8971

Framework versions

• Transformers 4.36.0.dev0
• Pytorch 2.1.2+cu121
• Datasets 2.14.6
• Tokenizers 0.14.0