G0514HMA20H

This model is a fine-tuned version of google/gemma-2b on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: -17.8888

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 16
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine_with_restarts
• lr_scheduler_warmup_steps: 60
• num_epochs: 3
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
0.7427	0.09	10	-0.3960
-1.7071	0.18	20	-3.6281
-5.2642	0.27	30	-7.4868
-9.0314	0.36	40	-11.0106
-12.2572	0.45	50	-13.9498
-14.8718	0.54	60	-15.9757
-16.4839	0.63	70	-16.9719
-17.1594	0.73	80	-17.3585
-17.4323	0.82	90	-17.5206
-17.5617	0.91	100	-17.6042
-17.6212	1.0	110	-17.6490
-17.6671	1.09	120	-17.6807
-17.6884	1.18	130	-17.6997
-17.7069	1.27	140	-17.7167
-17.7212	1.36	150	-17.7313
-17.7366	1.45	160	-17.7408
-17.7489	1.54	170	-17.7577
-17.7679	1.63	180	-17.7766
-17.7836	1.72	190	-17.7952
-17.8096	1.81	200	-17.8171
-17.8256	1.9	210	-17.8360
-17.844	1.99	220	-17.8501
-17.8551	2.08	230	-17.8607
-17.8656	2.18	240	-17.8707
-17.8732	2.27	250	-17.8780
-17.8763	2.36	260	-17.8821
-17.8785	2.45	270	-17.8839
-17.8842	2.54	280	-17.8864
-17.8859	2.63	290	-17.8876
-17.8859	2.72	300	-17.8882
-17.8836	2.81	310	-17.8885
-17.8875	2.9	320	-17.8888
-17.8874	2.99	330	-17.8888

Framework versions

• Transformers 4.36.0.dev0
• Pytorch 2.1.2+cu121
• Datasets 2.14.6
• Tokenizers 0.14.0