collapse_gemma-2-2b_hs2_replace_iter17_sftsd0

This model is a fine-tuned version of google/gemma-2-2b on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 2.6606
• Num Input Tokens Seen: 4619200

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 8e-06
• train_batch_size: 8
• eval_batch_size: 16
• seed: 0
• gradient_accumulation_steps: 16
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant_with_warmup
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
No log	0	0	1.3909	0
1.473	0.0511	5	1.2822	235136
0.7984	0.1021	10	1.3474	469808
0.5711	0.1532	15	1.5390	706816
0.2735	0.2042	20	1.7631	947744
0.1475	0.2553	25	2.0300	1178720
0.0929	0.3063	30	2.2727	1417608
0.0461	0.3574	35	2.4081	1660192
0.0308	0.4084	40	2.5069	1896288
0.0242	0.4595	45	2.5727	2132784
0.0244	0.5105	50	2.6064	2367208
0.0226	0.5616	55	2.6217	2602720
0.0225	0.6126	60	2.6156	2845336
0.0246	0.6637	65	2.6181	3087760
0.0217	0.7147	70	2.6219	3320336
0.0222	0.7658	75	2.6358	3565024
0.0287	0.8168	80	2.6363	3801992
0.0229	0.8679	85	2.6469	4048344
0.0236	0.9190	90	2.6532	4289752
0.0232	0.9700	95	2.6590	4521864

Framework versions

• Transformers 4.44.0
• Pytorch 2.4.0+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1