collapse_gemma-2-2b_hs2_replace_iter16_sftsd2

This model is a fine-tuned version of google/gemma-2-2b on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 2.6148
• Num Input Tokens Seen: 4649928

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 8e-06
• train_batch_size: 8
• eval_batch_size: 16
• seed: 2
• gradient_accumulation_steps: 16
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant_with_warmup
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
No log	0	0	1.3909	0
1.588	0.0511	5	1.2799	240344
0.8368	0.1021	10	1.3137	475832
0.4007	0.1532	15	1.5688	715888
0.2103	0.2042	20	1.7771	959272
0.0919	0.2553	25	2.1055	1202984
0.0579	0.3063	30	2.2288	1447360
0.0297	0.3574	35	2.4233	1690768
0.0258	0.4084	40	2.5438	1932088
0.0221	0.4595	45	2.5827	2172616
0.0207	0.5105	50	2.5997	2410040
0.022	0.5616	55	2.5889	2646200
0.0196	0.6126	60	2.6011	2888992
0.0203	0.6637	65	2.5980	3125144
0.0201	0.7147	70	2.6088	3358040
0.0197	0.7658	75	2.6171	3601728
0.0227	0.8168	80	2.6057	3835136
0.0201	0.8679	85	2.6070	4077024
0.0217	0.9190	90	2.6091	4307872
0.0194	0.9700	95	2.6178	4555016

Framework versions

• Transformers 4.44.0
• Pytorch 2.4.0+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1