collapse_gemma-2-2b_hs2_replace_iter11_sftsd2

This model is a fine-tuned version of google/gemma-2-2b on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 2.5696
• Num Input Tokens Seen: 4695928

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 8e-06
• train_batch_size: 8
• eval_batch_size: 16
• seed: 2
• gradient_accumulation_steps: 16
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant_with_warmup
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
No log	0	0	1.3909	0
1.6214	0.0511	5	1.2781	242552
0.8435	0.1021	10	1.2999	477288
0.4883	0.1532	15	1.5045	720128
0.2263	0.2042	20	1.7044	967664
0.114	0.2553	25	1.9947	1212888
0.0907	0.3063	30	2.2002	1453400
0.0368	0.3574	35	2.3314	1694024
0.0282	0.4084	40	2.4725	1936744
0.0231	0.4595	45	2.5584	2177368
0.0227	0.5105	50	2.5691	2417808
0.0262	0.5616	55	2.5540	2656048
0.0214	0.6126	60	2.5430	2897904
0.021	0.6637	65	2.5528	3141864
0.0233	0.7147	70	2.5605	3374832
0.0207	0.7658	75	2.5545	3623264
0.0218	0.8168	80	2.5579	3865080
0.0214	0.8679	85	2.5544	4106824
0.0208	0.9190	90	2.5564	4346216
0.0203	0.9700	95	2.5687	4597608

Framework versions

• Transformers 4.44.0
• Pytorch 2.4.0+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1