collapse_gemma-2-2b_hs2_replace_iter13_sftsd1

This model is a fine-tuned version of google/gemma-2-2b on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 2.5703
• Num Input Tokens Seen: 4616360

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 8e-06
• train_batch_size: 8
• eval_batch_size: 16
• seed: 1
• gradient_accumulation_steps: 16
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant_with_warmup
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
No log	0	0	1.3909	0
1.5085	0.0518	5	1.2783	240416
0.8501	0.1037	10	1.2793	483592
0.5486	0.1555	15	1.4415	723168
0.2963	0.2074	20	1.6758	961528
0.1623	0.2592	25	1.8930	1208112
0.0853	0.3111	30	2.1736	1446032
0.0375	0.3629	35	2.3490	1691608
0.0309	0.4148	40	2.5054	1936144
0.0246	0.4666	45	2.5678	2176536
0.0258	0.5185	50	2.5754	2418848
0.0254	0.5703	55	2.5852	2653968
0.0371	0.6222	60	2.5747	2900640
0.0236	0.6740	65	2.5400	3133040
0.0216	0.7259	70	2.5238	3373560
0.0218	0.7777	75	2.5299	3613712
0.0211	0.8296	80	2.5430	3853136
0.0229	0.8814	85	2.5534	4092912
0.0227	0.9332	90	2.5591	4325832
0.0217	0.9851	95	2.5698	4570496

Framework versions

• Transformers 4.44.0
• Pytorch 2.4.0+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1