collapse_gemma-2-2b_hs2_accumulatesubsample_iter16_sftsd0

This model is a fine-tuned version of google/gemma-2-2b on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.2147
• Num Input Tokens Seen: 5007000

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 8e-06
• train_batch_size: 8
• eval_batch_size: 16
• seed: 0
• gradient_accumulation_steps: 16
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant_with_warmup
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
No log	0	0	1.3909	0
1.3688	0.0535	5	1.2792	267216
1.1045	0.1071	10	1.2312	538752
0.9071	0.1606	15	1.2252	810008
0.8884	0.2142	20	1.2472	1090696
0.7495	0.2677	25	1.2613	1357168
0.6986	0.3213	30	1.2627	1622320
0.566	0.3748	35	1.2669	1885016
0.4901	0.4284	40	1.2516	2150224
0.5529	0.4819	45	1.2437	2426776
0.5463	0.5355	50	1.2177	2694864
0.5361	0.5890	55	1.2260	2967472
0.4028	0.6426	60	1.2219	3235816
0.4417	0.6961	65	1.2218	3506192
0.3597	0.7497	70	1.2340	3766528
0.4094	0.8032	75	1.2152	4037064
0.4751	0.8568	80	1.2227	4299936
0.3216	0.9103	85	1.2123	4567080
0.4108	0.9639	90	1.2150	4843480

Framework versions

• Transformers 4.44.0
• Pytorch 2.4.0+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1