collapse_gemma-2-2b_hs2_replace_iter14_sftsd1

This model is a fine-tuned version of google/gemma-2-2b on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 2.5773
• Num Input Tokens Seen: 4574776

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 8e-06
• train_batch_size: 8
• eval_batch_size: 16
• seed: 1
• gradient_accumulation_steps: 16
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant_with_warmup
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
No log	0	0	1.3909	0
1.5075	0.0511	5	1.2776	235544
0.836	0.1022	10	1.2777	467032
0.4955	0.1533	15	1.4886	700024
0.3103	0.2043	20	1.6793	936792
0.1031	0.2554	25	1.9368	1172032
0.0885	0.3065	30	2.1415	1401888
0.0543	0.3576	35	2.2858	1638624
0.0376	0.4087	40	2.3690	1882040
0.0231	0.4598	45	2.4836	2115960
0.0275	0.5109	50	2.5069	2358760
0.0255	0.5619	55	2.5359	2591288
0.0237	0.6130	60	2.5399	2832448
0.024	0.6641	65	2.5485	3068016
0.024	0.7152	70	2.5532	3301008
0.0249	0.7663	75	2.5420	3541120
0.0225	0.8174	80	2.5494	3772632
0.0232	0.8685	85	2.5649	4012576
0.0234	0.9195	90	2.5743	4257128
0.0228	0.9706	95	2.5775	4478968

Framework versions

• Transformers 4.44.0
• Pytorch 2.4.0+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1