qing-yao/babylm-ablated_seed-42_1e-3

babylm_seed-42_1e-3

This model was trained from scratch on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 3.0164
• Accuracy: 0.4206

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.001
• train_batch_size: 32
• eval_batch_size: 64
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 256
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 32000
• num_epochs: 20.0
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
6.1715	1.0	1803	4.2500	0.3062
4.0426	2.0	3606	3.7214	0.3486
3.5775	3.0	5409	3.4693	0.3724
3.3494	4.0	7212	3.3348	0.3847
3.2229	5.0	9015	3.2595	0.3912
3.1569	6.0	10818	3.2099	0.3959
3.0911	7.0	12621	3.1769	0.3996
3.0487	8.0	14424	3.1536	0.4015
3.0168	9.0	16227	3.1386	0.4034
2.9932	10.0	18030	3.1280	0.4049
2.948	11.0	19833	3.1185	0.4058
2.9377	12.0	21636	3.1100	0.4068
2.9329	13.0	23439	3.1038	0.4075
2.928	14.0	25242	3.1013	0.4081
2.9258	15.0	27045	3.0978	0.4083
2.8846	16.0	28848	3.0946	0.4086
2.8898	17.0	30651	3.0941	0.4088
2.8957	18.0	32454	3.0807	0.4106
2.8247	19.0	34257	3.0345	0.4165
2.6757	20.0	36060	3.0164	0.4206

Framework versions

• Transformers 4.45.1
• Pytorch 2.4.1+cu121
• Datasets 2.19.1
• Tokenizers 0.20.0