albert-xxlarge-v2-Adapters_2

This model is a fine-tuned version of albert/albert-xxlarge-v2 on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.5145

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 32
• eval_batch_size: 32
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: inverse_sqrt
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 5
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
1.1194	0.2291	50	1.1023
1.0831	0.4582	100	1.0770
1.0148	0.6873	150	0.9637
0.9064	0.9164	200	0.8538
0.7693	1.1455	250	0.7700
0.7445	1.3746	300	0.7098
0.7045	1.6037	350	0.6914
0.6733	1.8328	400	0.6374
0.6239	2.0619	450	0.6226
0.6208	2.2910	500	0.5913
0.6153	2.5200	550	0.5811
0.591	2.7491	600	0.5630
0.5894	2.9782	650	0.5500
0.5788	3.2073	700	0.5461
0.5187	3.4364	750	0.5353
0.5271	3.6655	800	0.5386
0.5812	3.8946	850	0.5309
0.5532	4.1237	900	0.5296
0.5602	4.3528	950	0.5204
0.5209	4.5819	1000	0.5144
0.5579	4.8110	1050	0.5145

Framework versions

• PEFT 0.11.1
• Transformers 4.42.4
• Pytorch 2.1.0
• Datasets 2.20.0
• Tokenizers 0.19.1