distilbert-base-uncased-finetuned-natural-questions

This model is a fine-tuned version of distilbert-base-uncased on the natural_questions dataset. It achieves the following results on the evaluation set:

• Loss: 0.6267

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 64
• eval_batch_size: 64
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 40

Training results

Training Loss	Epoch	Step	Validation Loss
2.0532	1.0	5104	0.2393
1.8912	2.0	10208	0.2284
1.7854	3.0	15312	0.2357
1.6856	4.0	20416	0.2487
1.5918	5.0	25520	0.2743
1.5067	6.0	30624	0.2586
1.4323	7.0	35728	0.2763
1.365	8.0	40832	0.2753
1.3162	9.0	45936	0.3200
1.281	10.0	51040	0.3127
1.308	11.0	57104	0.2947
1.241	12.0	62208	0.2941
1.1391	13.0	67312	0.3103
1.0334	14.0	72416	0.3694
0.9538	15.0	77520	0.3658
0.8749	16.0	82624	0.4009
0.8154	17.0	87728	0.3672
0.7533	18.0	92832	0.3675
0.7079	19.0	97936	0.4611
0.6658	20.0	103040	0.4222
0.595	21.0	108144	0.4095
0.5765	22.0	113248	0.4400
0.5259	23.0	118352	0.5109
0.4804	24.0	123456	0.4711
0.4389	25.0	128560	0.5072
0.4034	26.0	133664	0.5363
0.374	27.0	138768	0.5460
0.3434	28.0	143872	0.5627
0.3181	29.0	148976	0.5657
0.2971	30.0	154080	0.5819
0.275	31.0	159184	0.5649
0.2564	32.0	164288	0.6087
0.2431	33.0	169392	0.6137
0.2289	34.0	174496	0.6123
0.2151	35.0	179600	0.5979
0.2041	36.0	184704	0.6196
0.1922	37.0	189808	0.6191
0.1852	38.0	194912	0.6313
0.1718	39.0	200016	0.6234
0.1718	39.81	204160	0.6267

Framework versions

• Transformers 4.17.0
• Pytorch 1.10.0
• Datasets 1.18.4
• Tokenizers 0.11.6