pretoxtm-ner

This model is a fine-tuned version of dmis-lab/biobert-v1.1 on javicorvi/pretoxtm-dataset. It achieves the following results on the evaluation set:

• Loss: 0.2722
• Study Test: {'precision': 0.8222222222222222, 'recall': 0.8763157894736842, 'f1': 0.8484076433121018, 'number': 380}
• Manifestation: {'precision': 0.841025641025641, 'recall': 0.9265536723163842, 'f1': 0.8817204301075269, 'number': 177}
• Finding: {'precision': 0.7870485678704857, 'recall': 0.8154838709677419, 'f1': 0.8010139416983523, 'number': 775}
• Specimen: {'precision': 0.7793427230046949, 'recall': 0.8469387755102041, 'f1': 0.8117359413202934, 'number': 392}
• Dose: {'precision': 0.9595959595959596, 'recall': 0.9726962457337884, 'f1': 0.9661016949152542, 'number': 293}
• Dose Qualification: {'precision': 0.8787878787878788, 'recall': 0.8787878787878788, 'f1': 0.8787878787878788, 'number': 33}
• Sex: {'precision': 0.9279279279279279, 'recall': 0.9809523809523809, 'f1': 0.9537037037037037, 'number': 105}
• Group: {'precision': 0.8913043478260869, 'recall': 0.8817204301075269, 'f1': 0.8864864864864864, 'number': 93}
• Precision: 0.8298
• Recall: 0.8719
• F1: 0.8503
• Accuracy: 0.9530

Model description

PretoTM NER is a model developed to recognize relevant entities associated with treatment-related findings in preclinical toxicology.

Training and evaluation data

The model was trained on javicorvi/pretoxtm-dataset.

The dataset is divided in train, validation and test.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5.760003080365119e-05
• train_batch_size: 4
• eval_batch_size: 8
• seed: 4
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 7

Training results

Training Loss	Epoch	Step	Validation Loss	Study Test	Manifestation	Finding	Specimen	Dose	Dose Qualification	Sex	Group	Precision	Recall	F1	Accuracy
0.3365	1.0	514	0.2023	{'precision': 0.740909090909091, 'recall': 0.8578947368421053, 'f1': 0.7951219512195122, 'number': 380}	{'precision': 0.7874396135265701, 'recall': 0.9209039548022598, 'f1': 0.8489583333333334, 'number': 177}	{'precision': 0.7055214723926381, 'recall': 0.7419354838709677, 'f1': 0.7232704402515724, 'number': 775}	{'precision': 0.7312072892938497, 'recall': 0.8188775510204082, 'f1': 0.772563176895307, 'number': 392}	{'precision': 0.9243986254295533, 'recall': 0.9180887372013652, 'f1': 0.9212328767123289, 'number': 293}	{'precision': 0.8, 'recall': 0.8484848484848485, 'f1': 0.823529411764706, 'number': 33}	{'precision': 0.9074074074074074, 'recall': 0.9333333333333333, 'f1': 0.9201877934272301, 'number': 105}	{'precision': 0.7441860465116279, 'recall': 0.6881720430107527, 'f1': 0.7150837988826816, 'number': 93}	0.7617	0.8203	0.7899	0.9397
0.1325	2.0	1028	0.2012	{'precision': 0.7920792079207921, 'recall': 0.8421052631578947, 'f1': 0.8163265306122449, 'number': 380}	{'precision': 0.8488372093023255, 'recall': 0.8248587570621468, 'f1': 0.836676217765043, 'number': 177}	{'precision': 0.7163289630512515, 'recall': 0.775483870967742, 'f1': 0.7447335811648079, 'number': 775}	{'precision': 0.723175965665236, 'recall': 0.8596938775510204, 'f1': 0.7855477855477856, 'number': 392}	{'precision': 0.9013157894736842, 'recall': 0.9351535836177475, 'f1': 0.9179229480737018, 'number': 293}	{'precision': 0.8181818181818182, 'recall': 0.8181818181818182, 'f1': 0.8181818181818182, 'number': 33}	{'precision': 0.911504424778761, 'recall': 0.9809523809523809, 'f1': 0.944954128440367, 'number': 105}	{'precision': 0.8645833333333334, 'recall': 0.8924731182795699, 'f1': 0.8783068783068784, 'number': 93}	0.7792	0.8412	0.8090	0.9450
0.0733	3.0	1542	0.2101	{'precision': 0.7451403887688985, 'recall': 0.9078947368421053, 'f1': 0.8185053380782918, 'number': 380}	{'precision': 0.7922705314009661, 'recall': 0.9265536723163842, 'f1': 0.8541666666666666, 'number': 177}	{'precision': 0.7481840193704601, 'recall': 0.7974193548387096, 'f1': 0.7720174890693317, 'number': 775}	{'precision': 0.7676056338028169, 'recall': 0.8341836734693877, 'f1': 0.7995110024449877, 'number': 392}	{'precision': 0.9276315789473685, 'recall': 0.962457337883959, 'f1': 0.9447236180904522, 'number': 293}	{'precision': 0.7941176470588235, 'recall': 0.8181818181818182, 'f1': 0.8059701492537314, 'number': 33}	{'precision': 0.9292035398230089, 'recall': 1.0, 'f1': 0.9633027522935781, 'number': 105}	{'precision': 0.8695652173913043, 'recall': 0.8602150537634409, 'f1': 0.8648648648648649, 'number': 93}	0.7903	0.8665	0.8266	0.9484
0.0431	4.0	2056	0.2260	{'precision': 0.8477157360406091, 'recall': 0.8789473684210526, 'f1': 0.8630490956072351, 'number': 380}	{'precision': 0.8190954773869347, 'recall': 0.9209039548022598, 'f1': 0.8670212765957446, 'number': 177}	{'precision': 0.7653562653562653, 'recall': 0.8038709677419354, 'f1': 0.7841409691629955, 'number': 775}	{'precision': 0.7897196261682243, 'recall': 0.8622448979591837, 'f1': 0.824390243902439, 'number': 392}	{'precision': 0.9459459459459459, 'recall': 0.9556313993174061, 'f1': 0.9507640067911713, 'number': 293}	{'precision': 0.8055555555555556, 'recall': 0.8787878787878788, 'f1': 0.8405797101449276, 'number': 33}	{'precision': 0.9203539823008849, 'recall': 0.9904761904761905, 'f1': 0.9541284403669724, 'number': 105}	{'precision': 0.8842105263157894, 'recall': 0.9032258064516129, 'f1': 0.8936170212765957, 'number': 93}	0.8232	0.8697	0.8458	0.9515
0.0282	5.0	2570	0.2492	{'precision': 0.835820895522388, 'recall': 0.8842105263157894, 'f1': 0.8593350383631714, 'number': 380}	{'precision': 0.8333333333333334, 'recall': 0.9322033898305084, 'f1': 0.8800000000000001, 'number': 177}	{'precision': 0.7820197044334976, 'recall': 0.8193548387096774, 'f1': 0.800252047889099, 'number': 775}	{'precision': 0.785377358490566, 'recall': 0.8494897959183674, 'f1': 0.8161764705882352, 'number': 392}	{'precision': 0.9627118644067797, 'recall': 0.9692832764505119, 'f1': 0.9659863945578231, 'number': 293}	{'precision': 0.8235294117647058, 'recall': 0.8484848484848485, 'f1': 0.8358208955223881, 'number': 33}	{'precision': 0.9285714285714286, 'recall': 0.9904761904761905, 'f1': 0.9585253456221199, 'number': 105}	{'precision': 0.9120879120879121, 'recall': 0.8924731182795699, 'f1': 0.9021739130434783, 'number': 93}	0.8311	0.8754	0.8527	0.9528
0.0125	6.0	3084	0.2668	{'precision': 0.830423940149626, 'recall': 0.8763157894736842, 'f1': 0.852752880921895, 'number': 380}	{'precision': 0.839572192513369, 'recall': 0.8870056497175142, 'f1': 0.8626373626373628, 'number': 177}	{'precision': 0.7724477244772447, 'recall': 0.8103225806451613, 'f1': 0.7909319899244331, 'number': 775}	{'precision': 0.7617977528089888, 'recall': 0.8647959183673469, 'f1': 0.8100358422939069, 'number': 392}	{'precision': 0.9726962457337884, 'recall': 0.9726962457337884, 'f1': 0.9726962457337884, 'number': 293}	{'precision': 0.875, 'recall': 0.8484848484848485, 'f1': 0.8615384615384615, 'number': 33}	{'precision': 0.9279279279279279, 'recall': 0.9809523809523809, 'f1': 0.9537037037037037, 'number': 105}	{'precision': 0.8913043478260869, 'recall': 0.8817204301075269, 'f1': 0.8864864864864864, 'number': 93}	0.8235	0.8697	0.8460	0.9529
0.006	7.0	3598	0.2722	{'precision': 0.8222222222222222, 'recall': 0.8763157894736842, 'f1': 0.8484076433121018, 'number': 380}	{'precision': 0.841025641025641, 'recall': 0.9265536723163842, 'f1': 0.8817204301075269, 'number': 177}	{'precision': 0.7870485678704857, 'recall': 0.8154838709677419, 'f1': 0.8010139416983523, 'number': 775}	{'precision': 0.7793427230046949, 'recall': 0.8469387755102041, 'f1': 0.8117359413202934, 'number': 392}	{'precision': 0.9595959595959596, 'recall': 0.9726962457337884, 'f1': 0.9661016949152542, 'number': 293}	{'precision': 0.8787878787878788, 'recall': 0.8787878787878788, 'f1': 0.8787878787878788, 'number': 33}	{'precision': 0.9279279279279279, 'recall': 0.9809523809523809, 'f1': 0.9537037037037037, 'number': 105}	{'precision': 0.8913043478260869, 'recall': 0.8817204301075269, 'f1': 0.8864864864864864, 'number': 93}	0.8298	0.8719	0.8503	0.9530

Framework versions

• Transformers 4.39.3
• Pytorch 2.2.1+cu121
• Datasets 2.18.0
• Tokenizers 0.15.2