metadata

license: apache-2.0
base_model: distilbert-base-cased
tags:
  - generated_from_trainer
metrics:
  - precision
  - recall
  - f1
  - accuracy
model-index:
  - name: distilBert_NER_finer
    results: []
datasets:
  - nlpaueb/finer-139
language:
  - en
pipeline_tag: token-classification

distilBert_NER_finer

This model is a fine-tuned version of distilbert-base-cased on the Finer-139 dataset. It achieves the following results on the evaluation set:

Loss: 0.0198
Precision: 0.9445
Recall: 0.9640
F1: 0.9541
Accuracy: 0.9954

Training and evaluation data

The training data consists of the 4 most widely available ner_tags from the Finer-139 dataset. The training and the test data were curated from this source accordingly

Prediction procedure

from transformers import TAutoTokenizer
from optimum.onnxruntime import ORTModelForTokenClassification
import torch

def onnx_inference(checkpoint, test_data, export=False):
  test_text = " ".join(test_data['tokens'])
  print("Test Text: " + test_text)

  tokenizer = AutoTokenizer.from_pretrained(checkpoint)
  model = ORTModelForTokenClassification.from_pretrained(checkpoint, export=export)

  inputs = tokenizer(test_text, return_tensors="pt")
  outputs = model(**inputs).logits

  predictions = torch.argmax(outputs, dim=2)

  # Convert each tensor element to a scalar before calling .item()
  predicted_token_class = [label_list[int(t)] for t in predictions[0]]
  ner_tags = [label_list[int(t)] for t in test_data['ner_tags']]

  print("Original Tags: ")
  print(ner_tags)
  print("Predicted Tags: ")
  print(predicted_token_class)

onnx_model_path = "" #add the path

onnx_inference(onnx_model_path, test_data)

"""
Here the test_data should contain "tokens" and "ner_tags". This can be of type Dataset.
"""

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 2e-05
train_batch_size: 8
eval_batch_size: 8
seed: 42
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: linear
num_epochs: 3

Training results

Training Loss	Epoch	Step	Validation Loss	Precision	Recall	F1	Accuracy
0.0034	1.0	1620	0.0261	0.9167	0.9668	0.9411	0.9941
0.0031	2.0	3240	0.0182	0.9471	0.9651	0.9561	0.9956
0.0012	3.0	4860	0.0198	0.9445	0.9640	0.9541	0.9954

Framework versions

Transformers 4.38.2
Pytorch 2.2.1+cu121
Datasets 2.18.0
Tokenizers 0.15.2