Edit model card

deberta-v3-large-finetuned-squadv2

This model is a version of microsoft/deberta-v3-large fine-tuned on the SQuAD version 2.0 dataset. Fine-tuning & evaluation on a NVIDIA Titan RTX - 24GB GPU took 15 hours.

Results from 2023 ICLR paper, "DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing", by Pengcheng He, et. al.

  • 'EM' : 89.0
  • 'F1' : 91.5

Results calculated with:

metrics = evaluate.load("squad_v2")
squad_v2_metrics = metrics.compute(predictions = formatted_predictions, references = references)

for this fine-tuning:

  • 'exact' : 88.70,
  • 'f1' : 91.52,
  • 'total' : 11873,
  • 'HasAns_exact' : 83.70,
  • 'HasAns_f1' : 89.35,
  • 'HasAns_total' : 5928,
  • 'NoAns_exact' : 93.68,
  • 'NoAns_f1' : 93.68,
  • 'NoAns_total' : 5945,
  • 'best_exact' : 88.70,
  • 'best_exact_thresh' : 0.0,
  • 'best_f1' : 91.52,
  • 'best_f1_thresh' : 0.0}

Model description

For the authors' models, code & detailed information see: https://github.com/microsoft/DeBERTa

Intended uses

Extractive question answering on a given context

Fine-tuning hyperparameters

The following hyperparameters, as suggested by the 2023 ICLR paper noted above, were used during fine-tuning:

  • learning_rate : 1e-05
  • train_batch_size : 8
  • eval_batch_size : 8
  • seed : 42
  • gradient_accumulation_steps : 8
  • total_train_batch_size : 64
  • optimizer : Adam with betas = (0.9, 0.999) and epsilon = 1e-06
  • lr_scheduler_type : linear
  • lr_scheduler_warmup_steps : 1000
  • training_steps : 5200

Framework versions

  • Transformers : 4.35.0.dev0
  • Pytorch : 2.1.0+cu121
  • Datasets : 2.14.5
  • Tokenizers : 0.14.0

System

  • CPU : Intel(R) Core(TM) i9-9900K - 32GB RAM
  • Python version : 3.11.5 [GCC 11.2.0] (64-bit runtime)
  • Python platform : Linux-5.15.0-86-generic-x86_64-with-glibc2.35
  • GPU : NVIDIA TITAN RTX - 24GB Memory
  • CUDA runtime version : 12.1.105
  • Nvidia driver version : 535.113.01

Fine-tuning (Training) results before/after the best model (Step 3620)

Training Loss Epoch Step Validation Loss
0.5323 1.72 3500 0.5860
0.5129 1.73 3520 0.5656
0.5441 1.74 3540 0.5642
0.5624 1.75 3560 0.5873
0.4645 1.76 3580 0.5891
0.5577 1.77 3600 0.5816
0.5199 1.78 3620 0.5579
0.5061 1.79 3640 0.5837
0.484 1.79 3660 0.5721
0.5095 1.8 3680 0.5821
0.5342 1.81 3700 0.5602
Downloads last month
1,220
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Model tree for ahotrod/deberta-v3-large-finetuned-squadv2

Finetuned
(105)
this model

Dataset used to train ahotrod/deberta-v3-large-finetuned-squadv2

Evaluation results