FredNajjar/bigbird-QA-squad_v2.2

Fine-tuned google/bigbird-roberta-base model on the SQuAD 2.0 dataset for English extractive question answering.

Model Details

• Language Model: google/bigbird-roberta-base
• Language: English
• Task: Extractive Question Answering
• Data: SQuAD 2.0
• Infrastructure: 1x NVIDIA A100-SXM4-40GB

Training Hyperparameters

• Learning Rate: 3e-05
• Train Batch Size: 16
• Eval Batch Size: 8
• Seed: 42
• Gradient Accumulation Steps: 8
• Total Train Batch Size: 128
• Optimizer: Adam (betas=(0.9,0.999), epsilon=1e-08)
• LR Scheduler: Linear with 121 warmup steps
• Number of Epochs: 3

Results on SQuAD 2.0

• F1 Score: 81.39%
• Exact Match: 77.82%

Usage

from transformers import AutoModelForQuestionAnswering, AutoTokenizer, pipeline
model_name = "FredNajjar/bigbird-QA-squad_v2.2"
nlp = pipeline('question-answering', model=model_name, tokenizer=model_name)
QA_input = {
        'question': 'Your question here',
    'context': 'Your context here'
}
res = nlp(QA_input)

• Framework Versions:
  • Transformers: 4.34.0
  • Pytorch: 2.0.1+cu118
  • Datasets: 2.14.5
  • Tokenizers: 0.14.1

Limitations and Bias

This model inherits limitations and potential biases from the base BigBird model and the SQuAD 2.0 training data.

Contact

For inquiries, please reach out via LinkedIn.

---