Edit model card

Longformer base fine-tuned on MP-DocVQA

This is Longformer-base trained on SQuAD v1 from Valhalla hub and fine-tuned on Multipage DocVQA (MP-DocVQA) dataset.

This model was used as a baseline in Hierarchical multimodal transformers for Multi-Page DocVQA.

  • Results on the MP-DocVQA dataset are reported in Table 2.
  • Training hyperparameters can be found in Table 8 of Appendix D.

How to use

Inference

How to use this model to perform inference on a sample question and context in PyTorch:

from transformers import LongformerTokenizerFast, LongformerForQuestionAnswering

tokenizer = LongformerTokenizerFast.from_pretrained("rubentito/longformer-base-mpdocvqa")
model = LongformerForQuestionAnswering.from_pretrained("rubentito/longformer-base-mpdocvqa")

text = "Huggingface has democratized NLP. Huge thanks to Huggingface for this."
question = "What has Huggingface done?"

encoding = tokenizer(question, text, return_tensors="pt")
output = model(encoding["input_ids"], attention_mask=encoding["attention_mask"])

start_pos = torch.argmax(output.start_logits, dim=-1).item()
end_pos = torch.argmax(output.end_logits, dim=-1).item()

context_tokens = tokenizer.convert_ids_to_tokens(encoding["input_ids"][0].tolist())
answer_tokens = context_tokens[start_pos: end_pos + 1]
pred_answer = tokenizer.decode(tokenizer.convert_tokens_to_ids(answer_tokens))

Metrics

Average Normalized Levenshtein Similarity (ANLS)

The standard metric for text-based VQA tasks (ST-VQA and DocVQA). It evaluates the method's reasoning capabilities while smoothly penalizes OCR recognition errors. Check Scene Text Visual Question Answering for detailed information.

Answer Page Prediction Accuracy (APPA)

In the MP-DocVQA task, the models can provide the index of the page where the information required to answer the question is located. For this subtask accuracy is used to evaluate the predictions: i.e. if the predicted page is correct or not. Check Hierarchical multimodal transformers for Multi-Page DocVQA for detailed information.

Model results

Extended experimentation can be found in Table 2 of Hierarchical multimodal transformers for Multi-Page DocVQA. You can also check the live leaderboard at the RRC Portal.

Model HF name Parameters ANLS APPA
Bert large rubentito/bert-large-mpdocvqa 334M 0.4183 51.6177
Longformer base rubentito/longformer-base-mpdocvqa 148M 0.5287 71.1696
BigBird ITC base rubentito/bigbird-base-itc-mpdocvqa 131M 0.4929 67.5433
LayoutLMv3 base rubentito/layoutlmv3-base-mpdocvqa 125M 0.4538 51.9426
T5 base rubentito/t5-base-mpdocvqa 223M 0.5050 0.0000
Hi-VT5 rubentito/hivt5-base-mpdocvqa 316M 0.6201 79.23

Citation Information

@article{tito2022hierarchical,
  title={Hierarchical multimodal transformers for Multi-Page DocVQA},
  author={Tito, Rub{\`e}n and Karatzas, Dimosthenis and Valveny, Ernest},
  journal={arXiv preprint arXiv:2212.05935},
  year={2022}
}
Downloads last month
28
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Model tree for rubentito/longformer-base-mpdocvqa

Finetuned
(2)
this model

Dataset used to train rubentito/longformer-base-mpdocvqa