task_categories:question-answering size_categories:10K<n<100K language:en license:cc-by-4.0 region:us
DUDE requires models to reason and understand about document layouts in multi-page images/PDFs to answer questions about them.
Specifically, models need to incorporate a new modality of layout present in the images/PDFs and reason
over it to answer DUDE questions.
@inproceedings{dude2023icdar,
    title={ICDAR 2023 Challenge on Document UnderstanDing of Everything (DUDE)},
    author={Van Landeghem, Jordy et . al.},
    booktitle={Proceedings of the ICDAR},
    year={2023}
}
---
license: cc-by-4.0
task_categories:
- question-answering
language:
- en
pretty_name: DUDE
size_categories:
- 10K<n<100K
---


## Loading the dataset with a specific configuration

There are 3 different OCR versions to choose from with their original format or standardized DUE format, as well as the option to load the documents as filepaths or as binaries (PDF).
To load a specific configuration, pass a config from one of the following:

```python
#{bin_}{Amazon,Azure,Tesseract}_{original,due}
['Amazon_due', 'Amazon_original', 'Azure_due', 'Azure_original', 'Tesseract_due', 'Tesseract_original', 
'bin_Amazon_due', 'bin_Amazon_original', 'bin_Azure_due', 'bin_Azure_original', 'bin_Tesseract_due', 'bin_Tesseract_original']
```

Loading the dataset:
```python
from datasets import load_dataset

ds = load_dataset("jordyvl/DUDE_loader", 'Amazon_original')
```

This dataset repository contains helper functions to convert the dataset to ImDB (image database) format. 
We advise to clone the repository and run it according to your preferences (OCR version, lowercasing, ...).
When running the above data loading script, you should be able to find the extracted binaries under the [HF_CACHE](https://huggingface.co/docs/datasets/cache):  
`HF_CACHE/datasets/downloads/extracted/<hash>/DUDE_train-val-test_binaries`, which can be reused for the `data_dir` argument.

For example: 

```bash
python3 DUDE_imdb_loader.py \
--data_dir ~/.cache/huggingface/datasets/downloads/extracted/7adde0ed7b0150b7f6b32e52bcad452991fde0f3407c8a87e74b1cb475edaa5b/DUDE_train-val-test_binaries/
```

For baselines, we recommend having a look at the [MP-DocVQA repository](https://github.com/rubenpt91/MP-DocVQA-Framework)

We strongly encourage you to benchmark your best models and submit test set predictions on the [DUDE competition leaderboard](https://rrc.cvc.uab.es/?ch=23)
To help with test set predictions, we have included a sample submission file `RRC_DUDE_testset_submission_example.json`.

