task_categories:table-question-answering task_categories:question-answering size_categories:100K<n<1M language:en license:cc-by-sa-3.0 documents tables VQA region:us
---
license: cc-by-sa-3.0
task_categories:
- table-question-answering
- question-answering
language:
- en
tags:
- documents
- tables
- VQA
pretty_name: WikiDT
size_categories:
- 100K<n<1M
---
# WikiDT: Wikipedia Table Document dataset for table extraction and visual question answering

## Dataset Description

- **Homepage:** 
- **Repository:** 
- **Paper:** 
- **Leaderboard:** 
- **Point of Contact:** 

### Dataset Summary

The WikiDT contains multi-level annotations and labels for the question-answering task based on images. Meanwhile, as the questions are answered from some table on the image, and WikiDT provides the table annotation to facilitate the diagnosis of the models and decompose the problem, WikiDT can be also directly used as a 
table recognition dataset.

The dataset contains 16,887 Wikipedia screenshot, which are segmented to 54,032 subpages since the full screenshots are potentially long. In total, there's 159,905 tables in the dataset. The number of question-answer samples is 70,652. Each QA sample contains triplets of <question, answer, full-page screenshot filename>, and is additionally annotated with retrieval labels (which subpage, and which table). 53,698 QA samples also have SQL annotation. 

For each subpage, OCR and table extraction annotations from two sources are available. While rendering the screenshots, the ground truth table annotation is recorded. Meanwhile, to make the dataset realistic, we also requested OCR and table extraction from [Amazon Textract](https://aws.amazon.com/textract/) for each subpage (results obtained during Feb.28, 2023 - Mar.6, 2023). 

### Languages

English

## Dataset Structure

Once downloaded, the WikiDT has the following parts. The downloaded files are around 77GB. Please ensure you have at least 160GB since we will be extract individual files from the tars.

```
.
├── WikiTableExtraction
│   ├── detection.partaa
│   ├── detection.partab
│   ├── detection.partac
│   ├── detection.partad
│   ├── detection.partae
│   ├── detection.partaf
│   ├── detection.partag
│   ├── structure.partaa
│   ├── structure.partab
│   ├── structure.partac
│   ├── structure.partad
│   └── structure.partae
├── images.partaa
├── images.partab
├── images.partac
├── images.partad
├── images.partae
├── images.partaf
├── images.partag
├── images.partah
├── images.partai
├── ocr.tar
├── samples
│   ├── test.json
│   ├── train.json
│   └── val.json
└── tsv.tar
```

Please concat the part files and extract them into respective folder. For example, 
run
```
cd WikiTableExtraction/
cat detection.parta* | tar x
```
to extract the `detection` folder.
Once you extracted all the tar files, the WikiDT dataset has the following file structure.

```sh
+--WikiDT-dataset
|  +--WikiTableExtraction
|  |  +--detection
|  |  |  +--images  # sub page images
|  |  |  +--train  # xml table bbox annotation
|  |  |  +--test  # xml table bbox annotation
|  |  |  +--val  # xml table bbox annotation
|  |  |  images_filelist.txt  # index of 54,032 images
|  |  |  test_filelist.txt  # index of 5,410 test samples 
|  |  |  train_filelist.txt  # index of 43,248 train samples
|  |  |  val_filelist.txt  # index of 5,347 val samples
|  |  +--structure
|  |  |  +--images  # images cropped to table region
|  |  |  +--train  # xml table bbox annotation
|  |  |  +--test  # xml table bbox annotation
|  |  |  +--val  # xml table bbox annotation
|  |  |  images_filelist.txt  # index of 159,898 images
|  |  |  test_filelist.txt  # index of 15,989 test samples
|  |  |  train_filelist.txt  # index of 129,980 train samples
|  |  |  val_filelist.txt  # index of 15,991 val samples
|  +--samples  # in total 70,652 TableVQA samples from the three json files 
|  |  +--train.json  #
|  |  +--test.json  #
|  |  +--val.json  #
|  +--images  # full page image
|  +--ocr  #  text and bbox for the table content
|  |  +--textract # detected by Amazon Textract API
|  |  +--web # extracted from HTML information
|  +--tsv  # extracted table in tsv format
|  |  +--textract # detected by Amazon Textract API
|  |  +--web # extracted from HTML information
```

### Table VQA annotation example

Here is an example of an xml table bbox annotation from `WikiDT-dataset/samples/[train|test|val].json/`.

```
{'all_ocr_files_textract': ['ocr/textract/16301437_page_seg_0.json',
                            'ocr/textract/16301437_page_seg_1.json'],
 'all_ocr_files_web': ['ocr/web/16301437_page_seg_0.json',
                       'ocr/web/16301437_page_seg_1.json'],
 'all_table_files_textract': ['tsv/textract/16301437_page_0.tsv',
                              'tsv/textract/16301437_page_1.tsv'],
 'all_table_files_web': ['tsv/web/16301437_1.tsv', 'tsv/web/16301437_0.tsv'],
 'answer': [['don johnson buckeye st. classic']],
 'image': '16301437_page.png',
 'ocr_retrieval_file_textract': 'ocr/textract/16301437_page_seg_0.json',
 'ocr_retrieval_file_web': 'ocr/web/16301437_page_seg_0.json',
 'question': 'Name the Event which has a Score of 209-197?',
 'sample_id': '14190',
 'sql_str': "SELECT `event` FROM cur_table WHERE  `score` = '209-197'  ",
 'sub_page': ['16301437_page_seg_0.png', '16301437_page_seg_1.png'],
 'sub_page_retrieved': '16301437_page_seg_0.png',
 'subset': 'TFC',
 'table_id': '2-16301437-1',
 'table_retrieval_file_textract': 'tsv/textract/16301437_page_0.tsv',
 'table_retrieval_file_web': 'tsv/web/16301437_1.tsv'}
```


### Table Detection annotation example

Here is an example of an xml table bbox annotation from `WikiDT-dataset/WikiTableExtraction/structure/[train|test|val]/`.

```xml
<annotation>
  <folder />
  <filename>204_147_page_crop_5.png</filename>
  <source>WikiDT Dataset</source>
  <size>
    <width>788</width>
    <height>540.0</height>
    <depth>3</depth>
  </size>
  <object>
    <name>table</name>
    <rowspan />
    <colspan />
    <bndbox>
      <xmin>10</xmin>
      <ymin>10</ymin>
      <xmax>778</xmax>
      <ymax>530</ymax>
    </bndbox>
  </object>
  <object>
    <name>header row</name>
    <rowspan />
    <colspan />
    <bndbox>
      <xmin>10</xmin>
      <ymin>10</ymin>
      <xmax>778</xmax>
      <ymax>33</ymax>
    </bndbox>
  </object> 
  <object>
    <name>header cell</name>
    <rowspan />
    <colspan>10</colspan>
    <bndbox>
      <xmin>12</xmin>
      <ymin>35</ymin>
      <xmax>776</xmax>
      <ymax>58</ymax>
    </bndbox>
  </object>
  <object>
    <name>table row</name>
    <rowspan />
    <colspan />
    <bndbox>
      <xmin>10</xmin>
      <ymin>60</ymin>
      <xmax>778</xmax>
      <ymax>530</ymax>
    </bndbox>
  </object>
</annotation>
```

### Licensing Information

CC BY SA 3.0

### Contributors

[Hui Shi](mailto:hshi@ucsd.edu) (Work done during her internship at Amazon)

[Yusheng Xie](mailto:yushx@amazon.com) (corresponding person)

[Luis Goncalves](mailto:luisgonc@amazon.com) 
