license:cc-by-4.0 arxiv:2101.09465 region:us
---
license: cc-by-4.0
---
# WebSRC v1.0

WebSRC v1.0 is a dataset for reading comprehension on structural web pages.
The task is to answer questions about web pages, which requires a system to
have a comprehensive understanding of the spatial structure and logical
structure. WebSRC consists of 6.4K web pages and 400K question-answer pairs
about web pages. For each web page, we manually chose one segment from it
and saved the corresponding HTML code, screenshot, and metadata like
positions and sizes. Questions in WebSRC were created for each segment.
Answers are either text spans from web pages or yes/no. Taking the HTML
code, screenshot, metadata as well as question as input, a model is to
predict the answer from the web page. Our dataset is the first one that
provides HTML documents as well as images, and is larger in the number of
domains and queries.

For more details, please refer to our paper [WebSRC: A Dataset for Web-Based Structural Reading Comprehension](https://arxiv.org/abs/2101.09465).
The Leaderboard of WebSRC v1.0 can be found [here](https://x-lance.github.io/WebSRC/).

## Data Format Description

The dataset for each website will be stored in `dataset.csv` in the directory
`{domain-name}/{website-number}`. The corresponding raw data (including HTML
files, screenshots, bounding box coordinates, and page names and urls) is
stored in the `processed_data` folder in the same directory.

In `dataset.csv`, each row corresponds to one question-answer data point
except the header. The meanings of each column are as follows:
* `question`: a string, the question of this question-answer data point.
* `id`: a unique id for this question-answer data point. Each `id` has a length 14, the first two characters are the domain indicator, the following two number is the website name. The corresponding page id can be extracted by `id[2:9]`, for example, id "sp160000100001" means this line is created from the *sport* domain, website *16*, and the corresponding page is `1600001.html`.
* `element_id`: an integer, the tag id (corresponding to the tag's `tid` attribute in the HTML files) of the deepest tag in the DOM tree which contain all the answer. For yes/no question, there is no tag associated with the answer, so the `element_id` is -1.
* `answer_start`: an integer, the char offset of the answer from the start of the content of the tag specified by `element_id`. Note that before counting this number, we first eliminate all the inner tags in the specified tag and replace all the consecutive whitespaces with one space. For yes/no questions, `answer_start` is 1 for answer "yes" and 0 for answer "no".
* `answer`: a string, the answer of this question-answer data point.

## Data Statistics

We roughly divided the questions in WebSRC v1.0 into three categories: KV,
Compare, and Table. The detailed definitions can be found in our
[paper](https://arxiv.org/abs/2101.09465). The numbers of websites, webpages,
and QAs corresponding to the three categories are as follows:

 Type | # Websites | # Webpages | # QAs
 ---- | ---------- | ---------- | -----
 KV | 34 | 3,207 | 168,606
 Comparison | 15 | 1,339 | 68,578
 Table | 21 | 1,901 | 163,314

 The statistics of the dataset splits are as follows:
 
 Split | # Websites | # Webpages | # QAs
 ----- | ---------- | ---------- | -----
 Train | 50 | 4,549 | 307,315
 Dev | 10 | 913 | 52,826
 Test | 10 | 985 | 40,357

 ## Obtain Test Result

For test set evaluation, please send your prediction files to
zhao_mengxin@sjtu.edu.cn and chenlusz@sjtu.edu.cn with title "WebSRC Test:
\<your model name\>+\<your institution\>". For evaluation, the prediction
files should contain two files:

```jsonc
// prediction.json
// A json format file, keys are ids and values are the predicted answers (string).
{
  "sp160000100001": "predicted answer",
  "sp160000100002": "...",
  //...
}

// tag_prediction.json
// A json format file, keys are ids and values are the predicted tag tid (int)
{
  "sp160000100001": -1,
  "sp160000100002": -1,
  //...
}
```

We encourage to submit results from **at least three runs with different random
seeds** to reduce the uncertainty of the experiments. Please place prediction files
for each run in different directories and submit a zipped file. The average test
result will be sent by email.



## Reference

If you use any source codes or datasets included in this repository in your work,
please cite the corresponding papers. The bibtex are listed below:
```text
@inproceedings{chen-etal-2021-websrc,
    title = "{W}eb{SRC}: A Dataset for Web-Based Structural Reading Comprehension",
    author = "Chen, Xingyu  and
      Zhao, Zihan  and
      Chen, Lu  and
      Ji, JiaBao  and
      Zhang, Danyang  and
      Luo, Ao  and
      Xiong, Yuxuan  and
      Yu, Kai",
    booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2021",
    address = "Online and Punta Cana, Dominican Republic",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.emnlp-main.343",
    pages = "4173--4185",
    abstract = "Web search is an essential way for humans to obtain information, but it{'}s still a great challenge for machines to understand the contents of web pages. In this paper, we introduce the task of web-based structural reading comprehension. Given a web page and a question about it, the task is to find an answer from the web page. This task requires a system not only to understand the semantics of texts but also the structure of the web page. Moreover, we proposed WebSRC, a novel Web-based Structural Reading Comprehension dataset. WebSRC consists of 400K question-answer pairs, which are collected from 6.4K web pages with corresponding HTML source code, screenshots, and metadata. Each question in WebSRC requires a certain structural understanding of a web page to answer, and the answer is either a text span on the web page or yes/no. We evaluate various strong baselines on our dataset to show the difficulty of our task. We also investigate the usefulness of structural information and visual features. Our dataset and baselines have been publicly available.",
}
```
