annotations_creators:crowdsourced language_creators:found multilinguality:monolingual source_datasets:original language:en license:cc-by-4.0 table-annotation region:us
# Understanding the semantics of table elements is a prerequisite for many data integration and data discovery tasks. Table annotation is the task of labeling table elements with terms from a given vocabulary. This paper presents the WDC Schema.org Table Annotation Benchmark (SOTAB) for comparing the performance of table annotation systems. SOTAB covers the column type annotation (CTA) and columns property annotation (CPA) tasks. SOTAB provides ∼50,000 annotated tables for each of the tasks containing Schema.org data from different websites. The tables cover 17 different types of entities such as movie, event, local business, recipe, job posting, or product. The tables stem from the WDC Schema.org Table Corpus which was created by extracting Schema.org annotations from the Common Crawl. Consequently, the labels used for annotating columns in SOTAB are part of the Schema.org vocabulary. The benchmark covers 91 types for CTA and 176 properties for CPA distributed across textual, numerical and date/time columns. The tables are split into fixed training, validation and test sets. The test sets are further divided into subsets focusing on specific challenges, such as columns with missing values or different value formats, in order to allow a more fine-grained comparison of annotation systems. The evaluation of SOTAB using Doduo and TURL shows that the benchmark is difficult to solve for current state-of-the-art systems.
#
# @inproceedings{madoc63868, pages = {14--19}, booktitle = {SemTab 2022 : Proceedings of the Semantic Web Challenge on Tabular Data to Knowledge Graph Matching, co-located with the 21st International semantic Web Conference (ISWC 2022), virtual conference, October 23-27, 2022}, address = {Aachen, Germany}, editor = {Vasilis Efthymiou and Ernesto Jim{\'e}nez-Ruiz and Jiaoyan Chen and Vincenzo Cutrona and Oktie Hassanzadeh and Juan Sequeda and Kavitha Srinivas and Nora Abdelmageed and Madelon Hulsebos}, journal = {CEUR Workshop Proceedings}, year = {2022}, title = {SOTAB: The WDC Schema.org table annotation benchmark}, publisher = {RWTH Aachen}, language = {Englisch}, author = {Keti Korini and Ralph Peeters and Christian Bizer}, volume = {3320}, abstract = {Understanding the semantics of table elements is a prerequisite for many data integration and data discovery tasks. Table annotation is the task of labeling table elements with terms from a given vocabulary. This paper presents the WDC Schema.org Table Annotation Benchmark (SOTAB) for comparing the performance of table annotation systems. SOTAB covers the column type annotation (CTA) and columns property annotation (CPA) tasks. SOTAB provides {$\sim$}50,000 annotated tables for each of the tasks containing Schema.org data from different websites. The tables cover 17 different types of entities such as movie, event, local business, recipe, job posting, or product. The tables stem from the WDC Schema.org Table Corpus which was created by extracting Schema.org annotations from the Common Crawl. Consequently, the labels used for annotating columns in SOTAB are part of the Schema.org vocabulary. The benchmark covers 91 types for CTA and 176 properties for CPA distributed across textual, numerical and date/time columns. The tables are split into fixed training, validation and test sets. The test sets are further divided into subsets focusing on specific challenges, such as columns with missing values or different value formats, in order to allow a more fine-grained comparison of annotation systems. The evaluation of SOTAB using Doduo and TURL shows that the benchmark is difficult to solve for current state-of-the-art systems.}, url = {https://madoc.bib.uni-mannheim.de/63868/} }
#
---
annotations_creators:
- crowdsourced
language_creators:
- found
language:
- en
license:
- cc-by-4.0
multilinguality:
- monolingual
paperswithcode_id: null
pretty_name: SOTAB_CTA
source_datasets:
- original
task_ids: []
tags:
- table-annotation
dataset_info:
- config_name: 
  features:
  # - name: id
  #   dtype: int32
  - name: column_index
    dtype: int32
  - name: label
    dtype: string
  - name: table
    struct:
    - name: header
      sequence: string
    - name: rows
      sequence:
        sequence: string
    - name: name
      dtype: string
  splits:
  - name: train
    num_bytes: 
    num_examples: 130471
  - name: test
    num_bytes: 
    num_examples: 15040
  - name: validation
    num_bytes: 
    num_examples: 16840
  download_size: 
  dataset_size: 162351
---

# Dataset Card for SOTAB

## Table of Contents
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks](#supported-tasks-and-leaderboards)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-instances)
  - [Data Splits](#data-instances)
- [Dataset Creation](#dataset-creation)
  - [Curation Rationale](#curation-rationale)
  - [Source Data](#source-data)
  - [Annotations](#annotations)
  - [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
  - [Social Impact of Dataset](#social-impact-of-dataset)
  - [Discussion of Biases](#discussion-of-biases)
  - [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
  - [Dataset Curators](#dataset-curators)
  - [Licensing Information](#licensing-information)
  - [Citation Information](#citation-information)

## Dataset Description

- **Homepage:** [SOTAB homepage](https://webdatacommons.org/structureddata/sotab/)
- **Repository:** [SOTAB repository](https://github.com/wbsg-uni-mannheim/wdc-sotab)
- **Paper:** [SOTAB: The WDC Schema.org Table Annotation Benchmark](https://ceur-ws.org/Vol-3320/paper1.pdf)
- **Leaderboard:** [SOTAB leaderboard on PaperWithCode](https://paperswithcode.com/paper/sotab-the-wdc-schema-org-table-annotation)
- **Point of Contact:** [Needs More Information]

### Dataset Summary

The SOTAB dataset is a large-scale dataset for the task of column type annotation on semi-structured tables.

### Supported Tasks and Leaderboards

table-annotation, column-type-annotation

### Languages

en

## Dataset Structure

### Data Instances

#### default

- **Size of downloaded dataset files:** 
- **Size of the generated dataset:** 
- **Total amount of disk used:** 

An example of 'validation' looks as follows:
```
{
    "id": 0,
    "column_index": 3,
    "label": "currency",
    "table": {
        "name": "Book_7sat.co.uk_September2020_CTA.json.gz",
        "header": ["col1", "col2", ...]
        "rows": [
           ["2001", "2", "USL A-League", ...],
           ["2002", "2", "USL A-League", ...], 
           ...
        ]
    }
}
```

### Data Fields

The data fields are the same among all splits.

#### default
- `id`: a `int32` feature.
- `column_index`: a `string` feature.
- `label`: a `string` feature.
- `table`: a dictionary feature containing:
  - `rows`: a `list` of `string` features.
  - `rows`: a `list` of `list` of `string` features.
  - `name`: a `string` feature.

### Data Splits

| name  |train|validation|test |
|-------|-----:|---------:|----:|
|default|130471|     16840|15040|

## Dataset Creation

### Curation Rationale

[Needs More Information]

### Source Data

#### Initial Data Collection and Normalization

[Needs More Information]

#### Who are the source language producers?

[Needs More Information]

### Annotations

#### Annotation process

[Needs More Information]

#### Who are the annotators?

[Needs More Information]

### Personal and Sensitive Information

[Needs More Information]

## Considerations for Using the Data

### Social Impact of Dataset

[Needs More Information]

### Discussion of Biases

[Needs More Information]

### Other Known Limitations

[Needs More Information]

## Additional Information

### Dataset Curators

Panupong Pasupat and Percy Liang

### Licensing Information

Creative Commons Attribution Share Alike 4.0 International

### Citation Information

```

```

### Contributions

Thanks to [@ShivangiBithel](https://github.com/shivangibithel) for adding this dataset.