size_categories:100M<n<1B license:cc-by-sa-4.0 arxiv:2103.01913 region:us
---
dataset_info:
  features:
  - name: text_id
    dtype: string
  - name: page_url
    dtype: string
  - name: page_title
    dtype: string
  - name: section_title
    dtype: string
  - name: context_page_description
    dtype: string
  - name: context_section_description
    dtype: string
  - name: media
    sequence: string
  - name: hierachy
    sequence: string
  - name: category
    sequence: string
  - name: source_id
    dtype: string
  splits:
  - name: train
    num_bytes: 14378574060.336058
    num_examples: 10134744
  download_size: 6408012391
  dataset_size: 14378574060.336058
license: cc-by-sa-4.0
size_categories:
- 100M<n<1B
---
# Dataset Card for "AToMiC-Texts-Mapped"

## Dataset Description

- **Homepage:** [AToMiC homepage](https://trec-atomic.github.io/)
- **Source:** [WIT](https://github.com/google-research-datasets/wit)
- **Paper:** [WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning](https://arxiv.org/abs/2103.01913)

### Languages

This dataset only contains English in Wikipedia (parsed from the 20221101 XML dump).

### Data Instances

Each instance is a section of a Wikipedia page. We also provide its page-level information, and associated information such as categories and media.
The `source_id` can be mapped back to the instance in the original [WIT instance](https://github.com/google-research-datasets/wit/blob/main/DATA.md).
Notice that the WIT dataset is crawled from the earlier version of Wikipedia (2020-08-30).
The WIT dataset is mapped to the new dump by pure BM25 matching with [Anserini](https://github.com/castorini/anserini).

### Intended Usage

1. Text collection for Image-to-Text retrieval
2. Language model pretraining
3. Document classification

### Licensing Information

[CC BY-SA 4.0 international license](https://creativecommons.org/licenses/by-sa/4.0/)

### Citation Information

TBA

### Acknowledgement

Thanks to:
[mwparserfromhell](https://github.com/earwig/mwparserfromhell)
[Datasets](https://github.com/huggingface/datasets)
[Anserini](https://github.com/castorini/anserini)

[More Information needed](https://github.com/huggingface/datasets/blob/main/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)