task_categories:text-classification task_ids:multi-class-classification annotations_creators:found language_creators:found multilinguality:monolingual size_categories:10K<n<100K source_datasets:original language:en license:unknown region:us
The 20 Newsgroups data set is a collection of approximately 20,000 newsgroup documents, partitioned (nearly) evenly across
20 different newsgroups. The 20 newsgroups collection has become a popular data set for experiments in text applications of
machine learning techniques, such as text classification and text clustering.
@inproceedings{Lang95,
    author = {Ken Lang},
    title = {Newsweeder: Learning to filter netnews}
    year = {1995}
    booktitle = {Proceedings of the Twelfth International Conference on Machine Learning}
    pages = {331-339}
    }
---
annotations_creators:
- found
language:
- en
language_creators:
- found
license:
- unknown
multilinguality:
- monolingual
pretty_name: 20 Newsgroups
size_categories:
- 10K<n<100K
source_datasets:
- original
task_categories:
- text-classification
task_ids:
- multi-class-classification
paperswithcode_id: 20-newsgroups
dataset_info:
- config_name: 18828_alt.atheism
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 1669511
    num_examples: 799
  download_size: 14666916
  dataset_size: 1669511
- config_name: 18828_comp.graphics
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 1661199
    num_examples: 973
  download_size: 14666916
  dataset_size: 1661199
- config_name: 18828_comp.os.ms-windows.misc
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 2378739
    num_examples: 985
  download_size: 14666916
  dataset_size: 2378739
- config_name: 18828_comp.sys.ibm.pc.hardware
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 1185187
    num_examples: 982
  download_size: 14666916
  dataset_size: 1185187
- config_name: 18828_comp.sys.mac.hardware
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 1056264
    num_examples: 961
  download_size: 14666916
  dataset_size: 1056264
- config_name: 18828_comp.windows.x
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 1876297
    num_examples: 980
  download_size: 14666916
  dataset_size: 1876297
- config_name: 18828_misc.forsale
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 925124
    num_examples: 972
  download_size: 14666916
  dataset_size: 925124
- config_name: 18828_rec.autos
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 1295307
    num_examples: 990
  download_size: 14666916
  dataset_size: 1295307
- config_name: 18828_rec.motorcycles
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 1206491
    num_examples: 994
  download_size: 14666916
  dataset_size: 1206491
- config_name: 18828_rec.sport.baseball
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 1369551
    num_examples: 994
  download_size: 14666916
  dataset_size: 1369551
- config_name: 18828_rec.sport.hockey
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 1758094
    num_examples: 999
  download_size: 14666916
  dataset_size: 1758094
- config_name: 18828_sci.crypt
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 2050727
    num_examples: 991
  download_size: 14666916
  dataset_size: 2050727
- config_name: 18828_sci.electronics
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 1237175
    num_examples: 981
  download_size: 14666916
  dataset_size: 1237175
- config_name: 18828_sci.med
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 1886363
    num_examples: 990
  download_size: 14666916
  dataset_size: 1886363
- config_name: 18828_sci.space
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 1812803
    num_examples: 987
  download_size: 14666916
  dataset_size: 1812803
- config_name: 18828_soc.religion.christian
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 2307486
    num_examples: 997
  download_size: 14666916
  dataset_size: 2307486
- config_name: 18828_talk.politics.guns
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 1922992
    num_examples: 910
  download_size: 14666916
  dataset_size: 1922992
- config_name: 18828_talk.politics.mideast
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 2910324
    num_examples: 940
  download_size: 14666916
  dataset_size: 2910324
- config_name: 18828_talk.politics.misc
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 2102809
    num_examples: 775
  download_size: 14666916
  dataset_size: 2102809
- config_name: 18828_talk.religion.misc
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 1374261
    num_examples: 628
  download_size: 14666916
  dataset_size: 1374261
- config_name: 19997_alt.atheism
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 2562277
    num_examples: 1000
  download_size: 17332201
  dataset_size: 2562277
- config_name: 19997_comp.graphics
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 2181673
    num_examples: 1000
  download_size: 17332201
  dataset_size: 2181673
- config_name: 19997_comp.os.ms-windows.misc
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 2898760
    num_examples: 1000
  download_size: 17332201
  dataset_size: 2898760
- config_name: 19997_comp.sys.ibm.pc.hardware
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 1671166
    num_examples: 1000
  download_size: 17332201
  dataset_size: 1671166
- config_name: 19997_comp.sys.mac.hardware
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 1580881
    num_examples: 1000
  download_size: 17332201
  dataset_size: 1580881
- config_name: 19997_comp.windows.x
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 2418273
    num_examples: 1000
  download_size: 17332201
  dataset_size: 2418273
- config_name: 19997_misc.forsale
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 1412012
    num_examples: 1000
  download_size: 17332201
  dataset_size: 1412012
- config_name: 19997_rec.autos
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 1780502
    num_examples: 1000
  download_size: 17332201
  dataset_size: 1780502
- config_name: 19997_rec.motorcycles
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 1677964
    num_examples: 1000
  download_size: 17332201
  dataset_size: 1677964
- config_name: 19997_rec.sport.baseball
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 1835432
    num_examples: 1000
  download_size: 17332201
  dataset_size: 1835432
- config_name: 19997_rec.sport.hockey
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 2207282
    num_examples: 1000
  download_size: 17332201
  dataset_size: 2207282
- config_name: 19997_sci.crypt
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 2607835
    num_examples: 1000
  download_size: 17332201
  dataset_size: 2607835
- config_name: 19997_sci.electronics
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 1732199
    num_examples: 1000
  download_size: 17332201
  dataset_size: 1732199
- config_name: 19997_sci.med
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 2388789
    num_examples: 1000
  download_size: 17332201
  dataset_size: 2388789
- config_name: 19997_sci.space
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 2351411
    num_examples: 1000
  download_size: 17332201
  dataset_size: 2351411
- config_name: 19997_soc.religion.christian
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 2743018
    num_examples: 997
  download_size: 17332201
  dataset_size: 2743018
- config_name: 19997_talk.politics.guns
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 2639343
    num_examples: 1000
  download_size: 17332201
  dataset_size: 2639343
- config_name: 19997_talk.politics.mideast
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 3695931
    num_examples: 1000
  download_size: 17332201
  dataset_size: 3695931
- config_name: 19997_talk.politics.misc
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 3169183
    num_examples: 1000
  download_size: 17332201
  dataset_size: 3169183
- config_name: 19997_talk.religion.misc
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 2658700
    num_examples: 1000
  download_size: 17332201
  dataset_size: 2658700
- config_name: bydate_alt.atheism
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 1042224
    num_examples: 480
  - name: test
    num_bytes: 702920
    num_examples: 319
  download_size: 14464277
  dataset_size: 1745144
- config_name: bydate_comp.graphics
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 911665
    num_examples: 584
  - name: test
    num_bytes: 849632
    num_examples: 389
  download_size: 14464277
  dataset_size: 1761297
- config_name: bydate_comp.os.ms-windows.misc
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 1770988
    num_examples: 591
  - name: test
    num_bytes: 706676
    num_examples: 394
  download_size: 14464277
  dataset_size: 2477664
- config_name: bydate_comp.sys.ibm.pc.hardware
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 800446
    num_examples: 590
  - name: test
    num_bytes: 485310
    num_examples: 392
  download_size: 14464277
  dataset_size: 1285756
- config_name: bydate_comp.sys.mac.hardware
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 696311
    num_examples: 578
  - name: test
    num_bytes: 468791
    num_examples: 385
  download_size: 14464277
  dataset_size: 1165102
- config_name: bydate_comp.windows.x
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 1243463
    num_examples: 593
  - name: test
    num_bytes: 795366
    num_examples: 395
  download_size: 14464277
  dataset_size: 2038829
- config_name: bydate_misc.forsale
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 611210
    num_examples: 585
  - name: test
    num_bytes: 415902
    num_examples: 390
  download_size: 14464277
  dataset_size: 1027112
- config_name: bydate_rec.autos
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 860646
    num_examples: 594
  - name: test
    num_bytes: 535378
    num_examples: 396
  download_size: 14464277
  dataset_size: 1396024
- config_name: bydate_rec.motorcycles
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 811151
    num_examples: 598
  - name: test
    num_bytes: 497735
    num_examples: 398
  download_size: 14464277
  dataset_size: 1308886
- config_name: bydate_rec.sport.baseball
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 850740
    num_examples: 597
  - name: test
    num_bytes: 618609
    num_examples: 397
  download_size: 14464277
  dataset_size: 1469349
- config_name: bydate_rec.sport.hockey
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 1189652
    num_examples: 600
  - name: test
    num_bytes: 666358
    num_examples: 399
  download_size: 14464277
  dataset_size: 1856010
- config_name: bydate_sci.crypt
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 1502448
    num_examples: 595
  - name: test
    num_bytes: 657727
    num_examples: 396
  download_size: 14464277
  dataset_size: 2160175
- config_name: bydate_sci.electronics
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 814856
    num_examples: 591
  - name: test
    num_bytes: 523095
    num_examples: 393
  download_size: 14464277
  dataset_size: 1337951
- config_name: bydate_sci.med
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 1195201
    num_examples: 594
  - name: test
    num_bytes: 791826
    num_examples: 396
  download_size: 14464277
  dataset_size: 1987027
- config_name: bydate_sci.space
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 1197965
    num_examples: 593
  - name: test
    num_bytes: 721771
    num_examples: 394
  download_size: 14464277
  dataset_size: 1919736
- config_name: bydate_soc.religion.christian
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 1358047
    num_examples: 599
  - name: test
    num_bytes: 1003668
    num_examples: 398
  download_size: 14464277
  dataset_size: 2361715
- config_name: bydate_talk.politics.guns
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 1313019
    num_examples: 546
  - name: test
    num_bytes: 701477
    num_examples: 364
  download_size: 14464277
  dataset_size: 2014496
- config_name: bydate_talk.politics.mideast
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 1765833
    num_examples: 564
  - name: test
    num_bytes: 1236435
    num_examples: 376
  download_size: 14464277
  dataset_size: 3002268
- config_name: bydate_talk.politics.misc
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 1328057
    num_examples: 465
  - name: test
    num_bytes: 853395
    num_examples: 310
  download_size: 14464277
  dataset_size: 2181452
- config_name: bydate_talk.religion.misc
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 835761
    num_examples: 377
  - name: test
    num_bytes: 598452
    num_examples: 251
  download_size: 14464277
  dataset_size: 1434213
---

# Dataset Card for "newsgroup"

## Table of Contents
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
  - [Curation Rationale](#curation-rationale)
  - [Source Data](#source-data)
  - [Annotations](#annotations)
  - [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
  - [Social Impact of Dataset](#social-impact-of-dataset)
  - [Discussion of Biases](#discussion-of-biases)
  - [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
  - [Dataset Curators](#dataset-curators)
  - [Licensing Information](#licensing-information)
  - [Citation Information](#citation-information)
  - [Contributions](#contributions)

## Dataset Description

- **Homepage:** [http://qwone.com/~jason/20Newsgroups/](http://qwone.com/~jason/20Newsgroups/)
- **Repository:** [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
- **Paper:** [NewsWeeder: Learning to Filter Netnews](https://doi.org/10.1016/B978-1-55860-377-6.50048-7)
- **Point of Contact:** [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
- **Size of downloaded dataset files:** 929.27 MB
- **Size of the generated dataset:** 124.41 MB
- **Total amount of disk used:** 1.05 GB

### Dataset Summary

The 20 Newsgroups data set is a collection of approximately 20,000 newsgroup documents, partitioned (nearly) evenly across
20 different newsgroups. To the best of my knowledge, it was originally collected by Ken Lang, probably for his Newsweeder:
Learning to filter netnews paper, though he does not explicitly mention this collection. The 20 newsgroups collection has become
a popular data set for experiments in text applications of machine learning techniques, such as text classification and text clustering.

does not include cross-posts and includes only the "From" and "Subject" headers.

### Supported Tasks and Leaderboards

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Languages

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

## Dataset Structure

### Data Instances

#### 18828_alt.atheism

- **Size of downloaded dataset files:** 14.67 MB
- **Size of the generated dataset:** 1.67 MB
- **Total amount of disk used:** 16.34 MB

An example of 'train' looks as follows.
```

```

#### 18828_comp.graphics

- **Size of downloaded dataset files:** 14.67 MB
- **Size of the generated dataset:** 1.66 MB
- **Total amount of disk used:** 16.33 MB

An example of 'train' looks as follows.
```

```

#### 18828_comp.os.ms-windows.misc

- **Size of downloaded dataset files:** 14.67 MB
- **Size of the generated dataset:** 2.38 MB
- **Total amount of disk used:** 17.05 MB

An example of 'train' looks as follows.
```

```

#### 18828_comp.sys.ibm.pc.hardware

- **Size of downloaded dataset files:** 14.67 MB
- **Size of the generated dataset:** 1.18 MB
- **Total amount of disk used:** 15.85 MB

An example of 'train' looks as follows.
```

```

#### 18828_comp.sys.mac.hardware

- **Size of downloaded dataset files:** 14.67 MB
- **Size of the generated dataset:** 1.06 MB
- **Total amount of disk used:** 15.73 MB

An example of 'train' looks as follows.
```

```

### Data Fields

The data fields are the same among all splits.

#### 18828_alt.atheism
- `text`: a `string` feature.

#### 18828_comp.graphics
- `text`: a `string` feature.

#### 18828_comp.os.ms-windows.misc
- `text`: a `string` feature.

#### 18828_comp.sys.ibm.pc.hardware
- `text`: a `string` feature.

#### 18828_comp.sys.mac.hardware
- `text`: a `string` feature.

### Data Splits

|             name             |train|
|------------------------------|----:|
|18828_alt.atheism             |  799|
|18828_comp.graphics           |  973|
|18828_comp.os.ms-windows.misc |  985|
|18828_comp.sys.ibm.pc.hardware|  982|
|18828_comp.sys.mac.hardware   |  961|

## Dataset Creation

### Curation Rationale

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Source Data

#### Initial Data Collection and Normalization

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

#### Who are the source language producers?

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Annotations

#### Annotation process

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

#### Who are the annotators?

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Personal and Sensitive Information

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

## Considerations for Using the Data

### Social Impact of Dataset

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Discussion of Biases

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Other Known Limitations

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

## Additional Information

### Dataset Curators

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Licensing Information

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Citation Information

```
@incollection{LANG1995331,
title = {NewsWeeder: Learning to Filter Netnews},
editor = {Armand Prieditis and Stuart Russell},
booktitle = {Machine Learning Proceedings 1995},
publisher = {Morgan Kaufmann},
address = {San Francisco (CA)},
pages = {331-339},
year = {1995},
isbn = {978-1-55860-377-6},
doi = {https://doi.org/10.1016/B978-1-55860-377-6.50048-7},
url = {https://www.sciencedirect.com/science/article/pii/B9781558603776500487},
author = {Ken Lang},
}
```

### Contributions

Thanks to [@mariamabarham](https://github.com/mariamabarham), [@thomwolf](https://github.com/thomwolf), [@lhoestq](https://github.com/lhoestq) for adding this dataset.