task_categories:sentence-similarity size_categories:100M<n<1B language:en license:cc-by-2.0 doi:10.57967/hf/0751 region:us
---
license: cc-by-2.0
task_categories:
- sentence-similarity
language:
- en
pretty_name: HEADLINES
size_categories:
- 100M<n<1B
---

# Dataset Card for HEADLINES

![headline_image_with_title.jpg](https://s3.amazonaws.com/moonup/production/uploads/61654589b5ec555e8e9c203a/fJiAy43PFD3FBa1aMSzHz.jpeg)


## Dataset Description

- **Homepage:** [Dell Research homepage](https://dell-research-harvard.github.io/)
- **Repository:** [Github repository](https://github.com/dell-research-harvard)
- **Paper:** [arxiv submission](https://arxiv.org/abs/tbd)
- **Point of Contact:** [Melissa Dell](mailto:melissadell@fas.harvard.edu)

#### Dataset Summary
HEADLINES is a massive English-language semantic similarity dataset, containing 396,001,930 pairs of different headlines for the same newspaper article, taken from historical U.S. newspapers, covering the period 1920-1989. 

#### Languages
The text in the dataset is in English. 


## Dataset Structure
Each year in the dataset is divided into a distinct file (eg. 1952_headlines.json), giving a total of 70 files.  

The data is presented in the form of clusters, rather than pairs to eliminate duplication of text data and minimise the storage size of the datasets. Below we give an example of how to convert the dataset into pairs.

#### Dataset Instances

An example from the HEADLINES dataset looks like:  

```python
{
    "headline": "FRENCH AND BRITISH BATTLESHIPS IN MEXICAN WATERS",
    "group_id": 4
    "date": "May-14-1920",
    "state": "kansas",
}
```

#### Dataset Fields

- `headline`: headline text.
- `date`: the date of publication of the newspaper article, as a string in the form mmm-DD-YYYY.
- `state`: state of the newspaper that published the headline.
- `group_id`: a number that is shared with all other headlines for the same article. This number is unique across all year files. 


## Usage

The whole dataset can be easily downloaded using the `datasets` library. 

```
from datasets import load_dataset
dataset_dict = load_dataset('dell-research-harvard/headlines-semantic-similarity')
```


If you just want to load specific files, you can specify these in the command.

```
from datasets import load_dataset
load_dataset(
    'dell-research-harvard/headlines-semantic-similarity', 
    data_files=["1929_headlines.json", "1989_headlines.json"]
)
```




## Dataset Creation

### Source Data
The dataset was constructed using a large corpus of newly digitized articles from off-copyright, local U.S. newspapers. 
Many of these newspapers reprint articles from newswires, such as the Associated Press, but the headlines are written locally. 
The dataset comprises different headlines for the same article. 

#### Initial Data Collection and Normalization

To construct HEADLINES, we digitize front pages of off-copyright newspaper page scans, localizing and OCRing individual content regions like headlines and articles. The headlines, bylines, and article texts that form full articles span multiple bounding boxes - often arranged with complex layouts - and we associate them using a model that combines layout information and language understanding. Then, we use neural methods to accurately predict which articles come from the same underlying source, in the presence of noise and abridgement.

We remove all headline pairs that are below a Levenshtein edit distance, divided by the min length in the pair, of 0.1 from each other, with the aim of removing pairs that are exact duplicates up to OCR noise.

#### Who are the source language producers?

The text data was originally produced by journalists of local U.S. newspapers. 

### Annotations

The dataset does not contain any additional annotations.

### Personal and Sensitive Information
The dataset may contain information about individuals, to the extent that this is covered in the headlines of news stories. However we make no additional information about individuals publicly available. 


### Data Description
The dataset contains 396,001,930 positive semantic similarity pairs, from 1920 to 1989. 

![image (3).png](https://s3.amazonaws.com/moonup/production/uploads/61654589b5ec555e8e9c203a/vKeR-SEEfYte6ZZbdpaq3.png)

It contains headlines from all 50 states. 

![map (1).png](https://s3.amazonaws.com/moonup/production/uploads/61654589b5ec555e8e9c203a/0WMdO8Fo1nfYiId4SlWaL.png)



## Considerations for Using the Data

### Social Impact of Dataset

The purpose of this dataset is to widen the range of language and topics for training semantic similarity models. 

This will facilitate the study of semantic change across space and time. 

Specific biases in the dataset are considered in the next section. 


### Discussion of Biases

The headlines in the dataset may reflect attitudes and values from the period in which they were written, 1920-1989. This may include instances of racism, sexism and homophobia. 

We also note that given that all the newspapers considered are from the U.S., the data is likely to present a Western perspective on the news stories of the day. 

### Other Known Limitations

As the dataset is sourced from digitalised text, it contains some OCR errors. 



## Additional information

### Licensing Information
HEADLINES is released under the Creative Commons CC-BY 2.0 license. 

### Dataset curators 
This dataset was created by Emily Silcock and Melissa Dell. For more information, see [Dell Research Harvard](https://dell-research-harvard.github.io/). 

### Citation information
Citation coming soon. 
