task_categories:other task_ids:natural-language-inference task_ids:semantic-similarity-scoring task_ids:text-scoring annotations_creators:unknown language_creators:unknown multilinguality:monolingual size_categories:100K<n<1M source_datasets:unknown language:en license:mit deduplication region:us
---
annotations_creators:
- unknown
language_creators:
- unknown
language:
- en
license:
- mit
multilinguality:
- monolingual
size_categories:
- 100K<n<1M
source_datasets:
- unknown
task_categories:
- other
task_ids:
- natural-language-inference
- semantic-similarity-scoring
- text-scoring
pretty_name: CORE Deduplication of Scholarly Documents
tags:
- deduplication
---

# Dataset Card for CORE Deduplication

## Dataset Description

- **Homepage:** [https://core.ac.uk/about/research-outputs](https://core.ac.uk/about/research-outputs)
- **Repository:** [https://core.ac.uk/datasets/core_2020-05-10_deduplication.zip](https://core.ac.uk/datasets/core_2020-05-10_deduplication.zip)
- **Paper:** [Deduplication of Scholarly Documents using Locality Sensitive Hashing and Word Embeddings](http://oro.open.ac.uk/id/eprint/70519)
- **Point of Contact:** [CORE Team](https://core.ac.uk/about#contact)
- **Size of downloaded dataset files:** 204 MB

### Dataset Summary

CORE 2020 Deduplication dataset (https://core.ac.uk/documentation/dataset) contains 100K scholarly documents labeled as duplicates/non-duplicates.

### Languages

The dataset language is English (BCP-47 `en`)

### Citation Information

```
@inproceedings{dedup2020,
  title={Deduplication of Scholarly Documents using Locality Sensitive Hashing and Word Embeddings},
  author={Gyawali, Bikash and Anastasiou, Lucas and Knoth, Petr},
  booktitle = {Proceedings of 12th Language Resources and Evaluation Conference},
  month = may,
  year = 2020,
  publisher = {France European Language Resources Association},
  pages = {894-903}
}
```
