task_categories:graph-ml annotations_creators:crowdsourced annotations_creators:expert-generated size_categories:100M<n<1B source_datasets:wikidata language:en license:cc-by-sa-3.0 knowledge-graph rdf triples region:us
---
language:
- en
license: cc-by-sa-3.0
license_link: https://creativecommons.org/licenses/by-sa/3.0/
tags:
- knowledge-graph
- rdf
- triples
annotations_creators:
- crowdsourced
- expert-generated
source_datasets:
- wikidata
pretty_name: YAGO 4.5 (EN)
size_categories:
- 100M<n<1B
task_categories:
- graph-ml
dataset_info:
  features:
    - name: subject
      dtype: string
    - name: predicate
      dtype: string
    - name: object
      dtype: string
  config_name: default
  splits:
    - name: train
      num_bytes: 42709902295
      num_examples: 249675587
  dataset_size: 42709902295
viewer: false
---

# YAGO 4.5 Dataset (English subset for LLM fine-tuning)

To utilize the YAGO 4.5 (EN) Dataset, users should ensure they have the following prerequisites installed:

### Software
- Python (Tested with 3.10)
- [Hugging Face Datasets
  Library](https://huggingface.co/docs/datasets/): Required for loading and processing the dataset.
  ```sh
  pip install datasets
  pip install rdflib
  ```

### Hardware
* Sufficient Storage: The dataset is approximately 43 GB, ensure you
  have enough storage space to download and extract the dataset.
* Multi-core Processor: For efficient data loading and processing, a
  multi-core processor is recommended. The more threads the faster you
  can load the dataset.

## Dataset Description
This dataset contains triples filtered from yago-facts.ttl and
yago-beyond-wikipedia.ttl in the YAGO 4.5 dataset. The SPARQL query
used to filter the triples is in `filter.sparql`. This represents
a subset of the YAGO 4.5 dataset maintaining only English labels. 

I remapped some schema.org properties to
`http://yago-knowledge.org/resource/` which were not present in the
schema.org vocabulary.  I also removed schema:sameAs and owl:sameAs
relations from this dataset, as well as triples with xsd:anyURI object
literals, as my goal is to use this dataset for fine-tuning a large
language model for knowledge graph completion and I do not want
to train the base model to predict these kind of relations.

### Overview

YAGO 4.5 is the latest version of the YAGO knowledge base. It is
based on Wikidata — the largest public general-purpose knowledge
base. YAGO refines the data as follows:

* All entity identifiers and property identifiers are human-readable.
* The top-level classes come from schema.org — a standard repertoire
  of classes and properties maintained by Google and others. The lower
  level classes are a careful selection of the Wikidata taxonomy.
* The properties come from schema.org.
* YAGO 4.5 contains semantic constraints in the form of SHACL. These
  constraints keep the data clean, and allow for logical reasoning on
  YAGO. 


### Dataset Structure
The dataset is structured as follows:

- **yago-taxonomy.ttl:** Contains the `rdfs:subClassOf` relations
  for YAGO and the prefix mappings for the N-Triples.
- **facts.tar.gz:** Compressed file containing chunks of the
  dataset in N-Triples format, representing the factual knowledge in
  YAGO. 

### Features

Each RDF triple in the dataset is represented with the following features:

- **subject:** The subject of the triple, representing the entity.
- **predicate:** The predicate of the triple, representing the
  relationship between the subject and object.
- **object:** The object of the triple, representing the entity or
  value linked by the predicate.

### Chunks

The dataset is logically divided into multiple chunks, each containing
a subset of RDF triples. Users can load specific chunks or the entire
dataset based on their requirements.

## Usage

### Loading the Dataset

The dataset can be loaded using the Hugging Face `datasets` library as follows:

```python
from datasets import load_dataset

dataset = load_dataset('wikipunk/yago45en', num_proc=4, split='train')
```

``` python
# Accessing the first row of the dataset
first_row = dataset[0]

# Output: {'subject': '<http://yago-knowledge.org/resource/Sdsscgb_11322_U002E_4_Q85387516>',
#          'predicate': '<http://www.w3.org/2000/01/rdf-schema#comment>',
#          'object': '"galaxy"@en'}
```

## Additional Information

### Licensing

The YAGO 4.5 dataset is available under the [Creative Commons Attribution-ShareAlike 3.0 license](https://creativecommons.org/licenses/by-sa/3.0/).

### Citation

If you use the YAGO 4.5 dataset in your work, please cite the
following publication:

```bibtex
@article{suchanek2023integrating,
  title={Integrating the Wikidata Taxonomy into YAGO},
  author={Suchanek, Fabian M and Alam, Mehwish and Bonald, Thomas and Paris, Pierre-Henri and Soria, Jules},
  journal={arXiv preprint arXiv:2308.11884},
  year={2023}
}
```
