<jupyter_start><jupyter_text>Generating RDF TriplesThis notebook shows how to generate RDF triples according ot the conventions used in Wikidata. We have a Jupyter notebook that takes as input a KGTK file and generates as output a turtle file that can be loaded in a triple store. The notebook can also deploy a docker image with an instance of the Wikidata Blazegraph, but it takes more time than we want to wait while doing the tutorial.We will need to do a bit of data preparation to extract the `datatype` edges, a required input to the triple generation notebook.<jupyter_code>import sys  
sys.path.insert(0, 'tutorial')
from tutorial_setup import *<jupyter_output>ALIAS: "/Users/pedroszekely/Downloads/kgtk-tutorial/miniwikidata/aliases.en.tsv.gz"
ALL: "/Users/pedroszekely/Downloads/kgtk-tutorial/miniwikidata/all.tsv.gz"
CLAIMS: "/Users/pedroszekely/Downloads/kgtk-tutorial/miniwikidata/claims.tsv.gz"
DESCRIPTION: "/Users/pedroszekely/Downloads/kgtk-tutorial/miniwikidata/descriptions.en.tsv.gz"
EXAMPLES_DIR: "/Users/pedroszekely/Documents/GitHub/kgtk/examples"
GE: "/Users/pedroszekely/Downloads/kgtk-tutorial/temp/graph-embedding"
ISA: "/Users/pedroszekely/Downloads/kgtk-tutorial/miniwikidata/derived.isa.tsv.gz"
ITEM: "/Users/pedroszekely/Downloads/kgtk-tutorial/miniwikidata/claims.wikibase-item.tsv.gz"
KGTK_PATH: "/Users/pedroszekely/Documents/GitHub/kgtk"
LABEL: "/Users/pedroszekely/Downloads/kgtk-tutorial/miniwikidata/labels.en.tsv.gz"
OUT: "/Users/pedroszekely/Downloads/kgtk-tutorial/output"
P279: "/Users/pedroszekely/Downloads/kgtk-tutorial/miniwikidata/derived.P279.tsv.gz"
P279STAR: "/Users/pedroszekely/Downloads/kgtk-tutorial/miniwikidata/de[...]<jupyter_text>Sort the file with all edges and qualifiers for the subgraph Sort the file by identifier, which will place all the qualifiers of a claim immediately after the claim<jupyter_code>!$kgtk sort -i "$TEMP"/all_and_qualifiers.tsv.gz -o "$OUT"/all_and_qualifiers.sorted.1.tsv.gz<jupyter_output><empty_output><jupyter_text>Our file has `datatype` edges that specify the datatype of each property, including the ones that KGTK adds (unfortunately, at this time we have two properties, with and without dash):<jupyter_code>lines = !$kgtk filter -i "$OUT"/all_and_qualifiers.sorted.1.tsv.gz -p ';data_type,datatype;' 
kgtk_to_dataframe(lines)<jupyter_output><empty_output><jupyter_text>We don't want the data type edges in our triples, so we remove them:<jupyter_code>!$kgtk filter -i "$OUT"/all_and_qualifiers.sorted.1.tsv.gz -p ';data_type,datatype;' \
--invert -o "$OUT"/all_and_qualifiers.sorted.tsv.gz<jupyter_output><empty_output><jupyter_text>Consolidate all the data type edges as we need to give them as argument for triple generation<jupyter_code>!$kgtk filter -i "$TEMP"/kgtk.properties.tsv -p ';data_type;' \
-o "$TEMP/datatypes.kgtk.properties.tsv.gz"

!$kgtk cat -i "$WIKIDATA"/metadata.property.datatypes.tsv.gz  "$TEMP"/datatypes.kgtk.properties.tsv.gz \
-o "$OUT"/all.metadata.property.datatypes.tsv

!gzip -f "$OUT"/all.metadata.property.datatypes.tsv<jupyter_output><empty_output><jupyter_text>Look at the file:<jupyter_code>lines = !zcat < "$OUT"/all.metadata.property.datatypes.tsv.gz 
kgtk_to_dataframe(lines)<jupyter_output><empty_output><jupyter_text>Execute the Triple Generation notebook<jupyter_code>pm.execute_notebook(
    os.environ["USECASE_DIR"] + "/Generate-Triples-And-Load-Blazegraph.ipynb",
    os.environ["TEMP"] + "/Generate-Triples-And-Load-Blazegraph-OUT.ipynb",
    parameters=dict(
        kgtk_path = os.environ["OUT"],
        kgtk_file_name = "all_and_qualifiers.sorted.tsv.gz",
        properties_file_path = os.environ["OUT"] + "/all.metadata.property.datatypes.tsv.gz",
        create_image = False,
        load_triples = False
    )
)
;<jupyter_output><empty_output><jupyter_text>Take a peek at the output triples file<jupyter_code>!gzcat "$OUT"/all.ttl.gz | head -n 50<jupyter_output>@prefix wikibase: <http://wikiba.se/ontology#> .
@prefix wd: <http://www.wikidata.org/entity/> .
@prefix wdt: <http://www.wikidata.org/prop/direct/> .
@prefix wdtn: <http://www.wikidata.org/prop/direct-normalized/> .
@prefix wdno: <http://www.wikidata.org/prop/novalue/> .
@prefix wds: <http://www.wikidata.org/entity/statement/> .
@prefix wdv: <http://www.wikidata.org/value/> .
@prefix wdref: <http://www.wikidata.org/reference/> .
@prefix p: <http://www.wikidata.org/prop/> .
@prefix pr: <http://www.wikidata.org/prop/reference/> .
@prefix prv: <http://www.wikidata.org/prop/reference/value/> .
@prefix prn: <http://www.wikidata.org/prop/reference/value-normalized/> .
@prefix ps: <http://www.wikidata.org/prop/statement/> .
@prefix psv: <http://www.wikidata.org/prop/statement/value/> .
@prefix psn: <http://www.wikidata.org/prop/statement/value-normalized/> .
@prefix pq: <http://www.wikidata.org/prop/qualifier/> .
@prefix pqv: <http://www.wikidata.org/prop/qualifier/value/> .
@prefix pqn: <ht[...]
{'fraction_non_alphanumeric': 0.14213094753965028, 'fraction_numerical': 0.0030500203334688897, 'mean_word_length': 5.125778331257783, 'pattern_counts': {'":': 0, '<': 50, '<?xml version=': 0, '>': 48, 'https://': 0, 'lorem ipsum': 0, 'www.': 16, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '198216', 'license': 'MIT', 'n_tokens_mistral': 1888, 'n_tokens_neox': 1826, 'n_words': 341, 'path': 'tutorial-old/4 Generate Triples.ipynb', 'repo_name': 'robuso/kgtk', 'size': '4918'}