<jupyter_start><jupyter_text>Computes Graph and Text Embeddings, Elasticsearch Ready KGTK File and RDF Triples for BlazegraphThis notebook computes the following:- `complEx` graph embeddings- `transE` graph embeddings- `BERT` text embeddings- `elasticsearch` ready KGTK edge for [KGTK Search](https://kgtk.isi.edu/search/)- `elasticsearch` ready KGTK edge file for Table Linker- `RDF Triples` to be loaded into blazegraphInputs:- `item_file`: the subset of the `claims_file` consistin of edges for property of data type `wikibase-item`- `label_file`, `alias_file` and `description_file` containing labels, aliases and descriptions. It is assume that these files contain the labels, aliases and descriptions of all nodes appearing in the claims file. Users may provide these files for specific languages only.  Batch InvocationExample batch command. The second argument is a notebook where the output will be stored. You can load it to see progress.```papermill Embeddings-Elasticsearch-&-Triples.ipynb Embeddings-Elasticsearch-&-Triples.out.ipynb \-p claims_file /Volumes/GoogleDrive/Shared\ drives/KGTK-public-graphs/wikidata-20200803-v4/all.tsv.gz \-p label_file /Volumes/GoogleDrive/Shared\ drives/KGTK-public-graphs/wikidata-20200803-v4/part.label.en.tsv.gz \-p item_file /Volumes/GoogleDrive/Shared\ drives/KGTK-public-graphs/wikidata-20200803-v4/part.wikibase-item.tsv.gz \-p property_item_file = /Volumes/GoogleDrive/Shared\ drives/KGTK-public-graphs/wikidata-20200803-v4/part.property.wikibase-item.tsv.gz \-p output_path  \-p output_folder useful_files_v4 \-p temp_folder temp.useful_files_v4 \-p delete_database no -p languages es,ru,zh-cn```<jupyter_code>import os
import sys

import pandas as pd
 
from kgtk.configure_kgtk_notebooks import ConfigureKGTK
from kgtk.functions import kgtk, kypher
# Parameters

input_path = "/data/amandeep/wikidata-20220505-dwd-v4"
output_path = "/data/amandeep/wikidata-20220505-dwd-v4"
kgtk_path = "/Users/amandeep/Github/kgtk"

graph_cache_path = None

project_name = "embeddings-elasticsearch-triples"

languages = 'en,ru,es,zh-cn,de,it,nl,pl,fr,pt,sv'

files = 'label_all,alias_all,description_all'
compute_embeddings = False
generate_triples = False
generate_kgtk_search = True
datatype_property = "datatype"
files = files.split(',')
ck = ConfigureKGTK(files, kgtk_path=kgtk_path)
ck.configure_kgtk(input_graph_path=input_path,
                  output_path=output_path,
                  project_name=project_name,
                 graph_cache_path=graph_cache_path)
ck.print_env_variables()<jupyter_output>USE_CASES_DIR: /Users/amandeep/Github/kgtk/use-cases
TEMP: /data/amandeep/wikidata-20220505-dwd-v4/embeddings-elasticsearch-triples/temp.embeddings-elasticsearch-triples
EXAMPLES_DIR: /Users/amandeep/Github/kgtk/examples
KGTK_LABEL_FILE: /data/amandeep/wikidata-20220505-dwd-v4/labels.en.tsv.gz
GRAPH: /data/amandeep/wikidata-20220505-dwd-v4
kypher: kgtk query --graph-cache /data/amandeep/wikidata-20220505-dwd-v4/embeddings-elasticsearch-triples/temp.embeddings-elasticsearch-triples/wikidata.sqlite3.db
KGTK_OPTION_DEBUG: false
kgtk: kgtk
KGTK_GRAPH_CACHE: /data/amandeep/wikidata-20220505-dwd-v4/embeddings-elasticsearch-triples/temp.embeddings-elasticsearch-triples/wikidata.sqlite3.db
OUT: /data/amandeep/wikidata-20220505-dwd-v4/embeddings-elasticsearch-triples
STORE: /data/amandeep/wikidata-20220505-dwd-v4/embeddings-elasticsearch-triples/temp.embeddings-elasticsearch-triples/wikidata.sqlite3.db
label_all: /data/amandeep/wikidata-20220505-dwd-v4/labels.tsv.gz
alias_all: /data/amandeep/wi[...]<jupyter_text>Graph Embeddings  complEx<jupyter_code>if compute_embeddings:
    complex_temp_folder = f"{wikidata_root_folder}/temp.graph-embeddings.complex"
if compute_embeddings:
    !mkdir -p {complex_temp_folder}
if compute_embeddings:
    os.environ['TEMP_COMPLEX'] = complex_temp_folder
if compute_embeddings:
    !kgtk graph-embeddings --verbose -i "$ITEMS" \
    -o $OUT/wikidatadwd.complEx.graph-embeddings.txt \
    --retain_temporary_data True \
    --operator ComplEx \
    --workers 24 \
    --log $TEMP_COMPLEX/ge.complex.log \
    -T $TEMP_COMPLEX \
    -ot w2v \
    -e 600<jupyter_output><empty_output><jupyter_text>transE<jupyter_code>if compute_embeddings:
    transe_temp_folder = f"{wikidata_root_folder}/temp.graph-embeddings.transe"
if compute_embeddings:
    !mkdir -p {transe_temp_folder}
if compute_embeddings:
    os.environ['TEMP_TRANSE'] = transe_temp_folder
if compute_embeddings:
    !$kgtk graph-embeddings --verbose -i "$ITEMS" \
    -o $OUT/wikidatadwd.transE.graph-embeddings.txt \
    --retain_temporary_data True \
    --operator TransE \
    --workers 24 \
    --log $TEMP_TRANSE/ge.transE.log \
    -T $TEMP_TRANSE \
    -ot w2v \
    -e 600<jupyter_output><empty_output><jupyter_text>BERT Embeddings<jupyter_code>if compute_embeddings:
    !$kgtk text-embedding -i $ALL   \
    --model roberta-large-nli-mean-tokens   \
    --property-labels-file $LABELS_EN   \
    --isa-properties P31 P279 P106 P39 P1382 P373 P452 \
    --save-embedding-sentence > $OUT/wikidatadwd-text-embeddings-all.tsv<jupyter_output><empty_output><jupyter_text>Build KGTK edge file for KGTK Search<jupyter_code>if generate_kgtk_search:
    kgtk("""cat -i $GRAPH/all.tsv.gz 
                -i $GRAPH/derived.isastar.tsv.gz 
                -i $GRAPH/metadata.pagerank.undirected.tsv.gz
                -i $GRAPH/metadata.pagerank.directed.tsv.gz
                -o $TEMP/wikidata.dwd.all.kgtk.search.unsorted.tsv.gz""")
if generate_kgtk_search:
    kgtk(f"""sort -i $TEMP/wikidata.dwd.all.kgtk.search.unsorted.tsv.gz
                --columns node1
                --extra '--parallel 24 --buffer-size 30% --temporary-directory {os.environ['TEMP']}'
                -o $OUT/wikidata.dwd.all.kgtk.search.sorted.tsv.gz""")
if generate_kgtk_search:
    kgtk(f"""--debug build-kgtk-search-input --input-file "$OUT"/wikidata.dwd.all.kgtk.search.sorted.tsv.gz
    --output-file "$OUT"/wikidata.dwd.all.kgtk.search.sorted.jl 
    --label-properties label 
    --alias-properties alias 
    --extra-alias-properties P1448,P1705,P1477,P1810,P742,P1449 
    --description-properties description 
    --pagerank-properties Pundirected_pagerank 
    --languages {languages}
    --mapping-file "$OUT"/wikidata_dwd_v3_mapping.json 
    --property-datatype-file "$GRAPH"/metadata.property.datatypes.tsv.gz""")<jupyter_output><empty_output><jupyter_text>Build KGTK edge file for Triple generation<jupyter_code>if generate_triples:
    !$kgtk cat \
    -i $OUT/wikidata.dwd.all.kgtk.search.sorted.tsv.gz \
    -i $OUT/derived.isa.tsv.gz \
    -i $OUT/derived.P279star.tsv.gz \
    -i $OUT/metadata.in_degree.tsv.gz \
    -i $OUT/metadata.out_degree.tsv.gz \
    -o $TEMP/wikidata.dwd.all.kgtk.triples.1.tsv.gz
if generate_triples:
    !$kgtk add-id -i $TEMP/wikidata.dwd.all.kgtk.triples.1.tsv.gz \
    --id-style wikidata \
    -o $TEMP/wikidata.dwd.all.kgtk.triples.2.tsv.gz
if generate_triples:
    !$kgtk sort -i $TEMP/wikidata.dwd.all.kgtk.triples.2.tsv.gz \
    --columns node1 \
     --extra '--parallel 24 --buffer-size 30% --temporary-directory ' + temp_folder_path \
    -o $OUT/wikidata.dwd.all.kgtk.triples.sorted.tsv.gz<jupyter_output><empty_output><jupyter_text>Split the triples file to parallelize triple generation<jupyter_code>if generate_triples:
    !mkdir -p $OUT/kgtk_triples_split
if generate_triples:
    !$kgtk split -i $OUT/wikidata.dwd.all.kgtk.triples.sorted.tsv.gz \
    --output-path $OUT/kgtk_triples_split \
    --gzipped-output --lines 10000000 \
    --file-prefix kgtk_triples
if generate_triples:
    !curl https://raw.githubusercontent.com/usc-isi-i2/kgtk/dev/kgtk-properties/kgtk.properties.tsv -o $TEMP/kgtk-properties.tsv
if generate_triples:
    kgtk(f"""filter -p ";{datatype_property};" -i $TEMP/kgtk-properties.tsv -o $TEMP/kgtk-properties.datatype.tsv.gz""")
if generate_triples:
    !$kgtk cat -i $TEMP/kgtk-properties.datatype.tsv.gz $OUT/metadata.property.datatypes.tsv.gz -o $OUT/metadata.property.datatypes.augmented.tsv.gz
if generate_triples:
    ls $OUT/kgtk_triples_split/*.tsv.gz | parallel -j 18  'kgtk --debug generate-wikidata-triples -lp label -ap alias -dp description -pf $OUT/metadata.property.datatypes.augmented.tsv.gz --output-n-lines 100000 --generate-truthy --warning --use-id --log-path $TEMP/generate_triples_log.txt --error-action log -i {} -o {.}.ttl'<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12532699167657552, 'fraction_numerical': 0.026278240190249703, 'mean_word_length': 3.9360328638497655, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 29, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '542050', 'license': 'MIT', 'n_tokens_mistral': 3380, 'n_tokens_neox': 3155, 'n_words': 510, 'path': 'use-cases/create_wikidata/Embeddings-Elasticsearch-Triples.ipynb', 'repo_name': 'usc-isi-i2/kgtk-tutorial-files', 'size': '8410'}