<jupyter_start><jupyter_text>Compute ComplEx Graph Embeddings In this notebook we will compute graph embeddings using `kgtk graph-embeddings` command for the `arnold` subgraph and demonstrate a few applications.First step is to augment the graph with `derived.P31x.tsv` file which considers occupations for humans as `instance of (P31)`<jupyter_code>!kgtk cat -i $item \
-i $GRAPH/derived.P31x.tsv \
-o $GRAPH/claims.wikibase-item.augmented.tsv.gz<jupyter_output><empty_output><jupyter_text>Compute the `ComplEx` graph embeddings with vectors of dimension `30`<jupyter_code>!kgtk --debug graph-embeddings \
-op ComplEx \
--dimension 30 \
-ot kgtk \
-i $GRAPH/claims.wikibase-item.augmented.tsv.gz \
-o $OUT/arnold.embeddings.augmented.30.tsv \
--log $TEMP/ge.log.txt<jupyter_output>In Processing, Please go to /Volumes/saggu-ssd/projects/tutorial-graph-embeddings/temp.tutorial-graph-embeddings/ge.log.txt to check details
Processed Finished.<jupyter_text>The output is in `kgtk` format. Convert it to `word2vec` format for `gensim` similarity computation<jupyter_code>def convert_kgtk_to_w2v(input_path, output_path):
    """
    Convert a KGTK file (node1/label/node2) that contains embeddings to the w2v format
    """
    vector_count = 0
    vector_length = 0

    # Read the file once to count the lines as we need to put them at the top of the w2v file
    with open(input_path, "r") as kgtk_file:
        next(kgtk_file)
        for line in kgtk_file:
            items = line.split("\t")

            if vector_count == 0:
                vector_length = len(items[2].split(","))
            vector_count += 1
        kgtk_file.close()

    with open(output_path, "w") as w2v_file:
        w2v_file.write("{} {}\n".format(vector_count, vector_length))
        with open(input_path, "r") as kgtk_file:
            next(kgtk_file)
            for line in kgtk_file:
                items = line.split("\t")
                qnode = items[0]
                vector = items[2].replace(",", " ")
                w2v_file.write(qnode + " " + vector)
            kgtk_file.close()
        w2v_file.close()
convert_kgtk_to_w2v(f"{os.environ['OUT']}/arnold.embeddings.augmented.30.tsv", f"{os.environ['OUT']}/arnold.embeddings.augmented.30.w2v.tsv")<jupyter_output><empty_output><jupyter_text>Load the vectos into `gensim`<jupyter_code>ge_vectors = KeyedVectors.load_word2vec_format(f"{os.environ['OUT']}/arnold.embeddings.augmented.30.w2v.tsv", binary=False)<jupyter_output><empty_output><jupyter_text>Define a function to compute the `topn` similar vectors, and get the labels and descriptions of the matching Qnodes.<jupyter_code>def kgtk_most_similar(
    vectors,
    positive,
    relation_label="similarity_score",
    kg_path=None,
    add_label_description=True,
    output_path=None,
    topn=25,
):
    """"""
    result = []
    if add_label_description and kg_path:
        fp = tempfile.NamedTemporaryFile(
            mode="w", suffix=".tsv", delete=False, encoding="utf-8"
        )
        fp.write("node1\tlabel\tnode2\n")
        for (qnode, similarity) in vectors.most_similar(positive=positive, topn=topn):
            fp.write("{}\t{}\t{}\n".format(qnode, relation_label, similarity))
        filename = fp.name
        fp.close()

        os.environ["_temp_file"] = filename

        result = !$kypher -i label -i description -i "$_temp_file" --as sim \
--match 'sim: (n1)-[]->(similarity), label: (n1)-[]->(lab), description: (n1)-[]->(des)' \
--return 'distinct n1 as node1, similarity as node2, "similarity" as label, lab as `node1;label`, des as `node1;description`' \
--order-by 'cast(similarity, float) desc' 
        
        os.remove(filename)
        
    else:
        result.append("node1\tlabel\tnode2\n")
        for (qnode, similarity) in vectors.most_similar(positive=positive, topn=topn):
            result.append("{}\t{}\t{}\n".format(qnode, relation_label, similarity))

    if output_path:
        handle = open(output_path, "w")
        for line in result:
            handle.write(line)
            handle.write("\n")
        handle.close()
    else:
        columns = result[0].split("\t")
        data = []
        for line in result[1:]:
            data.append(line.split("\t"))
        return pd.DataFrame(data, columns=columns)<jupyter_output><empty_output><jupyter_text>Find the most similar 10 Qnodes to `Q2685` (<NAME>)<jupyter_code>kgtk_most_similar(ge_vectors, positive=['Q2685'], kg_path = os.environ['TEMP'], topn=10)<jupyter_output><empty_output><jupyter_text>Find the most similar 10 Qnodes to `Q162255` (Terminator,the movie)<jupyter_code>kgtk_most_similar(ge_vectors, positive=['Q162255'], kg_path = os.environ['TEMP'], topn=10)
v1 = ge_vectors['Q873'] # <NAME>
v2 = ge_vectors['Q2685'] # <NAME>
v3 = ge_vectors['Q103148'] # Lahn River
v4 = ge_vectors['Q104123'] # Pulp Fiction<jupyter_output><empty_output><jupyter_text>Compute average of `<NAME>` and `<NAME>`, find most similar Qnodes<jupyter_code>aver = np.mean([v1, v2], axis=0 )
kgtk_most_similar(ge_vectors, positive=aver, kg_path = os.environ['TEMP'], topn=10)<jupyter_output><empty_output><jupyter_text>Compute average of `<NAME>ep` and `Lahn River`, find most similar Qnodes<jupyter_code>aver = np.mean([v3, v1], axis=0 )
kgtk_most_similar(ge_vectors, positive=aver, kg_path = os.environ['TEMP'], topn=10)<jupyter_output><empty_output><jupyter_text>Compute average of `Lahn River` and `Pulp Fiction`, find most similar Qnodes<jupyter_code>aver = np.mean([v3, v4], axis=0)
kgtk_most_similar(ge_vectors, positive=aver, kg_path = os.environ['TEMP'], topn=10)<jupyter_output><empty_output><jupyter_text>Compute average of `Meryl Streep` and `Pulp Fiction`, find most similar Qnodes<jupyter_code>aver = np.mean([v4, v1], axis=0)
kgtk_most_similar(ge_vectors, positive=aver, kg_path = os.environ['TEMP'], topn=10)<jupyter_output><empty_output><jupyter_text>Prepare files for Google Projector In this section, we will prepare `vectors` and `metadata` files for google projector.We are focusing on the following types:- `Q11424` (film)- `Q33999` (actor)- `Q4022` (river)- `Q82955` (politician)First step is to create a file with the following information ,1. node1 :- Qnode2. node1;label :- label for node13. node2 :- `instance of` for node14. node2;label :- label for node25. embedding :- ComplEx graph embedding vector for node1<jupyter_code>%%time
kgtk(""" query -i $GRAPH/claims.wikibase-item.augmented.tsv.gz 
         -i p279star 
         -i label 
         -i $OUT/arnold.embeddings.augmented.30.tsv 
         -i $GRAPH/derived.P31x.tsv 
         --match 'item: (n1)-[]->(), 
             P31x: (n1)-[]->(c), 
             p279star: (c)-[]->(class), 
             label: (n1)-[]->(n1_label), 
             label: (class)-[]->(class_label), embeddings: (n1)-[]->(embedding)'
        --where 'class in ["Q11424", "Q33999", "Q4022", "Q82955"]' 
        --return 'distinct n1, kgtk_lqstring_text(n1_label) as `node1;label`, group_concat(distinct class) as node2, group_concat(distinct kgtk_lqstring_text(class_label)) as `node2;label`, embedding as embedding' \
        -o $TEMP/arnold.embeddings.google.projector.tsv
""")<jupyter_output>CPU times: user 4.07 ms, sys: 11.8 ms, total: 15.9 ms
Wall time: 3.73 s<jupyter_text>Take a peek at the file<jupyter_code>!head -2 $TEMP/arnold.embeddings.google.projector.tsv<jupyter_output>node1	node1;label	node2	node2;label	embedding
Q1000881	Erlau	Q4022	river	-0.278222889,0.298910052,-0.058227286,-0.707951486,-0.300852597,-0.292677164,0.347793698,0.180076435,0.370242327,0.807168007,-0.382684618,0.173900068,0.034428731,-1.144109964,-0.129670262,-0.191929370,-0.716327786,0.031291030,0.180736080,-0.297246993,-0.714032531,-0.364734739,0.823571563,0.272169203,0.657331109,0.312936962,-0.594842017,-0.129835084,0.980592489,-0.074916743<jupyter_text>Define a function to build the required files for google projector<jupyter_code>def build_embedding_projector_metadata(gp_embeddings_path, metadata_path, vectors_path):
    metadata_file = open(metadata_path, "w")
    metadata_file.write("tag\ttype\ttype_label\n")

    vectors_file = open(vectors_path, "w")

    with open(gp_embeddings_path) as qnodes_file:
        next(qnodes_file)
        for line in qnodes_file:
            vals = line.split('\t')
            qnode = vals[0]
            qnode_label = vals[1]
            ftype_label = vals[3]
            embeddings = "\t".join(vals[4].strip().split(","))

            tag = "{} ({})".format(qnode, qnode_label)

            if qnode.startswith("Q"):
                metadata_file.write("{}\t{}\t{}\n".format(tag, qnode, ftype_label))
                vectors_file.write(embeddings)
                vectors_file.write('\n')

    metadata_file.close()
    vectors_file.close()
build_embedding_projector_metadata(f"{os.environ['TEMP']}/arnold.embeddings.google.projector.tsv",
                                  f"{os.environ['OUT']}/arnold.metadata.30.tsv",
                                  f"{os.environ['OUT']}/arnold.vectors.30.tsv")<jupyter_output><empty_output><jupyter_text>Peek at the metadata file<jupyter_code>!head $OUT/arnold.metadata.30.tsv<jupyter_output>tag	type	type_label
Q1000881 (Erlau)	Q1000881	river
Q1001872 (Buersbach)	Q1001872	river
Q1004531 (Bullets Over Broadway)	Q1004531	film
Q1009788 (The Conversation)	Q1009788	film
Q1010099 (Get Carter)	Q1010099	film
Q1012216 (Gorillas in the Mist)	Q1012216	film
Q101410 (<NAME>)	Q101410	politician
Q101797 (<NAME>)	Q101797	actor
Q1018487 (Bye Bye Birdie)	Q1018487	film<jupyter_text>Peek at the vectors file<jupyter_code>!head -2 $OUT/arnold.vectors.30.tsv<jupyter_output>-0.278222889	0.298910052	-0.058227286	-0.707951486	-0.300852597	-0.292677164	0.347793698	0.180076435	0.370242327	0.807168007	-0.382684618	0.173900068	0.034428731	-1.144109964	-0.129670262	-0.191929370	-0.716327786	0.031291030	0.180736080	-0.297246993	-0.714032531	-0.364734739	0.823571563	0.272169203	0.657331109	0.312936962	-0.594842017	-0.129835084	0.980592489	-0.074916743
-0.292729884	0.365311205	0.049547877	-0.266633153	-0.517612636	-0.337442279	0.316574246	-0.161643475	0.366951913	1.027803302	-0.281238228	-0.533051729	0.022873841	-1.064334035	0.336856425	0.353341699	-0.658462703	0.076028503	0.257462531	-0.099187657	-0.353778273	-0.526077926	0.809671104	-0.297059596	0.488877356	-0.055506136	-0.431686580	-0.327586085	0.996009290	0.028423335
{'fraction_non_alphanumeric': 0.1366885625965997, 'fraction_numerical': 0.12200540958268934, 'mean_word_length': 3.8153488372093025, 'pattern_counts': {'":': 0, '<': 68, '<?xml version=': 0, '>': 77, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '325980', 'license': 'MIT', 'n_tokens_mistral': 4658, 'n_tokens_neox': 3890, 'n_words': 747, 'path': 'tutorial/kg-graph-embeddings.ipynb', 'repo_name': 'deeplearningnrs/kgtk-notebooks', 'size': '10428'}