<jupyter_start><jupyter_text>--- Python Notebook to read all the text files from a text dataset into Neo4j database (syntactilally-based, SVO)--- 1. List all the text files in the sub-directories your dataset.2. Read all the files.3. Create nodes, where n(nodes) = n(files)4. Dump the text files into individual nodes where every node is a document using Graphaware's NLP pipeline.5. Sample scripts for entity extraction--- Importsglob --> for iterating through the folders and sub-folders<jupyter_code>import glob
import csv<jupyter_output><empty_output><jupyter_text>Specifying the path for the files, the wildcards at the end of the path denote that all the files from all the subdirectories from the bbc folder will be accessed.<jupyter_code>folder_path = '/home/earendil/Desktop/ML_playground/Neo4j-NLP/data/biomarker/*'<jupyter_output><empty_output><jupyter_text>glob will help iterate through the entire folder path, allowing wildcards<jupyter_code>from py2neo import *
from neo4j import GraphDatabase
uri = "bolt://localhost:7687"
driver = GraphDatabase.driver(uri, auth=("neo4j", "1qazxsw2"))
authenticate("localhost:7474", "neo4j", "1qazxsw2")
graph = Graph("http://localhost:7474/db/data/")

#graph = Graph(host='localhost', user='neo4j',password='password')
tx = graph.begin()
for filename in glob.glob(folder_path):
    with open(filename, 'r') as f: 
        file_contents = f.read() 
        Nodes = Node("Article",Text=str(file_contents),path=filename)
        print(Nodes)
        graph.create(Nodes)
        tx.merge(Nodes)<jupyter_output><empty_output><jupyter_text>CALL ga.nlp.processor.addPipeline({name:"pipeline",textProcessor: 'com.graphaware.nlp.processor.stanford.ee.processor.EnterpriseStanfordTextProcessor',processingSteps: {tokenize:true, ner:true, dependencies:true, relations:true, open:true, sentiment:true}})<jupyter_code>#NEO4J scripts (non-Python code)

CALL ga.nlp.processor.addPipeline({
    name: 'pipeline2', 
    textProcessor: 'com.graphaware.nlp.processor.stanford.StanfordTextProcessor', 
    processingSteps: {tokenizerAndSentiment:true, ner: true, dependency: true}})

#NEO4J scripts (non-Python code)

CALL apoc.periodic.iterate(
'MATCH (n:Article) RETURN n',
'CALL ga.nlp.annotate({
        	text: n.Text,
        	id: id(n),
        	pipeline: "pipeline2",
        	checkLanguage:false
})
YIELD result MERGE (n)-[:HAS_ANNOTATED_TEXT]->(result)',
{batchSize:1, iterateList:false})

#NEO4J scripts (non-Python code)

MATCH (s:TagOccurrence)<-[]-(a:Sentence)-[]->(v:TagOccurrence),
(a)-[]->(o:TagOccurrence)
WHERE s.pos IN [['NNP']] AND v.pos IN [['VBZ']] AND o.pos IN [['NN']]
RETURN DISTINCT s.value, v.value, o.value
#NEO4J scripts (non-Python code)

MATCH (s:TagOccurrence)<-[]-(a:Sentence)-[]->(v:TagOccurrence),
(a)-[]->(o:TagOccurrence)
WHERE s.pos IN [['NNP']] AND v.pos IN [['VBZ']] AND o.pos IN [['NN']] AND abs(v.startPosition-s.endPosition)<10 AND abs(o.startPosition-v.endPosition)<10 
RETURN DISTINCT s.value, v.value, o.value, v.startPosition-s.endPosition
#NEO4J scripts (non-Python code)

MATCH p= (ar:Article)-[:HAS_ANNOTATED_TEXT]->(an:AnnotatedText)-[:CONTAINS_SENTENCE]->(se:Sentence)-[:SENTENCE_TAG_OCCURRENCE]-(s:TagOccurrence)-[:NSUBJ]-(v:TagOccurrence)-[:DOBJ]-(o:TagOccurrence)
OPTIONAL MATCH (o:TagOccurrence)-[:COMPOUND]-(co:TagOccurrence)
OPTIONAL MATCH (o:TagOccurrence)-[:AMOD]-(am:TagOccurrence)
OPTIONAL MATCH (o:TagOccurrence)-[:NMOD]-(nm:TagOccurrence)
OPTIONAL MATCH (o:TagOccurrence)-[:NMOD]-(nm:TagOccurrence)
OPTIONAL MATCH (nm:TagOccurrence)-[:APPOS]-(apr:TagOccurrence)


RETURN se.text as Text, s.value as Subject, v.value as Predicate, am.value as Desc1, nm.value as Desc2,co.value as Desc3,  apr.value as Prop,  o.value as Object LIMIT 200
MATCH p= (ar:Article)-[:HAS_ANNOTATED_TEXT]->(an:AnnotatedText)-[:CONTAINS_SENTENCE]->(se:Sentence)-[:SENTENCE_TAG_OCCURRENCE]-(s:TagOccurrence)-[:NSUBJ]-(v:TagOccurrence)-[:DOBJ]-(o:TagOccurrence)
OPTIONAL MATCH (o:TagOccurrence)-[:COMPOUND]-(co:TagOccurrence)
OPTIONAL MATCH (o:TagOccurrence)-[:AMOD]-(am:TagOccurrence)
OPTIONAL MATCH (o:TagOccurrence)-[:NMOD]-(nm:TagOccurrence)
OPTIONAL MATCH (o:TagOccurrence)-[:NMOD]-(nm:TagOccurrence)
OPTIONAL MATCH (nm:TagOccurrence)-[:APPOS]-(apr:TagOccurrence)

CALL apoc.create.relationship(s, toString(v.value), {}, o) YIELD rel

RETURN se.text as Text, s.value as Subject, v.value as Predicate, am.value as Desc1, nm.value as Desc2,co.value as Desc3,  apr.value as Prop,  o.value as Object LIMIT 200
MATCH a = ()-[:explore]-()
MATCH b = ()-[:combine]-()
MATCH c = ()-[:discover]-()
MATCH d = ()-[:changed]-()
MATCH e = ()-[:link]-()

RETURN a,b,c,d,e
MATCH p= (ar:Article)-[:HAS_ANNOTATED_TEXT]->(an:AnnotatedText)-[:CONTAINS_SENTENCE]->(se:Sentence)-[:SENTENCE_TAG_OCCURRENCE]-(s:TagOccurrence)-[:NSUBJ]-(v:TagOccurrence)-[:DOBJ]-(o:TagOccurrence)
OPTIONAL MATCH (o:TagOccurrence)-[:AMOD]-(am:TagOccurrence)
OPTIONAL MATCH (o:TagOccurrence)-[:NMOD]-(nm:TagOccurrence)
OPTIONAL MATCH (nm:TagOccurrence)-[:APPOS]-(apr:TagOccurrence)
OPTIONAL MATCH (s:TagOccurrence)-[:COMPOUND]-(coms:TagOccurrence)
OPTIONAL MATCH (o:TagOccurrence)-[:COMPOUND]-(como:TagOccurrence)

RETURN se.id as SentenceID, 
toString(COALESCE(coms.value+' ', ' '))+
toString(s.value) as Subject,
v.value as Predicate, 
toString(COALESCE(am.value+' ',' '))+
toString(COALESCE(nm.value+' ', ' '))+
toString(COALESCE(apr.value+' ',' '))+
toString(COALESCE(como.value+' ',' '))+ 
toString(o.value) as Object 
LIMIT 200<jupyter_output><empty_output><jupyter_text>-------------- --- OpenIE for automated triple extraction based on open information extraction triples--- 0. Coref resolution reads a file and substitutes the antecedents in place, producing a resolved text.1. OpenIE will read the text and extract triples using Stanford Core NLP2. The output is a list of dicts containing "subject-relation-object" triples.3. Due to the OIE implementation, similar nodes and edges (almost identical meanings) are present, we can remove them see below, but we may also miss information this way.4. Transfer these triples to Neo4j as nodes-edges.---<jupyter_code>from openie import StanfordOpenIE

with StanfordOpenIE() as client:
    
    with open('/home/earendil/Desktop/ML_playground/Neo4j-NLP/data/siris/tresspass.txt', 'r', encoding='utf8') as r:
        corpus = r.read().replace('\n', ' ').replace('\r', '')

    triples_corpus = client.annotate(corpus, 
                            properties={
                            "annotators":"tokenize,ssplit,pos,lemma,depparse,ner,coref,mention,natlog,openie",
                            "outputFormat": "json",
                            "openie.format": "ollie",
                            "openie.resolve_coref": "false",                         
                                 })
    
    print('Corpus: %s [...].' % corpus[0:80])
    print('Found %s triples in the corpus.' % len(triples_corpus))
    for triple in triples_corpus:
        print('|-', triple)

#Neural coreference resolution https://stackoverflow.com/questions/50004797/anaphora-resolution-in-stanford-nlp-using-python
#NEEDS SPACY 2.0.13
#Don't run this, run the AllenNLP script below instead
import spacy
nlp = spacy.load('en_coref_md')
doc = nlp(corpus)
corpus_coref = doc._.coref_resolved
triples_corpus = client.annotate(corpus_coref, 
                    properties={
                    "annotators":"tokenize,ssplit,pos,lemma,depparse,ner,coref,mention,natlog,openie",
                    "outputFormat": "json",
                    "openie.format": "ollie",
                    "openie.resolve_coref": "false",                         
                         })

print('Corpus: %s [...].' % corpus_coref[0:80])
print('Found %s triples in the corpus.' % len(triples_corpus))
for triple in triples_corpus:
    print('|-', triple)
# Neural Coref using Allennlp
#NEEDS SPACY 2.2.0
from allennlp.predictors import CorefPredictor
predictor = CorefPredictor.from_path("https://s3-us-west-2.amazonaws.com/allennlp/models/coref-model-2018.02.05.tar.gz")

corpus_coref = predictor.coref_resolved(document=corpus)

triples_corpus = client.annotate(corpus_coref, 
                    properties={
                    "annotators":"tokenize,ssplit,pos,lemma,depparse,ner,coref,mention,natlog,openie",
                    "outputFormat": "json",
                    "openie.format": "ollie",
                    "openie.resolve_coref": "false",                         
                         })

print('Corpus: %s [...].' % corpus_coref[0:80])
print('Found %s triples in the corpus.' % len(triples_corpus))
for triple in triples_corpus:
    print('|-', triple)
#remove duplicate triples (i.e. where some objects are subsets of others ) 

#MUST BE RUN, but not currently used. To use it substitute triples with unique_triples on the code blocks

triples = sorted(triples_corpus, key = lambda i: len(i['object']),reverse=True)

duplicates= []
unique_triples = []
c=0
for d in triples:
    t = tuple(d.items())
    if (triples[c]["relation"]) not in duplicates:
        duplicates.append(triples[c]["relation"])
        unique_triples.append(d)
    c+=1
print("===============")

print('Number of triples generated by OpenIE:',len(triples))
print('Number of triples after removing duplicates:',len(unique_triples))
#connect to Neo4j DB
from py2neo import *
from neo4j import GraphDatabase
uri = "bolt://localhost:7687"
authenticate("localhost:7474", "neo4j", "1qazxsw2")

driver = GraphDatabase.driver(uri, auth=("neo4j", "1qazxsw2"))
graph = Graph("http://localhost:7474/db/data/")

# create nodes and relations from extracted triples
tx = graph.begin()
for i in range(len(triples)):
    u1 = Node("Subject", name=triples[i]['subject'], corpus='CORDIS', entry='TRESSPASS')
    u2 = Node("Object", name=triples[i]['object'],corpus='CORDIS', entry='TRESSPASS')
    graph.merge(u1 | u2)
    relation = Relationship(u1,triples[i]['relation'], u2)
    graph.merge(relation)

    
# create nodes for the Corpus and the Entry if it doesnt exist already (merge instead of create)
u1 = Node("Entry", name='TRESSPASS', corpus='CORDIS')
u2 = Node("Corpus", name='CORDIS')
graph.merge(u1 | u2)
relation = Relationship(u2,'has_entry', u1)
graph.merge(relation)

#Gather all the triples under the same node
tx.run("MATCH (sub:Subject), (p:Entry {name: 'TRESSPASS' }) WHERE sub.entry IN ['TRESSPASS'] WITH p, COLLECT(sub) AS subs CREATE (p)-[:contains]->(c: Combo {name:'Triples-TRESSPASS'}) FOREACH(s IN subs | CREATE (c)-[:has_triple]->(s)) RETURN *")
tx.process()
tx.commit()<jupyter_output><empty_output><jupyter_text>--- Misc individual scripts for coref resolution (used above)---<jupyter_code>#Neural coreference resolution https://stackoverflow.com/questions/50004797/anaphora-resolution-in-stanford-nlp-using-python

import spacy

nlp = spacy.load('en_coref_md')

doc = nlp(corpus)

print(doc._.coref_clusters)

print(doc._.coref_resolved)
#Co-reference resolution using AthNLP

from allennlp.predictors import CorefPredictor
predictor = CorefPredictor.from_path("https://s3-us-west-2.amazonaws.com/allennlp/models/coref-model-2018.02.05.tar.gz")

print(predictor.coref_resolved(document=corpus))

#Co-reference resolution using StanfordCoreNLP (TODO)

def resolve(corenlp_output):
    """ Transfer the word form of the antecedent to its associated pronominal anaphor(s) """
    for coref in corenlp_output['corefs']:
        mentions = corenlp_output['corefs'][coref]
        antecedent = mentions[0]  # the antecedent is the first mention in the coreference chain
        for j in range(1, len(mentions)):
            mention = mentions[j]
            if mention['type'] == 'PRONOMINAL':
                # get the attributes of the target mention in the corresponding sentence
                target_sentence = mention['sentNum']
                target_token = mention['startIndex'] - 1
                # transfer the antecedent's word form to the appropriate token in the sentence
                corenlp_output['sentences'][target_sentence - 1]['tokens'][target_token]['word'] = antecedent['text']


def print_resolved(corenlp_output):
    """ Print the "resolved" output """
    possessives = ['hers', 'his', 'their', 'theirs']
    for sentence in corenlp_output['sentences']:
        for token in sentence['tokens']:
            output_word = token['word']
            # check lemmas as well as tags for possessive pronouns in case of tagging errors
            if token['lemma'] in possessives or token['pos'] == 'PRP$':
                output_word += "'s"  # add the possessive morpheme
            output_word += token['after']
            print(output_word, end='')


output = client.annotate(corpus, properties= {'annotators':'dcoref','outputFormat':'json','ner.useSUTime':'false'})
print(output)
resolve(output)

print('Original:', text)
print('Resolved: ', end='')
print_resolved(output)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14688081936685288, 'fraction_numerical': 0.011949099937926753, 'mean_word_length': 4.226682887266829, 'pattern_counts': {'":': 12, '<': 29, '<?xml version=': 0, '>': 39, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '321358', 'license': 'MIT', 'n_tokens_mistral': 4503, 'n_tokens_neox': 4334, 'n_words': 970, 'path': 'src/DocumentIngestion.ipynb', 'repo_name': 'lighteternal/Automated-Knowledge-Graph-Construction', 'size': '12888'}