<jupyter_start><jupyter_text>Noun chunksBasic usage of spaCy from its web page. https://spacy.io/usage/linguistic-featuresdependency-parse> Noun chunks are “base noun phrases” – flat phrases that have a noun as their head. You can think of noun chunks as a noun plus the words describing the noun – for example, “the lavish green grass” or “the world’s largest tech fund”. To get the noun chunks in a document, simply iterate over Doc.noun_chunks.<jupyter_code>nlp = spacy.load("en_core_web_sm")
doc = nlp("Autonomous cars shift insurance liability toward manufacturers")
for chunk in doc.noun_chunks:
    print(chunk.text, chunk.root.text, chunk.root.dep_,
            chunk.root.head.text)<jupyter_output>Autonomous cars cars nsubj shift
insurance liability liability dobj shift
manufacturers manufacturers pobj toward<jupyter_text>Navigating the Parse Tree> spaCy uses the terms head and child to describe the words connected by a single arc in the dependency tree. The term dep is used for the arc label, which describes the type of syntactic relation that connects the child to the head. As with other attributes, the value of .dep is a hash value. You can get the string value with .dep_.<jupyter_code>doc = nlp("Autonomous cars shift insurance liability toward manufacturers")
for token in doc:
    print(token.text, token.dep_, token.head.text, token.head.pos_,
            [child for child in token.children])<jupyter_output>Autonomous amod cars NOUN []
cars nsubj shift VERB [Autonomous]
shift ROOT shift VERB [cars, liability, toward]
insurance compound liability NOUN []
liability dobj shift VERB [insurance]
toward prep shift VERB [manufacturers]
manufacturers pobj toward ADP []<jupyter_text>Visualizing Dependencies<jupyter_code>from spacy import displacy
doc = nlp("Autonomous cars shift insurance liability toward manufacturers")
# Since this is an interactive Jupyter environment, we can use displacy.render here
displacy.render(doc, style='dep')<jupyter_output><empty_output><jupyter_text>40. Read the Parsed Result as Words<jupyter_code># https://stackoverflow.com/questions/1535327/how-to-print-instances-of-a-class-using-print

class Word(object):
    def __init__(self, text, lemma, pos):
        self.text = text
        self.lemma = lemma
        self.pos = pos
    
    def __str__(self):
        return self.text
    
    def __repr__(self):
        return self.__str__()
with open('./ai.en.txt') as f:
    text = f.read()
    text = text.replace('."', '".')
    text = text.replace('e.g.', 'eg')
    text = text.replace('U.S.', 'US')
    text = text.replace('U.S', 'US')
    text = text.replace('R.U.R.', 'RUR')
    text = text.replace('c.', 'c')
    text = text.replace('No.', 'No')
    text = text.split('\n')
    text = list(filter(lambda x: x, text))
sentences = []

for t in text:
    for sentence in t.split('.'):
        s = sentence.strip()
        if s:
            sentences.append(s)
for s in sentences:
    doc = nlp(s)
    words = [Word(token.text, token.lemma_, token.pos_) for token in doc]
    
    print(words)
    break<jupyter_output>[In, computer, science, ,, artificial, intelligence, (, AI, ), ,, sometimes, called, machine, intelligence, ,, is, intelligence, demonstrated, by, machines, ,, in, contrast, to, the, natural, intelligence, displayed, by, humans, and, animals]<jupyter_text>41. Depencency Tree<jupyter_code>class Word(object):
    def __init__(self, text, lemma, pos, head, dep, children):
        self.token = token
        self.text = text
        self.lemma = lemma
        self.pos = pos
        self.head = head
        self.dep = dep
        self.children = children
    
    def __str__(self):
        return self.text
    
    def __repr__(self):
        return self.__str__()
my_sentences = []
for s in sentences:
    doc = nlp(s)
    
    words = [Word(token.text, token.lemma_, token.pos_, token.head, token.dep_, list(token.children)) for token in doc]
    
    my_sentences.append(words)<jupyter_output><empty_output><jupyter_text>42. Root Words<jupyter_code>docs = [nlp(s) for s in sentences]

for doc in docs:
    print(doc)
    displacy.render(doc, style='dep')
    
    for token in doc:
        if token.dep_ == 'ROOT':
            print(token, token.dep_)
    break<jupyter_output>In computer science, artificial intelligence (AI), sometimes called machine intelligence, is intelligence demonstrated by machines, in contrast to the natural intelligence displayed by humans and animals<jupyter_text>43. Verbs and Dependents<jupyter_code>for doc in docs:
    for token in doc:
        if token.pos_ == "VERB":
            print(token, "\n")
            for child in token.children:
                # print(list(token.children))
                if child.pos_ == "NOUN":
                    print("  ", child)
    
    break<jupyter_output>called 

   intelligence
   intelligence
is 

   intelligence
demonstrated 

displayed<jupyter_text>45. Subject, Verb, Object<jupyter_code>doc = nlp("<NAME> invented the perceptron")
displacy.render(doc, style='dep')
class Triplet(object):
    def __init__(self, token):
        self.token = token
        self.subject = None
        self.object = None
        
        for child in self.token.children:
            if child.dep_ == "nsubj":
                self.subject = child
            elif child.dep_ == "dobj":
                self.object = child
        
        self.subject_chunk = None
        self.object_chunk = None
        return
    
    def __str__(self):
        subject = str(self.subject) if self.subject_chunk is None else str(self.subject_chunk)
        predicate = str(self.token)
        object_ = str(self.object) if self.object_chunk is None else str(self.object_chunk)
        return "(subject, predicate, object) = (%s, %s, %s)" % (subject, predicate, object_)
    
    def __repr__(self):
        return self.__str__()
     
for token in doc:
    if token.pos_ == "VERB":
        triplet = Triplet(token)
        print(triplet)<jupyter_output>(subject, predicate, object) = (Rosenblatt, invented, perceptron)<jupyter_text>46 Expanding Subjects and Objects<jupyter_code>doc = nlp("<NAME> invented the perceptron")
for token in doc:
    if token.pos_ == "VERB":
        triplet = Triplet(token)
        print(triplet)
for chunk in doc.noun_chunks:
    for token in chunk:
        # print(token, triplet.subject)
        if token == triplet.subject:
            triplet.subject_chunk = chunk

print(triplet)<jupyter_output>(subject, predicate, object) = (<NAME>, invented, perceptron)<jupyter_text>47. Triple from a Passive Voice Sentence<jupyter_code>doc = nlp("Artificial intelligence was founded as an academic discipline in 1955")
for token in doc:
    print(token, token.pos_)
displacy.render(doc, style='dep')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10575931654146413, 'fraction_numerical': 0.0036824274561791134, 'mean_word_length': 3.2464040025015635, 'pattern_counts': {'":': 6, '<': 37, '<?xml version=': 0, '>': 39, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '402147', 'license': 'MIT', 'n_tokens_mistral': 2141, 'n_tokens_neox': 2052, 'n_words': 642, 'path': 'NLP/DependencyParsing/DependencyParsing.ipynb', 'repo_name': 'SMU-HCI-Lab/tutorials', 'size': '6819'}