<jupyter_start><jupyter_text>Objectives - Identify noun phrase chunks using POS tags- Extract information from noun phrases in the Penn Treebank  Dowloading Lexicons<jupyter_code>import nltk
nltk.download("punkt")
nltk.download("treebank")
nltk.download("averaged_perceptron_tagger")
nltk.download("wordnet")<jupyter_output>[nltk_data] Downloading package punkt to
[nltk_data]     /Users/varadrajrameshpoojary/nltk_data...
[nltk_data]   Package punkt is already up-to-date!
[nltk_data] Downloading package treebank to
[nltk_data]     /Users/varadrajrameshpoojary/nltk_data...
[nltk_data]   Package treebank is already up-to-date!
[nltk_data] Downloading package averaged_perceptron_tagger to
[nltk_data]     /Users/varadrajrameshpoojary/nltk_data...
[nltk_data]   Package averaged_perceptron_tagger is already up-to-
[nltk_data]       date!
[nltk_data] Downloading package wordnet to
[nltk_data]     /Users/varadrajrameshpoojary/nltk_data...
[nltk_data]   Package wordnet is already up-to-date!<jupyter_text>Code to access relevant modules (you can add to this as needed):<jupyter_code>from nltk.corpus import treebank
from nltk import word_tokenize, pos_tag, RegexpParser
from nltk.tree import Tree
from nltk.chunk.util import ChunkScore
from nltk.stem import WordNetLemmatizer<jupyter_output><empty_output><jupyter_text>Simple NP chunking We start by building a basic NP chunker. A simple approach to the task of NP chunking is to assume that a sequence of words is an NP if * it contains only determiners, nouns, pronouns, and adjectives,* and it contains at least one noun or pronoun. The first letters of relevant POS tags are provided for you in the sets `NP_POS` and `NP_HEAD_POS`. We write a function which takes a raw sentence (a string) and 1. tokenizes and POS tags it using NLTK,1. finds all contiguous sequences of words that fit the above description, and returns them.<jupyter_code>NP_POS = {"DT", "NN", "JJ", "PR"}  # these are the first two letters of the POS that you should consider potential parts of NP chunks 
NP_HEAD_POS = {"NN", "PR"}  # each chunk must have at least one of these

def get_chunks(sentence):
    '''Extracts noun phrases from a sentence corresponding to the part-of-speech tags in optional_POS,
    requiring at least one of the POS tags in required_POS. Returns the chunks as a list of strings'''
    # your code here
    chunks=[]
    list_1=[]
    tags= set()
    tagged_sent = pos_tag(word_tokenize(sentence))
    for word,tag in tagged_sent:
        if tag[:2] in NP_POS:
            list_1.append(word)
            tags.add(tag[:2])
        else:
            if len(tags & NP_HEAD_POS)>0:
                chunks.append(" ".join(list_1))
            list_1= []
            tags= set()
    if len(tags & NP_HEAD_POS)>0:
        chunks.append(" ".join(list_1))
    return chunks<jupyter_output><empty_output><jupyter_text>Here are a few examples which show you the input format for `get_chunks` and the intended output format.<jupyter_code>assert(sorted(get_chunks("the quick brown fox jumped over the lazy dog"))) == sorted(["the quick brown fox", "the lazy dog"])
assert(get_chunks("life is good")) == ["life"]
assert(get_chunks("life is good and chickens are tasty")) == ["life","chickens"]
print("Success!")<jupyter_output>Success!<jupyter_text>Regex chunking Created three different NLTK regex noun chunkers using the `RegexpParser` class:1. `simple_chunk` which exactly duplicates the logic from above.<jupyter_code>tagged_sent = pos_tag(word_tokenize("I gave John my old Globe and Mail"))
simple_chunk = RegexpParser("NP: {(<DT.*|NN.*|JJ.*|PR.*>*)(<NN.*|PR.*>)(<DT.*|NN.*|JJ.*|PR.*>*)}")
print(simple_chunk.parse(tagged_sent))<jupyter_output>(S
  (NP I/PRP)
  gave/VBD
  (NP John/NNP my/PRP$ old/JJ Globe/NNP)
  and/CC
  (NP Mail/NNP))<jupyter_text>2. `ordered_chunk` which captures the standard English NP word order, defined by the following properties: * The syntactic head of an NP is either a personal pronoun, common noun or proper noun. Every NP has to contain at least one of these. Note that there can be more. * If the head is a noun, it can be preceded by a determiner (also called an article) as in _the dog_ or a possessive pronoun as in _my dogs_.  * If the head is a noun, it can be preceded by one or more adjectives as in _beautiful weather_. * If a determiner or possessive pronoun occurs, it has to be the first token of the NP.   * If the syntactic head is a noun, it can be preceded by an adjective as in _the grey dog_ and _grey dogs_.<jupyter_code>tagged_sent = pos_tag(word_tokenize("I gave John my old Globe and Mail"))
ordered_chunk = RegexpParser("NP: {(<DT.*|PRP\$>?<JJ.*>*<NN.*>+)|(<PRP>+)}")
print(ordered_chunk.parse(tagged_sent))<jupyter_output>(S
  (NP I/PRP)
  gave/VBD
  (NP John/NNP)
  (NP my/PRP$ old/JJ Globe/NNP)
  and/CC
  (NP Mail/NNP))<jupyter_text>3. `conj_chunk` which allows for coordination of two NPs matching `ordered_chunk` using a coordinate conjunction `CC`. Note that often there is only one determiner in a coordinated NP as in "the Globe and Mail", however, "the Globe and the Mail" is also grammatical.<jupyter_code>tagged_sent = pos_tag(word_tokenize("I gave John my old Globe and MAIL and mail"))
conj_chunk = RegexpParser("NP: {((<DT.*|PRP\$>?<JJ.*>*<NN.*>+)|(<PRP>+))(<CC>((<DT.*|PRP\$>?<JJ.*>*<NN.*>+)|(<PRP>+)))*}")
print(conj_chunk.parse(tagged_sent))
sent = "I gave John my old Globe and Mail"
assert (str(simple_chunk.parse(pos_tag(word_tokenize(sent)))) == str(Tree.fromstring("(S (NP I/PRP) gave/VBD (NP John/NNP my/PRP$ old/JJ Globe/NNP) and/CC (NP Mail/NNP))")))
assert (str(ordered_chunk.parse(pos_tag(word_tokenize(sent)))) == str(Tree.fromstring("(S (NP I/PRP) gave/VBD (NP John/NNP) (NP my/PRP$ old/JJ Globe/NNP) and/CC (NP Mail/NNP))")))
assert (str(conj_chunk.parse(pos_tag(word_tokenize(sent)))) == str(Tree.fromstring("(S (NP I/PRP) gave/VBD (NP John/NNP) (NP my/PRP$ old/JJ Globe/NNP and/CC Mail/NNP))")))
print("Success!")<jupyter_output>Success!<jupyter_text>Chunking evaluation and improvementEvaluation our regular expression chunkers by comparing their output to gold standard chunks extrated from the Penn Treebank.First, we will create a new test set for our chunkers by pulling out noun phrases from the Penn Treebank. We start by creating a function `convert_to_chunk` which converts standard syntactic trees into shallow chunk trees, where all phrases except `NP` have been flattened.    Your `convert_to_chunk` function should take a list of syntax trees as input and return a list of chunk trees as output. Here is an example of a syntax tree and the corresponding chunk tree:<jupyter_code>def is_wanted_NP(tree):
    '''returns False if the NLTK tree of a NP has either other NPs or traces ("*") within it'''
    if tree.label() != "NP":
        return False
    
    subtrees = list(tree.subtrees())[1:]
    if any([subtree.label().startswith("NP") for subtree in subtrees]):
        return False
    # your code here

    if "*" in tree.leaves():
        return False
    return True

def convert_to_chunk_(tree,chunks):
    '''Recursively finds any shallow NPs in the tree, converting the parse into the NLTK chunk format.
       The list of chunks is returned'''
    # your code here

    for inner_tree in tree:
        if is_wanted_NP(inner_tree):
            chunks.append(Tree(inner_tree.label(), inner_tree.pos()))
        elif inner_tree.height()==2:
            chunks.append(inner_tree.pos()[0])
        else:
            chunks = (convert_to_chunk_(inner_tree, chunks))
    return chunks

tree = Tree.fromstring("(S (NP (DT the) (NN dog)) (VP (VBD saw) (NP (DT the) (NN cat))))")
assert(convert_to_chunk_(tree,[]) == [Tree('NP', [('the', 'DT'), ('dog', 'NN')]), ('saw', 'VBD'), Tree('NP', [('the', 'DT'),('cat','NN')])])

def convert_to_chunk(tree):
    return Tree("S",convert_to_chunk_(tree,[]))


treebank_test = []

for parsed_sent in treebank.parsed_sents():
    treebank_test.append(convert_to_chunk(parsed_sent))<jupyter_output><empty_output><jupyter_text>Now, we evaluate the three regex chunkers from Exercise 2 using the built-in NLTK chunk evaluation system.<jupyter_code>print("simple chunk")
print(simple_chunk.evaluate(treebank_test))
print("ordered chunk")
print(ordered_chunk.evaluate(treebank_test))
print("conj chunk")
print(conj_chunk.evaluate(treebank_test))<jupyter_output>simple chunk
ChunkParse score:
    IOB Accuracy:  75.4%%
    Precision:     50.3%%
    Recall:        65.4%%
    F-Measure:     56.9%%
ordered chunk
ChunkParse score:
    IOB Accuracy:  75.5%%
    Precision:     50.6%%
    Recall:        66.8%%
    F-Measure:     57.6%%
conj chunk
ChunkParse score:
    IOB Accuracy:  75.3%%
    Precision:     50.8%%
    Recall:        64.4%%
    F-Measure:     56.8%%<jupyter_text>looking for errors<jupyter_code>dev_set = treebank.tagged_sents()[:50]
test_set = treebank.tagged_sents()[50:]
for tagged, gold_tree in zip(dev_set,treebank_test):
    sys_tree = ordered_chunk.parse(tagged)
    print("SYS:",sys_tree)
    print("GOLD:",gold_tree)<jupyter_output>SYS: (S
  (NP Pierre/NNP Vinken/NNP)
  ,/,
  61/CD
  (NP years/NNS)
  old/JJ
  ,/,
  will/MD
  join/VB
  (NP the/DT board/NN)
  as/IN
  (NP a/DT nonexecutive/JJ director/NN Nov./NNP)
  29/CD
  ./.)
GOLD: (S
  (NP Pierre/NNP Vinken/NNP)
  ,/,
  (NP 61/CD years/NNS)
  old/JJ
  ,/,
  will/MD
  join/VB
  (NP the/DT board/NN)
  as/IN
  (NP a/DT nonexecutive/JJ director/NN)
  Nov./NNP
  29/CD
  ./.)
SYS: (S
  (NP Mr./NNP Vinken/NNP)
  is/VBZ
  (NP chairman/NN)
  of/IN
  (NP Elsevier/NNP N.V./NNP)
  ,/,
  (NP the/DT Dutch/NNP)
  publishing/VBG
  (NP group/NN)
  ./.)
GOLD: (S
  Mr./NNP
  Vinken/NNP
  is/VBZ
  (NP chairman/NN)
  of/IN
  (NP Elsevier/NNP N.V./NNP)
  ,/,
  (NP the/DT Dutch/NNP publishing/VBG group/NN)
  ./.)
SYS: (S
  (NP Rudolph/NNP Agnew/NNP)
  ,/,
  55/CD
  (NP years/NNS)
  old/JJ
  and/CC
  (NP former/JJ chairman/NN)
  of/IN
  (NP Consolidated/NNP Gold/NNP Fields/NNP PLC/NNP)
  ,/,
  was/VBD
  named/VBN
  *-1/-NONE-
  (NP a/DT nonexecutive/JJ director/NN)
  of/IN
  (NP this/D[...]<jupyter_text>The cd or the cardinal number tag can appear before noun and that must be included in our Parser.<jupyter_code>from nltk.corpus import brown

chunk = RegexpParser("NP: {((<DT|PRP\$>?<JJ.*>*<CD.*|NN.*>+)|(<PRP>))(<CC>((<DT|PRP\$>?<JJ.*>*<CD.*|NN.*>+)|(<PRP>)))*}")
test_set = []

for parsed_sent in treebank.parsed_sents()[50:]:
    test_set.append(convert_to_chunk(parsed_sent))

print(chunk.evaluate(test_set))<jupyter_output>ChunkParse score:
    IOB Accuracy:  78.1%%
    Precision:     52.9%%
    Recall:        71.2%%
    F-Measure:     60.7%%<jupyter_text>Identifying predicates and objectsWe now build a function which extracts predicate-object pairs from syntax trees. For example, for the sentence _I bought the toys_ , your function should identify that the predicate of the sentence is _bought_ and its object is _toys_ , the function should then return the pair `("buy", "toy")`. First, we write a recursive function `get_head` which takes two arguments: `phrase` and `phrase_type` as input. The `phrase` argument is an NLTK tree representing either an NP or a VP, and `phrase_type` is either `"N"` or `"V"` for NPs and VPs, respectively. The function should return the **lemmatized** syntactic head of `phrase`. For example, given the following NLTK syntax tree as input```(NP  (DT the)  (JJ grey)  (NN dogs) )```The function should return `dog`. We assume that the head is either the right-most token with the appropriate POS `V.*` or `N.*`, or the syntactic head of the right-most child phrase having type `NP.*` or `VP.*` depending on `phrase_type`. This means that we may need to call `get_head` recursively. For example, for ```(NP  (DT the)  (JJ second)  (NN incentive)  (NN plan))```should return `"plan"` which is the right-most noun. As another example, consider ```(NP  (DT the)  (JJ blue)  (NN bird) (CC and) (NP    (DT the)   (JJ yellow)   (NN butterfly) ))```Here we return "`butterfly`" which is the head of the right-most child NP.<jupyter_code># lemmatizer.lemmatize(word,pos) returns the lemma for word. 
# pos should be 'n' for nouns and 'v' for verbs.
lemmatizer = WordNetLemmatizer()

head_list = []
def get_head(phrase, phrase_type):
    '''returns the lemmatized lexical head assuming the provided phrase_type ("N","V",etc.)'''
    head = None
    # your code here
    get_head_(phrase, head, phrase_type)
    if len(head_list) == 0:
        return head

    return lemmatizer.lemmatize(head_list[-1],phrase_type.lower())

def get_head_(phrase, head, phrase_type):
    '''returns the lemmatized lexical head assuming the provided phrase_type ("N","V",etc.)'''
    if phrase.label()[:1] == phrase_type and phrase.height() == 2:
        head = phrase.pos()[0][0]
        head_list.append(phrase.pos()[0][0])
    elif phrase.label()[:1] == phrase_type and phrase.height() > 2:
        for sub in phrase:
            get_head_(sub, head, phrase_type)
assert (get_head(Tree.fromstring("(NP (DT the) (JJ second) (NN incentive) (NN plan))"), "N") == "plan")
assert (get_head(Tree.fromstring("(NP-SUBJ (NP (DT the) (NNS policies)) (PP (IN of) (NP (NN tomorrow))))"), "N") == "policy")
assert (get_head(Tree.fromstring("(VP (VBN offered) (NP (NNS advertisers)))"),"V") == "offer")
print("Success!")<jupyter_output>Success!
{'fraction_non_alphanumeric': 0.13073859794124276, 'fraction_numerical': 0.006987752648583665, 'mean_word_length': 4.131071703932151, 'pattern_counts': {'":': 1, '<': 65, '<?xml version=': 0, '>': 68, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '8578', 'license': 'MIT', 'n_tokens_mistral': 4784, 'n_tokens_neox': 4573, 'n_words': 1442, 'path': 'Chunking.ipynb', 'repo_name': 'VaradrajPoojari/NLP_Chunking', 'size': '13309'}