<jupyter_start><jupyter_text>For Mac - Download NLTK window pop up - click below to expand and run<jupyter_code>import ssl

try:
    _create_unverified_https_context = ssl._create_unverified_context
except AttributeError:
    pass
else:
    ssl._create_default_https_context = _create_unverified_https_context

nltk.download()
def lexical_diversity(text):
    return len(set(text)) / len(text)

def scale(array, scale_type='standard'):
    if scale_type == 'standard': #mean zero, std = 1
        return (array - np.mean(array))/np.std(array)
    if scale_type == 'min_max':
        return (array - np.min(array))/(np.max(array) - np.min(array))

def clean_text(text):
    text = text.lower()
    text_sub = re.sub('[^A-Za-z0-9]+', ' ', text)
    display(md(f'{text_sub[:100]}'))
    tokens = word_tokenize(text_sub)
    display(md(f'*Raw tokens found: {len(tokens):,.0f}*'))
    display(md(f'*Raw Vocab Size: {len(set(tokens)):,.0f}*'))
    nltk_text = nltk.Text(tokens)
    stop_words = set(stopwords.words('english'))
    filtered_text = ' '.join(
        [w for w in nltk_text if not w in stop_words])
    tokens_filtered = word_tokenize(filtered_text)
    nltk_text_filtered = nltk.Text(tokens_filtered)
    return nltk_text_filtered
gutenberg.fileids()
gutenberg.raw('shakespeare-hamlet.txt')
gutenberg.raw('shakespeare-macbeth.txt')<jupyter_output><empty_output><jupyter_text>***1: Run one of the part-of-speech (POS) taggers available in Python.*** ***(a) Find the longest sentence you can, longer than 10 words, that the POS tagger tags correctly. Show the input and output.****We'll train our custom punkt sentence tokenizer on Shakespeare's "Hamlet" and test it out on "MacBeth" and then use the standard NLTK pos tagger to get the parts of speech*<jupyter_code>train_text = gutenberg.raw('shakespeare-hamlet.txt')
sample_text = gutenberg.raw('shakespeare-macbeth.txt')
custom_sent_tokenizer = PunktSentenceTokenizer(train_text)
tokenized = custom_sent_tokenizer.tokenize(sample_text)
tokenized_filter = [
    x for x in tokenized if len(nltk.word_tokenize(x)) == 11 and x[0] != '['
    #and len(nltk.word_tokenize(x)) > 10
]
sentence_1 = tokenized_filter[:1]
try:
    for i in sentence_1:
        words = nltk.word_tokenize(i)
        tagged = nltk.pos_tag(words)
        print(f'sentence {i}')
        print(f'length {len(words)}')
except Exception as e:
    print(str(e))


#process_content()<jupyter_output>sentence Dismay'd not this our Captaines, Macbeth and
Banquoh?
length 11<jupyter_text>*Works very well. Identifies all proper nouns, the future tense of "'d" in "Dismay'd" which is cool. Gets the plural correct on Captaines.*  OS tag list:- CC	coordinating conjunction- CD	cardinal digit- DT	determiner (the, a, an)- EX	existential there (like: "there is" ... think of it like "there exists")- FW	foreign word- IN	preposition/subordinating conjunction- JJ	adjective	'big'- JJR	adjective, comparative	'bigger'- JJS	adjective, superlative	'biggest'- LS	list marker	1)- MD	modal	could, will- NN	noun, singular 'desk'- NNS	noun plural	'desks'- NNP	proper noun, singular	'Harrison'- NNPS	proper noun, plural	'Americans'- PDT	predeterminer	'all the kids'- POS	possessive ending	parent\'s- PRP	personal pronoun	I, he, she- PRP	possessive pronoun	my, his, hers- RB	adverb	very, silently,- RBR	adverb, comparative	better- RBS	adverb, superlative	best- RP	particle	give up- TO	to	go 'to' the store.- UH	interjection	errrrrrrrm- VB	verb, base form	take- VBD	verb, past tense	took- VBG	verb, gerund/present participle	taking- VBN	verb, past participle	taken- VBP	verb, sing. present, non-3d	take- VBZ	verb, 3rd person sing. present	takes- WDT	wh-determiner	which- WP	wh-pronoun	who, what- WP	possessive wh-pronoun	whose- WRB	wh-abverb	where, when<jupyter_code>tokenized_filter = [
    x for x in tokenized if len(nltk.word_tokenize(x)) < 10 and x[0] != '['
    #and len(nltk.word_tokenize(x)) > 10
]
sentence_2 = tokenized_filter[4:5]
try:
    for i in sentence_2:
        words = nltk.word_tokenize(i)
        tagged = nltk.pos_tag(words)
        print(f'sentence {i}')
        print(f'length {len(words)}')
        print(tagged)
except Exception as e:
    print(str(e))<jupyter_output>sentence When shall we three meet againe?
length 7
[('When', 'WRB'), ('shall', 'MD'), ('we', 'PRP'), ('three', 'CD'), ('meet', 'FW'), ('againe', 'NN'), ('?', '.')]<jupyter_text>***(b) Find the shortest sentence you can, shorter than 10 words, that the POS tagger fails to tag 100 percent correctly. Show the input and output. Explain your conjecture as to why the tagger might have been less than perfect with this sentence.****This is a very difficult one for the tagger. The old english form of the averb again - "againe" - is incorrectly classified as a NN (noun). Technically, this word isn't part of the English corpus so it's not surprising that the POS tokenizer fails to recognize the word.*  ***2: Run a different POS tagger in Python. Process the same two sentences from question 1.***  *Switching to the standard NLTK Word Tokenizer and a custom Regex tagger*<jupyter_code>patterns = [
    (r'.*ing$', 'VBG'),                # gerunds
    (r'.*ed$', 'VBD'),                 # simple past
    (r'.*es$', 'VBZ'),                 # 3rd singular present
    (r'.*ould$', 'MD'),                # modals
    (r'.*\'s$', 'NN$'),                # possessive nouns
    (r'.*s$', 'NNS'),                  # plural nouns
    (r'^-?[0-9]+(\.[0-9]+)?$', 'CD'),  # cardinal numbers
    (r'.*', 'NN')                      # nouns (default)
]
regexp_tagger = nltk.RegexpTagger(patterns)
try:
    for i in sentence_1:
        words = nltk.word_tokenize(i)
        tagged = regexp_tagger.tag(words)
        print(f'sentence {i}')
        print(f'length {len(words)}')
        print(tagged)
except Exception as e:
    print(str(e))
try:
    for i in sentence_2:
        words = nltk.word_tokenize(i)
        tagged = regexp_tagger.tag(words)
        print(f'sentence {i}')
        print(f'length {len(words)}')
        print(tagged)
except Exception as e:
    print(str(e))<jupyter_output>sentence When shall we three meet againe?
length 7
[('When', 'NN'), ('shall', 'NN'), ('we', 'NN'), ('three', 'NN'), ('meet', 'NN'), ('againe', 'NN'), ('?', 'NN')]<jupyter_text>***(a) Does it produce the same or different output?****Very different, as expected - it's a simplistic POS tagger that relies on a limited number of regex patterns to generate tags and the default is NN. The fact that it doesn't work on old english isn't at all surprising.****(b) Explain any differences as best you can.****The best approach in this case would be a combination of a regex tagger and perhaps the standard POS tagger available in NLTK. Adding specific regex patterns for names and other idiosyncratic items such as "againe" would create more accurate tagger for the old-english dialect familiar to those from Shaksepearen times.* ***3: In a news article from this week’s news, find a random sentence of at least 10 words.******(a) Looking at the Penn tag set, manually POS tag the sentence yourself.***- 'The': DT -  'head': NN-  'of': IN-  'the': DT-  'Oregon': NNP-  'Health': NNP-  'Authority': NNP-  'on': IN-  'Tuesday': NNP-  'said': VBD-  'the': DT-  'state': NN-  'is': VBZ-  'seeing':VBG-  'a': DT-  'significant': JJ-  'spike': NN-  'in': IN-  'new': JJ-  'coronavirus': NNP-  'cases':NNS-  'since':IN-  'autumn':NN-  'began':VBD- '. ***(b) Now run the same sentences through both taggers that you implemented for questions 1 and 2. Did either of the taggers produce the same results as you had created manually?***<jupyter_code>tokenized = custom_sent_tokenizer.tokenize(
    '''The head of the Oregon Health Authority on Tuesday said the state is seeing a significant spike in new coronavirus cases since autumn began.'''
)
#tokenized
tagged = nltk.pos_tag(tokenized)
tagged
tokenized = custom_sent_tokenizer.tokenize(
    '''The head of the Oregon Health Authority on Tuesday said the state is seeing a significant spike in new coronavirus cases since autumn began.'''
)
try:
    for i in tokenized:
        words = nltk.word_tokenize(i)
        tagged = regexp_tagger.tag(words)
        print(f'sentence {i}')
        print(f'length {len(words)}')
        print(tagged)
except Exception as e:
    print(str(e))<jupyter_output>sentence The head of the Oregon Health Authority on Tuesday said the state is seeing a significant spike in new coronavirus cases since autumn began.
length 25
[('The', 'NN'), ('head', 'NN'), ('of', 'NN'), ('the', 'NN'), ('Oregon', 'NN'), ('Health', 'NN'), ('Authority', 'NN'), ('on', 'NN'), ('Tuesday', 'NN'), ('said', 'NN'), ('the', 'NN'), ('state', 'NN'), ('is', 'NNS'), ('seeing', 'VBG'), ('a', 'NN'), ('significant', 'NN'), ('spike', 'NN'), ('in', 'NN'), ('new', 'NN'), ('coronavirus', 'NNS'), ('cases', 'VBZ'), ('since', 'NN'), ('autumn', 'NN'), ('began', 'NN'), ('.', 'NN')]
{'fraction_non_alphanumeric': 0.14316239316239315, 'fraction_numerical': 0.006297795771479982, 'mean_word_length': 3.943301834352418, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 19, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '654143', 'license': 'MIT', 'n_tokens_mistral': 3114, 'n_tokens_neox': 2943, 'n_words': 1009, 'path': 'HW1/HW4.ipynb', 'repo_name': 'fastest-parrot/NLP', 'size': '8892'}