<jupyter_start><jupyter_text>Introduction to Data Science – Text Munging Exercises*COMP 5360 / MATH 4100, University of Utah, http://datasciencecourse.net/*   NLP Exercise 1.1: Frequent WordsFind the most frequently used words in Moby Dick which are not stopwords and not punctuation. Hint: [`str.isalpha()`](https://docs.python.org/3/library/stdtypes.htmlstr.isalpha) could be useful here.<jupyter_code>import nltk
from nltk.corpus import stopwords
stopwords = nltk.corpus.stopwords.words('english')
from nltk.book import *
frequency_dist = FreqDist(text1)
print(frequency_dist)

most_common = frequency_dist.most_common(500)

filtered_words = [word_tuple for word_tuple in most_common if word_tuple[0].lower() not in stopwords]
filtered_words = [word_tuple for word_tuple in filtered_words if word_tuple[0].isalpha()]
filtered_words[0:50]<jupyter_output><FreqDist with 19317 samples and 260819 outcomes><jupyter_text>Exercise 2.1You're an evil Spammer who's observed that many people try to obfuscate their e-mail using this notation: "`alex at utah dot edu`". Below are three examples of such e-mails text. Try to extract "alex at utah dot edu", etc. Start with the first string. Then extend your regular expression to work on all of them at the same time. Note that the second and third are slightly harder to do!<jupyter_code>import re
html_smart = "You can reach me: alex at utah dot edu"
html_smart2 = "You can reach me: alex dot lex at utah dot edu"
html_smart3 = "You can reach me: alex dot lex at sci dot utah dot edu"
def testRegex(regex):
    for html in (html_smart, html_smart2, html_smart3):
        print(re.search(regex, html).group())
# TODO write your regex here
mail_regex = "\w+\sat\s\w+\sdot\s\w+"
testRegex(mail_regex)
better_regex = "((\w+\s)+(\sdot)*)+at\s\w+\sdot\s\w+"
testRegex(better_regex)
best_regex = "((\w+\s)+(\sdot)*)+at(\s\w+\sdot)+\s\w+"
testRegex(best_regex)<jupyter_output>alex at utah dot edu
alex dot lex at utah dot edu
alex dot lex at sci dot utah dot edu<jupyter_text>Exercise 2.2: Find AdverbsWrite a regular expression that finds all adverbs in a sentence. Adverbs are characterized by ending in "ly".<jupyter_code>text = "He was carefully disguised but captured quickly by police."
re.findall(r"\w+ly", text)<jupyter_output><empty_output><jupyter_text>Exercise 2.3: Phone NumbersExtract the phone numbers that follow a (xxx) xxx-xxxx pattern from the text:<jupyter_code>phone_numbers = "(857) 131-2235, (801) 134-2215, but this one (12) 13044441 shouldnt match. Also, this is common in twelve (12) countries and one (1) state"
re.findall(r"\([0-9]{3}\)\s[0-9]{3}-[0-9]{4}", phone_numbers)<jupyter_output><empty_output><jupyter_text>Exercise 2.4: HTML ContentExtract the content between the `` and `` tags but not the other tags:<jupyter_code>html_tags = "This is <b>important</b> and <u>very</u><i>timely</i>"
re.findall(r"<[bi]>(.*?)<\/[bi]>", html_tags)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11834924965893588, 'fraction_numerical': 0.02899045020463847, 'mean_word_length': 5.576233183856503, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 28, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '112271', 'license': 'MIT', 'n_tokens_mistral': 1074, 'n_tokens_neox': 994, 'n_words': 325, 'path': '17-nlp_regex/lecture-17-exercise-solutions.ipynb', 'repo_name': 'uwsd/2018-datascience-lectures', 'size': '2932'}