<jupyter_start><jupyter_text>Week 9 Discussion Infographic* [History of Infographics](http://infowetrust.com/scroll/) Links* [Python's Regular Expression HOWTO](https://docs.python.org/3/howto/regex.htmlregex-howto)* [RegExr](https://regexr.com/) -- test regex online  Regular ExpressionsA _regular expression_, or regex, is a language for expressing patterns in strings.Regular expressions are not Python-specific! They are also supported in R, Java, Perl, etc...In Python, regex is supported by the built-in [`re` module][re] as well as the [`regex` package][regex] and some of the Pandas [`.str` methods][pandas-str].Regular expressions are __slow and brittle__. Use them as a last resort! Try to solve problems with [string methods][str] or [an appropriate parser][lxml] instead. There's even a [famous SO post][so-regex-html] about this.[re]: https://docs.python.org/3/library/re.html[regex]: https://pypi.python.org/pypi/regex/[pandas-str]: https://pandas.pydata.org/pandas-docs/stable/text.html[str]: https://docs.python.org/3/library/stdtypes.htmlstring-methods[lxml]: http://lxml.de/[so-regex-html]: https://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags/17324541732454  MetacharactersA regular expression can describe a complicated pattern in just a few characters because some non-alphabet characters have special meanings.These special characters are called metacharacters.Metacharacter | Meaning------------- | -------`.`           | any 1 character (a _wildcard_)`[ ]`         | any 1 character listed (a _set_)`[^ ]`        | any 1 character except those listed (a _set complement_)`^`           | start of string`$`           | end of stringThe `re.findall()` function returns a list of each part of the string that matched the pattern. Experimenting with `re.findall()` is a good way to learn regular expressions.<jupyter_code>import re

# re.findall(PATTERN, STRING)
re.findall(".", "xyz")<jupyter_output><empty_output><jupyter_text>Square brackets `[ ]` mark a set. A set matches any 1 of the symbols inside.<jupyter_code>re.findall("[abc]", "caxyb")<jupyter_output><empty_output><jupyter_text>Putting `^` at the beginning of a set matches the _set complement_ (any character except the ones in the set):<jupyter_code>re.findall("[^12]", "31214")<jupyter_output><empty_output><jupyter_text>Sets can contain ranges of letters or numbers:<jupyter_code>re.findall("[h-z0-9]", "0abcdefghi")<jupyter_output><empty_output><jupyter_text>To include a literal dash `-` in a set, don't put it between letters:<jupyter_code>re.findall("[a-z]", "-")
re.findall("[a-z-]", "-")<jupyter_output><empty_output><jupyter_text>Use `^` and `$` to mark the start and end of a the string, respectively:<jupyter_code>re.findall("^a", "aba")<jupyter_output><empty_output><jupyter_text>Escape SequencesThe backslash `\` has a special meaning in Python strings: it marks the beginning of an escape sequence.Escape sequences are necessary to write characters that don't appear on the keyboard. For instance, a newline is `\n`:<jupyter_code>print("Hello Goodbye")
print("Hello\nGoodbye")<jupyter_output>Hello
Goodbye<jupyter_text>Because backslash marks the beginning of an escape sequence, if we want to write a literal backslash, we have to write `\\`:<jupyter_code>print("\\")<jupyter_output>\<jupyter_text>This is inconvenient since regex uses backslash to make a metacharacter into a literal character.So if we wanted to match a literal `.`, we'd have to write:<jupyter_code>pattern = "\\."
print(pattern)
re.findall(pattern, "x.y")<jupyter_output>\.<jupyter_text>Even worse, to match a literal backslash, we'd have to write `"\\\\"`!Python provides _raw strings_ to fix this problem. In a raw string, backslash has no special meaning for Python (but it still has a special meaning for regex.To make a raw string, put the letter `r` before the quotes:<jupyter_code>pattern = r"\."
print(pattern)
re.findall(pattern, "x.y")<jupyter_output>\.<jupyter_text>QuantifiersYou can describe repeated characters with quantifiers:Metacharacter | Meaning------------- | -------`*`           | repeat previous character 0 or more times`+`           | repeat previous character 1 or more times`?`           | repeat previous character 0 or 1 times<jupyter_code>re.findall("a", "aaa")
re.findall("a+", "aaa")
re.findall("xy?z", "xz")
re.findall("xy?z", "xyz")<jupyter_output><empty_output><jupyter_text>GroupsYou can make a group with parentheses `( )`.Groups can be repeated just like single characters.The `re.findall()` function has special handling for groups, so instead we use `re.search()` here:<jupyter_code>re.search("x(abc)+y", "xabcabcabcy")<jupyter_output><empty_output><jupyter_text>Natural Language Processing (NLP)Basic NLP workflow:1. __Tokenize__ -- split text into words2. __Remove Noise__ (optional) -- remove stop words, convert words to lemmas, correct spelling, ...3. __Vectorize__ -- compute term frequencies, tf-idfs, or some other statistic4. __Analyze__This workflow is the same regardless of what language you're using.  In PythonPython has lots of packages for natural language processing:* [TextBlob][] is the most Pythonic NLP package. Good for learning and prototyping.* [NLTK][] is the most comprehensive NLP package. Good for _learning_ NLP, but runs slowly.* [spaCy][] is the fastest NLP package.* [Stanford's Core NLP][CoreNLP] is the cutting edge of NLP research. It's written in Java, but several Python packages provide an interface. The [pynlp][] and [stanford-corenlp][] packages look promising.* [SyntaxNet][] is Google's NLP package. Good if you're already using TensorFlow.* [Pattern][] combines web scraping and NLP.* [gensim][] is good for topic modelling (a specific NLP method).Nick recommends __TextBlob__ with __NLTK__ for beginners, and __spaCy__ or __Core NLP__ for serious projects.[TextBlob]: http://textblob.readthedocs.io/en/dev/[NLTK]: https://www.nltk.org/[spaCy]: https://spacy.io/[CoreNLP]: https://stanfordnlp.github.io/CoreNLP/[pynlp]: https://github.com/sina-al/pynlp[stanford-corenlp]: https://github.com/Lynten/stanford-corenlp[SyntaxNet]: https://github.com/tensorflow/models/tree/master/research/syntaxnet[Pattern]: https://www.clips.uantwerpen.be/pattern[gensim]: https://radimrehurek.com/gensim/<jupyter_code># Set up NLTK packages used by TextBlob.
import nltk

nltk.download("punkt")
nltk.download("averaged_perceptron_tagger")
nltk.download("brown")
nltk.download("wordnet")
nltk.download("stopwords")
from textblob import TextBlob

text = "The quick brown foxes jumped lightly over the lazy dog."

blob = TextBlob(text)
blob.words<jupyter_output><empty_output><jupyter_text>Removing NoiseSometimes it's useful to remove words that appear frequently, like "the". These are called _stopwords_.Removing stopwords is something you need to think about carefully. Does it make sense for the problem you're trying to solve?Also, the NLTK stopwords are not necessarily the right stopwords for your problem<jupyter_code>from nltk.corpus import stopwords

stopwords = stopwords.words("english")

new_text = " ".join(w for w in blob.words if w.lower() not in stopwords)
blob = TextBlob(new_text)
blob<jupyter_output><empty_output><jupyter_text>In addition to removing stopwords, you might want to _lemmatize_ the words.Lemmatization converts words to a single inflection. After lemmatizing, plurals and verb tenses are eliminated.A related procedure, _stemming_, also converts words to a single inflection, but uses programmatic rules rather than a dictionary. Stemming is simpler but less accurate.As an example, if we lemmatize all of the words, "foxes" becomes "fox":<jupyter_code>blob.words.lemmatize()<jupyter_output><empty_output><jupyter_text>Why wasn't "jumped" converted to "jump"?By default, NLTK's lemmatizer only looks for nouns. We need to tell it which words are verbs.<jupyter_code>blob.words[3]
blob.words[3].lemmatize("v")<jupyter_output><empty_output><jupyter_text>Fortunately, TextBlob can detect parts of speech (POS):<jupyter_code>blob.pos_tags<jupyter_output><empty_output><jupyter_text>These are [Brown POS tags][brown], but the lemmatizer uses WordNet POS tags. Use this function to convert:[brown]: https://en.wikipedia.org/wiki/Brown_CorpusPart-of-speech_tags_used<jupyter_code>from nltk.corpus import wordnet

def wordnet_pos(tag):
    """Map a Brown POS tag to a WordNet POS tag."""
    
    table = {"N": wordnet.NOUN, "V": wordnet.VERB, "R": wordnet.ADV, "J": wordnet.ADJ}
    
    # Default to a noun.
    return table.get(tag[0], wordnet.NOUN)<jupyter_output><empty_output><jupyter_text>Now we can get the tags and use them to lemmatize:<jupyter_code>tags = [wordnet_pos(x[1]) for x in blob.pos_tags]
tags
new_text = " ".join(x.lemmatize(t) for x, t in zip(blob.words, tags))
blob = TextBlob(new_text)
blob<jupyter_output><empty_output><jupyter_text>Term FrequenciesHow can we quantify a text document so we can do statistics?In a _bag of words_ model, we assume that the order of the words in a document doesn't matter.Then we can quantify the document by counting how many times each word appears. This is called a _word vector_.Surprisingly, the bag of words model works well in most cases.<jupyter_code>blob.word_counts<jupyter_output><empty_output><jupyter_text>We can check how similar two documents are by computing the dot product of their word vectors.The more words they have in common, the larger the dot product will be.<jupyter_code>docs = ["the quick brown fox", "the fast brown fox"]

def term_frequencies(doc):
    return {w: x / doc.words for w, x in doc.word_counts}
for doc in docs:
    doc = TextBlob(doc)
    {w: tf(doc) for w in doc.words}
import pandas as pd

docs = [
    "The quick brown foxes jumped lightly over the lazy dog.",
    "Brown ducks lay eggs once a year and are omnivorous.",
    "The fleet brown foxes jumped over the dogs."
]

# Get word counts.
docs_df = pd.DataFrame(TextBlob(d).word_counts for d in docs).T
docs_df = docs_df.fillna(0)
docs_df.head()
# Compute frequencies.
tf = docs_df / docs_df.sum()
tf.head()
sum(tf[0] * tf[1])
sum(tf[0] * tf[2])<jupyter_output><empty_output><jupyter_text>One problem with term frequencies is that some terms have high frequencies simply because they appear frequently in the language. These words can cause a high similarity score for documents that are otherwise different._Term frequency-inverse document frequency_ (tf-idf) statistics solve this problem. There are several different tf-idf statistics.The _smoothed term frequency-inverse document frequency_ (smoothed tf-idf), for a term $t$ and document $d$, is given by$$\operatorname{tf-idf}(t, d) = \operatorname{tf}(t, d) \cdot \log \left( \frac{N}{1 + n_t} \right)$$where $N$ is the total number of documents and $n_t$ is the number of documents that contain $t$. In practice, it's easiest to use a package to compute tf-idf:<jupyter_code>from sklearn.feature_extraction.text import TfidfVectorizer

tf_idf = TfidfVectorizer().fit_transform(docs)

# Use .A to display a sparse matrix.
(tf_idf * tf_idf.T).A<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11384532179556517, 'fraction_numerical': 0.005137912385073012, 'mean_word_length': 5.314741035856573, 'pattern_counts': {'":': 5, '<': 85, '<?xml version=': 0, '>': 85, 'https://': 16, 'lorem ipsum': 0, 'www.': 2, 'xml': 3}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '22', 'dir': 'jupyter-structured-clean-dedup', 'id': '205862', 'license': 'BSD-3-Clause-Clear', 'n_tokens_mistral': 3610, 'n_tokens_neox': 3443, 'n_words': 1171, 'path': 'sta141b/2018/discussion09.ipynb', 'repo_name': 'clarkfitzg/teaching-notes', 'size': '11094'}