<jupyter_start><jupyter_text>Problem-solving notebook  Exercise: combinations of first and last namesGiven the following list of first and last names, write a nested Python loop to print every combination of first and last name.<jupyter_code>first=['Xue', 'Mary', 'Robert']
last=['Li', 'Smith', 'Dixon']<jupyter_output><empty_output><jupyter_text>The first step is to understand the problem, so let's figure out what the output looks like:```Xue LiX<NAME>X<NAME>Mary Li<NAME> <NAME>Robert Li<NAME> <NAME>```If we were trying to do this for two arbitrary lists, we should think about the edge cases, such as when one or both of the lists are empty.  Let's assume that we fully understand the problem now.The next step is to try to solve the problem, which we must be able to manually. Unless you are an experienced programmer, it's best not to think about coding at this point. I literally think about moving my fingers or data around. In this case, I use a finger on each list and then for every top pointer finger I move my finger through all of the bottom fingers:When you hear the "all combinations" pattern, think of nested loops.  Moving a finger through a list also screams loop! Here's one possible imitation:<jupyter_code>for f in first:
    for l in last:
        print(f+' '+l)<jupyter_output><NAME>
<NAME>
<NAME>
<NAME>
<NAME>
<NAME>
<NAME>
<NAME>
<NAME><jupyter_text>How would you solve this with a list comprehension (to get a list of instead of printing all combinations)?<jupyter_code>[f+' '+l for f in first for l in last]<jupyter_output><empty_output><jupyter_text>Exercise: checking for valid parenthesesLet's try to solve a problem from leetcode.com called[Valid Parentheses](https://leetcode.com/problems/valid-parentheses/).  That website has lots and lots of sample problems to solve and will help you get better at programming, and is very useful when you  start practicing for interviews.  The problem description is (taken verbatim from the site):*Given a string s containing just the characters '(', ')', '{', '}', '[' and ']', determine if the input string is valid.**An input string is valid if:*1. *Open brackets must be closed by the same type of brackets.*2. *Open brackets must be closed in the correct order.*Per our process, we have to understand the problem and my first question involves the set of valid characters. Looking more closely at the problem description, I see that only `()[]{}` characters are allowed. Now, let's come up with a bunch of examples that illustrate the goal:            InputValid        ()true    []true    {}true    ()[]true    ([])true    {({})}true    {)false    {})false    {()false    (false        If I did this manually, I would move my finger down the list of characters and perform an operation that made sure that closing symbols come in the right order and that all groupings are closed at the end:  First attemptMy first thought was to increment in decrement account for each kind of grouping. As I encountered an opening symbol, I would bump that count. As I encountered a closing symbol, I would decrement that count. After walking through the entire string, the counts should all be zero indicating nothing is still open. This doesn't work. Why not?<jupyter_code>from lolviz import *
objviz({'(':0,'[':0,'{':0})  # a first attempt at tracking<jupyter_output><empty_output><jupyter_text>Let's see what happens with a bad example: `([)`. When we get to the `)`, the counts are<jupyter_code>objviz({'(':1,'[':1,'{':0})  # a first attempt at tracking<jupyter_output><empty_output><jupyter_text>We need to drop the count of the `)` but how do we know that it is in the right order? We don't.   The right approachIt's not just the character count that matters, but the sequence of open and closes. When we see a close character, the most recent opening character must be the associated grouping symbol. Because these groupings can be nested, it complicates matters. We need a way to track the opening characters we've seen before and in their order of appearance.  Since I know that the solution requires a [stack data structure](https://docs.python.org/3/tutorial/datastructures.htmlusing-lists-as-stacks) for tracking symbols, I'm going to draw it as such:At this point, we see the first closing symbol and so we compare the closing symbol with the top of the stack to see if it's the most recently opened grouping. It is, so we pop off the `(` from the stack and move to the next character with her finger.Take a look at the situation when our finger moves to the `}`. We can see that there is a clear mismatch between the closing character under the finger and what we are expecting, thanks to the stack of open grouping symbols. Now that we have a working process, we can think about actually implementing it.  I will play around with some code to make sure I can make it work and then worry about putting it into a nice function. This is a typical workflow for programmers. Explore then cleanup.<jupyter_code>symbols = [] # our stack of symbols
openers = {')':'(', ']':'[', '}':'{'}
s = ")]}"
for c in s:
    if c in "([{":
        symbols.append(c)
    else: # verify we have the right opening symbol
        if len(symbols)==0:
            print(f"Char '{c}' doesn't have an opening char; symbols={','.join(symbols)}")
            continue
        opener = symbols.pop()
        if openers[c] != opener:
            print(f"mismatched '{c}' doesn't match '{opener}'; symbols={','.join(symbols)}")
if len(symbols)>0:
    print(f"Unclosed groupings: {symbols}")<jupyter_output>Char ')' doesn't have an opening char; symbols=
Char ']' doesn't have an opening char; symbols=
Char '}' doesn't have an opening char; symbols=<jupyter_text>After trying various input strings, I added a number of messages to catch errors. Now we can create a clean function that returns true or false as to whether the groupings are balanced:<jupyter_code>def isValid(s: str) -> bool:
    symbols = [] # our stack of symbols
    openers = {')':'(', ']':'[', '}':'{'}
    for c in s:
        if c in "([{":
            symbols.append(c)
        else: # verify we have the right opening symbol
            if len(symbols)==0:
                return False
            opener = symbols.pop()
            if openers[c] != opener:
                return False
    if len(symbols)>0:
        return False
    return True<jupyter_output><empty_output><jupyter_text>We have to try the function out with a bunch of test cases to make sure it works, including some edge cases:<jupyter_code>isValid('')
isValid('()')
isValid('{()')
isValid('(]')
isValid('{[()]}')
isValid('()[]{}')
isValid('(')<jupyter_output><empty_output><jupyter_text>Submitting to leetcode indicates that the runtime is faster than 85% of submitted solutions.  Exercise: determining which ad to show on a websiteImagine you work for a company that has a website and you'd like to be able to show ads on the various webpages on the website.  Customers that want to purchase ads by a certain fraction of the ad "impressions".  For example, company X might buy 30%, company Y 60%, and company Z 10% of the ad space. Your job is to randomly show the company's ads, but in the correct proportions.To solve this problem, but it helps to visualize the relationship between the companies; we might draw a bar chart on paper that looks like this:<jupyter_code>import matplotlib.pyplot as plt
%config InlineBackend.figure_format = 'retina'  # gets sharper images
plt.figure(figsize=(2,2))
plt.bar(["X","Y","Z"],[30,60,10])
plt.xlabel("Company")
plt.ylabel("Ad percentage")
plt.show()<jupyter_output><empty_output><jupyter_text>We need to roll some random computer dice with three sides that are biased in this way. Another way to look at this as throwing darts at the wall and having most of the darts end up in the Y column, 30% in the X column, etc... Without altering gravity, that's hard to do. An easier way is to stretch things out so that the width of each company's region is proportional to its percentage, which then flattens out the heights of the bars:<!-- this doesn't work for some reason--><jupyter_code>from IPython.display import Image
Image(filename='images/ads-regions.png', width=300)<jupyter_output><empty_output><jupyter_text>If we throw darts at this overall region, X will get 30%, Y will get 60%, and Z will get 10%. So this tells us an approach: pick a uniform random number and figure out which region it falls in and report that company's ad.So now we know how to solve it, let's code it up. Obviously we have some research to do because we need to figure out how to get a random number. There are a number of ways, but let's use numpy:<jupyter_code>import numpy as np

np.random.random(), np.random.random(), np.random.random()<jupyter_output><empty_output><jupyter_text>Let's make a function that returns a company name with every call. To keep things simple, let's do it for the fixed three companies we've been talking about:<jupyter_code>def select(companies):
    i = np.random.random() # throw a dart
    if i>=0 and i<.30:
        return companies[0]
    if i>=.30 and i<.90:
        return companies[1]
    return companies[2]
ads = [select(companies=["X","Y","Z"]) for i in range(1000)]
ads[:15]<jupyter_output><empty_output><jupyter_text>As part of our problem-solving strategy, we also need to verify things, so let's do a histogram to verify the percentages look right:<jupyter_code>plt.figure(figsize=(2,2))
plt.hist(ads)
plt.xlabel("Company")
plt.ylabel("Ad percentage")
plt.show()<jupyter_output><empty_output><jupyter_text>Exercise: computing histogramsIn the previous exercise, we used the histogram functionality of the plotting library to compute the number of X, Y, and Z company values in the `ads` list. Now, let's figure out a way to efficiently keep a count for the company names in the list, without relying on built-in functionality.  To make it more interesting, we're opening up to more than just the three companies.<jupyter_code>companies=["Google","Apple","FaceBook","HP","SAP"]
n = len(companies)<jupyter_output><empty_output><jupyter_text>As a subexercise, how would we get a uniform random sample of 5000 company names from this list? To get you familiar with the vector-based approach of numpy, let's ask for a list of integer indexes into the company's list of size 5000. The index values should be integers and in the range `[0,n)`:<jupyter_code>idxs = np.random.randint(low=0,high=n,size=5000)
idxs[0:50]<jupyter_output><empty_output><jupyter_text>Let's also convert the list to a numpy array, which then allows us to use a list of integers as an index:<jupyter_code>companies = np.array(companies)
names = companies[idxs]
names[0:15]<jupyter_output><empty_output><jupyter_text>We could also do this with a list comprehension in pure Python, but it's a good idea to get familiar with numpy.Ok, now we have a list of companies and would like to count the number of occurrences of each. If we wanted to use numpy, we could do the following:<jupyter_code>np.unique(names, return_counts=True)<jupyter_output><empty_output><jupyter_text>Instead, we want to implement this ourselves.   We are given a list of company names with multiple occurrences of each.  To solve this, we don't think about code and think about how we would do this by hand. The first thing I would do is to create a table with a company name and account to the right, something like this:<jupyter_code>from lolviz import *
objviz({c:0 for c in companies})<jupyter_output><empty_output><jupyter_text>Then, I would move my finger down the list of company names and the count in my table for the company pointed to by my finger:That sounds like a good approach and therefore we can code this up.  Every time I think of moving my finger down a list, I think of a for loop of some kind.   When I think of a table mapping company name to account, I think of a dictionary.<jupyter_code>counts = {c:0 for c in companies}
for name in names:
    counts[name] += 1
objviz(counts)<jupyter_output><empty_output><jupyter_text>We would expect the numbers to be roughly the same, which they are. If we increase the sample size to 10,000, the counts would be even closer.  Exercise: encrypting textIn this exercise, let's learn a little bit about security and encryption by implementing a [cipher encryption function](https://en.wikipedia.org/wiki/Cipher). Given a string, the function must return an encrypted version and the key (the key is the reverse mapping of the cipher and is needed to decrypt). The cipher is a dictionary the maps character to a new character. A cipher encryption works by replacing each character in the plaintext with a character from a key.  For example, if we have the following partial key, we'd encrypt `cat` as `nsz` and `taca` as `zsns`.<jupyter_code>objviz({'a':'s', 'c':'n', 't':'z'})<jupyter_output><empty_output><jupyter_text>Starting the design process at the highest level, it seems like we have two steps: 1. create a cipher that maps all lowercase letters to a random unique character; create the key by reversing the cipher2. create a new string by walking the original and replacing each character with the character from the keyStep one is nontrivial, so we need to break that process down into some smaller chunks. Working backwards, the problem is straightforward if we have a list of characters a..z and a shuffled version of a..z.  If that's the case, we can simply move a finger through the a..z and the shuffled version, combining them into a dictionary.  To get a list of a..z, I happen to know we can get it easily from Python in `string.ascii_lowercase` so let's not worry about that. How do we get a shuffled version? There are a number of solutions using list comprehensions but it's easier if we use a built-in shuffling function in numpy.  Given a vector/list of elements, `np.random.shuffle()` will shuffle them up in place (meaning it will alter the list to pass and instead of creating a new one and returning it).<jupyter_code>import string
import numpy as np

vocab = string.ascii_lowercase + ' ' # allow spaces as well
shuffled_chars = list(vocab)
np.random.shuffle( shuffled_chars )<jupyter_output><empty_output><jupyter_text>Once we have the shuffled list, use zip() and a dictionary comprehension to glue them together. (A dictionary comprehension is the same as a list comprehension except you get a dictionary instead of a list.)<jupyter_code>cipher = {c:k for c,k in zip(vocab,shuffled_chars)}
objviz(cipher)<jupyter_output><empty_output><jupyter_text>The key is the reverse of the cipher so let's just walk that dictionary and flip the key and value:<jupyter_code>key = {v:k for k,v in cipher.items()}
objviz(key)<jupyter_output><empty_output><jupyter_text>That completes the first step. For the second step, we have to create a new string from the original plaintext based on the cipher. That's a matter of walking the characters, looking them up in the cipher, and adding the translated character to a new string.<jupyter_code>s = "my password is <PASSWORD>"
e = ""
for c in s:
    e += cipher[c]
# same as: e = ''.join([cipher[c] for c in s])
e<jupyter_output><empty_output><jupyter_text>To go backwards to the plaintext:<jupyter_code>s = ''.join([key[c] for c in e])
s<jupyter_output><empty_output><jupyter_text>That appears to work so let's create two clean functions:<jupyter_code>def encrypt(s) -> (dict,dict):
    vocab = string.ascii_lowercase + ' ' # allow spaces as well
    shuffled_chars = list(vocab)
    np.random.shuffle( shuffled_chars )
    cipher = {c:k for c,k in zip(vocab,shuffled_chars)}
    key = {v:k for k,v in cipher.items()}
    e = ''.join([cipher[c] for c in s])
    return e, key

def decrypt(e, key) -> str:
    return ''.join([key[c] for c in e])<jupyter_output><empty_output><jupyter_text>Let's give it a try with some sample strings:<jupyter_code>e,key = encrypt("hi mom")
decrypt(e,key)
e,key = encrypt("")
decrypt(e,key)
e,key = encrypt("apple cat banana")
decrypt(e,key)<jupyter_output><empty_output><jupyter_text>Here's a cool Python syntax trick.  If you have a tuple `t`, then `*t` unpacks that into an argument list and a function call, which lets us tighten up the syntax like this:<jupyter_code>decrypt( *encrypt("apple cat banana") )
decrypt( *encrypt("i like toast") )<jupyter_output><empty_output><jupyter_text>This encryption seems great! How could somebody guess our randomized mapping of characters? Unfortunately, these simple ciphers fall immediately to character frequency analysis, as long as we have enough characters in the sample to get an accurate picture of character frequencies. For example, in English the letter `e` has the highest frequency so we replace the character with the highest frequency in the encrypted text with `e` and so on down the frequency list.  As we go along, we look for words in English dictionary or sequences that are close to endless words.  It's like a game of Jeopardy where we try to guess a phrase given some of the characters. We can also look at most common word pairs like `th` to tease out the original plaintext. If you'd like to read a fascinating and well-written (nonacademic) book on cryptography from ancient times to modern times, check out [The Code Book](https://www.amazon.com/Code-Book-Science-Secrecy-Cryptography/dp/0385495323) by <NAME>.   Exercise: Download and filter CSV data from the webAs we will see in the data acquisitions class, a very common pattern is to collect some data from the web and process it in some way.  Let's get started  by downloading a CSV file from the web that contains information on [cities in various states](https://people.sc.fsu.edu/~jburkardt/data/csv/cities.csv) from the US. We could simply download that file onto our local disk, but let's do this automatically with a program. The next step is to filter out any cities that are not in Washington state, code `WA`.   Let's finish up by saving those `WA` rows back to a text file, `/tmp/wa_cities.csv`, in CSV format. We'll avoid using pandas initially and then do it again with pandas.To get started, think about the overall sequence of steps needed to solve this problem at a high level; totally ignore the coding issues. In my mind, there are three things to do:1. Download the CSV file as a string2. Filter for just the `WA` states3. Write those rows to a new text fileThere are a number of details we need to flesh out for each step, but it's important to start at a high level before drilling down into the details.For step one, a quick Google search reveals that there is a package called `requests` for Python that allows us to download files easily so let's do that and take a look at what we get:<jupyter_code>import requests
r = requests.get("https://people.sc.fsu.edu/~jburkardt/data/csv/cities.csv")
contents = r.text
print(contents[0:300])<jupyter_output>"LatD", "LatM", "LatS", "NS", "LonD", "LonM", "LonS", "EW", "City", "State"
   41,    5,   59, "N",     80,   39,    0, "W", "Youngstown", OH
   42,   52,   48, "N",     97,   23,   23, "W", "Yankton", SD
   46,   35,   59, "N",    120,   30,   36, "W", "Yakima", WA
   42,   16,   12, "N",     71,<jupyter_text>There are a lot of quotation marks in there that are unnecessary and we can remove.  We did not know this detail when planning the overall process, but now we know and so let's clean up:<jupyter_code>contents = contents.replace('"', '')
print(contents[0:300])<jupyter_output>LatD, LatM, LatS, NS, LonD, LonM, LonS, EW, City, State
   41,    5,   59, N,     80,   39,    0, W, Youngstown, OH
   42,   52,   48, N,     97,   23,   23, W, Yankton, SD
   46,   35,   59, N,    120,   30,   36, W, Yakima, WA
   42,   16,   12, N,     71,   48,    0, W, Worcester, MA
   43,   37,<jupyter_text>For step two, we need to filter for just those cities in Washington state but right now all we have is a string. Let's split that string into a list of strings, with one row per string and separate out the first row as the list of columns:<jupyter_code>lines = contents.split('\n')
columns = lines[0]
lines = lines[1:]
lines[:5]<jupyter_output><empty_output><jupyter_text>Once we have that list, we can use a list comprehension to filter the rows. Note that we don't have to split up the individual rows into fields or columns because a simple check for string `WA` in each row is sufficient:<jupyter_code>wa = [row for row in lines if 'WA' in row]
wa<jupyter_output><empty_output><jupyter_text>For step three, we can open up a text file as we normally do and iterate through the rows of data (starting out by writing the list of columns):<jupyter_code>with open("/tmp/wa_cities.csv", "w") as f:
    f.write(columns+"\n")
    for row in wa:
        f.write(row+"\n")<jupyter_output><empty_output><jupyter_text>Pandas alternativePandas is also network aware and so we can give it a URL instead of a file name to read in a CSV file, which it processes heavily into a table rather than a simple list of strings as we did previously<jupyter_code>import pandas as pd
import csv
df = pd.read_csv("https://people.sc.fsu.edu/~jburkardt/data/csv/cities.csv",
                 delim_whitespace=True)
df.head(5)<jupyter_output><empty_output><jupyter_text>For step two, it's easy to filter using a query like syntax.  Expression `df['State']=='WA'` gives us a vector of Boolean values indicating whether a row satisfies the conditions:<jupyter_code>df['State']=='WA'<jupyter_output><empty_output><jupyter_text>That vector of Booleans is usable as an index into the rows of the data frame:<jupyter_code>df_wa = df[df['State']=='WA']
df_wa<jupyter_output><empty_output><jupyter_text>Step three is also much easier using pandas:<jupyter_code>df_wa.to_csv("/tmp/wa_cities2.csv", index=False)<jupyter_output><empty_output><jupyter_text>The final phase of our problem-solving template is to verify results, which we can do by comparing the two files we wrote to make sure they are the same.  Here I'm using the `diff` tool from the commandline; getting no output indicates there are no differences, which is what we want.<jupyter_code>! diff /tmp/wa_cities*.csv<jupyter_output>1,7c1,7
< LatD, LatM, LatS, NS, LonD, LonM, LonS, EW, City, State
<    46,   35,   59, N,    120,   30,   36, W, Yakima, WA
<    47,   25,   11, N,    120,   19,   11, W, Wenatchee, WA
<    46,    4,   11, N,    118,   19,   48, W, Walla Walla, WA
<    47,   14,   24, N,    122,   25,   48, W, Tacoma, WA
<    47,   40,   11, N,    117,   24,   36, W, Spokane, WA
<    47,   35,   59, N,    122,   19,   48, W, Seattle, WA
---
> "LatD,","LatM,","LatS,","NS,","LonD,","LonM,","LonS,","EW,","City,",State
> "46,","35,","59,","N,","120,","30,","36,","W,","Yakima,",WA
> "47,","25,","11,","N,","120,","19,","11,","W,","Wenatchee,",WA
> "46,","4,","11,","N,","118,","19,","48,","W,","Walla Walla,",WA
> "47,","14,","24,","N,","122,","25,","48,","W,","Tacoma,",WA
> "47,","40,","11,","N,","117,","24,","36,","W,","Spokane,",WA
> "47,","35,","59,","N,","122,","19,","48,","W,","Seattle,",WA<jupyter_text>Exercise: simple EDA of the text corpusThe goal of this exercise is to load all of the text files in a directory and compute an overall vocabulary and count per word. This would be the first step in understanding a corpus of text we wanted to analyze. For example, what is the most common word used in English?  It's almost certainly `the`, right? Another interesting question is the length of the average word and the size of the overall vocabulary. Took a look at the [Berlitz](https://github.com/parrt/msds501/tree/master/data/berlitz1) data directory that's part of this course is repository. You will see a series of text files. Here's a terminal command to find all files that have the word Israel and the name:<jupyter_code>! ls ../data/berlitz1/*Israel*<jupyter_output>../data/berlitz1/HandRIsrael.txt   ../data/berlitz1/WhatToIsrael.txt
../data/berlitz1/HistoryIsrael.txt ../data/berlitz1/WhereToIsrael.txt
../data/berlitz1/IntroIsrael.txt<jupyter_text>The overall steps needed to solve this problem are something like:1.  Get a list of all files in a directory2.  Walked a list of files and extract to the text, appending it to an overall string3.  Normalize the contents string by converting to lowercase and calling `split()` to break up the string into a list of words4.  Report the total number of words, number of unique words, average word length, and perhaps the top 20 most common wordsA bit of googling will lead you to `os.listdir()` for step 1 and step number 2 is just a for-each loop over that list that uses the "fetch file contents" pattern we've seen before. Using the `+` operator, we can add strings to an overall `contents` variable. Step 3 is just to function calls to `lower()` and `split()`. Depending on our knowledge of available libraries, step 4 could be a lot of work or it very simple; let's code the other parts first.<jupyter_code>import os

contents = ""
for filename in os.listdir("../data/berlitz1"):
    with open(f"../data/berlitz1/{filename}", "r") as f:
        s = f.read()
        contents += " " # add space so words don't get stuck together
        contents += s
contents = contents.lower()
words = contents.split()
print(f"There are {len(words)} words in the contents")<jupyter_output>There are 520878 words in the contents<jupyter_text>To get the unique words, we can either use numpy or `set(...)`. It looks like there are 57,894 words in the vocabulary:<jupyter_code>len(np.unique(words)), len(set(words))<jupyter_output><empty_output><jupyter_text>Let's work backwards: To compute the average word length, we need a list of the word lengths and can then just compute the mean using numpy.  To get the word lengths, we can use a list comprehension to walk through the words. The code then looks like this:<jupyter_code>avg_len = np.mean([len(w) for w in words])
avg_len<jupyter_output><empty_output><jupyter_text>Finally, we need to count the number of occurrences of the words, kind of like we did for the histogram exercise above. Since you know how to do that with your own code now, let's learn to use a built-in `Counter` object that does all the work for us:<jupyter_code>from collections import Counter

ctr = Counter(words)
c = ctr.most_common(20)
c<jupyter_output><empty_output><jupyter_text>The solution to this problem is not too hard given all of the libraries we have available to us.Let's also plot the counts to see the relationship between frequencies and just how fast it drops off.<jupyter_code>c = np.array(c)
wds = c[:,0]
counts = c[:,1].astype(int)  # numpy converted c to strings
fig, ax = plt.subplots(1,1,figsize=(8.5,2.5))
barcontainers = ax.bar(x=wds, height=counts, color='#FEE08F')

# make it look good
ax.spines['top'].set_visible(False)
ax.spines['right'].set_visible(False)
ax.spines['left'].set_linewidth(.5)
ax.spines['bottom'].set_linewidth(.5)
for rect in barcontainers.patches:
    rect.set_linewidth(.5)
    rect.set_edgecolor('grey')

plt.show()<jupyter_output><empty_output><jupyter_text>Exercise: Extract person and place names from a document**Warning**: On M1 macs, use `conda install spacy` not `pip install spacy`. See [issues with M1 and spaCy](https://github.com/explosion/spaCy/issues/7962).Imagine trying to analyze text document. One of the things you might want to know is who is referenced within. This is actually an extremely hard problem to solve, but fortunately the people at [spaCy](https://spacy.io) have provided a library that does a very good job. All we have to do is put the glue around it. To install it, we need a library and their model of English: ```bash$ pip install spacy$ python -m spacy download en_core_web_sm...```If we have a library that can identify the names within a document, the overall process is straightforward:1. Load the text of a file into a string variable2. Give the string to the spaCy library and ask it to analyze the text3. Walk the list of names and add them to a set, which will create a unique collectionOur implementation starts with an import and loading of the English model:<jupyter_code>import spacy
nlp = spacy.load("en_core_web_sm")<jupyter_output><empty_output><jupyter_text>Next, we load the text of a sample file and give it to the `nlp` object that represents the English model:<jupyter_code>with open(f"../data/berlitz1/WhatToJamaica.txt", "r") as f:
    s = f.read()
doc = nlp(s)<jupyter_output><empty_output><jupyter_text>The `doc` object represents the analyze document and `doc.ents` represents a list of entities found during analysis. We can walk those entities with a for-each loop and, if the type of the entity is a person, we can add it to our unique set.  From the documentation of spaCy, I saw that `entity.label_` gives us the associated entity type and `entity.text` gives us the text of that entity:<jupyter_code>names = set()
for entity in doc.ents:
    if entity.label_=="PERSON":
        names.add(entity.text)
sorted(names)[:20]<jupyter_output><empty_output><jupyter_text>That does a reasonable job, but also seems to include a lot of organizations and locations. For fun, we can also look for organizations and geopolitical entities<jupyter_code>orgs = set()
for entity in doc.ents:
    if entity.label_=="ORG" or entity.label_=="GPE":
        orgs.add(entity.text)
sorted(orgs)[:20]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.095496417604913, 'fraction_numerical': 0.014568406687137495, 'mean_word_length': 4.577735490009514, 'pattern_counts': {'":': 4, '<': 213, '<?xml version=': 0, '>': 218, 'https://': 10, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '48', 'dir': 'jupyter-structured-clean-dedup', 'id': '301866', 'license': 'MIT', 'n_tokens_mistral': 9116, 'n_tokens_neox': 8602, 'n_words': 3986, 'path': 'notes/problem-solving.ipynb', 'repo_name': 'Linlin15963/msds501', 'size': '29369'}