<jupyter_start><jupyter_text>Build your own concordanceIn the lecture, we discussed how it took 500 Dominican munks to write the first concordance of the Latin bible, and how it took <NAME> 10 years to write the first concordance of the Hebrew bible. With Python, it only takes a matter of seconds to find words in a text, along with the surrounding words.In this exercise, first run the code and check that everything works. Then, try modifying the code. Do at least to the part where it says **Stop now if you want!** Do as much or as little of the other exercises as you can / want to.Run each cell in this notebook at a time, in order. If something in one cell doesn't work right, it might be because you have overwritten a variable, so try going back and running all the previous cells again.The point of the exercise isn't for you to understand all the commands fully; the idea is to start getting you familiar with how jupyter notebooks work, with the concept of copying and modifying code, and hopefully give you a little taste of the power of Python.<jupyter_code># install the natural language toolkit package (nltk), which has a copy of several texts, including the King James Bible
%pip install nltk
# import the nltk package so that it is accessible to Python, and download a collection of texts from Project Gutenberg
import nltk
nltk.download('gutenberg')
# Create a variable called "bible" which contains the text of the King James bible.
bible = nltk.corpus.gutenberg.raw('bible-kjv.txt')

# make all characters lowercase
bible = bible.lower()

# remove the "\n" characters, which indicate line breaks in the text (newlines)
bible = bible.replace('\n', ' ')

# split up the text into a long list of individual words
bible = bible.split(' ')
# make a variable called "concordance", and fill it with every occurrence of the phrase "this world", and a few words preceeding and following "this world"
concordance = []
for i, val in enumerate(bible):
    if val == "world":
        if bible[i-1] == "this":
            concordance.append(str(' '.join(bible[i-5:i+5])))
# take a look at what the algorithm has found
concordance<jupyter_output><empty_output><jupyter_text>Let's try again, but this time let's just search for "world" by itself, not "this world".<jupyter_code>concordance = []
for i, val in enumerate(bible):
    if val == "world":
        concordance.append(str(' '.join(bible[i-5:i+5])))
# take a look at what the algorithm has found
concordance<jupyter_output><empty_output><jupyter_text>Now, in the cell below, modify the code to search for a different word.<jupyter_code># add your modified code here and run the cell...<jupyter_output><empty_output><jupyter_text>The nltk package has the full text of several other classic books. You can see what they are called by running the command in the cell below:<jupyter_code>nltk.corpus.gutenberg.fileids()<jupyter_output><empty_output><jupyter_text>Stop now if you want!If you have come this far, and got everything to work, you can stop if you want! The goal of this assignment is to successfully run and modify some Python code to achieve a goal. But..... if you want to challenge yourself, here are a few more things you can try. In each case, I have given you a little bit of starter code to get you going. Challenge 1: build your own concordancePick a different book and a different word, or pair of words. Copy and paste from the code above to write some Python code that searches the book of your choice for the word or pair of words of your choice. Put this code in the cell below. By the way, some of the texts use the characters "\r" for "carriage return" instead of "\n" for "newline". You can remove these the same way that you remove the "\n" characters.<jupyter_code># add your code to search for a word or pair of words in a different book here<jupyter_output><empty_output><jupyter_text>Challenge 2: compare lengths of booksWe can use the command `len` to find how many items there are in a list. E.g., to find the number of words in the list called `bible`, above, we can write: `len(bible)`. Use the starter code below to find out which book in the books included in `nltk` has the most words.<jupyter_code># solution 1: print all the titles and numbers of words
# starter code:

books = nltk.corpus.gutenberg.fileids()

for title in books:
    book = nltk.corpus.gutenberg.raw(title)

# more advanced, for those with some Python experience, or those who want to google around..
# solution 2: make a list of titles and a list of wordcounts, zip them together, then sort them based on wordcount
# starter code:

books = nltk.corpus.gutenberg.fileids()

titles = []
numwords = []
for title in books:
    book = nltk.corpus.gutenberg.raw(title)<jupyter_output><empty_output><jupyter_text>Challenge 3: what are the most frequent words?`nltk` has a built-in function called `FreqDist` which counts up how many times each word in a text occurs. So, if you have a list called `words` which contains all the words in a book, you find the frequencies of all of them by writing `freq = nltk.FreqDist(words)`. You can then get the e.g. ten most common words by writing `freq.most_common(10)`. What are the ten most common words in Jane Austen's "Emma"? What about <NAME>'s "Moby Dick"?<jupyter_code># starter code:

book = nltk.corpus.gutenberg.raw('austen-emma.txt')
words = book.lower()<jupyter_output><empty_output><jupyter_text>Challenge 4: Remove stopwordsOften, the most frequent words are not the most interesting ones. Words like "a" and "the" are so common in English, that they don't really tell us much about the text. That is why we often remove "stopwords", that is, a list of the most common words in English, before e.g. counting frequencies. Below is some starter code to remove stopwords. Use these snippets to see what the most common words in Emma and Moby Dick are after removing these most common words.Hint: In Moby Dick, you will also have to remove the string `\r`, in addition to removing `\n`.<jupyter_code># list of stopwords
stopwords = ["", "i", "me", "my", "myself", "we", "our", "ours", "ourselves", "you", "your", "yours", "yourself", "yourselves", "he", "him", "his", "himself", "she", "her", "hers", "herself", "it", "its", "itself", "they", "them", "their", "theirs", "themselves", "what", "which", "who", "whom", "this", "that", "these", "those", "am", "is", "are", "was", "were", "be", "been", "being", "have", "has", "had", "having", "do", "does", "did", "doing", "a", "an", "the", "and", "but", "if", "or", "because", "as", "until", "while", "of", "at", "by", "for", "with", "about", "against", "between", "into", "through", "during", "before", "after", "above", "below", "to", "from", "up", "down", "in", "out", "on", "off", "over", "under", "again", "further", "then", "once", "here", "there", "when", "where", "why", "how", "all", "any", "both", "each", "few", "more", "most", "other", "some", "such", "no", "nor", "not", "only", "own", "same", "so", "than", "too", "very", "s", "t", "can", "will", "just", "don", "should", "now"]
# starter code:

book = nltk.corpus.gutenberg.raw('austen-emma.txt')
words = book.lower()


# code to remove stopwords.
words = [word for word in words if word not in stopwords]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11594602038006059, 'fraction_numerical': 0.0024786560176259984, 'mean_word_length': 4.773449920508744, 'pattern_counts': {'":': 3, '<': 35, '<?xml version=': 0, '>': 35, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '127509', 'license': 'MIT', 'n_tokens_mistral': 2234, 'n_tokens_neox': 2163, 'n_words': 1054, 'path': '2021/Notebooks/Lab_02.ipynb', 'repo_name': 'ethanweed/ExPsyLing', 'size': '7286'}