<jupyter_start><jupyter_text>Appendix A - Learning NLTKThe Natural Language Toolkit is a powerful method for language analysis. With it you can determine many key details of a sentence which will be important in decomposing information for this project. This Appendix follows the exercises in the [NLTK online book](http://www.nltk.org/book/) which is an open access resource allowing one to learn the NLTK toolset.  Chapter 1<jupyter_code>import nltk
from nltk.book import *<jupyter_output><empty_output><jupyter_text>☼ Try using the Python interpreter as a calculator, and typing expressions like 12 / (4 + 1).<jupyter_code>12 / (4 + 1)<jupyter_output><empty_output><jupyter_text>☼ Given an alphabet of 26 letters, there are 26 to the power 10, or 26 ** 10, ten-letter strings we can form. That works out to 141167095653376. How many hundred-letter strings are possible?<jupyter_code>26 ** 100<jupyter_output><empty_output><jupyter_text>☼ The Python multiplication operation can be applied to lists. What happens when you type ['Monty', 'Python'] \* 20, or 3 \* sent1?<jupyter_code>print(['Monty', 'Python'] * 5)
print(3*sent1)<jupyter_output>['Monty', 'Python', 'Monty', 'Python', 'Monty', 'Python', 'Monty', 'Python', 'Monty', 'Python']
['Call', 'me', 'Ishmael', '.', 'Call', 'me', 'Ishmael', '.', 'Call', 'me', 'Ishmael', '.']<jupyter_text>☼ Review 1 on computing with language. How many words are there in text2? How many distinct words are there?<jupyter_code>len(text2)<jupyter_output><empty_output><jupyter_text>☼ Compare the lexical diversity scores for humor and romance fiction in 1.1. Which genre is more lexically diverse? Humor ☼ Produce a dispersion plot of the four main protagonists in Sense and Sensibility: Elinor, Marianne, Edward, and Willoughby. What can you observe about the different roles played by the males and females in this novel? Can you identify the couples?<jupyter_code>text2.dispersion_plot(['Elinor', 'Marianne', 'Edward', 'Willoughby'])<jupyter_output><empty_output><jupyter_text>☼ Find the collocations in text5.<jupyter_code>from nltk import bigrams
list(bigrams(text5))[:10]<jupyter_output><empty_output><jupyter_text>☼ Consider the following Python expression: len(set(text4)). State the purpose of this expression. Describe the two steps involved in performing this computation. The expression creates a set of all the individual words of text 4. This is then counted by `len` which outputs the number of words as seen below.<jupyter_code>len(set(text4))<jupyter_output><empty_output><jupyter_text>a. Define a string and assign it to a variable, e.g., my_string = 'My String' (but put something more interesting in the string). Print the contents of this variable in two ways, first by simply typing the variable name and pressing enter, then by using the  print statement.<jupyter_code>my_string = 'A much funner string than in the example'
my_string
print(my_string)<jupyter_output>A much funner string than in the example<jupyter_text>b. Try adding the string to itself using my_string + my_string, or multiplying it by a number, e.g., my_string * 3. Notice that the strings are joined together without any spaces. How could you fix this?<jupyter_code>my_string + my_string
my_string * 3<jupyter_output><empty_output><jupyter_text>☼ Define a variable my_sent to be a list of words, using the syntax my_sent = ["My", "sent"] (but with your own words, or a favorite saying).<jupyter_code>my_sent = ["My", "sent"]<jupyter_output><empty_output><jupyter_text>Use ' '.join(my_sent) to convert this into a string.Use split() to split the string back into the list form you had to start with.☼ Define several variables containing lists of words, e.g., phrase1, phrase2, and so on. Join them together in various combinations (using the plus operator) to form whole sentences. What is the relationship between len(phrase1 + phrase2) and  len(phrase1) + len(phrase2)?<jupyter_code>joined_my_sent = ' '.join(my_sent)
list_my_sent = joined_my_sent.split()
print(joined_my_sent)
print(list_my_sent)<jupyter_output>My sent
['My', 'sent']<jupyter_text>☼ Consider the following two expressions, which have the same value. Which one will typically be more relevant in NLP? Why?"Monty Python"[6:12]["Monty", "Python"][1] The second as language is typically used in words, not letters. ☼ We have seen how to represent a sentence as a list of words, where each word is a sequence of characters. What does sent1[2][2] do? Why? Experiment with other index values. It indexes through the word, followed by the list of characters ☼ The first sentence of text3 is provided to you in the variable sent3. The index of the in sent3 is 1, because sent3[1] gives us 'the'. What are the indexes of the two other occurrences of this word in sent3?<jupyter_code>[i for i in range(len(sent3)) if sent3[i] == 'the']<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09411037891268534, 'fraction_numerical': 0.017092257001647446, 'mean_word_length': 5.889361702127659, 'pattern_counts': {'":': 0, '<': 50, '<?xml version=': 0, '>': 50, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '638201', 'license': 'MIT', 'n_tokens_mistral': 1527, 'n_tokens_neox': 1461, 'n_words': 614, 'path': 'docs/jupyter-notebook/Appendix A - Learning NLTK.ipynb', 'repo_name': 'CraigLangford/twitter-summarizer', 'size': '4856'}