<jupyter_start><jupyter_text>1.1.Computing with Language: Texts and Words  Getting Started with Python<jupyter_code>#Check Python version
import sys
print(sys.version)
#Using Python as a calculator
1 + 5 * 2 - 3<jupyter_output><empty_output><jupyter_text>Getting Started with NLTK<jupyter_code># Before going further you should install NLTK
!pip install nltk
#Install the data required for the book
import nltk
#nltk.download()
#All the data we will need in this chapter
from nltk.book import *
#Find out about these texts,
text1<jupyter_output><empty_output><jupyter_text>Searching Text<jupyter_code>#Look up the word "monstrous" in Moby Dick
text1.concordance("monstrous")
#Appending the term similar to the name of the text in question
text1.similar("monstrous")
# The term ´common_contexts´ allows us to examine just the contexts that are shared by two or more words, such as monstrous and very
text2.common_contexts(["monstrous", "very"])
#we can also determine the location of a word in the text: how many words from the beginning it appears. This positional information can be displayed using a dispersion plot.
text4.dispersion_plot(["citizens", "democracy", "freedom", "duties", "America"])
#Try generating some random text
text3.generate()<jupyter_output>Building ngram index...
laid by her , and said unto Cain , Where art thou , and said , Go to ,
I will not do it for ten ' s sons ; we dreamed each man according to
their generatio the firstborn said unto Laban , Because I said , Nay ,
but Sarah shall her name be . , du<NAME> , duke Shobal , and Akan .
and looked upon my affliction . Bashemath Ishmael ' s blood , but Isra
for as a prince hast thou found of all the cattle in the valley , and
the wo The<jupyter_text>Counting Vocabulary<jupyter_code>#Length of a text from start to finish
len(text3)<jupyter_output><empty_output><jupyter_text>* Note:> *"A token is the technical name for a sequence of characters—such as hairy, his, or :)—that we want to treat as a group"*<jupyter_code>#Distinct words, or “word types.”
len(set(text3))<jupyter_output><empty_output><jupyter_text>* Note:> *"A word type is the form or spelling of the word independently of its specific occurrences in a text—that is, the word considered as a unique item of vocabulary"*<jupyter_code># Sorted list of vocabulary
sorted(set(text3))[5:15]
# Lexical richness
len(text3) / len(set(text3))
# Counting how often a word occurs in a text
text3.count("smote")
# Counting how often a word occurs in a text (percentage)
100 * text4.count('a') / len(text4)
# Define two new functions
def lexical_diversity(text):
    return len(text) / len(set(text))

def percentage(count, total):
    return 100 * count / total   
# Applying lexical_diversity function
lexical_diversity(text3)
# Applying percentage function
percentage(text3.count('a'), len(text3))<jupyter_output><empty_output><jupyter_text>1.2.Closer Look at Python: Texts as Lists of Words * Note:>*"We will think of a text as nothing more than a sequence of words and punctuation"*<jupyter_code>#Opening sentence of <NAME>
sent1 = ['Call', 'me', 'Ishmael', '.']
sent1
#Asking  for its length
len(sent1)
#Asking for lexical diversity
lexical_diversity(sent1)
#List - sum
['Monty', 'Python'] + ['and', 'the', 'Holy', 'Grail']
#List - append
sent1.append('lol')
sent1<jupyter_output><empty_output><jupyter_text>Indexing Lists<jupyter_code>#Index inside square brackets
text4[173]
#Given a word, find the index of when it first occurs
text4.index('awaken')
#Extracting manageable pieces of language from large texts
text5[16715:16720]
#Slicing
text2[141570:]<jupyter_output><empty_output><jupyter_text>Variables<jupyter_code>#Evaluating the expression, and save its result to the variable (assignment)
my_sent = ['Bravely', 'bold', 'Sir', 'Robin', ',', 'rode','forth', 'from', 'Camelot', '.']

noun_phrase = my_sent[1:4]
noun_phrase
wOrDs = sorted(noun_phrase)
wOrDs
#Using variables to hold intermediate steps of a computation
vocab = set(text1)
vocab_size = len(vocab)
vocab_size<jupyter_output><empty_output><jupyter_text>Strings<jupyter_code>name = 'Monty'
name[0]
# Slicing a string
name[:4]
# Perform multiplication and addition with strings ...
name*2
# and addition with strings
name+"1"<jupyter_output><empty_output><jupyter_text>1.3.Computing with Language: Simple Statistics<jupyter_code>#Review the previous section before continuing further
saying = ['After', 'all', 'is', 'said', 'and', 'done','more', 'is', 'said', 'than', 'done']
tokens = set(saying)
tokens = sorted(tokens)
tokens<jupyter_output><empty_output><jupyter_text>Frequency Distributions<jupyter_code>#Use a FreqDist to find the 50 most frequent words of Moby Dick
fdist1 = FreqDist(text1)
fdist1
vocabulary1 = list(fdist1.keys())
vocabulary1[0:9]
# Generating a cumulative frequency plot for these words
fdist1.plot(50, cumulative=True)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10507988529291275, 'fraction_numerical': 0.017410897173289634, 'mean_word_length': 6.2880597014925375, 'pattern_counts': {'":': 0, '<': 50, '<?xml version=': 0, '>': 53, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '315085', 'license': 'MIT', 'n_tokens_mistral': 1690, 'n_tokens_neox': 1604, 'n_words': 553, 'path': 'Python/Machine Learning With Python/Natural Language Processing/Analyzing Text with the Natural Language Toolkit/Chapter 01 - Language Processing and Python.ipynb', 'repo_name': 'FelipeRamosOliveira/Portfolio', 'size': '4886'}