<jupyter_start><jupyter_text>Day 66 sentence tokenize and word tokenize TokenizingBy tokenizing, you can conveniently split up text by word or by sentence. This will allow you to work with smaller pieces of text that are still relatively coherent and meaningful even outside of the context of the rest of the text. It’s your first step in turning unstructured data into structured data, which is easier to analyze.When you’re analyzing text, you’ll be tokenizing by word and tokenizing by sentence. Here’s what both types of tokenization bring to the table:**Tokenizing by word:** Words are like the atoms of natural language. They’re the smallest unit of meaning that still makes sense on its own. Tokenizing your text by word allows you to identify words that come up particularly often. For example, if you were analyzing a group of job ads, then you might find that the word “Python” comes up often. That could suggest high demand for Python knowledge, but you’d need to look deeper to know more.**Tokenizing by sentence:** When you tokenize by sentence, you can analyze how those words relate to one another and see more context. Are there a lot of negative words around the word “Python” because the hiring manager doesn’t like Python? Are there more terms from the domain of herpetology than the domain of software development, suggesting that you may be dealing with an entirely different kind of python than you were expecting?Here’s how to import the relevant parts of NLTK so you can tokenize by word and by sentence:<jupyter_code># import libraries

from nltk import word_tokenize,sent_tokenize<jupyter_output><empty_output><jupyter_text>Now that you’ve imported what you need, you can create a string to tokenize. Here’s a quote from Dune that you can use:<jupyter_code>example_string = """
 Muad'Dib learned rapidly because his first training was in how to learn.
 And the first lesson of all was the basic trust that he could learn.
 It's shocking to find how many people do not believe they can learn,
 and how many more believe learning to be difficult."""<jupyter_output><empty_output><jupyter_text>You can use sent_tokenize() to split up example_string into sentences:<jupyter_code># apply stentence tokenize to the variable

sent_tokenize(example_string)<jupyter_output><empty_output><jupyter_text>Tokenizing example_string by sentence gives you a list of three strings that are sentences:"Muad'Dib learned rapidly because his first training was in how to learn."'And the first lesson of all was the basic trust that he could learn.'"It's shocking to find how many people do not believe they can learn, and how many more believe learning to be difficult." Now try tokenizing example_string by word:<jupyter_code># try word tokenizer

word_tokenize(example_string)<jupyter_output><empty_output><jupyter_text>You got a list of strings that NLTK considers to be words, such as:"Muad'Dib"'training''how'But the following strings were also considered to be words:"'s"',''.'See how "It's" was split at the apostrophe to give you 'It' and "'s", but "Muad'Dib" was left whole? This happened because NLTK knows that 'It' and "'s" (a contraction of “is”) (It is => It's) are two distinct words, so it counted them separately. But "Muad'Dib" isn’t an accepted contraction like "It's", so it wasn’t read as two separate words and was left intact.  Filtering Stop WordsStop words are words that you want to ignore, so you filter them out of your text when you’re processing it. Very common words like 'in', 'is', and 'an' are often used as stop words since they don’t add a lot of meaning to a text in and of themselves.Here’s how to import the relevant parts of NLTK in order to filter out stop words:<jupyter_code># to install "stopwords"
import nltk
nltk.download("stopwords")
from nltk.corpus import stopwords
# from nltk.tokenize import word_tokenize<jupyter_output><empty_output><jupyter_text>Here’s a quote from Worf that you can filter:<jupyter_code>worf_quote = "Sir, I protest. I am not a merry man!"<jupyter_output><empty_output><jupyter_text>Now tokenize worf_quote by word and store the resulting list in words_in_quote:<jupyter_code>words_in_quote = word_tokenize(worf_quote)
words_in_quote
type(words_in_quote)<jupyter_output><empty_output><jupyter_text>You have a list of the words in worf_quote, so the next step is to create a set of stop words to filter words_in_quote. For this example, you’ll need to focus on stop words in "english":<jupyter_code>stopwords.words("english")
len(stopwords.words("english"))
set(stopwords.words("english"))
stop_words = set(stopwords.words("english"))
stop_words<jupyter_output><empty_output><jupyter_text>Next, create an empty list to hold the words that make it past the filter:<jupyter_code>filtered_list = []<jupyter_output><empty_output><jupyter_text>You created an empty list, filtered_list, to hold all the words in words_in_quote that aren’t stop words. Now you can use stop_words to filter words_in_quote:<jupyter_code>for word in words_in_quote:
    if word.casefold() not in stop_words:
        filtered_list.append(word)<jupyter_output><empty_output><jupyter_text>You iterated over words_in_quote with a for loop and added all the words that weren’t stop words to filtered_list. You used .casefold() on word so you could ignore whether the letters in word were uppercase or lowercase. This is worth doing because stopwords.words('english') includes only lowercase versions of stop words.<jupyter_code>filtered_list<jupyter_output><empty_output><jupyter_text>Alternative MethodAlternatively, you could use a list comprehension to make a list of all the words in your text that aren’t stop words:<jupyter_code>filtered_list = [
    word for word in words_in_quote if word.casefold() not in stop_words
]
filtered_list<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.061210453920220086, 'fraction_numerical': 0.000343878954607978, 'mean_word_length': 5.811475409836065, 'pattern_counts': {'":': 1, '<': 49, '<?xml version=': 0, '>': 50, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '139801', 'license': 'MIT', 'n_tokens_mistral': 1648, 'n_tokens_neox': 1611, 'n_words': 779, 'path': 'NLP/1.Tokenizing(Jan14,2022).ipynb', 'repo_name': 'SuryaReginaAA/Learn', 'size': '5816'}