<jupyter_start><jupyter_text>This section of the tutorial will teach you how to tag words with their parts of speech. First, we are going to open up our saved text that we retrieved earlier in the tutorial.<jupyter_code>import nltk
f = open("255s.txt","rU") #U is for Universal
raw = f.read()
f.close()<jupyter_output><empty_output><jupyter_text>Now, we will break up our text into sentences, and then words. After we do this, we tag all the words with their corresponding parts of speech.<jupyter_code>sentences = nltk.sent_tokenize(raw) #create list of sentences
sentences = [nltk.word_tokenize(sent) for sent in sentences] #convert each sentence into a list of words
sentences = [nltk.pos_tag(sent) for sent in sentences] #for each sentence, tag each word<jupyter_output><empty_output><jupyter_text>This leaves us with a nested array of (word, tag) pairs.<jupyter_code>sentences<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08232711306256861, 'fraction_numerical': 0.003293084522502744, 'mean_word_length': 6.181102362204724, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '482534', 'license': 'Unlicense', 'n_tokens_mistral': 287, 'n_tokens_neox': 283, 'n_words': 111, 'path': 'Parts of Speech.ipynb', 'repo_name': 'Jstanesa/Text-Mining-NLTK-Tutorial', 'size': '911'}