<jupyter_start><jupyter_text>Computing with Language: Statistics<jupyter_code>import nltk

# make sure that NLTK language resources have been downloaded 
# (see "NLTK Introduction" notebook)

from nltk.book import *<jupyter_output>*** Introductory Examples for the NLTK Book ***
Loading text1, ..., text9 and sent1, ..., sent9
Type the name of the text or sentence to view it.
Type: 'texts()' or 'sents()' to list the materials.
text1: <NAME> by <NAME> 1851
text2: Sense and Sensibility by <NAME> 1811
text3: The Book of Genesis
text4: Inaugural Address Corpus
text5: Chat Corpus
text6: Monty Python and the Holy Grail
text7: Wall Street Journal
text8: Personals Corpus
text9: The Man Who Was Thursday by <NAME> 1908<jupyter_text>[Word] Frequency DistributionsFreqDist is used to encode "frequency distributions", which count the number of times that each outcome of an experiment occurs.* In case of text, its frequency distribution will contains counts of all tokens that appear in the text.* Technically: FreqDist() creates a Python object (that holds information about a frequency distribution)<jupyter_code># frequency distribution of text1
fdist1 = FreqDist(text1)

print(fdist1)<jupyter_output><FreqDist with 19317 samples and 260819 outcomes><jupyter_text>**FreqDist** methods:* freq(sample) - returns the number of times "sample" appears in FreqDist* hapaxes() - a list of samples that appear only once* max() - the sample with the maximum number of occurences* plot() - plot a FreqDist chart* pprint() - "pretty print" the first items of FreqDistNLTK book: http://www.nltk.org/book/ch01.htmlcomputing-with-language-simple-statisticsFull list of methods: http://www.nltk.org/api/nltk.htmlnltk.probability.FreqDist<jupyter_code># print frequency distribution (top results)

fdist1.pprint()
# max()

fdist1.max()
# freq()

print("','  :", fdist1.freq(","))
print("whale:", fdist1.freq("whale"))<jupyter_output>','  : 0.07174707364110744
whale: 0.003473673313677301<jupyter_text>---Information about Python dictionaries: * ["Dictionaries and Structuring Data"](https://automatetheboringstuff.com/chapter5/)<jupyter_code># output of fdist1.pprint() looks like a Python "dictionary"

# can we look up its values by a given "key"?
fdist1["whale"]
# top 10 results (not that interesting for text)
# so nltk builds upon Counter type object
fdist1.most_common(10)
type(fdist1) #strongly suspect that theres is a Counter like dictionary lurking down there
%matplotlib inline

# plot the distribution
fdist1.plot(30)
import plotly.graph_objects as go
tuples = fdist1.most_common(50)
x = [t[0] for t in tuples]
y = [t[1] for t in tuples]

# fig = go.Figure(data=go.Scatter(x=x, y=y))
fig = go.Figure(data=go.Bar(x=x, y=y))
fig.show()
import string
tuples = fdist1.most_common(100)
tuples = [t for t in tuples if t[0] not in string.punctuation] # so we do not want any punctuation in results
x = [t[0] for t in tuples]
y = [t[1] for t in tuples]

# fig = go.Figure(data=go.Scatter(x=x, y=y))
fig = go.Figure(data=go.Bar(x=x, y=y))
fig.show()
# most_common() returs a list -> we can "slice" it

my_list = fdist1.most_common(100)

# results 50 through 59
my_list[50:60]
# least common results (first 10 examples)

fdist1.hapaxes()[:10]<jupyter_output><empty_output><jupyter_text>Words can appear both in lowercase and CapitalizedLet's fix our FreqDist:<jupyter_code># need to "lowercase" the text before passing it to FreqDist
#   - see example in https://www.nltk.org/api/nltk.html#nltk.probability.FreqDist

fdist2 = FreqDist(word.lower() for word in text1) # so we normalize to lowercase

# we're going through the list of tokens in text,
#  - returning (generating) lowercase versions of these tokens
#  - and passing the result to FreqDist
# initial:
print(fdist1.freq("whale"))
print(fdist1.freq("Whale"))
print()

# fixed:
print(fdist2.freq("whale"))<jupyter_output>0.003473673313677301
0.0010812095744558486

0.004700577795329328<jupyter_text>Cleaning data: removing stopwordsNLTK contains a corpus of *stopwords* - high-frequency words like "the", "to" and "also" - that we may want to filter out of a document before further processing.Stopwords usually have little lexical content, and their presence in a text fails to distinguish it from other texts.https://www.nltk.org/book/ch02wordlist-corpora<jupyter_code>from nltk.corpus import stopwords

# English stopwords
stop_words = stopwords.words("english")

stop_words[:8]
# let's start with text1 in lowercase

# return a list  
#   containing "word.lower()"
#     for every item (stored in variable "word")
#       in resource "text1"

text = [word.lower() for word in text1]

text[:7]<jupyter_output><empty_output><jupyter_text>**NLTK book: [4.2   Operating on Every Element](https://www.nltk.org/book/ch01operating-on-every-element)**This *pattern* – doing something (e.g. modifying) with every item in a sequence and returning a list of results – is called Python *list comprehension*:    `result_list = [item.do_something() for item in list]`List comprehensions may also contain conditions (only items matching the condition will be included in the resulting list):`result_list = [item.do_something() for item in list `**`if`**` condition]`It is very useful for filtering and modifying lists.<jupyter_code># we can filter either (a) text before calling FreqDist or (b) results of FreqDist.
# let's filter before calling FreqDist.

# create a set of stopwords (operations with sets are faster that with lists)
stop_set = set(stop_words)

# filter out stopwords (return only words not in the stoplist)
without_stopwords = [word for word in text if word not in stop_set]

text[:7]
# let's also filter out tokens that are not text or numbers

# Python has a built-in method .isalnum() that determines 
# if a string only consists of letters or digits:

# https://docs.python.org/3/library/stdtypes.html#str.isalnum

filtered = [word for word in without_stopwords if word.isalnum()]

filtered[:7]
# word frequency

freq = FreqDist(filtered)

freq.most_common(15)<jupyter_output><empty_output><jupyter_text>Exploring data: finding interesting wordsNLTK also includes a list of common English words. We can use it to find unusual or mis-spelt words in a text corpus.See also: https://www.nltk.org/book/ch02code-unusual<jupyter_code>common_word_list = nltk.corpus.words.words()

# convert word list to a set (+ convert words to lowercase)
word_set = set(word.lower() for word in common_word_list)

# filter out common words
uncommon = [word for word in filtered if word not in word_set]

uncommon[:7]
# word frequency

freq = FreqDist(uncommon)

freq.most_common(15)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10242141675439916, 'fraction_numerical': 0.031282899684163035, 'mean_word_length': 5.303317535545023, 'pattern_counts': {'":': 0, '<': 34, '<?xml version=': 0, '>': 35, 'https://': 6, 'lorem ipsum': 0, 'www.': 6, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '54835', 'license': 'MIT', 'n_tokens_mistral': 2304, 'n_tokens_neox': 2159, 'n_words': 760, 'path': 'nlp/NLTK - Statistics (with output).ipynb', 'repo_name': 'ValRCS/Python_DIP720_RTU_Spring_2021', 'size': '6678'}