<jupyter_start><jupyter_text>NLP Text Processing methods  NLTKWe can use some available data sets for educational purpose. One of such source are NLTK book corporas. To download it we need to do the following:<jupyter_code>import nltk
nltk.download('book')<jupyter_output>[nltk_data] Downloading collection 'book'
[nltk_data]    | 
[nltk_data]    | Downloading package abc to
[nltk_data]    |     /home/kprzystalski/nltk_data...
[nltk_data]    |   Unzipping corpora/abc.zip.
[nltk_data]    | Downloading package brown to
[nltk_data]    |     /home/kprzystalski/nltk_data...
[nltk_data]    |   Unzipping corpora/brown.zip.
[nltk_data]    | Downloading package chat80 to
[nltk_data]    |     /home/kprzystalski/nltk_data...
[nltk_data]    |   Unzipping corpora/chat80.zip.
[nltk_data]    | Downloading package cmudict to
[nltk_data]    |     /home/kprzystalski/nltk_data...
[nltk_data]    |   Unzipping corpora/cmudict.zip.
[nltk_data]    | Downloading package conll2000 to
[nltk_data]    |     /home/kprzystalski/nltk_data...
[nltk_data]    |   Unzipping corpora/conll2000.zip.
[nltk_data]    | Downloading package conll2002 to
[nltk_data]    |     /home/kprzystalski/nltk_data...
[nltk_data]    |   Unzipping corpora/conll2002.zip.
[nltk_data]    | Downloading package dep[...]<jupyter_text>To check what kind of books are available we need to import these and print the list as following:<jupyter_code>from nltk.book import *
texts()<jupyter_output>*** Introductory Examples for the NLTK Book ***
Loading text1, ..., text9 and sent1, ..., sent9
Type the name of the text or sentence to view it.
Type: 'texts()' or 'sents()' to list the materials.
text1: Moby Dick by <NAME> 1851
text2: Sense and Sensibility by Jane Austen 1811
text3: The Book of Genesis
text4: Inaugural Address Corpus
text5: Chat Corpus
text6: Monty Python and the Holy Grail
text7: Wall Street Journal
text8: Personals Corpus
text9: The Man Who Was Thursday by G . K . Chesterton 1908
text1: Moby Dick by <NAME> 1851
text2: Sense and Sensibility by Jane Austen 1811
text3: The Book of Genesis
text4: Inaugural Address Corpus
text5: Chat Corpus
text6: Monty Python and the Holy Grail
text7: Wall Street Journal
text8: Personals Corpus
text9: The Man Who Was Thursday by G . K . Chesterton 1908<jupyter_text>NLTK Text object does have some useful methods that you can find here: [NLTK Text Docs](http://www.nltk.org/api/nltk.htmlnltk.text.Text). Some are used below.  Word and sentences similarityWord does have different meanings. This makes the comparison and analysis a bit more complex.<jupyter_code>text6.concordance("King")

text6.count("King")<jupyter_output>Displaying 25 of 38 matches:
CENE 1 : [ wind ] [ clop clop clop ] KING ARTHUR : Whoa there ! [ clop clop cl
ragon , from the castle of Camelot . King of the Britons , defeator of the Sax
 CART - MASTER : I dunno . Must be a king . CUSTOMER : Why ? CART - MASTER : H
 all over him . SCENE 3 : [ thud ] [ King Arthur music ] [ thud thud thud ] [ 
 Arthur music ] [ thud thud thud ] [ King Arthur music stops ] ARTHUR : Old wo
e an inferior ! ARTHUR : Well , I am king ! DENNIS : Oh king , eh , very nice 
HUR : Well , I am king ! DENNIS : Oh king , eh , very nice . And how d ' you g
o you do , good lady . I am Arthur , King of the Britons . Who ' s castle is t
s . Who ' s castle is that ? WOMAN : King of the who ? ARTHUR : The Britons . 
. We are all Britons , and I am your king . WOMAN : I didn ' t know we had a k
g . WOMAN : I didn ' t know we had a king . I thought we were an autonomous co
ink he is ? Heh . ARTHUR : I am your king ! WOMAN : Well , I didn ' t vote for
 . WOMAN : Well , how d[...]<jupyter_text>We can also build a Corpus from a plain text. See the example below.<jupyter_code>from nltk.corpus import PlaintextCorpusReader
from nltk.tokenize.regexp import WordPunctTokenizer
from nltk.data import LazyLoader

trump = PlaintextCorpusReader('../datasets/','trump.txt',word_tokenizer=WordPunctTokenizer(),
             sent_tokenizer=LazyLoader(
                 'tokenizers/punkt/english.pickle'),
             encoding='utf8')
print(trump)
trump.words()<jupyter_output><PlaintextCorpusReader in '/home/kprzystalski/workshop/datasets'><jupyter_text>Corpuses contain large amount of texts, say multiple books, couple hundreds of articles from various magazines etc. NLTK has chosen a few books from over 25,000 pieces available in the whole dataset. We are going to pick Bible and Shakespeare's Macbeth in order to compare a few statistics between them. Let's take a look at other corporas and check the gutenberg and check the word distribution:<jupyter_code>import matplotlib.pyplot as plt
import seaborn as sns

sns.set_style('whitegrid') # general display style of seaborn
# how to display plot in notebook, jupyter's magic command
%matplotlib inline 

# Download necessary texts
nltk.download('gutenberg')
nltk.download('stopwords')

print("Books available in gutenberg dataset:")
nltk.corpus.gutenberg.fileids()
nltk.corpus.stopwords.words("english")
%matplotlib inline

def lexical_richness(text: str) -> float:
  # Unique words in text divided by the total amount of words
  words = nltk.Text(nltk.corpus.gutenberg.words(text))
  return len(set(words))/len(words)

macbeth = "shakespeare-macbeth.txt"
bible = "bible-kjv.txt"

print("Macbeth richness: {}".format(lexical_richness(macbeth)))
print("Bible richness: {}".format(lexical_richness(bible)))

# Removing stopwords like: , . and or
bible_words = [
    word
    for word in nltk.corpus.gutenberg.words(bible)
    if word not in nltk.corpus.stopwords.words("english") and word.isalpha()
]

words_distribution = nltk.probability.FreqDist(bible_words)
print("Words occuring most often in Bible:") 
display(words_distribution.plot(15))<jupyter_output>Macbeth richness: 0.17359550561797754
Bible richness: 0.01362385148626533
Words occuring most often in Bible:<jupyter_text>Regular expressionThe first solution that comes to our mind when it comes to text processing are regular expression. It is usually the best solution in software development. It is also used in many tools that we have mentioned before. Let's take an example to do a short recap how to use regular expressions in Python.<jupyter_code>example = "JU is a great company with many great developers."

import re

pattern = "\\s+"
words = re.split(pattern, example)
print(words)<jupyter_output>['JU', 'is', 'a', 'great', 'company', 'with', 'many', 'great', 'developers.']<jupyter_text>Python does have some methods for Strings that can do the same as above, but with less code.<jupyter_code>words = example.split(' ')
print(words)

text = "Well, what can I say about regexes? \
They are quite annoying to be honest, but once you get a grasp you're gonna be fine. \
Yeah, I could use some basic boring text, here you go if that's what you miss: \
Ala ma kota, kot ma Alę. m?ke a m1sta.e and see what happens... \
This sentence: 'Mr. & Mrs. Smith, been looking for easter egg here, failed."
text.split(' ')<jupyter_output>['JU', 'is', 'a', 'great', 'company', 'with', 'many', 'great', 'developers.']<jupyter_text>That was too naive, let's make something more intelligent:<jupyter_code>print("Using regex:")
display(re.split('(?<=[.!?]) +',text))<jupyter_output>Using regex:<jupyter_text>Our regex only splits the text if either of . ! ? is followed by one or more whitespace signs. This approach is quite powerful, yet limited (as could be seen), keep in mind it has it's uses but is far from perfect (later we will see more sophisticated approaches).All in all, when faced with a choice: do something using traditional methods or using machine learning, ALWAYS pick the first one.<jupyter_code># Examples of some other regex functions

text = re.sub(r"Mr. & Mrs. Smith", "Now works like a charm", text) #substitute string
found_text = re.findall(", been .*,", text) # find all occurences of string
display(found_text)

# findall returns list, we need to access it's element
re.sub(found_text[0], " but life", text)<jupyter_output><empty_output><jupyter_text>TokenizationNLTK is a more advanced tool than just regular expressions. We can easily tokenize sentences. Tokenization is a process of getting words from sentences. **Tokenizers** are used to divide a string into it's logical subsets. Above, we have used the sentence tokenizer. Tokenization on the other hand is a common name for dividing a text into separate words and can be done in NLTK using word_tokenizer:<jupyter_code>tokens = nltk.word_tokenize(example)
print("Tokens: " + str(tokens))<jupyter_output>Tokens: ['JU', 'is', 'a', 'great', 'company', 'with', 'many', 'great', 'developers', '.']<jupyter_text>After a little fun we can use nltk to separate text into sentences in 'smarter' fashion:<jupyter_code>from nltk.tokenize import sent_tokenize

nltk.download('punkt')

sent_tokenize(text)<jupyter_output>[nltk_data] Downloading package punkt to
[nltk_data]     /home/kprzystalski/nltk_data...
[nltk_data]   Package punkt is already up-to-date!<jupyter_text>__Tokenizers__ are used to divide a string into it's logical subsets. Above, wehave usedthe sentence tokenizer. __Tokenization__ on the other hand is a commonname for dividinga text into separate words and can be done in NLTK using__word_tokenizer__:<jupyter_code>from nltk.tokenize import word_tokenize, TweetTokenizer
from nltk.tokenize import TweetTokenizer # Yes, this tokenizer was based on Tweets

text = "This is a cooool #dummysmiley: :-) :-P <3 and some arrows < > -> <--"
print("Normal tokenizer:")
print(word_tokenize(text))

print("This tokenizer is better suited for emoji handling")
twitter_tokenizer = TweetTokenizer()
twitter_tokenizer.tokenize(text)<jupyter_output>Normal tokenizer:
['This', 'is', 'a', 'cooool', '#', 'dummysmiley', ':', ':', '-', ')', ':', '-P', '<', '3', 'and', 'some', 'arrows', '<', '>', '-', '>', '<', '--']
This tokenizer is better suited for emoji handling<jupyter_text>As you can see there are other tokenizers available and each is focused ondifferent'kind' of tokenization.They are usually implemented as a set ofcomplicated rules trailed down to certain language(e.g. will be different forpolish and english) and __ARE NOT__ a subset of machinelearning (inthis part of course at least).Here are some other NLTK's tokenizers you coulduse:- __PunktTokenizer__: been used above for sentence tokenization. This oneactually uses unsupervised algorithm to infer sentence boundaries.- __NISTTokenizer__: tokenizer for non-european text (say in Chinese).- __SExprTokenizer__: find parenthesized expressions in strings, see below:<jupyter_code>from nltk.tokenize import SExprTokenizer

SExprTokenizer().tokenize('(a b (c d)) e f (g)')<jupyter_output><empty_output><jupyter_text>TaggingWhat is great with NLTK is that we can tag each word. A tag give us a meaning of a word. We can think about it as a type of word.<jupyter_code>tags = nltk.pos_tag(tokens)
print("Tagged: " + str(tags))<jupyter_output>Tagged: [('JU', 'NNP'), ('is', 'VBZ'), ('a', 'DT'), ('great', 'JJ'), ('company', 'NN'), ('with', 'IN'), ('many', 'JJ'), ('great', 'JJ'), ('developers', 'NNS'), ('.', '.')]<jupyter_text>Some tags are shown in the table below.| tag  | short | description   ||------|-------|---------------|| DT   | determiner | all an another any both del each either every half la many much nary neither no some such that the them these this those || IN   | preposition or conjunction, subordinating | astride among uppon whether out inside pro despite on by throughout below within for towards near behind atop around if like until below next into if beside ... || JJ   | adjective or numeral, ordinal | third ill-mannered pre-war regrettable oiled calamitous first separable ectoplasmic battery-powered participatory fourth still-to-be-named multilingual multi-disciplinary ... || NN   | noun, common, singular or mass | common-carrier cabbage knuckle-duster Casino afghan shed thermostat investment slide humour falloff slick wind hyena override subhumanity machinist ... || NNP  | noun, proper, singular | <NAME> <NAME> Conchita Trumplane <NAME> <NAME> <NAME> ODI Darryl CTCA Shannon A.K.C. Meltex Liverpool ... || NNS  | noun, common, plural | undergraduates scotches bric-a-brac products bodyguards facets coasts divestitures storehouses designs clubs fragrances averages subjectivists apprehensions muses factory-jobs ... || VBZ  | verb, present tense, 3rd person singular | bases reconstructs marks mixes displeases seals carps weaves snatches slumps stretches authorizes smolders pictures emerges stockpiles seduces fizzes uses bolsters slaps speaks pleads ... |To get the full list of tags, use the following code:<jupyter_code>nltk.download('tagsets')
nltk.help.upenn_tagset()<jupyter_output>$: dollar
    $ -$ --$ A$ C$ HK$ M$ NZ$ S$ U.S.$ US$
'': closing quotation mark
    ' ''
(: opening parenthesis
    ( [ {
): closing parenthesis
    ) ] }
,: comma
    ,
--: dash
    --
.: sentence terminator
    . ! ?
:: colon or ellipsis
    : ; ...
CC: conjunction, coordinating
    & 'n and both but either et for less minus neither nor or plus so
    therefore times v. versus vs. whether yet
CD: numeral, cardinal
    mid-1890 nine-thirty forty-two one-tenth ten million 0.5 one forty-
    seven 1987 twenty '79 zero two 78-degrees eighty-four IX '60s .025
    fifteen 271,124 dozen quintillion DM2,000 ...
DT: determiner
    all an another any both del each either every half la many much nary
    neither no some such that the them these this those
EX: existential there
    there
FW: foreign word
    gemeinschaft hund ich jeux habeas <NAME> K'ang-si vous
    lutihaw alai je jour objets salutaris fille quibusdam pas trop Monte
    terram fiche oui corporis ...
IN: preposition or[...]<jupyter_text>We can find the token in NLTK's corporas too:<jupyter_code>text6.tokens<jupyter_output><empty_output><jupyter_text>Another feature of NLTK are the stopwords that are used in each language that NLTK supports:<jupyter_code>english_stopwords = set(nltk.corpus.stopwords.words('english'))
print(english_stopwords)<jupyter_output>{'himself', 'we', 'our', 'under', 'here', 'more', 'itself', "haven't", 'couldn', 'will', 'why', 'to', 'who', 'having', 'am', "mustn't", 'up', 'too', "shan't", "that'll", 'does', 'through', "aren't", 'have', 'but', 'when', 'needn', 'y', 'all', 'he', 'shan', 'by', 'from', 't', "mightn't", "shouldn't", "she's", 'wasn', 'no', 'o', 'shouldn', 'once', 'they', 'those', "you're", 'haven', 'll', 'yourself', 'in', 'their', 'if', 'most', "wouldn't", 'or', 'hadn', 'during', "didn't", "doesn't", 'my', 'can', "couldn't", 'these', 'wouldn', 'been', "it's", 'be', 'until', 'you', 'is', 'a', 'own', 'your', 'd', 'them', 'she', 'for', 'hers', 'isn', 'before', 'myself', 'i', 'did', "hasn't", 'mightn', 'there', 'some', 'how', "don't", 'theirs', "wasn't", 'which', 'had', 'further', 'so', 'below', 'hasn', 'are', 'being', 'do', 'it', 'any', 'same', 'its', 'into', 'on', 'whom', 'aren', 'against', 'other', 'doesn', 'while', 'with', "hadn't", 'down', 'above', 'was', 'don', 'each', 'should', 've', 'just', 'themsel[...]<jupyter_text>__POS-tagging__ is another method of describing our text. For each tokenized word we can obtain it's __Part Of Speech__. Those informations are frequently used throughout more advanced machine learning methods.Up until now everything was easy, fortunately this one is no exception.<jupyter_code># we have already imported nltk.tokenize.word_tokenizer

nltk.download('averaged_perceptron_tagger')

words = word_tokenize(
    "Another text. You know, this is the hardest part, coming up with all those texts. "
    "Lorem Ipsum or some random stuff is pretty boring IMO. "
    "You may feel like somebody didn't care (or at least I do)."
    "Or maybe it's just an act? I will not address this question."
)

nltk.pos_tag(words)<jupyter_output>[nltk_data] Downloading package averaged_perceptron_tagger to
[nltk_data]     /home/kprzystalski/nltk_data...
[nltk_data]   Package averaged_perceptron_tagger is already up-to-
[nltk_data]       date!<jupyter_text>Look at `act` and `address` towards the end of output.POS can be used to dismabiguate word meaning, `act` can be both noun (NN) and verb (VB), same goes for address. This tokenizer is __based on simple one layer neural network__ hence it takes context into account (words before and after an act). Context (N words before and after certain word) is called __N-gram__ in Machine Learning and we will make use of it more later in this course.This is more sophisticated approach and __should be__ used when more abstract reasoning is needed (like word disambiguation).With __NLTK__ we can create and train our own taggers, or choose a pre-made one and train is as seen below (this one takes the next word as it's context).<jupyter_code>import nltk
from nltk.corpus import brown 
nltk.download("brown")
 
brown_tagged_sents = brown.tagged_sents(categories="news")
brown_sents = brown.sents(categories="news") 
unigram_tagger = nltk.BigramTagger(brown_tagged_sents)
unigram_tagger.tag(brown_sents[2007])<jupyter_output>[nltk_data] Downloading package brown to
[nltk_data]     /home/kprzystalski/nltk_data...
[nltk_data]   Package brown is already up-to-date!<jupyter_text>UnicodeEach alphabet is different and some languages use letter that are not used in any other language. Unicode allows use to standardize the text. Let's see an of some german and portugese text below.<jupyter_code>sample = """Min<NAME>. O português é uma língua romântica"""

type(sample)<jupyter_output><empty_output><jupyter_text>If we convert it to ASCII it will look as following:<jupyter_code>ascii(sample)<jupyter_output><empty_output><jupyter_text>It is not what we expected. It's hard to work with such a text. We can go a step further and encode the string to be a unicode encoded one:<jupyter_code>sample.encode('utf8')<jupyter_output><empty_output><jupyter_text>We could also ignore the ASCII characters that we cannot do anything with in the further processing:<jupyter_code>sample.encode('ascii', 'ignore')<jupyter_output><empty_output><jupyter_text>Now we have all letter that make sense, but we have text that does not make any sense anymore. To solve it, we need to normalize it and only after the normalization remove the ascii signs.<jupyter_code>import unicodedata

unicodedata.normalize('NFKD', sample).encode('ascii','ignore')<jupyter_output><empty_output><jupyter_text>More about [Unicode normalization forms](http://unicode.org/reports/tr15/)  LemmatizationLemmatization is a process of getting one word for many forms of the word. Let's see an example:<jupyter_code>from nltk.stem import WordNetLemmatizer

wordnet_lemmatizer = WordNetLemmatizer()

print(wordnet_lemmatizer.lemmatize('do',pos='v'))
print(wordnet_lemmatizer.lemmatize('does',pos='v'))
print(wordnet_lemmatizer.lemmatize('doing',pos='v'))<jupyter_output>do
do
do<jupyter_text>We can set the main part of speech as one of the following:ADJ, ADJ_SAT, ADV, NOUN, VERB = 'a', 's', 'r', 'n', 'v'<jupyter_code>wordnet_lemmatizer.lemmatize('are',pos='v')<jupyter_output><empty_output><jupyter_text>StemmingStemming is similar to lemmatization, but the main difference is that is just reduce the word to it root. It gives in many cases different results than lemmatization as the second solution is based on part of speech.<jupyter_code>from nltk import PorterStemmer, LancasterStemmer, word_tokenize
from nltk.stem.snowball import SnowballStemmer

sample = "This is a new training about machine learning usage for chatbots"

tokens = word_tokenize(sample)

porter = PorterStemmer()
p_stem = [porter.stem(t) for t in tokens]
print(p_stem)

lancaster = LancasterStemmer()
l_stem = [lancaster.stem(t) for t in tokens]
print(l_stem)

snowball = SnowballStemmer('english')
s_stem = [snowball.stem(t) for t in tokens]
print(s_stem)<jupyter_output>['thi', 'is', 'a', 'new', 'train', 'about', 'machin', 'learn', 'usag', 'for', 'chatbot']
['thi', 'is', 'a', 'new', 'train', 'about', 'machin', 'learn', 'us', 'for', 'chatbot']
['this', 'is', 'a', 'new', 'train', 'about', 'machin', 'learn', 'usag', 'for', 'chatbot']<jupyter_text>As per usual there are other stemmers and lemmatizers available in NLTK. Someof them may be faster (PorterStemmer), but Snowball is considered de-factostandard.A few traits of each should be noticed:- Stemming is faster (itdoesn't have to consult dictionary and complicatedmorphological rules).-Stemming may not produce a dictionary word, see `studi` above.- Lemmatizationproduces smaller set of words. It is useful as we do not have to keeptrack ofall the different form of word (running, runner, runs etc.). More memoryefficient.- Lemmatization is not as crude, when it does not find the lemma itreturns the originalform. It __may__ produce more words this way, depends onthe kind of text we areprocessing.  Sentence extractionIn this section we show some features of SpaCy. One of the most commonly used is the sentence extraction. An example of Trump speech divided into sentences can be found below.<jupyter_code>import spacy

file = open("../datasets/trump.txt", "r",encoding='utf-8') 
trump = file.read() 

nlp = spacy.load("en_core_web_sm")
doc = nlp(trump)

for span in doc.sents:
    print("> ", span)<jupyter_output>>  Thank you very much.
>  Mr. Speaker, Mr. Vice President, Members of Congress, the First Lady of the United States, and citizens of America: Tonight, as we mark the conclusion of our celebration of Black History Month, we are reminded of our Nation's path towards civil rights and the work that still remains to be done.
>  Recent threats targeting Jewish community centers and vandalism of Jewish cemeteries, as well as last week's shooting in Kansas City, remind us that while we may be a nation divided on policies, we are a country that stands united in condemning hate and evil in all of its very ugly forms.
>  

>  Each American generation passes the torch of truth, liberty, and justice in an unbroken chain, all the way down to the present.
>  That torch is now in our hands, and we will use it to light up the world.
>  I am here tonight to deliver a message of unity and strength, and it is a message deeply delivered from my heart.
>  A new chapter of American greatness is now beginnin[...]<jupyter_text>Each sentence can next be tokenized and we can get a tag and pos of it.<jupyter_code>for span in doc.sents:
    for i in range(span.start, span.end):
        token = doc[i]
        print(i, token.text, token.tag_, token.pos_)<jupyter_output>0 Thank VBP VERB
1 you PRP PRON
2 very RB ADV
3 much RB ADV
4 . . PUNCT
5 Mr. NNP PROPN
6 Speaker NNP PROPN
7 , , PUNCT
8 Mr. NNP PROPN
9 Vice NNP PROPN
10 President NNP PROPN
11 , , PUNCT
12 Members NNS NOUN
13 of IN ADP
14 Congress NNP PROPN
15 , , PUNCT
16 the DT DET
17 First NNP PROPN
18 Lady NNP PROPN
19 of IN ADP
20 the DT DET
21 United NNP PROPN
22 States NNP PROPN
23 , , PUNCT
24 and CC CCONJ
25 citizens NNS NOUN
26 of IN ADP
27 America NNP PROPN
28 : : PUNCT
29 Tonight NN NOUN
30 , , PUNCT
31 as IN ADP
32 we PRP PRON
33 mark VBP VERB
34 the DT DET
35 conclusion NN NOUN
36 of IN ADP
37 our PRP$ PRON
38 celebration NN NOUN
39 of IN ADP
40 Black NNP PROPN
41 History NNP PROPN
42 Month NNP PROPN
43 , , PUNCT
44 we PRP PRON
45 are VBP AUX
46 reminded VBN VERB
47 of IN ADP
48 our PRP$ PRON
49 Nation NN NOUN
50 's POS PART
51 path NN NOUN
52 towards IN ADP
53 civil JJ ADJ
54 rights NNS NOUN
55 and CC CCONJ
56 the DT DET
57 work NN NOUN
58 that WDT DET
59 still RB ADV
60 remains VBZ V[...]<jupyter_text>Noun chunksChunking is a process of getting just specific part of speeches. The most popular is the noun chunking. It allow us to get the general meaning of what is the text about.<jupyter_code>for np in doc.noun_chunks:
    print(np)<jupyter_output>you
Mr. Vice President
Members
Congress
the First Lady
the United States
citizens
America
Tonight
we
the conclusion
our celebration
Black History Month
we
our Nation's path
civil rights
the work
Recent threats
Jewish community centers
vandalism
Jewish cemeteries
last week's shooting
Kansas City
us
we
a nation
policies
we
a country
hate
evil
its very ugly forms
Each American generation
the torch
truth
liberty
justice
an unbroken chain
the present
That torch
our hands
we
it
the world
I
a message
unity
strength
it
a message
my heart
A new chapter
American greatness
A new national pride
our Nation
a new surge
optimism
impossible dreams
our grasp
What
we
the renewal
the American spirit
Our allies
America
All the nations
the world
friend
foe
America
America
America
9 years
the United States
the 250th anniversary
our founding
the day
we
our independence
It
the great milestones
the history
the world
what
America
we
our 250th year
What kind
country
we
our children
I
the mistakes
recent decades
[...]<jupyter_text>Named entity recognitionThe last feature that we show here is the named entitiy resolution. It returns a NER of each word.<jupyter_code>for entity in doc.ents:
    print(entity.label_, entity.text)<jupyter_output>PERSON Speaker
ORG Congress
ORDINAL First
GPE the United States
GPE America
TIME Tonight
EVENT Black History Month
ORG Nation
NORP Jewish
NORP Jewish
DATE last week's
GPE Kansas City
NORP American
TIME tonight
NORP American
DATE today
NORP American
GPE America
GPE America
GPE America
GPE America
DATE 9 years
GPE the United States
ORDINAL 250th
DATE 250 years
DATE the day
GPE America
ORDINAL 250th
DATE recent decades
CARDINAL one
GPE Chicago
GPE Baltimore
GPE Detroit
CARDINAL trillions
MONEY trillions of dollars
DATE 2016
LOC Earth
CARDINAL thousands
CARDINAL one
GPE America
GPE America
NORP American
ORG Inauguration
ORG Ford
ORG Fiat Chrysler
ORG General Motors
GPE Sprint
ORG Softbank
ORG Lockheed
ORG Intel
MONEY billions and billions of dollars
GPE the United States
CARDINAL tens of thousands
NORP American
MONEY almost $3 trillion
DATE November 8
MONEY hundreds of millions of dollars
PRODUCT F-35
CARDINAL billions
DATE 5-year
PRODUCT Task Force
CARDINAL one
CARDINAL two
ORG Keystone
O[...]<jupyter_text>Bag of WordsMany machine learning methods cannot use strings as features, we have to encode it using numbers.We can easily do this using __Bag Of Words (BOW)__ technique and marvelous __sklearn__ library:<jupyter_code>from sklearn.feature_extraction.text import CountVectorizer

vectorizer = CountVectorizer()

corpus = [
     'Bag Of Words is based on counting',
     'words occurences throughout multiple documents.',
     'This is the third document.',
     'As you can see most of the words occur only once.',
     'This gives us a pretty sparse matrix, see below. Really, see below',
]

X = vectorizer.fit_transform(corpus)
print(X.toarray())<jupyter_output>[[0 1 1 0 0 1 0 0 0 1 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 1 0]
 [0 0 0 0 0 0 0 1 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 1 0]
 [0 0 0 0 0 0 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 0]
 [1 0 0 0 1 0 0 0 0 0 0 1 0 1 0 1 0 1 1 0 0 1 0 1 0 0 0 0 1 1]
 [0 0 0 2 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 1 1 2 1 0 0 1 0 1 0 0]]<jupyter_text>Each document is represented by the row. Values ranging from 0 to N represent whether and how many times the word occured in the document. You can see what word corresponds to which column by issuing get_feature_names() on vectorizer object.<jupyter_code>vectorizer.get_feature_names()<jupyter_output><empty_output><jupyter_text>This approach allows us to easily describe the whole corpora, but it lacks informations crucial for solving some tasks. TF-IDFLook at word is. It is used in most documents many times, yet it does not tell us anything about them. Let's think about sentiment analysis: if words like great or awesome occur frequently in comparison with another documents it may suggest positive attitude.TF-IDF is one way to encode this information and I'll walk you through it step by step.First part of TF-IDF is, yes, you guessed it, TF, which means Term Frequency. It can be calculated as:\begin{equation}tf_{ij}=\frac{n_{ij}}{\sum n_{ij}},\end{equation}where $n_{ij}$ is the number of occurence of word $i$ in document $j$.<jupyter_code>import numpy as np

corpus = [
'Tom has cat',
'Tom has fish',
'Tom is polish',
]

def tf(corpus):
    vec = CountVectorizer()
    bow_representation = vec.fit_transform(corpus)
    words_per_corpus = bow_representation.sum(axis=1)
    return np.divide(np.array(bow_representation.toarray()),np.array(words_per_corpus).reshape((5,))[:,None])<jupyter_output><empty_output><jupyter_text>For each document we count how many times it occurred (BoW implementaion) and divide by the count of all words in this document.Next part is IDF, which stands for Inverse Document Frequency:\begin{equation}idf=\log(\frac{N}{df_{t}}),\end{equation}where $N$ is the total number of documents and $df_{t}$ is number of documents containing $t$.<jupyter_code>def idf(corpus):
    document_count = len(corpus)
    bow_representation = CountVectorizer().fit_transform(corpus)
    return np.log(document_count / np.count_nonzero(bow_representation.toarray(), axis=0))<jupyter_output><empty_output><jupyter_text>First we calculate number of documents in corpus (number of rows in our case). Next, for each word, we calculate documents containing said word at least once.Taking logarithm allows us to dampen the effect of idf. For example, the difference between term occuring in 40 out of 50 documents and 45 out of 50 documents will be smaller than difference between 1/50 and 5/50. This puts a bigger emphasis on rarely occuring terms as they are more informative.Finally, for the whole thing to work, we simply multiply both:<jupyter_code>def tf_idf(corpus):
    return tf(corpus) * idf(corpus)<jupyter_output><empty_output><jupyter_text>Let's calculate it:<jupyter_code>corpus = [
     'Bag Of Words is based on counting',
     'words occurences throughout multiple documents.',
     'This is the third document.',
     'As you can see most of the words occur only once.',
     'This gives us a pretty sparse matrix, see below. Really, see below',
]

tfidf_result = tf_idf(corpus)

print(tfidf_result.shape)<jupyter_output>(5, 30)<jupyter_text>In Jupyter it's easier to display results with pandas:<jupyter_code>import pandas as pd
pd.DataFrame(tfidf_result).head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09399702331734744, 'fraction_numerical': 0.015544898296676038, 'mean_word_length': 4.722180166540499, 'pattern_counts': {'":': 0, '<': 146, '<?xml version=': 0, '>': 153, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 40, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '38', 'dir': 'jupyter-structured-clean-dedup', 'id': '529030', 'license': 'MIT', 'n_tokens_mistral': 10248, 'n_tokens_neox': 9531, 'n_words': 3947, 'path': 'ML1/nlp/102_NLP_Text_Processing.ipynb', 'repo_name': 'DevilWillReign/ML2022', 'size': '30347'}