<jupyter_start><jupyter_text>NLTK exampleExample on how to use natural language processing with python's NLTK package. NLTK is a tool kit for cleaning and prepping the data for analysis. http://www.nltk.org/Info on NLTK: http://www.nltk.org/book/<jupyter_code>import nltk<jupyter_output><empty_output><jupyter_text>Below will bring up a pop-up and you need to select all to download all packages.This downloaded stuff to /Users/korolo/nltk-dataThis does not need to be run again.<jupyter_code># nltk.download()
# this has been downloaded<jupyter_output><empty_output><jupyter_text>Tokenizing<jupyter_code>#from nltk.tockenize import sent_tockenize, word_tockenize
example_text = "This is a sentence number one. This is another sentence. Text could be boring sometimes. " + \
    "Don't read this! C. elegans is a small worm. Someone loves Mr. Johnson song by J.A.I.N."
nltk.sent_tokenize(example_text)
nltk.word_tokenize(example_text)<jupyter_output><empty_output><jupyter_text>Stop words removal<jupyter_code>from nltk.corpus import stopwords
example_text = "This is a sentence number one. This is another sentence. Text could be boring sometimes. " + \
    "Don't read this! <NAME>ans is a small worm. Someone loves Mr. Johnson song by J.A.I.N."
stop_words = set(stopwords.words("english"))
stop_words
# Can add other custom stop words here
words = nltk.word_tokenize(example_text)
words
filtered_text = []
for w in words:
    if w not in stop_words:
        filtered_text.append(w)
#filtered_text
## OR
filtered_text = [w for w in words if w not in stop_words]
filtered_text<jupyter_output><empty_output><jupyter_text>StemmingStemming is used to get to the root of the word. Used to get to the meaning.Stemming as below might not be necessary as a separate step in a text mining pipeline, since usually you would feed words through nltk and use wordnet, i.e. use lemmatazing<jupyter_code>from nltk.stem import PorterStemmer
ps = PorterStemmer()
example_words = ["sample", "sampling", "sampled"]
for w in example_words:
    print(ps.stem(w))
example_text ='''We have sampled a rather small C. elegans worm and were determined to find \\
out what unique DNA sample resulted from this sampling.'''
words = nltk.word_tokenize(example_text)
words
for w in words:
    print(ps.stem(w))<jupyter_output>We
have
sampl
a
rather
small
C.
elegan
worm
and
were
determin
to
find
\out
what
uniqu
dna
sampl
result
from
thi
sampl
.<jupyter_text>Part of speach taggingPOS tagging, or simply tagging, is known as breaking down the text into word classes, or lexical categories, or part-of-speach.For information on tagging, see http://www.nltk.org/book/ch05.htmlAlso, see a separate notebook, pos_tagging.ipynb for more detailed examples.<jupyter_code>from nltk.corpus import gutenberg
from nltk.tokenize import PunktSentenceTokenizer # unsupervied trained tokenizer. Can train it myself.
tokenizer_training_set = gutenberg.raw("austen-persuasion.txt")
sense_and_sensibility_text = gutenberg.raw("austen-sense.txt")

custom_austen_tokenizer = nltk.PunktSentenceTokenizer(tokenizer_training_set)
custom_austen_tokenizer
# This gives an error => gutenberg is not a tagged corpus
# gutenberg.tagged_words()
tokenised_sense_and_sensibility = custom_austen_tokenizer.tokenize(sense_and_sensibility_text)
tokenised_sense_and_sensibility
len(tokenised_sense_and_sensibility)
tokenised_sense_and_sensibility[2]
def process_text(text):
    tagged_text = []
    try:
        for s in text:
            words = nltk.word_tokenize(s)
            tagged = nltk.pos_tag(words)
            tagged_text.append(tagged)
    except Exception as e:
        print(e)
    
    return tagged_text
        
tagged_text = process_text(tokenised_sense_and_sensibility)<jupyter_output><empty_output><jupyter_text>The default pos_tag() model  is maxent_treebanck_pos_tagger, found in ~/nltk_data/taggers/. COde in nltk-master/nltk/tag/__init__.pyThe tagged training set would be a .txt file with the ext broken up one word per line and each line looking like this:    word POS-tag BIO-tag<jupyter_code>tagged_text<jupyter_output><empty_output><jupyter_text>To see the pos (part-of-speach) tagset and what the tags mean, do this<jupyter_code>nltk.help.upenn_tagset()<jupyter_output>$: dollar
    $ -$ --$ A$ C$ HK$ M$ NZ$ S$ U.S.$ US$
'': closing quotation mark
    ' ''
(: opening parenthesis
    ( [ {
): closing parenthesis
    ) ] }
,: comma
    ,
--: dash
    --
.: sentence terminator
    . ! ?
:: colon or ellipsis
    : ; ...
CC: conjunction, coordinating
    & 'n and both but either et for less minus neither nor or plus so
    therefore times v. versus vs. whether yet
CD: numeral, cardinal
    mid-1890 nine-thirty forty-two one-tenth ten million 0.5 one forty-
    seven 1987 twenty '79 zero two 78-degrees eighty-four IX '60s .025
    fifteen 271,124 dozen quintillion DM2,000 ...
DT: determiner
    all an another any both del each either every half la many much nary
    neither no some such that the them these this those
EX: existential there
    there
FW: foreign word
    gemeinschaft hund ich jeux habeas Haementeria Herr K'ang-si vous
    lutihaw alai je jour objets salutaris fille quibusdam pas trop Monte
    terram fiche oui corporis ...
IN: preposition or[...]<jupyter_text>Here is a short reference of the above command:`CC: conjunction, coordinatingCD: numeral, cardinalDT: determinerEX: existential thereFW: foreign wordIN: preposition or conjunction, subordinatingJJ: adjective or numeral, ordinalJJR: adjective, comparativeJJS: adjective, superlativeLS: list item markerMD: modal auxiliaryNN: noun, common, singular or massNNP: noun, proper, singularNNPS: noun, proper, pluralNNS: noun, common, pluralPDT: pre-determinerPOS: genitive markerPRP: pronoun, personalPRP$: pronoun, possessiveRB: adverbRBR: adverb, comparativeRBS: adverb, superlativeRP: particleSYM: symbolTO: "to" as preposition or infinitive markerUH: interjectionVB: verb, base formVBD: verb, past tenseVBG: verb, present participle or gerundVBN: verb, past participleVBP: verb, present tense, not 3rd person singularVBZ: verb, present tense, 3rd person singularWDT: WH-determinerWP: WH-pronoun    WP$: WH-pronoun, possessiveWRB: Wh-adverb`Tagged corpora use many different conventions for tagging words. Universal part-of-speach tagset is the simplified one. The one above is not universal, it is unsimplified. For more info see: http://www.nltk.org/book/ch05.html  ChunkingFiguring out the meaning of the sentense.What we do is combining part-of-speach tagging and regular expressions.We will chunk into noun phrases (noun and descriptive group of words surrounding the noun (should be touching each other))  First, a small example on how chunking works<jupyter_code>sample_text = '''The purpose of this assessment is to evaluate the plant health risk of pests associated with the \
importation of organic wheat (Triticum aestivum, T. durum), including triticale (xTriticosecale) (hereafter referred \
to as wheat unless addressed explicitly) from the Republic of Kazakhstan (Kazakhstan). This document is the risk \
assessment portion of the pest risk analysis process. The method used by the CFIA to initiate and conduct this pest \
risk assessment is consistent with international guidelines (IPPC 2016).\
This assessment has considered the pest status of bacteria and phytoplasmas, fungi, viruses, nematodes, insects, \
mites, molluscs and weeds reported for organic wheat in this region. Appendix 1 provides a list of pests potentially \
associated with organic wheat from Kazakhstan.'''
sample_text<jupyter_output><empty_output><jupyter_text>Tokenize text into words and tag the words as pos<jupyter_code>words = nltk.word_tokenize(sample_text)
tagged_words = nltk.pos_tag(words)
tagged_words<jupyter_output><empty_output><jupyter_text>Use regex to chunk the text Note that the following exception will occur if the regex string is not matched to anything in the texts:ValueError: Expected stage marker (eg NP:)<jupyter_code>chunk_expr = r'''Chunk: {<NN.?><NN.?>}'''
chunk_parser = nltk.RegexpParser(chunk_expr)
chunked = chunk_parser.parse(tagged_words)
chunked.draw()
#chunked<jupyter_output><empty_output><jupyter_text>Now a function that takes <NAME>'s tokenized "Sense and Sensibility" and does chunking.<jupyter_code>def chunk_tokenized_text(text):
    ''' text: should be tockenized
    '''
    chunked_text = []
    try:
        for s in text:
            words = nltk.word_tokenize(s)
            tagged = nltk.pos_tag(words)
            
            # Any adverb 0 or more, followed by a verb (0 or more) followed by required noun
            chunk_gram = r'''Chunk: {<RB.?>*<VB.?>*<NNP>+<NN>?}'''
            chunk_parser = nltk.RegexpParser(chunk_gram)
            chunked = chunk_parser.parse(tagged)
            
            chunked_text.append(chunked)
    except Exception as e:
        print(e)
    
    return chunked_text

chunked_sens_and_sensibility = chunk_tokenized_text(tokenised_sense_and_sensibility)


len(chunked_sens_and_sensibility)
chunked_sens_and_sensibility[0].draw()<jupyter_output><empty_output><jupyter_text>ChinkingChink smth from a chunk. I.e. chunk everything but smth....Chink is defined as anything between }{ , instead of {} for a chunk.<jupyter_code>sample_text = '''The purpose of this assessment is to evaluate the plant health risk of pests associated with the \
importation of organic wheat (Triticum aestivum, T. durum), including triticale (xTriticosecale) (hereafter referred \
to as wheat unless addressed explicitly) from the Republic of Kazakhstan (Kazakhstan). This document is the risk \
assessment portion of the pest risk analysis process. The method used by the CFIA to initiate and conduct this pest \
risk assessment is consistent with international guidelines (IPPC 2016).\
This assessment has considered the pest status of bacteria and phytoplasmas, fungi, viruses, nematodes, insects, \
mites, molluscs and weeds reported for organic wheat in this region. Appendix 1 provides a list of pests potentially \
associated with organic wheat from Kazakhstan.'''
words = nltk.word_tokenize(sample_text)
tagged_words = nltk.pos_tag(words)

chunk_expr = r'''Chunk: {<.*>+}
            }<VB.?|IN|DT|TO>{'''       ##  <=== this is where chinking is happening

chunk_parser = nltk.RegexpParser(chunk_expr)
chunked = chunk_parser.parse(tagged_words)
chunked.draw()<jupyter_output><empty_output><jupyter_text>Named Entity RecognitionNamed entities that will bediscovered depend on the corpus on which the chunker (ne_chunk) was trained. Note that false positive rate for NER with nltk is high, and general error rate is high, so might not be a good option for most cases. Description of the entities discoveredThere doesn't seem to be a command or a full list of entities that are used by the chunker, but there is a website with the main ones:http://www.nltk.org/book/ch07.htmltab-ne-typesHere they are:`ORGANIZATION Georgia-Pacific Corp., WHO PERSON       <NAME>, President Obama LOCATION     Murray River, Mount Everest DATE         June, 2008-06-29 TIME         two fifty a m, 1:30 p.m. MONEY        175 million Canadian Dollars, GBP 10.40 PERCENT      twenty pct, 18.75 % FACILITY     Washington Monument, Stonehenge GPE          South East Asia, Midlothian `<jupyter_code>sample_text = '''The purpose of this assessment is to evaluate the plant health risk of pests associated with the \
importation of organic wheat (Triticum aestivum, T. durum), including triticale (xTriticosecale) (hereafter referred \
to as wheat unless addressed explicitly) from the Republic of Kazakhstan (Kazakhstan). This document is the risk \
assessment portion of the pest risk analysis process. The method used by the CFIA to initiate and conduct this pest \
risk assessment is consistent with international guidelines (IPPC 2016).\
This assessment has considered the pest status of bacteria and phytoplasmas, fungi, viruses, nematodes, insects, \
mites, molluscs and weeds reported for organic wheat in this region. Appendix 1 provides a list of pests potentially \
associated with organic wheat from Kazakhstan.'''
words = nltk.word_tokenize(sample_text)
tagged_words = nltk.pos_tag(words)

named_entities = nltk.ne_chunk(tagged_words)
#named_entity = nltk.ne_chunk(tagged_words, binary = True)  # if you don't want LOCATION, GPE, etc. displayed, but just entities
named_entities.draw()<jupyter_output><empty_output><jupyter_text>Selecting only specific named entities (ex. just geographic locations)<jupyter_code>for subtree in named_entities.subtrees(filter=lambda x: x.label() == 'GPE'):
    print(subtree.leaves())<jupyter_output>[('Kazakhstan', 'NNP')]
[('Kazakhstan', 'NNP')]
[('Kazakhstan', 'NNP')]<jupyter_text>LemmatizingSimilar to stemming, but returns an actual word, not just root. By default works with nouns. If using other part of speach, use parameters. I.e. when lemmatizing a text, will need to use pos tagging and then feed the words with tags to lemmatizer.<jupyter_code>from nltk.stem import WordNetLemmatizer

lemmatizer = WordNetLemmatizer()

print(lemmatizer.lemmatize("bacteria"))   ## default of lemmatizer.lemmatize("bacteria", pos='n')
print(lemmatizer.lemmatize("bacterium"))
print(lemmatizer.lemmatize("mice"))
print(lemmatizer.lemmatize("better", pos='a'))
print(lemmatizer.lemmatize("best", pos='a'))
print(lemmatizer.lemmatize("running", pos='v'))
print(lemmatizer.lemmatize("were", pos='v'))<jupyter_output>bacteria
bacterium
mouse
good
best
run
be<jupyter_text>NLTK Corpora and viewing NLTK parametersInstalled on ~/nltk_data. See the data on the file system. NLTK has some capability to handle csv data. Otherwise, use pandas.Within NLTK, the corpus with the biggest capabilities is WordNet.Some NLTK corpora is tagged for part-of-speach tagging. You can use these corpora for training of the pos taggers. If the corpus is pos tagged, then you can do the following:`nltk.corpus..tagged_words()nltk.corpus..tagged_words(tagset='universal')`Not all corpora have the same tagset. See tagset help and readme for details.  See where nltk is installed<jupyter_code>nltk.__file__<jupyter_output><empty_output><jupyter_text>To see where nltk data is, in the nltk install dir (above), locate script called "data" and see the code for you OS. Default dir is there.  WordNet corpus<jupyter_code>from nltk.corpus import wordnet<jupyter_output><empty_output><jupyter_text>Check corpus's info (readme)<jupyter_code>nltk.corpus.wordnet.readme()<jupyter_output><empty_output><jupyter_text>Finding synonyms<jupyter_code>synonyms = wordnet.synsets("cloud")
synonyms
# Just the word
synonyms[6].lemmas()[0].name()
synonyms[0].definition()
synonyms[6].examples()
#### Finding all synonyms and antonyms with lemmas
all_synonyms = []
all_antonyms = []
for synset in wordnet.synsets("good"):
    for lemma in synset.lemmas():
        all_synonyms.append(lemma.name())
        if lemma.antonyms():
            for antonm in lemma.antonyms():
                all_antonyms.append(antonm.name())
            
set(all_synonyms)
set(all_antonyms)<jupyter_output><empty_output><jupyter_text>Semantic similarityUse synsets<jupyter_code>w1 = wordnet.synset("bacteria.n.01")
w2 = wordnet.synset("virus.n.01")

## Use Wu and Palmer semantic similarity score
w1.wup_similarity(w2)
## Does not work well on scientific identification
w1 = wordnet.synset("wheat.n.01")
w2 = wordnet.synset("triticum.n.01")
#w2 = wordnet.synset("durum.n.01")  ## better with durum
w1.wup_similarity(w2)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08413384050029012, 'fraction_numerical': 0.006511507962091419, 'mean_word_length': 4.916094584286804, 'pattern_counts': {'":': 0, '<': 93, '<?xml version=': 0, '>': 93, 'https://': 0, 'lorem ipsum': 0, 'www.': 5, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '434285', 'license': 'MIT', 'n_tokens_mistral': 5094, 'n_tokens_neox': 4780, 'n_words': 1653, 'path': 'nltk.ipynb', 'repo_name': 'oxyko/text-mining', 'size': '15521'}