<jupyter_start><jupyter_text>Natural Language Processing The essence of Natural Language Processing lies in making computers understand the natural language. That’s not an easy task though. Computers can understand the structured form of data like spreadsheets and the tables in the database, but human languages, texts, and voices form an unstructured category of data, and it gets difficult for the computer to understand it, and there arises the need for Natural Language Processing.  Importing Libraries<jupyter_code>import os
import nltk
import nltk.corpus
print(os.listdir(nltk.data.find("corpora")))
from nltk.corpus import brown
brown.words()
nltk.corpus.gutenberg.fileids()
hamlet = nltk.corpus.gutenberg.words('shakespeare-hamlet.txt')
hamlet
for word in hamlet[:500]:
    print(word, sep = ' ', end = ' ')<jupyter_output>[ The Tragedie of Hamlet by <NAME> 1599 ] Actus Primus . Scoena Prima . Enter Barnardo and Francisco two Centinels . Barnardo . Who ' s there ? Fran . Nay answer me : Stand & vnfold your selfe Bar . Long liue the King Fran . Barnardo ? Bar . He Fran . You come most carefully vpon your houre Bar . ' Tis now strook twelue , get thee to bed Francisco Fran . For this releefe much thankes : ' Tis bitter cold , And I am sicke at heart Barn . Haue you had quiet Guard ? Fran . Not a Mouse stirring Barn . Well , goodnight . If you do meet Horatio and Marcellus , the Riuals of my Watch , bid them make hast . Enter Horatio and Marcellus . Fran . I thinke I heare them . Stand : who ' s there ? Hor . Friends to this ground Mar . And Leige - men to the Dane Fran . Giue you good night Mar . O farwel honest Soldier , who hath relieu ' d you ? Fra . Barnardo ha ' s my place : giue you goodnight . Exit Fran . Mar . Holla Barnardo Bar . Say , what is Horatio there ? Hor . A peece of him Bar [...]<jupyter_text>Tokenization Tokenization is one of the most common tasks when it comes to working with text data. But what does the term ‘tokenization’ actually mean?Tokenization is essentially splitting a phrase, sentence, paragraph, or an entire text document into smaller units, such as individual words or terms. Each of these smaller units are called tokens.  Importing the library for word tokenization<jupyter_code>from nltk.tokenize import word_tokenize
AI ="""<NAME>, a brilliant mathematician, who broke the Nazi encryption machine Enigma, came up with a history-changing question, “Can machines think?” in 1950. The actual research began in 1956, at a conference held at Dartmouth College (a lot of the inventions have come into the picture, thanks to the Ivy League). A couple of attendees at the conference were the ones who came up with the idea and also the name “Artificial Intelligence”. But since the whole idea was new, people didn’t buy the idea and funding for further research was pulled off. This period, the 1950s – 1980s was called “AI Winter”. In the early 1980s however, the Japanese government saw a future in AI and started funding the field again. As this was interconnected to the electronics and computer science fields, there was a sudden spike in those as well. The first AI machine was introduced to the world in 1997; IBM’s Deep Blue became the first computer to beat a chess champion when it defeated Russian grandmaster <NAME>. And that, my dear readers, was the advent of a massive field called “AI”."""
type(AI)<jupyter_output><empty_output><jupyter_text>Performing word tokenization<jupyter_code>AI_tokens = word_tokenize(AI)
AI_tokens 
len(AI_tokens)<jupyter_output><empty_output><jupyter_text>For finding the occurance of lowercase words.,<jupyter_code>from nltk.probability import FreqDist
fdist = FreqDist()
for word in AI_tokens:
    fdist[word.lower()]+=1
fdist
fdist['the']
len(fdist)
fdist_top5 = fdist.most_common(5)
fdist_top5
from nltk.tokenize import blankline_tokenize
AI_blank = blankline_tokenize(AI)
AI_blank
len(AI_blank)<jupyter_output><empty_output><jupyter_text>Ngrams, Bigrams and Trigrams 1) Tokens of any number of consecutive written words are called Ngrams.2) Tokens of two consecutive written words are called Bigrams.3) Tokens of three consecutive written words are calles Trigrams.<jupyter_code>from nltk.util import bigrams, trigrams, ngrams
string = "The basic function of the algorithms of AI is data analysis. Let me put it this way. How do you think human beings learn new things? They observe. They observe and that’s how they learn. Machines learn the same way. "
quotes_tokens = nltk.word_tokenize(string)
quotes_tokens
quotes_bigrams = list(nltk.bigrams(quotes_tokens))
quotes_bigrams
quotes_trigrams = list(nltk.trigrams(quotes_tokens))
quotes_trigrams<jupyter_output><empty_output><jupyter_text>Stemming Stemming involves normalizing a word into its base or root form.Ex: Consider the words: Affect, Affection, Affected, Affecting.The base or root form of the above words is "Äffect."Note: The NLTK tool provides mainly three types of stemmers, namely:1) PorterStemmer2) LancasterStemmer3) SnowballStemmer<jupyter_code>from nltk.stem import PorterStemmer
pst = PorterStemmer()
pst.stem("Loving")
words_to_stem = ['Killing', 'Saving', 'Protecting', 'Served', 'loved']

for words in words_to_stem:
    print(words + ":" + pst.stem(words))
from nltk.stem import LancasterStemmer
lst = LancasterStemmer()

for words in words_to_stem:
    print(words + ":" + lst.stem(words))

from nltk.stem import SnowballStemmer
sbst = SnowballStemmer('english') 


for words in words_to_stem:
    print(words + ":" + sbst.stem(words))<jupyter_output>Killing:kill
Saving:save
Protecting:protect
Served:serv
loved:love<jupyter_text>Lemmatization 1) Groups together different inflicted forms of a word, called a lemma.2) Somehow similar to stemming as it maps words into one common root.3) The outcome of lemmatization is a proper word.for example, the words "going" and "gone" when lemmatized, should return "go" as the result.Lemmatization does acquire Wordnet database often for its functioning.<jupyter_code>from nltk.stem import wordnet
from nltk.stem import WordNetLemmatizer

word_lem = WordNetLemmatizer()
word_lem.lemmatize('corpora')
for words in words_to_stem:
    print(words + ":" + word_lem.lemmatize(words))
from nltk.corpus import stopwords

stopwords.words('english')
len(stopwords.words('english'))
fdist_top5
import re 
punctuation = re.compile(r'[-.?!:;()|0-9]')<jupyter_output><empty_output><jupyter_text>Parts of Speech Tagging<jupyter_code>sentence = "Mama is a natural when it comes to kicking ass."
sent_tokens = word_tokenize(sentence)
len(sent_tokens)
sent_tokens
len(sent_tokens)
for token in sent_tokens:
    print(nltk.pos_tag([token]))
sent2 = "Elise kicked Arno in the balls."
sen2_tokens = word_tokenize(sent2)

sen2_tokens
for token in sen2_tokens:
    print(nltk.pos_tag([token]))<jupyter_output>[('Elise', 'NN')]
[('kicked', 'VBN')]
[('Arno', 'NN')]
[('in', 'IN')]
[('the', 'DT')]
[('balls', 'NNS')]
[('.', '.')]<jupyter_text>Named Entity Recognition The detection of a named entity which could be either a movie, a monetary value, a location or even a person is called Named Entity Recognition.We import ne_chunk to perform Named Entity Recognition.<jupyter_code>from nltk import ne_chunk
line = "The US President resides in the White House."

line_token = word_tokenize(line)

line_tag = nltk.pos_tag(line_token)
line_ner = ne_chunk(line_tag)
line_ner<jupyter_output>The Ghostscript executable isn't found.
See http://web.mit.edu/ghostscript/www/Install.htm
If you're using a Mac, you can try installing
https://docs.brew.sh/Installation then `brew install ghostscript`
{'fraction_non_alphanumeric': 0.08064303597311899, 'fraction_numerical': 0.006852022664382659, 'mean_word_length': 5.314475873544093, 'pattern_counts': {'":': 4, '<': 36, '<?xml version=': 0, '>': 36, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '499862', 'license': 'Apache-2.0', 'n_tokens_mistral': 2421, 'n_tokens_neox': 2293, 'n_words': 927, 'path': 'Natural Language Processing.ipynb', 'repo_name': 'VMSthvik/MachineLearning', 'size': '7615'}