<jupyter_start><jupyter_text>Sentiment Analysis<jupyter_code>from textblob import TextBlob
texts=["The movie was good.", 
    "The movie was not good.",
    "I really think this product sucks.",
    "Really great product.",
    "I don't like this product"]
for t in texts:
    print(t, "==>", TextBlob(t).sentiment.polarity)
text=TextBlob("""The movie was good. The movie was not good. I really think this product sucks.
Really great product. I don't like this product""")
for s in text.sentences:
    print("=>", s)
for s in text.sentences:
    print(s, "==> ", s.sentiment.polarity)<jupyter_output><empty_output><jupyter_text>Creating our own classifierLets use [Sentiment Polarity Dataset 2.0](https://www.cs.cornell.edu/people/pabo/movie-review-data/), included in the `NLTK` library.It consists of 1000 positive and 1000 negative processed reviews. Introduced in Pang/Lee ACL 2004. Released June 2004.<jupyter_code>import nltk
from nltk.corpus import stopwords
from collections import defaultdict
from nltk import word_tokenize
import string

from nltk.corpus import movie_reviews as mr
print("The corpus contains %d reviews"% len(mr.fileids()))

for i in mr.fileids()[995:1005]: # Reviews 995 to 1005
    print(i, "==>", i.split('/')[0])<jupyter_output><empty_output><jupyter_text>Let's see the content of one of these reviews<jupyter_code>print(mr.raw(mr.fileids()[1]))<jupyter_output><empty_output><jupyter_text>Calculating the frequency of each word in the document ...<jupyter_code>from nltk.probability import FreqDist
FreqDist(mr.raw(mr.fileids()[1]).split())<jupyter_output><empty_output><jupyter_text>Lets take a look at the most frequent words in the corpus<jupyter_code>wordfreq = FreqDist()
for i in mr.fileids():
    wordfreq += FreqDist(w.lower() for w in mr.raw(i).split())<jupyter_output><empty_output><jupyter_text>The previous code has flaws because split() is a very basic way of finding the words. Let's use `word_tokenize()` or `mr.words()` instead...<jupyter_code>wordfreq = FreqDist()
for i in mr.fileids():
    wordfreq += FreqDist(w.lower() for w in mr.words(i))
print(wordfreq)
print(wordfreq.most_common(10))<jupyter_output><empty_output><jupyter_text>stop words and punctuation are causing trouble, lets remove them...<jupyter_code>stopw = stopwords.words('english')
wordfreq = FreqDist()
for i in mr.fileids():
    wordfreq += FreqDist(w.lower() for w in mr.words(i) if w.lower() not in stopw and w.lower() not in string.punctuation)
print(wordfreq)
print(wordfreq.most_common(10))<jupyter_output><empty_output><jupyter_text>Shuffling Lets shuffle the documents, otherwise they will remain sorted ["neg", "neg" ... "pos"]<jupyter_code>import random
docnames=mr.fileids()
random.shuffle(docnames)<jupyter_output><empty_output><jupyter_text>Lets split each document into words ...<jupyter_code>documents=[]
for i in docnames:
    y = i.split('/')[0]
    documents.append( ( mr.words(i) , y) )<jupyter_output><empty_output><jupyter_text>Let's take a look at our documents...<jupyter_code>for docs in documents[:5]:
    print(docs)<jupyter_output><empty_output><jupyter_text>Document representationNow, lets produce the final document representation, in the form of a Frequency Distribution ...First, without stop words and punctuation ... (you could use other technique, such as IDF)<jupyter_code>stopw = stopwords.words('english')
docrep=[]
for words,tag in documents:
    features = FreqDist(w for w in words if w.lower() not in stopw and w.lower() not in string.punctuation)
    docrep.append( (features, tag) )<jupyter_output><empty_output><jupyter_text>Let's take a look at our documents again...<jupyter_code>for doc in docrep[:5]:
    print(doc)<jupyter_output><empty_output><jupyter_text>NLTK classifier: Naive BayesDefining our training and test sets...<jupyter_code>numtrain = int(len(documents) * 80 / 100)  # number of training documents

train_set, test_set = docrep[:numtrain], docrep[numtrain:]

print(test_set[0])
from nltk.classify import NaiveBayesClassifier as nbc

classifier = nbc.train(train_set)

print("Accuracy:", nltk.classify.accuracy(classifier, test_set))

classifier.show_most_informative_features(5)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12359818659031258, 'fraction_numerical': 0.011691720353137676, 'mean_word_length': 6.045378151260504, 'pattern_counts': {'":': 0, '<': 53, '<?xml version=': 0, '>': 57, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '496443', 'license': 'MIT', 'n_tokens_mistral': 1479, 'n_tokens_neox': 1404, 'n_words': 385, 'path': '#8 NLP/SentimentAnalysis.ipynb', 'repo_name': 'Sphincz/dsml', 'size': '4191'}