<jupyter_start><jupyter_text>Predicting sentiment from product reviews Fire up GraphLab Create<jupyter_code>import graphlab<jupyter_output><empty_output><jupyter_text>Read some product review dataLoading reviews for a set of baby products.<jupyter_code>products = graphlab.SFrame('amazon_baby.gl/')<jupyter_output>This non-commercial license of GraphLab Create for academic use is assigned to <EMAIL> and will expire on June 12, 2018.<jupyter_text>Let's explore this data togetherData includes the product name, the review text and the rating of the review.<jupyter_code>products.head()<jupyter_output><empty_output><jupyter_text>Build the word count vector for each review<jupyter_code>products['word_count'] = graphlab.text_analytics.count_words(products['review'])
products.head()
graphlab.canvas.set_target('ipynb')
products['name'].show()<jupyter_output><empty_output><jupyter_text>Examining the reviews for most-sold product:  '<NAME>'<jupyter_code>giraffe_reviews = products[products['name'] == '<NAME>']
len(giraffe_reviews)
giraffe_reviews['rating'].show(view = 'Categorical')<jupyter_output><empty_output><jupyter_text>Build a sentiment classifier<jupyter_code>products['rating'].show(view = 'Categorical')<jupyter_output><empty_output><jupyter_text>Define what's a positive and a negative sentimentWe will ignore all reviews with rating = 3, since they tend to have a neutral sentiment.  Reviews with a rating of 4 or higher will be considered positive, while the ones with rating of 2 or lower will have a negative sentiment.<jupyter_code>#ignore all 3* reviews
products = products[products['rating'] != 3]
#positive sentiment = 4* or 5* reviews
products['sentiment'] = products['rating'] >= 4
products.head()<jupyter_output><empty_output><jupyter_text>Let's train the sentiment classifier<jupyter_code>train_data,test_data = products.random_split(.8, seed=0)
sentiment_model = graphlab.logistic_classifier.create(train_data,
                                                     target='sentiment',
                                                     features=['word_count'],
                                                     validation_set=test_data)<jupyter_output><empty_output><jupyter_text>Evaluate the sentiment model<jupyter_code>sentiment_model.evaluate(test_data, metric = 'roc_curve')
sentiment_model.show(view = 'Evaluation')<jupyter_output><empty_output><jupyter_text>Applying the learned model to understand sentiment for Giraffe<jupyter_code>giraffe_reviews['predicted_sentiment'] = sentiment_model.predict(giraffe_reviews, output_type = 'probability')
giraffe_reviews.head()<jupyter_output><empty_output><jupyter_text>Sort the reviews based on the predicted sentiment and explore<jupyter_code>giraffe_reviews = giraffe_reviews.sort('predicted_sentiment', ascending = False)
giraffe_reviews.head()<jupyter_output><empty_output><jupyter_text>Most positive reviews for the giraffe<jupyter_code>giraffe_reviews[0]['review']
giraffe_reviews[1]['review']<jupyter_output><empty_output><jupyter_text>Show most negative reviews for giraffe<jupyter_code>giraffe_reviews[-1]['review']
giraffe_reviews[-2]['review']<jupyter_output><empty_output><jupyter_text>Assignments Question 1<jupyter_code>def awesome_count(dic,wd):
    if wd in dic:
        return dic[wd]
    else:
        return 0

selected_words_count = dict()
selected_words = ['awesome', 'great', 'fantastic', 'amazing', 'love', 'horrible', 'bad', 'terrible', 'awful', 'wow', 'hate']

for w in selected_words:
    products[w] = products['word_count'].apply(lambda x: awesome_count(x,w))
    selected_words_count[w] = products[w].sum()
    
products.head()
selected_words_count
import operator
print 'Most used word:', max(selected_words_count.iteritems(), key = operator.itemgetter(1))[0]
print 'Least used word:', min(selected_words_count.iteritems(), key = operator.itemgetter(1))[0]<jupyter_output>Most used word: great
Least used word: wow<jupyter_text>Question 2<jupyter_code>train_data,test_data = products.random_split(.8, seed=0)
selected_words_model = graphlab.logistic_classifier.create(train_data,
                                                     target = 'sentiment',
                                                     features = selected_words,
                                                     validation_set = test_data)
selected_words_model
selected_words_model['coefficients']
v = list(selected_words_model['coefficients']['value'])
mx = v.index(max(v))
mn = v.index(min(v))
print mx, mn
mx_name = selected_words_model['coefficients']['name'][mx]
mn_name = selected_words_model['coefficients']['name'][mn]
print 'Word with max weight:', mx_name
print 'Word with min weight:', mn_name<jupyter_output>Word with max weight: love
Word with min weight: terrible<jupyter_text>Question 3<jupyter_code>selected_words_model.evaluate(test_data, metric='roc_curve')
selected_words_model.show(view='Evaluation')<jupyter_output><empty_output><jupyter_text>Question 4<jupyter_code>diaper_champ_reviews = products[products['name'] == 'Baby Trend Diaper Champ']
diaper_champ_reviews['predicted_sentiment'] = sentiment_model.predict(diaper_champ_reviews, output_type = 'probability')
diaper_champ_reviews.head()
diaper_champ_reviews = diaper_champ_reviews.sort('predicted_sentiment', ascending=False)
diaper_champ_reviews.head()
diaper_champ_reviews[0]['review']
diaper_champ_reviews[0]['predicted_sentiment']
diaper_champ_reviews['predicted_sentiment'] = selected_words_model.predict(diaper_champ_reviews, output_type = 'probability')
diaper_champ_reviews.head()
diaper_champ_reviews[0]['predicted_sentiment']
diaper_champ_reviews = diaper_champ_reviews.sort('predicted_sentiment', ascending=False)
diaper_champ_reviews
diaper_champ_reviews[0]['review']<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10945618288881191, 'fraction_numerical': 0.006061655697956356, 'mean_word_length': 6.227784730913642, 'pattern_counts': {'":': 0, '<': 69, '<?xml version=': 0, '>': 70, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '571744', 'license': 'MIT', 'n_tokens_mistral': 1862, 'n_tokens_neox': 1815, 'n_words': 342, 'path': 'Machine-Learning---Washington/course1/week3/Analyzing product sentiment.ipynb', 'repo_name': 'nikunjlad/Coursera', 'size': '5842'}