<jupyter_start><jupyter_text>AirBNB Sentiment Analysis [Airbnb](https://www.airbnb.com/) is a popular online marketplace that connects people who rent their space to people who are looking for accomodations in that areaCustomers can leave reviews for the rented space and sometimes bad ones can scare people away.We'll analyze how's the review looks like, from Airbnb public data.  Business UnderstandingBad review could turn customers away.It's now common for a customer to do their own research before deciding a buy. Some aspects they like to compare are price, product review, brand, etc. During their own research, consumer able to look for positive or negative reviews.A study founds that online reviews affect [67%](https://soundwave.co.uk/blog/study-finds-67-of-consumers-are-influenced-by-online-reviews) of consumer purchasing decision. Positive reviews would increase the reputation, and able to drive more sales. Meanwhile, negative reviews could be detrimental to business. Revenue Loss, Undermine brand image, drive away customer visit, and low search traffic.Thus, it's essential for the business to find bad review and take action to improve the product/serviceThese are the questions that we wish to answer in this analysis : - How to identify positive and negative reviews? - What are the popular words from the customers? - How to detect negative reviews and does it robust enough to detect the sentiment?  Data UnderstandingThe data taken from Airbnb for 3 cities (Boston, Seattle, Amsterdam)- City : [Boston](https://www.kaggle.com/airbnb/boston), [Seattle](https://www.kaggle.com/airbnb/seattle/data), [Amsterdam](http://insideairbnb.com/get-the-data.html)- List : Contains the description of review and ratings- Reviews: contains ID for reviewer and comments     In this analysis, first, we will find the overall rating of the service and find the popular review words.Then we will use the sentiment analysis technique to find bad reviewsThe sentiment analysis uses NLTK, which is a powerful language tool to analyze text data. It covers many language forms to assess the sentiment review in this analysis.To see how nltk performs in a non-English language, we are adding dutch language in Amsterdam Airbnb.  Import & Overview of dataset<jupyter_code>import pandas as pd 
#Load Dataset
seattle_raw_list = pd.read_csv("/Users/yogisharosarumaha/Udacity Project 1/airbnb_seattle_data/listings.csv") 
seattle_raw_review = pd.read_csv("/Users/yogisharosarumaha/Udacity Project 1/airbnb_seattle_data/reviews.csv") 

boston_raw_list = pd.read_csv("/Users/yogisharosarumaha/Udacity Project 1/airbnb_boston_data/listings.csv") 
boston_raw_review = pd.read_csv("/Users/yogisharosarumaha/Udacity Project 1/airbnb_boston_data/reviews.csv") 

amsterdam_raw_list = pd.read_csv("/Users/yogisharosarumaha/Udacity Project 1/airbnb_amsterdam_data/listings.csv") 
amsterdam_raw_review = pd.read_csv("/Users/yogisharosarumaha/Udacity Project 1/airbnb_amsterdam_data/reviews.csv") 
#Look at the first rows from Listings and Review dataset
display(seattle_raw_list.head(3),seattle_raw_review.head(3))<jupyter_output><empty_output><jupyter_text>Combine dataset<jupyter_code>#Label each area
seattle_raw_list['area'] = 'seattle'
seattle_raw_review['area'] = 'seattle'

boston_raw_list['area'] = 'boston'
boston_raw_review['area'] = 'boston'

amsterdam_raw_list['area'] = 'amsterdam'
amsterdam_raw_review['area'] = 'amsterdam'

#Union all the cities dataset
concat_list_df = pd.concat([seattle_raw_list,boston_raw_list,amsterdam_raw_list], ignore_index=True)
concat_review_df = pd.concat([seattle_raw_review,boston_raw_review,amsterdam_raw_review], ignore_index=True)<jupyter_output><empty_output><jupyter_text>Airbnb Stay have Positive Review Customers in 3 different areas have great room stay experiece with ratings ~90This is common for the reviews to have more positive than negative in services apps, as they keep improving over time.<jupyter_code>concat_list_df.groupby('area')["review_scores_rating"].describe().transpose()<jupyter_output><empty_output><jupyter_text>All service aspects have Positive Review Airbnb have various aspects of customer experience such as check-in experience, room cleanliness,  location, communication, and also value.In the chart, all aspects also have positive rating business should maintain these effort to keep customers coming inThere's still improvement to be made from the bad ratingsWe'll take a look at these bad reviews later and what kind of unpleasant experience they had during the stay<jupyter_code>import matplotlib.pyplot as plt

fig, ax = plt.subplots(2, 3)
fig.tight_layout()

ax[0, 0].hist(concat_list_df["review_scores_value"]) #row=0, col=0
ax[0, 0].set_title('Value')
ax[1, 0].hist(concat_list_df["review_scores_checkin"]) #row=1, col=0
ax[1, 0].set_title('Check In')
ax[0, 1].hist(concat_list_df["review_scores_cleanliness"]) #row=0, col=1
ax[0, 1].set_title('Cleanliness')
ax[1, 1].hist(concat_list_df["review_scores_communication"]) #row=1, col=1
ax[1, 1].set_title('Communication')
ax[1, 2].hist(concat_list_df["review_scores_location"]) #row=1, col=2
ax[1 ,2].set_title('Location')
plt.show()<jupyter_output>/Users/yogisharosarumaha/anaconda3/lib/python3.7/site-packages/numpy/lib/histograms.py:839: RuntimeWarning: invalid value encountered in greater_equal
  keep = (tmp_a >= first_edge)
/Users/yogisharosarumaha/anaconda3/lib/python3.7/site-packages/numpy/lib/histograms.py:840: RuntimeWarning: invalid value encountered in less_equal
  keep &= (tmp_a <= last_edge)<jupyter_text>What are customers mostly saying We have checked that most customers have a great experience during the stay. In all aspects, the scores was pretty high.It's interesting to also look at the most frequent words from the comments.  Fill Missing Reviews as Blank<jupyter_code>#count missing values
concat_review_df['comments'].isnull().sum()<jupyter_output><empty_output><jupyter_text>There are 339 rows that have missing values and we fill it with blanks.If we did not perform Missing Value Treatment (MVT), performing word count will occur error.It's safe to fill it with blanks, since customer did not leave review about their stay.<jupyter_code>import itertools
import collections

def convert(lst): 
    return ([i for item in lst for i in item.lower().split()]) 

all_words_review = convert(concat_review_df['comments'].fillna("")) #Fill in missing reviews with blank<jupyter_output><empty_output><jupyter_text>The texts also lowered to avoid overlapping words that have similar intention.Then, split each word to count their frequency  Word Frequency<jupyter_code>count_word = collections.Counter(all_words_review) #Count word frequency

clean_word_count = pd.DataFrame(count_word.most_common(5),
                             columns=['words', 'count'])

clean_word_count.head()<jupyter_output><empty_output><jupyter_text>The word frequency runs well, but we cannot understand what these words mean.We wouldn't want these words showed up and we can remove them with `nltk` library  Removing Stopwords Stopwords are common in any language.But we should remove them when analyzing text data, as this doesn't add much value for our analysis<jupyter_code>import nltk
from nltk.corpus import stopwords
stop_words = set(stopwords.words(['dutch','english']))

# View a few words from the set
list(stop_words)[0:10]<jupyter_output><empty_output><jupyter_text>Here's some examples of english and dutch stopwords and we'll remove them<jupyter_code>words_review_cleaned = [word for word in all_words_review if word not in stop_words]
words_review_count = collections.Counter(words_review_cleaned)

word_review_count_df = pd.DataFrame(words_review_count.most_common(15),
                             columns=['words', 'count'])
fig, ax = plt.subplots(figsize=(8, 8))

# Plot horizontal bar graph
word_review_count_df.sort_values(by='count').plot.barh(x='words',
                      y='count',
                      ax=ax,
                      color="purple")

ax.set_title("Most Frequent Words (All Review)")

plt.show()<jupyter_output><empty_output><jupyter_text>Notice that the result improved by producing more meaningful wordsAs the rating suggests, most review expresses positive service.Some of the aspects was mentionted, such as  _"Great"_  , _"Location"_ , _"Recommend"_ , _"Clean"_ , _"Close"_  Sentiment Analysis to detect good/bad Reviews Sentiment analysis is the process of identifying people's attitude and emotional states from a language.In the application, it able to classify good/bad reviews.It is a technique uses for natural language processing to interpret and classify emotions.The output would summarize the positive, negative, or neutral sentiment of these reviews.Since sentiment analysis is a [complex subject](http://www.360doc.com/content/16/0426/18/2459_554012119.shtml) , the accuracy may not be perfect.But it's easier for us to find a bad review using this approach when there's a lot of dataWe will be using `nltk` from python to analyze the data<jupyter_code># Snippet from :
# https://www.kaggle.com/residentmario/sentiment-analysis-and-collocation-of-reviews

from nltk.sentiment.vader import SentimentIntensityAnalyzer
sid = SentimentIntensityAnalyzer()
for sentence in concat_review_df['comments'].values[1:3]:
    print(sentence)
    ss = sid.polarity_scores(sentence)
    for k in sorted(ss):
        print('{0}: {1}, '.format(k, ss[k]), end='')
    print()<jupyter_output>Kelly has a great room in a very central location. 
Beautiful building , architecture and a style that we really like. 
We felt guite at home here and wish we had spent more time.
Went for a walk and found Seattle Center with a major food festival in progress. What a treat.
Visited the Space Needle and the Chihuly Glass exhibit. Then Pikes Place Market. WOW.  Thanks for a great stay.
compound: 0.9872, neg: 0.0, neu: 0.609, pos: 0.391, 
Very spacious apartment, and in a great neighborhood.  This is the kind of apartment I wish I had!

Didn't really get to meet Kelly until I was on my out, but she was always readily available by phone. 

I believe the only "issue" (if you want to call it that) was finding a place to park, but I sincerely doubt its easy to park anywhere in a residential area after 5 pm on a Friday
compound: 0.8718, neg: 0.043, neu: 0.772, pos: 0.185,<jupyter_text>Let's look at some of the comment example, and how `nltk` determines a review`compound` is the normalized weighted score.The metric is useful when determining a uni-dimensional sentiment over a sentence.A positive or negative sentiment being +1 as the most positive and -1 as the most negativeThe `pos`, `neu`, and `neg` scores are ratios for proportions of text that fall in each category (the values will add up to 1 if summed) . These metrics are useful if we want a multi-dimensional sentiment over a sentence.More about the [scoring](https://github.com/cjhutto/vaderSentiment)<jupyter_code>from nltk.corpus import stopwords   # stopwords to detect language
from nltk import wordpunct_tokenize # function to split up our words

def get_language_likelihood(input_text):
    """Return a dictionary of languages and their likelihood of being the 
    natural language of the input text
    """
 
    input_text = input_text.lower()
    input_words = wordpunct_tokenize(input_text)
 
    language_likelihood = {}
    total_matches = 0
    for language in stopwords._fileids:
        language_likelihood[language] = len(set(input_words) &
                set(stopwords.words(language)))
 
    return language_likelihood
 
def get_language(input_text):
    """Return the most likely language of the given text
    """ 
    likelihoods = get_language_likelihood(input_text)
    return sorted(likelihoods, key=likelihoods.get, reverse=True)[0]

reviews_f = [r for r in concat_review_df['comments'] if pd.notnull(r) and get_language(r) in ['dutch','english']]

pscores = [sid.polarity_scores(comment) for comment in reviews_f]<jupyter_output><empty_output><jupyter_text>After taking all sentiment scores from the comments,Let's take a look at the distributions  Compound Scores<jupyter_code>#Histogram plot neutral
pd.Series([score['compound'] for score in pscores]).plot(kind='hist')

plt.title('Compound Scores')
plt.xlabel('Scores')
plt.ylabel('frequency')<jupyter_output><empty_output><jupyter_text>Neutral Scores<jupyter_code>#Histogram plot neutral
pd.Series([score['neu'] for score in pscores]).plot(kind='hist')

plt.title('Neutral Scores')
plt.xlabel('Scores')
plt.ylabel('frequency')<jupyter_output><empty_output><jupyter_text>Positive Scores<jupyter_code>#Histogram plot positive
pd.Series([score['pos'] for score in pscores]).plot(kind='hist')

plt.title('Positive Scores')
plt.xlabel('Scores')
plt.ylabel('frequency')<jupyter_output><empty_output><jupyter_text>Negative Scores Surprisingly, the negative scores was mostly zero.This implies that indeed the experience was pleasant for most Airbnb stay.<jupyter_code>#Histogram plot negative
pd.Series([score['neg'] for score in pscores]).plot(kind='hist', bins=100)

plt.title('Negative Scores')
plt.xlabel('Scores')
plt.ylabel('frequency')
#Create dataframe for sentiment analysis
scored_reviews = pd.DataFrame()
scored_reviews['review'] = [r for r in reviews_f if get_language(r) in ['dutch','english']]
scored_reviews['compound'] = [score['compound'] for score in pscores]
scored_reviews['negativity'] = [score['neg'] for score in pscores]
scored_reviews['neutrality'] = [score['neu'] for score in pscores]
scored_reviews['positivity'] = [score['pos'] for score in pscores]<jupyter_output><empty_output><jupyter_text>Inspecting Bad review<jupyter_code>130700/len(scored_reviews)
scored_reviews.query('positivity > 0')<jupyter_output><empty_output><jupyter_text>n =500376; (96,5% users)Not all review is positive, some have a negative compound that expresses an unpleasant stay.<jupyter_code>scored_reviews.query('negativity > 0')<jupyter_output><empty_output><jupyter_text>n =130700; (25% users)Also, some of the negative scores were not bad at all. Row 517987 has a high compound score, and the review seems positive.<jupyter_code>scored_reviews.query('negativity > positivity')
scored_reviews.iloc[517953]['review']<jupyter_output><empty_output><jupyter_text>Compund: 0.1883, Pos: 0.061, Neu: 0.852 , Neg: 0.08 (n=6276, 1,2%  of all)If we closely read the review, It strongly influences readers to not recommend the place. But, sentiment scores tend to be neutral with a slight difference between positive & negative. The compound score has a better explanation as the score is closer to zero.Notice that only a small fraction of reviews affected (1,2% of all).<jupyter_code>scored_reviews.query('negativity > positivity').query('compound > 0.2')
scored_reviews.iloc[500214]['review']<jupyter_output><empty_output><jupyter_text>Compound: 0.2667, Pos: 0.416, Neu: 0.288 , Neg: 0.416 (n=135, 0,02% of all)It's not accurate to score this review as negative, as there is no bad comment about it. Started out as a positive review, the follow up sentence was mostly negative.This was accurate, being positivity (0.083) is lower than negativity (0.209) metric.<jupyter_code>scored_reviews.iloc[517895]
scored_reviews.iloc[517895]['review']<jupyter_output><empty_output><jupyter_text>The compound metric itself may not all accurate. It detects the sentence as badif we translate this,it basically saying that the location of the stay was good with helpful staff.Thus , we should also look into negativity and neutrality to justify bad review<jupyter_code>scored_reviews.query('negativity > positivity').query('compound < -0.8')
scored_reviews.iloc[516995]['review']<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09056171188383645, 'fraction_numerical': 0.016558400203795696, 'mean_word_length': 5.818497611810682, 'pattern_counts': {'":': 0, '<': 85, '<?xml version=': 0, '>': 90, 'https://': 6, 'lorem ipsum': 0, 'www.': 5, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '601842', 'license': 'FTL', 'n_tokens_mistral': 4960, 'n_tokens_neox': 4634, 'n_words': 1603, 'path': 'airbnb_sentiment_analysis.ipynb', 'repo_name': 'sarumaha/airbnb_sentiment_analysis', 'size': '15702'}