<jupyter_start><jupyter_text>Covid Candles – Sentiment Analysis----**Author:** <NAME>**[GitHub](https://github.com/SimonAytes)****[Website](https://www.saytes.io)**  Import Packages<jupyter_code>import pandas as pd
import csv
import os
import nltk
from nltk.sentiment import SentimentIntensityAnalyzer
import matplotlib
# Download NLTK packages
nltk.download([
     "names",
     "stopwords",
     "state_union",
     "twitter_samples",
     "movie_reviews",
     "averaged_perceptron_tagger",
     "vader_lexicon",
     "punkt"])
# Get path to the repo directory
dir_path = "/".join(os.getcwd().split("/")[0:-1])<jupyter_output><empty_output><jupyter_text>1. Load pre-processed data----<jupyter_code># Load in previously gathered reviews
reviews = pd.read_csv(open(dir_path + "/data/raw/reviews.csv"))<jupyter_output><empty_output><jupyter_text>2. Explore Data----  2.1 View rating distribution<jupyter_code># View the distributions of ratings
reviews['rating'].value_counts().plot(kind='barh')<jupyter_output><empty_output><jupyter_text>2.2 View date distribution<jupyter_code># View the distributions of dates
reviews['date'].value_counts()<jupyter_output><empty_output><jupyter_text>3. Sentiment Analysis----  3.1 Configure sentiment analyzer<jupyter_code># Create a sentiment analyzer
sentiment_analyzer = SentimentIntensityAnalyzer()

# Create an empty list to house sentiments
sentiment = []<jupyter_output><empty_output><jupyter_text>3.2 Analyze review sentimentEach review will be classified as: (1) positive, (2) negative, (3) neutral, or (4) na.<jupyter_code># Iterate through each row of "reviews"
for index, row in reviews.iterrows():
    # Try to analyze the sentiment (if not na)
    try:
        # Analyze the review
        t = sentiment_analyzer.polarity_scores(row['review'])

        # Assign sentiment based on compound sentiment value
        # If it is positive, mark positive
        if t['compound'] > 0:
            sentiment.append("positive")
        # If it is negative, mark negative
        elif t['compound'] < 0:
            sentiment.append("negative")
        # Else, mark neutral
        else:
            sentiment.append("neutral")

    # Mark all failed attempts as "na"
    except:
        sentiment.append("na")
# Create new column in "reviews" that contains the sentiment of each review
reviews['sentiment'] = sentiment<jupyter_output><empty_output><jupyter_text>3.3 View review sentiment distribution<jupyter_code># View the distribution of sentiments
reviews['sentiment'].value_counts().plot(kind='barh')<jupyter_output><empty_output><jupyter_text>3.4 Output tagged reviews<jupyter_code>reviews.to_csv((dir_path + "/data/interim/reviews-classified.csv"), index = False)<jupyter_output><empty_output><jupyter_text>4. Process sentiment data----<jupyter_code>negative_reviews = pd.DataFrame()<jupyter_output><empty_output><jupyter_text>4.1 Filter by sentiment<jupyter_code># Filter all negative reviews
negative_reviews = reviews[reviews['sentiment'] == "negative"]<jupyter_output><empty_output><jupyter_text>4.2 Filter by rating<jupyter_code># Filter all ratings less than 3 (1 or 2)
negative_reviews = negative_reviews[negative_reviews['rating'] < 3.0]<jupyter_output><empty_output><jupyter_text>4.3 Filter by mention of smell<jupyter_code># Gather all negative reviews that mention smell, scent, or fragrance
negative_reviews_smell = negative_reviews.loc[negative_reviews['review'].str.contains('smell'),:]<jupyter_output><empty_output><jupyter_text>4.4 Sort reviews by date<jupyter_code># Sort the results by date
negative_reviews_smell = negative_reviews_smell.sort_values(by="date")<jupyter_output><empty_output><jupyter_text>4.5 Output processed data<jupyter_code>negative_reviews_smell.to_csv((dir_path + "/data/interim/negative-reviews-smell.csv"), index = False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1209992193598751, 'fraction_numerical': 0.01014832162373146, 'mean_word_length': 5.767605633802817, 'pattern_counts': {'":': 0, '<': 60, '<?xml version=': 0, '>': 59, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '437229', 'license': 'MIT', 'n_tokens_mistral': 1277, 'n_tokens_neox': 1244, 'n_words': 271, 'path': 'notebooks/2-sentiment-analysis.ipynb', 'repo_name': 'SimonAytes/covid-candles', 'size': '3848'}