<jupyter_start><jupyter_text>2. Feature Engineering and EDA**<NAME>**  **BrainStation, Data Science**  **Previous Notebook: 1. Data Loading, Preprocessing and Cleaning****Next Notebook: 3. Modeling**In the previous notebook, we completed some initial cleaning on the dataset. In this notebook, we will be performing some feature engineering, and some exploratory data analysis. Notably, I want to explore the following questions.1. What is the distribution of the ratings?2. What are the summary statistics of our features in the dataset?3. Is there any correlation between our continuous input variables and the target variable?4. What are the most common words used across all the reviews in the dataset?First, lets import our packages and read in the cleaned data.<jupyter_code># Import standard data science packages
import pandas as pd 
import numpy as np 
import matplotlib.pyplot as plt 
from scipy import stats

# Utility packages
from wordcloud import WordCloud
import joblib
# Read in preprocessed and cleaned data
data = joblib.load('data/cleaned_data.pkl')

# Inspect the data
data.head()<jupyter_output><empty_output><jupyter_text>***  Count the number of words in each reviewI suspect that reviewers will write more and rave on about highly rated whiskys. We will define a function that counts the number of words in each review and add that to our dataset.<jupyter_code># Define function and lambda function to count the number of words in each review
def review_length(text):
    '''
    Takes in a string and counts the number of words in the string
    '''
    num_words = len(text.split(' '))
    return num_words

length_of_review = lambda x: review_length(x)

data['review_length'] = data['cleaned_reviews'].apply(length_of_review)

data.head()
# Save data with engineered feature to disk
joblib.dump(data, 'data/data_with_engineered_feature.pkl')<jupyter_output><empty_output><jupyter_text>***  Distribution of the ratings<jupyter_code># Plot the distribution of the scores
ratings = data['review.point'].value_counts().sort_index()

# plot rating vs frequency
plt.figure(figsize=(15,10))
plt.bar(ratings.index, ratings)
plt.title('Distribution of whisky ratings')
plt.xlabel('Rating')
plt.ylabel('Frequency')
plt.xticks(ratings.index)
plt.show()<jupyter_output><empty_output><jupyter_text>The distribution is farily normal, but it appears left-skewed (long-tail towards low scoring whiskys).  ***  Summary statistics<jupyter_code># Describe the data
data.describe().T<jupyter_output><empty_output><jupyter_text>- The average rating for the whiskys in the dataset is 86.7. The range of ratings is between 63 and 97. - There is a large variance in the price of whiskys- The average review is only 72 words in length***  Checking for correlations<jupyter_code># Check the correlation coefficient and p-value between each continuous variable and the target variable
for feature in ['price_string', 'review_length']:
    r = stats.pearsonr(data['review.point'], data[feature])[0]
    p = stats.pearsonr(data['review.point'], data[feature])[1]
    print(f"Feature: {feature} ... r={r} ... p={p}")

# Extract continuous variables from the dataset
cont_var = data[['review.point', 'price_string', 'review_length']]

# Correlation matrix 
cont_var.corr().style.background_gradient('coolwarm')<jupyter_output>Feature: price_string ... r=0.12808032456684176 ... p=1.1266543441467823e-09
Feature: review_length ... r=0.12732671116219402 ... p=1.4110170846947998e-09<jupyter_text>There appears to be a significant weak positive correlation between both the length of the review, and the price of the whisky with the rating of the whisky. ***  What are the most common words used in reviews?One way to visualize the most commonly used words in the corpus is to use a word cloud. The bigger the word appears in the word cloud, the more often the word is used across all reviews in the dataset.<jupyter_code># combine all reviews into a text to create wordcloud
text = " ".join(review for review in data['cleaned_reviews'])
print(f'There are {len(text)} words in the combination of all reviews')

# create the wordcloud
plt.figure(figsize=(15,10))
wordcloud = WordCloud(background_color='white').generate(text)
plt.imshow(wordcloud, interpolation='bilinear')
plt.axis('off')
plt.show()<jupyter_output>There are 948503 words in the combination of all reviews
{'fraction_non_alphanumeric': 0.09051329055912008, 'fraction_numerical': 0.024289642529789185, 'mean_word_length': 5.5344311377245505, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '608158', 'license': 'MIT', 'n_tokens_mistral': 1327, 'n_tokens_neox': 1232, 'n_words': 495, 'path': '2_EDA_and_Feature_Engineering.ipynb', 'repo_name': 'andrew-dang/whisky_predictions', 'size': '4369'}