<jupyter_start><jupyter_text>Introduction Sephora is a French retailer of personal care and beauty products. Customers can leave reviews for products and also place a heart next to a product they want.  For this project, I wanted to look at customer behavior and what influences ratings and popularity. Link to dataset origin: https://www.kaggle.com/raghadalharbi/all-products-available-on-sephora-website/version/1It was collected by <NAME> for their Data Science Immersive Course with General Assembly First part: Data Cleaning and Data Checks  Import Libraries<jupyter_code>from google.colab import drive
drive.mount('/content/gdrive')
# Import pandas and read in our GDrive-mounted csv

import pandas as pd

sephora_df = pd.read_csv('/content/gdrive/My Drive/Colab Datasets/sephora_updated.csv')
sephora_df.head()<jupyter_output><empty_output><jupyter_text>Data Wrangling & Checking for Nulls<jupyter_code>sephora_df.info()
sephora_df.shape
sephora_df.dtypes
sephora_df.isnull().sum()
#Filter out columns that are not going to be used 
sephora_df_drop = sephora_df.drop(['id', 'URL', 'details', 'ingredients'], axis=1)
sephora_df_drop.head()
#Dropping products that have no ratings
sephora_df_new = sephora_df_drop[sephora_df_drop['rating']>0]
sephora_df_new.rating.unique()
#Explore each category type for products in dataset 
#This showcases which are the most populated type in categories
pd.set_option('display.max_rows', 500)
sephora_df_new.category.value_counts()<jupyter_output><empty_output><jupyter_text>Data Exploration and Visulizations<jupyter_code>from sqlalchemy import create_engine
import pandas as pd
import matplotlib.pyplot as plt

import seaborn as sns
sns.set_style("white")

#it is set to ignore warning
import warnings
warnings.filterwarnings("ignore")<jupyter_output><empty_output><jupyter_text>Visulization of Price<jupyter_code>#First, let's find the median and max of price for a hypothesis 
sephora_df_new['price'].describe()


#Let's look at the density of price 
sns.distplot(x=sephora_df_new['price'], bins=50)
plt.title('Price Density of Products at Sephora')
plt.xlabel('Price')
plt.ylabel('Count(Density)')
plt.show()<jupyter_output><empty_output><jupyter_text>Most of the prices range from 0 - 250. There is a product that is close to 549 USD. Visulization of *Ratings* and *Loves*<jupyter_code>sephora_df_new['rating'].describe()
plt.figure(figsize=(10,10))
ax = sns.barplot(x='rating', y='love', data=sephora_df_new)
plt.title('Ratings and the amount of loves')
plt.xlabel('Ratings')
plt.ylabel('Loves')
plt.show()<jupyter_output><empty_output><jupyter_text>From the bar graph, you can see that products that with a rating between 3.5 to 4.5 have a large amount of loves when combined.<jupyter_code>sephora_df_new['number_of_reviews'].describe()
#Remove review outliers greater than 19000 since most products have a review count of less than 5000
sephora_remove_review_outliers = sephora_df_new[sephora_df_new ['number_of_reviews'] <= 19000]
sns.distplot(x=sephora_remove_review_outliers['number_of_reviews'], bins=25)
plt.title('Number of Reviews')
plt.xlabel('Amount of Reviews')
plt.ylabel('Count (Density)')
plt.show()<jupyter_output><empty_output><jupyter_text>Second Part: Hypothesis Testing Hypothesis 1: I hypothesize that there is a correlation between products with higher *ratings* and the amount of *love* received from customers At Sephora, products are reviewed and receive a rating ranging from 1.0 to 5.0 from customers. Customers can also place a heart next to an item they desire. The loved product will be saved in a list for the customer and other customers can see how many loves a product has. Do customers rate products higher if they have more love and are popular?<jupyter_code>#The average ratings is situated between 3.5 to 5.0
sns.catplot (x = sephora_df_new["rating"], y = sephora_df_new ["love"], data = sephora_df_new, jitter = True)
plt.title('Correlation between Ratings and Loves')
plt.show()
from scipy import stats
stats.pearsonr(sephora_df_new['rating'], sephora_df_new['love'])
#there is a positive and significant coorelation between rating and loves<jupyter_output><empty_output><jupyter_text>Hypothesis 2: I hypothesize that products priced lower or equal to 50 USD are rated higher than items that are priced higher than 50 USD Sephora sells a variety of products along a varity of price points. When a customer purchases an item, they can leave an honest review and rating of that product. With the current pandemic and a loss of income in households, are customers still purchasing higher end products or leaning towards lower priced products?  The mean for price is close to 50 US ($49.50)<jupyter_code>sephora_df_new['price'].describe()<jupyter_output><empty_output><jupyter_text>Seperate products into two seperate dataframes according to price.<jupyter_code>#We end up with 6184 items that are less than 50 USD
lower_priced_products = sephora_df_new[['price', 'rating']].iloc[(sephora_df_new['price']<=50.0).values]
lower_priced_products.info()
#There are 2582 products  
higher_priced_products = sephora_df_new[['price', 'rating']].iloc[(sephora_df_new['price']>50.0).values]
higher_priced_products.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 2582 entries, 0 to 9149
Data columns (total 2 columns):
 #   Column  Non-Null Count  Dtype  
---  ------  --------------  -----  
 0   price   2582 non-null   float64
 1   rating  2582 non-null   float64
dtypes: float64(2)
memory usage: 60.5 KB<jupyter_text>Visualize lower priced products, higher priced products and their ratings<jupyter_code>#Using a Categorical plot, we can see that lower priced products have an average rating between 3.0 to 5.0
sns.catplot(x = lower_priced_products ["rating"], y = lower_priced_products ["price"], kind = "swarm", data = lower_priced_products)
plt.title ('Lower Priced Products and their Ratings vs. Price')
plt.xlabel('Ratings')
plt.ylabel('Price')
plt.show()
lower_priced_products.rating.value_counts()
#Products (<$50) rated higher than 4.0 total 5,022 out of 6,084 (82%)
sns.lineplot(x=higher_priced_products["rating"], y=higher_priced_products["price"], ci= None, data=higher_priced_products)
plt.title ('Higher Priced Products: Ratings vs. Price')
plt.xlabel('Ratings')
plt.ylabel('Price')
plt.show()
higher_priced_products.rating.value_counts()
#Products (>$50) rated higher than 4.0 total 2,230 out of 2,582 (86%)<jupyter_output><empty_output><jupyter_text>For each dataframe, a .describe will show the average rating before running a statistical test<jupyter_code>lower_priced_products.rating.describe()
#Lower priced products have an average rating of 4.1
higher_priced_products.rating.describe()
#Higher priced products have an average rating of 4.2
stats.ttest_ind(lower_priced_products['rating'], higher_priced_products['rating'])
#the p-value is less than 0.5 which means there is a statistical signifigance between products. The original hypothesis is false since the average rating of products over 50USD is 4.2<jupyter_output><empty_output><jupyter_text>Hypothesis 3: I hypothesize that there are more *loves* for makeup products for the face versus fragrance products. On Sephora's website, customers can put a heart next to product they like. These hearts are *loves*. As the beauty industry has evolved to include Youtubers and their makeup tutorials, are fragrance products even desired by customers? Are customers desiring more makeup products and brands over fragrance products? Face products and fragrance products will be split into two dataframes. Face products excludes lip and eye products.<jupyter_code>#Each category for products offered 
sephora_df_new.category.unique()<jupyter_output><empty_output><jupyter_text>Create a new dataframe for fragrance products<jupyter_code>Fragrance_List = ['Fragrance','Cologne', 'Perfume', 'Body Mist & Hair Mist', 'Body Lotions & Body Oils','Body Sprays & Deodrants', 'Perfume Gift Sets', 'Rollerballs & Travel Size', 'After Shave','Deodorant & Antiperspirant']
sephora_fragrance = sephora_df_new[sephora_df_new['category'].isin(Fragrance_List)]
sephora_fragrance.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 1297 entries, 0 to 9159
Data columns (total 17 columns):
 #   Column                  Non-Null Count  Dtype  
---  ------                  --------------  -----  
 0   brand                   1297 non-null   object 
 1   category                1297 non-null   object 
 2   name                    1297 non-null   object 
 3   size                    1297 non-null   object 
 4   rating                  1297 non-null   float64
 5   number_of_reviews       1297 non-null   int64  
 6   love                    1297 non-null   int64  
 7   price                   1297 non-null   float64
 8   value_price             1297 non-null   float64
 9   MarketingFlags          1297 non-null   bool   
 10  MarketingFlags_content  1297 non-null   object 
 11  options                 1297 non-null   object 
 12  how_to_use              1297 non-null   object 
 13  online_only             1297 non-null   int64  
 14  exclusive               1297 non-null   [...]<jupyter_text>Create another dataframe for makeup products for the face<jupyter_code>Face_List = ['Eye Palettes', 'Highlighter', 'Cheek Palettes', 'Face Primer', 'Color Correct', 'Tinted Moisturizer', 'Concealer', 'BB & CC Cream', 'Foundation', 'Setting Spray & Powder', 'Contour', 'Bronzer', 'Blush', 'Makeup', 'Makeup Palettes']
sephora_face = sephora_df_new[sephora_df_new['category'].isin(Face_List)]
sephora_face.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 1226 entries, 57 to 9156
Data columns (total 17 columns):
 #   Column                  Non-Null Count  Dtype  
---  ------                  --------------  -----  
 0   brand                   1226 non-null   object 
 1   category                1226 non-null   object 
 2   name                    1226 non-null   object 
 3   size                    1226 non-null   object 
 4   rating                  1226 non-null   float64
 5   number_of_reviews       1226 non-null   int64  
 6   love                    1226 non-null   int64  
 7   price                   1226 non-null   float64
 8   value_price             1226 non-null   float64
 9   MarketingFlags          1226 non-null   bool   
 10  MarketingFlags_content  1226 non-null   object 
 11  options                 1226 non-null   object 
 12  how_to_use              1226 non-null   object 
 13  online_only             1226 non-null   int64  
 14  exclusive               1226 non-null  [...]<jupyter_text>Explore each new dataframe.<jupyter_code>#There are 1297 fragrance products offered at Sephora 
#The average amount of loves for fragrance products is around 5816
sephora_fragrance['love'].describe()
#Face products total 1226 which is 71 less than fragrance 
#The average amount of loves for face products is around 37,246
sephora_face['love'].describe()<jupyter_output><empty_output><jupyter_text>Distribution of loves among fragrance products and their categories<jupyter_code>sns.relplot(x = sephora_fragrance["love"], y = sephora_fragrance["category"], size = sephora_fragrance["love"], sizes=(20,250), data = sephora_fragrance )
plt.title ('Fragrance Products: Their categories and amount of loves')
plt.xlabel('Loves')
plt.ylabel('Category')
plt.gcf().set_size_inches(15, 5)
plt.show()
#Let's look at how amount of loves are distributed among makeup products for the face and their categories
sns.relplot(x = sephora_face["love"], y = sephora_face["category"], size = sephora_face["love"], sizes=(20,250),hue = sephora_face["category"], data = sephora_face)
plt.title ('Face Products: Their categories and amount of loves')
plt.xlabel('Loves')
plt.ylabel('Category')
plt.gcf().set_size_inches(15, 5)
plt.show()
stats.ttest_ind(sephora_face['love'], sephora_fragrance['love'])
#The p-value is less than 0.5 which means there is a statistical signfigance between the fragrance products and face products.<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09231152027867628, 'fraction_numerical': 0.03300986978518703, 'mean_word_length': 4.302550571679859, 'pattern_counts': {'":': 0, '<': 69, '<?xml version=': 0, '>': 69, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '239131', 'license': 'CC-BY-3.0', 'n_tokens_mistral': 4088, 'n_tokens_neox': 3634, 'n_words': 1208, 'path': 'Sephora_&_Customer_Behavior_Capstone_3_Kat_Goodman_update.ipynb', 'repo_name': 'katgoodman/kat', 'size': '12067'}