<jupyter_start><jupyter_text>Women Shoe Prices> An exploretory data analysis on women shoe price- toc: false - badges: true- comments: true- author: <NAME>- categories: [EDA, Python, Data Vidualization]- annotations: true- image: images- hide: false  Women's Shoe Prices IntroductionThis is a list of 10,000 women's shoes and their product information provided by Datafiniti's Product Database.The dataset includes shoe name, brand, price, and more. Each shoe will have an entry for each price found for it and some shoes may have multiple entries.  Import libraries<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

%matplotlib inline<jupyter_output><empty_output><jupyter_text>Loading Data<jupyter_code># load our dataset 
WomenShoe = pd.read_csv('Women_Shoe_Price.csv', parse_dates =
                        ['dateAdded', 'dateUpdated', 'prices.dateAdded', 'prices.dateSeen'] )

#dimension of our dataset
WomenShoe.shape<jupyter_output><empty_output><jupyter_text>Data Cleaning  Missing Values<jupyter_code># check the percent of missing values in each column. 
miss_val = (WomenShoe.isnull().sum()/len(WomenShoe)*100).sort_values(ascending=False)
miss_val[miss_val>0]
#Find the columns without missing value
(miss_val[miss_val==0].index)
#Drop thoses columns with high missing values and that are not useful in this analysis. 
WomenShoe = WomenShoe.drop(columns=
                           ['id', 'asins', 'count', 'descriptions', 
                            'dimension','ean', 'flavors', 'isbn','manufacturer', 
                            'manufacturerNumber', 'merchants', 'keys', 'imageURLs', 
                            'prices.availability', 'prices.color', 'prices.condition', 
                            'prices.count', 'prices.flavor', 'prices.merchant', 'prices.offer', 
                            'prices.returnPolicy', 'prices.shipping', 'prices.size', 'prices.source', 
                            'prices.warranty', 'quantities', 'reviews', 'skus', 'upc', 'websiteIDs', 
                            'weight', 'prices.sourceURLs','prices.dateAdded', 'prices.dateSeen', 
                            'dateUpdated', 'sizes', 'sourceURLs'] )
#Check the new dimension of the dataset
WomenShoe.shape
# Rename some of the columns 
WomenShoe = WomenShoe.rename(columns={"prices.amountMin": "MinPrice", 
                                      "prices.amountMax": "MaxPrice", "prices.currency": "currency"})
WomenShoe.head(8)<jupyter_output><empty_output><jupyter_text>Correct Data TypesActually, the data types for dateAdded, MinPrice, MaxPrice, and prices.isSale are all object when I first process this dataset. I tried to convert MinPrice and MaxPrice to float, but it didn't work. I looked at the data in Excel, I found there are so many rows with data that are overlapped with other columns, so I had to go over those 19,034 rows to find and correct the data manually.<jupyter_code>WomenShoe.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 19034 entries, 0 to 19033
Data columns (total 10 columns):
brand            18401 non-null object
categories       19034 non-null object
colors           11882 non-null object
dateAdded        19034 non-null datetime64[ns, UTC]
features         14099 non-null object
name             19034 non-null object
MinPrice         19034 non-null float64
MaxPrice         19034 non-null float64
currency         19034 non-null object
prices.isSale    19034 non-null bool
dtypes: bool(1), datetime64[ns, UTC](1), float64(2), object(6)
memory usage: 1.3+ MB<jupyter_text>Currency Change<jupyter_code>#add one column that is the aerage price of MinPrice and MaxPrice 
WomenShoe['average_price'] = (WomenShoe['MinPrice'] + WomenShoe['MaxPrice'])/2
#drop MinPrice and MaxPrice columns
WomenShoe.drop(columns=['MinPrice', 'MaxPrice'], axis=1, inplace=True)

WomenShoe.head() #9 columns left
# find the number of rows for each currency
WomenShoe.currency.value_counts()
# A function to convert each currency to USD. 
def curr_change(x):
    if x.currency == 'AUD':
        x.average_price = x.average_price * 0.73
    if x.currency == 'CAD':
        x.average_price = x.average_price * 0.76
    if x.currency == 'EUR':
        x.average_price = x.average_price * 1.19
    if x.currency == 'GBP':
        x.average_price = x.average_price * 1.33
    
    return x
WomenShoe[WomenShoe['currency'] != "USD"].head()
# Apply the currency change function to the datafram. 

WomenShoe = WomenShoe.apply(curr_change, axis=1)
currency_change = WomenShoe[WomenShoe['currency'] != "USD"]
currency_change.head()<jupyter_output><empty_output><jupyter_text>Uniform Brand Names<jupyter_code>WomenShoe.brand.value_counts()
# A dictionary for uniform the brand names
brand_map = {'a2 by aerosoles':'aerosoles', "what's what by aerosoles":'aerosoles',
'adidas outdoor':'adidas',
'adriana new york':'adriana',
'alexander mcqueen by puma':'puma', 'alexander mcqueen':'puma',
'alpine':'alpine swiss',
'anne klein ak':'anne klein', 'anne klein sport':'anne klein',
'annie shoes':'annie',
'athena':'<NAME>',
'babe.':'babe',
'baretraps':'bare traps',
'bcbg max azria':'bcbg', 'bcbg paris':'bcbg', 'bcbgeneration':'bcbg',
'beacon shoes':'beacon',
'bebe sport':'bebe', 'bebe stu':'bebe',
'belle by sigerson morrison':'belle', 'belle sigerson morrison':'belle',
'bernie mev':'bernie', 'bernie mev.':'bernie',
'bettye by bettye muller':'bettye muller', 'bettye by bettye muller ':'bettye muller',
"breckelle's":'breckelles',
'callaway footwear':'callaway',
'calvin klein ck':'<NAME>', 'calvin klein jeans':'<NAME>lein',
'carlos by <NAME>':'<NAME>',
'charles by char<NAME>avid':'charels david',
'see by chloe':'chloe',
'clarks artisan':'clarks', 'clarks artisan collection':'clarks', 'clarks collection':'clarks',
'cobb hill by new balance':'new balance', 'cobb hill':'new balance',
'maria sharapova by cole haan': 'cole hann',
"corky's":'corkys', "corky's footwear":'corkys', "corkys footwear, inc.":'corkys',
'dearforms':'dearfoams', 'df by dearfoams':'dearfoams',
'derek lam 10 crosby':'derek lam',
'diba.true':'dibatrue',
'dolce&gabbana':'dolce and gabbana', 'dolce & gabbana':'dolce and gabbana', 'dolce by mojo moxy':'dolce and gabbana',
  'dolce vita':'dolce and gabbana', 'dv8 by dolce vita':'dolce and gabbana', 'dv by dolce vita':'dolce and gabbana',
"dr. scholl's":'dr scholls',
"dr. martens air wair":'drmartens',
'drew shoe':'drew',
'easy spirit e360':'easy spirit', 'easy spirit.':'easy spirit',
'ellie shoes':'ellie',
'emu australia':'emu',
'fergie footwear':'fergie',
'forever collectible':'forever', 'forever link':'forever', 'fourever funky':'forever',
'sarto by <NAME>':'fr<NAME>to',
'ferriniusa':'ferrini',
'fitflop':'fit flop',
'funtasma by pleaser':'funtasma',
'g by guess':'guess',
'gc shoes':'gc',
'genuine grip footwear':'genuine grip',
"hogan by tod's":'hogan',
'soft style by hush puppies':'hush puppies',
'ilse jacobsen hornbaek':'ilse jacobson',
'isaacmizrahi':'<NAME>',
'italian shoe makers':'italian comfort',
'j.renee':'j. renee',
'jbu by jambu':'jambu',
'josefseibel':'josef siebel',
'justin blair':'justin', 'justin boots':'justin', 'justin gypsy':'justin',
'kate spade new york':'kate spade',
'kenneth cole reaction':'kenneth cole', 'kenneth cole ny':'kenneth cole', 'kenneth cole new york':'kenneth cole',
 'unlisted kenneth cole':'kenneth cole',
'lamo sheepskin inc':'lamo',
'lifestride':'lifes tride',
'luoluo':'luo luo',
'marc fisher ltd':'marc fisher',
'mia heritage':'mia',
'<NAME>':'<NAME>', '<NAME>':'<NAME>',
'mobils by mephisto':'mephisto',
'top moda':'moda', 'moda essentials':'moda', 'everybody by bz moda':'moda',
'muk luks a la mode':'muk luks',
'munro american':'munro',
'naot footwear':'naot',
'new@titude':'new attitude', 'new@ttitude':'new attitude',
'nina originals':'nina',
'nine west vintage america collection':'nine west',
'nufoot���':'nufoot',
'pleaser shoes':'pleaser', 'pleaser usa, inc.':'pleaser', 'pleaserusa':'pleaser',
'rachel':'rachel roy', 'rachel rachel roy ':'rachel roy',
'lauren by <NAME>':'<NAME>', '<NAME>':'<NAME>', '<NAME>':'<NAME>',
 '<NAME>':'<NAME>','<NAME> supply':'<NAME>',
'rieker-antistress':'rieker',
'rocket dog brands llc':'rocket dog',
'sanita clogs':'sanita',
'ferragamo':'salvatore ferragamo',
'skechers usa':'skechers',
'sperry top sider':'sperry', 'sperry top-sider':'sperry',
"l'artiste by spring step":'spring step', "flexus by spring step":'spring step', "patrizia by spring step ":'spring step',
 "patrizia pepe":'spring step', "patrizia":'spring step',
'steven steve madden':'steve madden',
'style & co.':'style and co',
'timberland earthkeepers':'timberland', 'timberland pro':'timberland',
'toms shoes':'toms',
'tony lama boot co.':'tony lama',
'totes isotoner':'totes',
'trotter':'trotters',
'ugg australia':'ugg',
'famous name brand':'unbranded', 'generic':'unbranded', 'generic surplus':'unbranded', 'non-branded':'unbranded',
 'not applicable':'unbranded', 'not rated':'unbranded', 'lucky  brand':'unbranded', 'lucky brand':'unbranded',
 'very fine dance shoes':'unbranded',
'valentino noir':'valentino',
'victoria k.':'victoria',
'vince camuto':'vince',
'vionic by orthaheel':'vionic', 'vionic with orthaheel technology':'vionic',
'elites by walking cradles':'walking cradles', 'elites':'walking cradles','mark lemp by walking cradles':'walking cradles',
 'rose petals by walking cradles':'walking cradles', 'the walking cradle company':'walking cradles'
}<jupyter_output><empty_output><jupyter_text>Reference this note book https://www.kaggle.com/ashishg21/data-cleaning-and-some-analysis-shoe-prices.<jupyter_code># convert all brand names to lower case
WomenShoe['brand'] = WomenShoe['brand'].str.lower()
# replace the brand names with the dictionary
WomenShoe['brand_clean'] = WomenShoe['brand'].replace(brand_map)
WomenShoe.brand_clean.value_counts()<jupyter_output><empty_output><jupyter_text>EDA  Color & Price Analysis  What are the 10 most popular colors for women's shoe?<jupyter_code>#find the number of rows for each color
WomenShoe.colors.value_counts()
color_rank = WomenShoe.colors.value_counts().head(10)
# append 10 colors with the highest count in to a list
rows = []
for i in range(10):
    rows.append([color_rank.index[i], color_rank[i]])

# convert the list to a pandas dataframe
Col_Rank = pd.DataFrame(rows, columns=["color", "count"])
# A bar chart for the 10 colors with the highest count in the dataset
plt.style.use('ggplot')
fig, ax = plt.subplots(figsize=(12, 5))
Col_Rank = Col_Rank.sort_values('count')
ax.barh(Col_Rank['color'], Col_Rank['count'], color=
        ['pink', 'green', 'silver', 'red', 'gray', 'white', 'blue', 'beige', 'brown', 'black']) 
ax.set_ylabel("Color")
ax.set_xlabel("Count")
ax.set_title("Top 10 Color Count")
plt.show()<jupyter_output><empty_output><jupyter_text>Does the mediem price of each color also follow the color rank in the plot above?<jupyter_code># Create dataframes for each color 
Black = WomenShoe[WomenShoe['colors'] == "Black"]
Brown = WomenShoe[WomenShoe['colors'] == "Brown"]
Beige = WomenShoe[WomenShoe['colors'] == "Beige"]
Blue = WomenShoe[WomenShoe['colors'] == "Blue"]
White = WomenShoe[WomenShoe['colors'] == "White"]
Gray = WomenShoe[WomenShoe['colors'] == "Gray"]
# Make a copy of dataframes for each color 
Black1 = WomenShoe[WomenShoe['colors'] == "Black"]
Brown1 = WomenShoe[WomenShoe['colors'] == "Brown"]
Beige1 = WomenShoe[WomenShoe['colors'] == "Beige"]
Blue1 = WomenShoe[WomenShoe['colors'] == "Blue"]
White1 = WomenShoe[WomenShoe['colors'] == "White"]
Gray1 = WomenShoe[WomenShoe['colors'] == "Gray"]
# define a function to find the outliers
def outlier(x):
    y = (x['average_price'].quantile(0.75) - x.average_price.quantile(0.25)) * 1.5 + x.average_price.quantile(0.75)
    return y
print(outlier(Black))
print(outlier(Brown))
print(outlier(Beige))
print(outlier(Blue))
print(outlier(White))
print(outlier(Gray))

#Remove the outliers for black shoes
Black.drop(Black[Black['average_price'] > 167].index, inplace = True)
#Remove the outliers for Brown shoes
Brown.drop(Brown[Brown['average_price'] > 205].index, inplace = True)
#Remove the outliers for Beige shoes
Beige.drop(Beige[Beige['average_price'] > 154].index, inplace = True)
#Remove the outliers for Blue shoes
Blue.drop(Blue[Blue['average_price'] > 121].index, inplace = True)
#Remove the outliers for White shoes
White.drop(White[White['average_price'] > 145].index, inplace = True)
#Remove the outliers for Beige shoes
Gray.drop(Gray[Gray['average_price'] > 191].index, inplace = True)

# the boxplot for the top 6 colors
plt.style.use('ggplot')
fig, ax = plt.subplots(figsize=(15, 7)) 
ax.boxplot([Black["average_price"], Brown["average_price"], Beige["average_price"], 
            Blue["average_price"], White["average_price"], Gray["average_price"]]) 
ax.set_xticklabels(["Black", "Brown", "Beige", "Blue", "White", "Gray"])
ax.set_ylabel("Price") 
ax.set_xlabel("Color") 
ax.set_title('Box Plot for the Top 6 Colors')
plt.show()<jupyter_output><empty_output><jupyter_text>How prices are distributed in the top six colors?<jupyter_code># use 3 x 2 subplots to plot six histgrams for the top six colors
plt.style.use('ggplot')
fig, ax = plt.subplots(3, 2, figsize=(15, 8))
ax[0, 0].hist(Black1["average_price"], bins=100, range=(0,500), color = 'black')
ax[0,0].set_title('Black Shoes')
ax[0,0].set_ylabel('Frequency')

ax[0, 1].hist(Brown1["average_price"], bins=100, range=(0,500), color = 'Brown')
ax[0,1].set_title('Brown Shoes')

ax[1, 0].hist(Beige1["average_price"], bins=100, range=(0,500), color = 'Beige')
ax[1,0].set_title('Beige Shoes')
ax[1,0].set_ylabel('Frequency')

ax[1, 1].hist(Blue1["average_price"], bins=100, range=(0,500), color = 'Blue')
ax[1,1].set_title('Blue Shoes')

ax[2, 0].hist(White1["average_price"], bins=100, range=(0,500), color = 'white')
ax[2,0].set_title('White Shoes')
ax[2,0].set_xlabel('Price')
ax[2,0].set_ylabel('Frequency')

ax[2, 1].hist(Gray1["average_price"], bins=100, range=(0,500), color = 'gray')
ax[2,1].set_title('Gray Shoes')
ax[2,1].set_xlabel('Price')

plt.tight_layout()#Get rid of overlaps
plt.show()
#chi-squared distribution<jupyter_output><empty_output><jupyter_text>What is the average price for the top 10 colors in decending order?<jupyter_code>#append the top 10 colors to a list 
y = WomenShoe.colors.value_counts().head(10)
Colors = []
for i in range (10):
    Colors.append([y.index[i], y[i]])

# filter the origanial dataframe and keep all the rows that has those top ten colors
dfc = []
for i in range (len(Colors)):
    dfc.append(Colors[i][0])
    color_df = WomenShoe[WomenShoe['colors'].isin(dfc)]
#sort the mean price for each color shoes in decending order.
color_df.groupby('colors')['average_price'].mean().sort_values(ascending=False)<jupyter_output><empty_output><jupyter_text>Brand & Price Analysis  Which brand have the highest price?<jupyter_code>#append the brands with at least 5 rows in our dataset to a list
x = WomenShoe.brand_clean.value_counts()
brands = []
for i in range (1897):
    if x[i] > 5:
        brands.append([x.index[i], x[i]])

#filter the origanial dataframe and keep all the brands that have at least 5 rows in the dataset
df = []
for i in range (len(brands)):
    df.append(brands[i][0])
    WS_Brand = WomenShoe[WomenShoe['brand_clean'].isin(df)]


#Calculate the average price for each brand and only keep the top 20 brands
df_brand = WS_Brand.groupby('brand_clean')['average_price'].mean().sort_values(ascending=False).head(20)
Over5 = []
for i in range (len(df_brand)):
    Over5.append([df_brand.index[i], df_brand[i]])
    
Over5_df = pd.DataFrame(Over5, columns=["brand", "average_price"])
#draw a bar chart for 20 brands with the highest average price
plt.style.use('ggplot')
fig, ax = plt.subplots(figsize=(12, 10))
Over5_df = Over5_df.sort_values('average_price')
ax.barh(Over5_df['brand'], Over5_df['average_price'], color= "blue") 
ax.set_ylabel("Brands")
ax.set_xlabel("Average Price")
ax.set_title("Top 20 brands with Count over 5")
plt.show()
#append the brands with at least 50 rows in our dataset to a list
brands = []
for i in range (1897):
    if x[i] > 50:
        brands.append([x.index[i], x[i]])

#filter the origanial dataframe and keep all the brands that have at least 50 rows in the dataset
df = []
for i in range (len(brands)):
    df.append(brands[i][0])
    WS_Brand = WomenShoe[WomenShoe['brand_clean'].isin(df)]

    
#Calculate the average price for each brand and only keep the top 20 brand
df_brand = WS_Brand.groupby('brand_clean')['average_price'].mean().sort_values(ascending=False).head(20)
Over5 = []
for i in range (len(df_brand)):
    Over5.append([df_brand.index[i], df_brand[i]])
Over5_df = pd.DataFrame(Over5, columns=["brand", "average_price"])

#draw a bar chart for 20 brands with the highest average price
plt.style.use('ggplot')
my_cmap = plt.get_cmap("viridis")
fig, ax = plt.subplots(figsize=(12, 10))
Over5_df = Over5_df.sort_values('average_price')
ax.barh(Over5_df['brand'], Over5_df['average_price'], color= my_cmap.colors ) 
ax.set_ylabel("Brands")
ax.set_xlabel("Average Price")
ax.set_title("Top 20 brands with Count over 50")
plt.show()<jupyter_output><empty_output><jupyter_text>Specific Brand & Price Analysis  Which ones have the widest distribution of prices? Is there a typical price distribution across brands or within specific brands?<jupyter_code># a histgram to show the price distribution across all brands
plt.style.use('ggplot')
fig, ax = plt.subplots()
ax.hist(WomenShoe["average_price"], bins=50, range=(0,800), color = 'orange')
ax.set_title('Price Distribution Across Brands')
ax.set_xlabel("Price")
ax.set_ylabel("Frequency")
plt.show()
# Create dataframes for those six brands that I was interested in 
Ralph_lauren = WomenShoe[WomenShoe['brand_clean'] == "<NAME>"]
Ugg = WomenShoe[WomenShoe['brand_clean'] == "ugg"]
Nike = WomenShoe[WomenShoe['brand_clean'] == "nike"]
Puma = WomenShoe[WomenShoe['brand_clean'] == "puma"]
Toms = WomenShoe[WomenShoe['brand_clean'] == "toms"]
Vans = WomenShoe[WomenShoe['brand_clean'] == "vans"]
# use 3 x 2 subplots to plot the price distribution for Ralph_lauren, Ugg, Nike, Puma, Toms, and Vans using histgrams.
plt.style.use('ggplot')
fig, ax = plt.subplots(3, 2, figsize=(15, 8))
ax[0, 0].hist(Ralph_lauren["average_price"], bins=50, range=(0,1500), color = 'cadetblue')
ax[0,0].set_title('<NAME>')
ax[0,0].set_ylabel("Frequency")


ax[0, 1].hist(Ugg["average_price"], bins=50, range=(0,500), color = 'lightblue')
ax[0,1].set_title('Ugg')

ax[1, 0].hist(Nike["average_price"], bins=50, range=(0,500), color = 'skyblue')
ax[1,0].set_title('Nike')
ax[1,0].set_ylabel("Frequency")

ax[1, 1].hist(Puma["average_price"], bins=50, range=(0,500), color = 'steelblue')
ax[1,1].set_title('Puma')

ax[2, 0].hist(Toms["average_price"], bins=50, range=(0,500), color = 'dodgerblue')
ax[2,0].set_title('Toms')
ax[2,0].set_ylabel("Frequency")
ax[2,0].set_xlabel("Average Price")

ax[2, 1].hist(Vans["average_price"], bins=50, range=(0,500), color = 'deepskyblue')
ax[2,1].set_title('Vans')
ax[2,1].set_xlabel("Average Price")

plt.tight_layout()#Get rid of overlaps
plt.show()
#density plot for Ralph Lauren
Ralph_lauren['average_price'].plot(kind='density', color = 'cadetblue')
plt.title('Price Distribution for <NAME>')
plt.xlabel('Price')
plt.show()
#density plot for Ugg
Ugg['average_price'].plot(kind='density', color = 'lightblue')
plt.title('Price Distribution for Ugg')
plt.xlabel('Price')
plt.show()
#density plot for Nike
Nike['average_price'].plot(kind='density', color = 'skyblue')
plt.title('Price Distribution for Nike')
plt.xlabel('Price')
plt.show()
#density plot for Puma
Puma['average_price'].plot(kind='density', color = 'steelblue')
plt.title('Price Distribution for Puma')
plt.xlabel('Price')
plt.show()
#density plot for Toms
Toms['average_price'].plot(kind='density', color = 'dodgerblue')
plt.title('Price Distribution for Toms')
plt.xlabel('Price')
plt.show()
#density plot for Vans
Vans['average_price'].plot(kind='density', color = 'deepskyblue')
plt.title('Price Distribution for Vans')
plt.xlabel('Price')
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1628230616302187, 'fraction_numerical': 0.020825049701789264, 'mean_word_length': 5.9382758620689655, 'pattern_counts': {'":': 16, '<': 80, '<?xml version=': 0, '>': 90, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '79162', 'license': 'Apache-2.0', 'n_tokens_mistral': 7649, 'n_tokens_neox': 6983, 'n_words': 1804, 'path': '_notebooks/2020-12-01-Women-Shoes.ipynb', 'repo_name': 'Shudi-Zhao/My_Data_Science_Blog', 'size': '20303'}