<jupyter_start><jupyter_text>Library imports<jupyter_code>#usual imports
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt

#Outliers 
from scipy import stats
data = pd.read_csv('data/raw/Walmart_Store_sales.csv')
data.head()
print(f'There is {data.shape[0]} rows and {data.shape[1]} columns in this dataset')
print(f'Columns in this dataset {list(data.columns)}')
print(f'Overall missing values: \n{100*data.isnull().sum()/data.shape[0]}')
data.dtypes
data['Store'].value_counts()<jupyter_output><empty_output><jupyter_text>There is no missing values on the stores, we can convert it to an integer<jupyter_code>data.dtypes
data['Store'] = data['Store'].astype(int)
data.head(1)
data['Holiday_Flag'].isnull().sum()
data[data['Holiday_Flag'].isnull()]<jupyter_output><empty_output><jupyter_text>Re-assess Holiday_Flag column<jupyter_code>#Holidays = 27/08/2010 <NAME> Day in Texas will be transformed to 1 in Holiday_Flag
#The rest of the dates are not holidays according to the national register so we'll transform them to 0
data.loc[data.Date == "27-08-2010", "Holiday_Flag"] = 1.0
data['Holiday_Flag'].isnull().sum()
data['Holiday_Flag'] = data['Holiday_Flag'].replace(np.nan, 1.0)
data['Holiday_Flag'].isnull().sum()
days = data['Holiday_Flag'].value_counts()[0]
holidays = data['Holiday_Flag'].value_counts()[1]

print(f'There is {days} normal days in the dataset')
print(f'There is {holidays} holidays in the dataset')
data = data.dropna(subset=['Weekly_Sales'])
#Drop the nan values in Temperature/Fuel Price/CPI/Unemployment
data = data.dropna(subset=['Temperature','Fuel_Price','CPI','Unemployment'])
data.describe()<jupyter_output><empty_output><jupyter_text>Computing confidence interval taking 99,73%<jupyter_code>y23 = [data.Temperature.mean()-3*data.Temperature.std(),data.Temperature.mean()+3*data.Temperature.std()]
y24 = [data.Fuel_Price.mean()-3*data.Fuel_Price.std(),data.Fuel_Price.mean()+3*data.Fuel_Price.std()]
y25 = [data.CPI.mean()-3*data.CPI.std(),data.CPI.mean()+3*data.CPI.std()]
y26 = [data.Unemployment.mean()-3*data.Unemployment.std(),data.Unemployment.mean()+3*data.Unemployment.std()]
print(f'Any values outside of this interval within Temperature will be removed {y23}')
print(f'Any values outside of this interval within Fuel price will be removed {y24}')
print(f'Any values outside of this interval within CPI will be removed {y25}')
print(f'Any values outside of this interval within Unemployment will be removed {y26}')
#Therefore we have to focus on the outliers in unemployment 
data.drop(data[data.Unemployment > y26[1]].index, inplace=True)
data.describe()
print(f'After cleaning, there is only {data.shape[0]} rows left, we dropped {(100-((data.shape[0])/150)*100)}%')
data.head()<jupyter_output><empty_output><jupyter_text>EDA<jupyter_code>store_sales = data.groupby('Store')['Weekly_Sales'].sum()
first_shop = store_sales.sort_values(ascending = False).index[0]
print(f'The shop n°{first_shop} sold the most over one week')
plt.figure(figsize = (10, 5))
g = sns.barplot(data = data, x = 'Store', y = 'Weekly_Sales', color = 'RED')
g.set_title("Weekly sales per store")
plt.show()
#Even though the store #13 holds the record in the biggest weekly sale, 
#it isn't the best performance overall, #4 seems to be performing better (We have to take into account its 10^6)
data.corr()
plt.figure(figsize = (16,10))
sns.heatmap(data.corr(), cmap = 'Reds', annot = True)
plt.show()<jupyter_output><empty_output><jupyter_text>As we're speaking of sales, we can take into account two factors, the first would be the week aka date, when there are sales. The second factor could be the quarters as any financial firm would assess their performance.However, we dropped the date column, it could be a way of fine tuning our model, this could be explored in the future<jupyter_code>data = data[['Store','Weekly_Sales','Holiday_Flag','Temperature','Fuel_Price','CPI','Unemployment']]
#data.to_csv('./data/walmart_store_cleaned.csv', index = False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1443044230294045, 'fraction_numerical': 0.021497405485544848, 'mean_word_length': 7.0, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 27, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '62171', 'license': 'MIT', 'n_tokens_mistral': 1447, 'n_tokens_neox': 1363, 'n_words': 362, 'path': '03-Walmart Sales/walmart_store.ipynb', 'repo_name': 'anatolereffet/Jedha_portfolio', 'size': '4062'}