<jupyter_start><jupyter_text>**My main objective**: finish the first project cycle deploying a solution to the CFO so he can make his decisions.  0.0. IMPORTS<jupyter_code>import math
import pylab
import warnings
import inflection # used when renaming columns in subsection 1.1

import numpy               as np
import pandas              as pd
import seaborn             as sns
import matplotlib.pyplot   as plt
import matplotlib.gridspec as gs

from scipy import stats as ss
from IPython.display       import Image
from IPython.core.display  import HTML

warnings.filterwarnings( 'ignore' )<jupyter_output><empty_output><jupyter_text>Helper Functions Here there are some functions that will be helpful in the project<jupyter_code># setting some notebook display as default
def jupyter_settings():
    %matplotlib inline
    %pylab inline
    
    display( HTML( '<style>.container { width:100% !important; }</style>') )
    pd.options.display.max_columns = None
    pd.options.display.max_rows = None
    pd.set_option( 'display.expand_frame_repr', False )
    
    sns.set()
jupyter_settings()
pd.set_option( 'display.float_format', lambda x: '%.2f' % x )

# setting plot parameters as default
plt.rcParams[ 'figure.figsize' ] = [25, 12]
plt.rcParams[ 'font.size' ] = 24
sns.set_style( "white" )<jupyter_output><empty_output><jupyter_text>0.2. Loading Data<jupyter_code>df_sales_raw = pd.read_csv('../data/train.csv', low_memory = False)
df_store_raw = pd.read_csv('../data/store.csv', low_memory = False) # other store features

# merging datasets
df_raw = pd.merge(df_sales_raw, df_store_raw, how='left', on='Store')<jupyter_output><empty_output><jupyter_text>1.0. STEP 1: DATA DESCRIPTION<jupyter_code>df1 = df_raw.copy()<jupyter_output><empty_output><jupyter_text>1.1. Rename Columns We basically rename them to better manipulate and visualize the data, since the names become shortly and intuitive after renaming<jupyter_code>cols_old = ['Store', 'DayOfWeek', 'Date', 'Sales', 'Customers', 'Open', 'Promo','StateHoliday', 'SchoolHoliday', 'StoreType',
            'Assortment', 'CompetitionDistance', 'CompetitionOpenSinceMonth', 'CompetitionOpenSinceYear', 'Promo2', 
            'Promo2SinceWeek', 'Promo2SinceYear', 'PromoInterval']

# creating new names with a snakecase pattern
snakecase = lambda x: inflection.underscore( x )

# creating a list containing the new column names
cols_new = list(map(snakecase, cols_old))

# renaming the dataframe columns
df1.columns = cols_new<jupyter_output><empty_output><jupyter_text>1.2. Data Dimensions<jupyter_code>print(f'Number of Rows: {df1.shape[0]}')
print(f'Number of Columns: {df1.shape[1]}')<jupyter_output>Number of Rows: 1017209
Number of Columns: 18<jupyter_text>1.3. Data Types<jupyter_code>df1['date'] = pd.to_datetime( df1['date'] ) # changing 'date' column from object to pandas datetime
df1.dtypes<jupyter_output><empty_output><jupyter_text>1.4. Checking for NANs<jupyter_code>df1.isna().sum()<jupyter_output><empty_output><jupyter_text>1.5. Filling out NANs **competition_distance:**I'll assume that if the value is missing it's because there'no competitor close to the store. Thus if I input a distance greater than the maximum distance of the column it's the same as there'no competitor close.The maximum distance is 75.860m. So, if the value is missing, I will input 200.000m<jupyter_code>df1['competition_distance'] = df1['competition_distance'].apply( lambda x: 200000.0 if math.isnan( x ) else x )<jupyter_output><empty_output><jupyter_text>**competition_open_since_month** and **competition_open_since_year:**There can be two reasons for missing values in this column:- the store doesn't really have a close competitor.- The store does have a close competitor, but we don't know when it was opened. Maybe the competitor existed before or after the store opening.So, what I'll do: if the store has a NA in this column, I'll copy the date from this line to the competition_since_month. The main reason to do it is because I believe that this column is important to the model.For now, it's not a big problem because I'm working in cycles. Thus, if this assumption doesn't help in the modeling performance, I handle it in the next CRISP cycle.<jupyter_code># competition_open_since_month
df1['competition_open_since_month'].fillna( df1['date'].dt.month, inplace=True )
# competition_open_since_year
df1['competition_open_since_year'].fillna( df1['date'].dt.year, inplace=True )<jupyter_output><empty_output><jupyter_text>**promo2_since_week** and **promo2_since_year**:If there are missing values in these columns, it's because the store is not participating in promo2.So, we'll fill it out with the present date.<jupyter_code>df1['promo2_since_week'].fillna( df1['date'].dt.isocalendar().week, inplace=True )
df1['promo2_since_year'].fillna( df1['date'].dt.year, inplace=True )<jupyter_output><empty_output><jupyter_text>**promo_interval**:<jupyter_code>month_map = {1: 'Jan',  2: 'Feb',  3: 'Mar',  4: 'Apr',  5: 'May',  6: 'Jun',  7: 'Jul',  8: 'Aug',  9: 'Sep',  10: 'Oct', 11: 'Nov', 12: 'Dec'}

df1['promo_interval'].fillna('', inplace=True) 

# getting month abreviation
df1['month_map'] = df1['date'].dt.strftime( "%b" ) 

df1['is_promo'] = df1[['promo_interval', 'month_map']].apply(lambda x: x['month_map'] in x['promo_interval'].split( ',' ), axis='columns').astype('int')<jupyter_output><empty_output><jupyter_text>1.6. Change Data Types Some features has float as type but they should be integer. So, they're going to be changed.<jupyter_code>df1['competition_open_since_month'] = df1['competition_open_since_month'].astype( int )
df1['competition_open_since_year'] = df1['competition_open_since_year'].astype( int )
df1['promo2_since_week'] = df1['promo2_since_week'].astype( int )
df1['promo2_since_year'] = df1['promo2_since_year'].astype( int )<jupyter_output><empty_output><jupyter_text>1.7. Descriptive Statistics This subsection provides both a business notion (with metrics of dispersion and central tendency) and a detection of error in the data<jupyter_code>df1.dtypes
# dividing data in numerical and categorical
num_attributes = df1.select_dtypes( include = ['int32', 'int64', 'float64'] )
cat_attributes = df1.select_dtypes( exclude = ['int32', 'int64', 'float64', 'datetime64[ns]'] )<jupyter_output><empty_output><jupyter_text>1.7.1. Numerical Attributes<jupyter_code># Central Tendendy - mean, median
ct1 = pd.DataFrame( num_attributes.apply( np.mean ) ).T
ct2 = pd.DataFrame( num_attributes.apply( np.median ) ).T

# Dispersion - std, min, max, range, skewness, kurtosis
d1 = pd.DataFrame( num_attributes.apply( np.std ) ).T 
d2 = pd.DataFrame( num_attributes.apply( min ) ).T 
d3 = pd.DataFrame( num_attributes.apply( max ) ).T 
d4 = pd.DataFrame(num_attributes.quantile( .25 )).T
d5 = pd.DataFrame(num_attributes.quantile( .75 )).T
d6 = pd.DataFrame( num_attributes.apply( lambda x: x.max() - x.min() ) ).T 
d7 = pd.DataFrame( num_attributes.apply( lambda x: x.skew() ) ).T 
d8 = pd.DataFrame( num_attributes.apply( lambda x: x.kurtosis() ) ).T


# concatenating them and printing
m = pd.concat( [d2, d3, d6, d4,  ct2, d5, ct1, d1, d7, d8] ).T.reset_index()
m.columns = ['attributes', 'min', 'max', 'range', '25%','median','75%', 'mean', 'std', 'skewness', 'kurtosis'] # renaming columns
m
plt.figure(figsize=(20, 8))

plt.subplot( 1,3,1)
sns.distplot( df1['sales'] )
plt.title( 'Sales Distribution' )

plt.subplot( 1,3,2 )
sns.distplot( df1['competition_distance'] )
plt.title( 'Competition Distance Distribution' );

plt.subplot( 1,3,3 )
sns.distplot( df1['customers'] )
plt.title( 'Customers Distribution' );<jupyter_output><empty_output><jupyter_text>In EDA Section I'll check if sales distribution is or not normal.  1.7.1. Categorical Attributes<jupyter_code># taking a first look at the levels of each categorical variables
cat_attributes.apply( lambda x: x.unique().shape[0] )
aux = df1[( df1['state_holiday'] != '0' ) & ( df1['sales'] > 0 )]

plt.subplot( 1, 3, 1 )
sns.boxplot( x = 'state_holiday', y ='sales', data = aux )

plt.subplot( 1, 3, 2 )
sns.boxplot( x = 'store_type', y ='sales', data = aux )

plt.subplot( 1, 3, 3 )
sns.boxplot( x = 'assortment', y = 'sales', data = aux );<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1328390860476422, 'fraction_numerical': 0.02442877977637336, 'mean_word_length': 5.673965936739659, 'pattern_counts': {'":': 0, '<': 66, '<?xml version=': 0, '>': 67, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '301741', 'license': 'MIT', 'n_tokens_mistral': 2987, 'n_tokens_neox': 2842, 'n_words': 785, 'path': 'notebooks/m02_v01_store_sales_prediction.ipynb', 'repo_name': 'KattsonBastos/rossmann_sales_prediction', 'size': '8228'}