<jupyter_start><jupyter_text>Summary The main purpose of this project is that we want to figure out what features matter most in retail business and there is two parts of it:real retail shop and online retail.In order to have a close look to real retail,We pick the dataset of Walmart sale forecasting.There are three csv :stores,features and sales.And train&test dataset is already offered.We also can make some prediction of week sales based on median of the weekly sales grouped by the Type of Store, Store & Department number, Month and Holiday dummy.<jupyter_code>import pandas as pd 
import numpy as np
from numpy import *
import gc
import time
import warnings
import os
from datetime import datetime
from datetime import timedelta
from scipy.misc import imread
from scipy import sparse
import scipy.stats as ss
import math
import matplotlib
import matplotlib.pyplot as plt
import seaborn as sns
import plotly
import plotly.offline as py
import plotly.tools as tls
import plotly.graph_objs as go
import plotly.tools as tls
from sklearn.ensemble import RandomForestRegressor
from sklearn.utils.validation import check_X_y, check_is_fitted
from sklearn.linear_model import LogisticRegression
from sklearn import metrics
from sklearn.metrics import log_loss
from sklearn.model_selection import StratifiedKFold
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import accuracy_score, confusion_matrix
from scipy import sparse
import keras
from keras.layers.core import Dense, Activation, Dropout
from keras.layers.recurrent import LSTM
from keras.layers import Bidirectional
from keras.models import Sequential
from keras import regularizers
from keras import optimizers
from sklearn.metrics import mean_squared_error<jupyter_output>Using TensorFlow backend.<jupyter_text>Input the csv.The lable divide is to make sure which part of data is from test/train dataset.<jupyter_code>features=pd.read_csv('Features data set.csv')
sales=pd.read_csv('sales data-set.csv')
stores=pd.read_csv('stores data-set.csv')
train1=pd.read_csv('train1.csv')
test1=pd.read_csv('test1.csv')
df=stores
df2=sales<jupyter_output><empty_output><jupyter_text>As a matter of fact,there are many factor effect on the retail sale.I have collected many features like CPI,ISHLIDAY and Emloyee rate.To have further reaserch,Let's use merge function to create a join of all three data sets.<jupyter_code>train1['Divide'] = 'Value_Train'
test1['Divide'] = 'Value_test'
length_train = len(train1) # Count how many train data we have
df = pd.concat([train1,test1],axis=0) # Merge the two dataset into one<jupyter_output><empty_output><jupyter_text>MarkDown1-5 is anonymized data that is related to promotional markdowns that Walmart is running and since we don't know what it is so let's drop it first.But if other features make little contribution to price forecasting,we should pick up MarkDowns again.<jupyter_code>df.head()<jupyter_output><empty_output><jupyter_text>Using the autocorrelation function caculate the correlation with diffrent features.First thing we know is that the weekly sales is mostly depend on size of the store.Dept also have some positive impact on sales.<jupyter_code>sns.set(style="white")
corr = df.corr()
plt.figure(figsize=(10,10))
sns.heatmap(corr,annot=True, fmt=".3f",xticklabels=corr.columns.values,yticklabels=corr.columns.values)
plt.show()<jupyter_output><empty_output><jupyter_text>Except Week_Sales itself,actually stores is just the name of diffentrent shops.Besides,the impact of Is_holiday is not that obvious.Before creating the predict model,some pre-process is needed.<jupyter_code>%matplotlib inline
df.loc[df['Weekly_Sales'] >300000]
df.loc[df['Weekly_Sales'] >240000,"Date"].value_counts()
df.fillna(0, inplace=True)<jupyter_output><empty_output><jupyter_text>Change the format of the value<jupyter_code>#df['Type'] = 'Type_' + df['Type'].map(str)
df['Store'] = 'Store_' + df['Store'].map(str)
df['Dept'] = 'Dept_' + df['Dept'].map(str)
df['IsHoliday'] = 'IsHoliday_' + df['IsHoliday'].map(str)
#Convert categorical variable into dummy/indicator variables.type_dummies = pd.get_dummies(df['Type'])
store_dummies = pd.get_dummies(df['Store'])
dept_dummies = pd.get_dummies(df['Dept'])
holiday_dummies = pd.get_dummies(df['IsHoliday'])<jupyter_output><empty_output><jupyter_text>The value of time itself is string and it can't use directly since it makes no sense saying July make the price uprising.What we do here is to parse the date(String) to datetime and find the Promotion Holiday(black friday and christmas) in them,which may make contribution to price change.<jupyter_code>df['DateType'] = [datetime.strptime(date,'%Y-%m-%d').date() for date in df['Date'].astype(str).values.tolist()]
df['Month'] = [date.month for date in df['DateType']]
df['Month'] = 'Month_' + df['Month'].map(str)
Month_dummies = pd.get_dummies(df['Month'] )
#Regular Expression.The code in the brackets after "where"is the group and "|" shows each specific string we want.
df['Black_Friday'] = np.where((df['DateType']==datetime(2010, 11, 26).date()) | (df['DateType']==datetime(2011, 11, 25).date()), 'yes', 'no')
df['Pre_christmas'] = np.where((df['DateType']==datetime(2010, 12, 23).date()) | (df['DateType']==datetime(2010, 12, 24).date()) | (df['DateType']==datetime(2011, 12, 23).date()) | (df['DateType']==datetime(2011, 12, 24).date()), 'yes', 'no')
df['Black_Friday'] = 'Black_Friday_' + df['Black_Friday'].map(str)
df['Pre_christmas'] = 'Pre_christmas_' + df['Pre_christmas'].map(str)
Black_Friday_dummies = pd.get_dummies(df['Black_Friday'] )
Pre_christmas_dummies = pd.get_dummies(df['Pre_christmas'] )
df = pd.concat([df,holiday_dummies,Pre_christmas_dummies,Black_Friday_dummies],axis=1)<jupyter_output><empty_output><jupyter_text>Caculate the average sale and create the medians column.<jupyter_code>medians = pd.DataFrame({'The_Median' :df.loc[df['Divide']=='Value_Train'].groupby(by=['Dept','Store','Month','IsHoliday'])['Weekly_Sales'].median()}).reset_index()
medians.head()
df = df.merge(medians, how = 'outer', on = ['Dept','Store','Month','IsHoliday'])
df['The_Median'].fillna(df['The_Median'].loc[df['Divide']=='Value_Train'].median(), inplace=True) 
df['Key'] = df['Dept'].map(str)+df['Store'].map(str)+df['Date'].map(str)+df['IsHoliday'].map(str)<jupyter_output><empty_output><jupyter_text>As you can see in the following diagram,there boolean value "0" and "1" to show it is holiday or not and meadian value is added.<jupyter_code>df.head()<jupyter_output><empty_output><jupyter_text>In some period of time we can take the sale as a continues data.So it is a little bit like LSTM that we decide to take the last week sale as one of the features.If the sales data is inaccessible the column will be filled with median data.<jupyter_code>df['DateLagged'] = df['DateType']- timedelta(days=7)
df.head()
sorted_df = df.sort_values(['Store', 'Dept','DateType'], ascending=[1, 1,1])
sorted_df = sorted_df.reset_index(drop=True)<jupyter_output><empty_output><jupyter_text>Sorting makes lagge process easy and the following loop is to set the laggedSales value.The percentage shows how much progress we have made.<jupyter_code>sorted_df['LaggedSales'] = np.nan
sorted_df['LaggedAvailable'] = np.nan 
last=df.loc[0] 
row_len = sorted_df.shape[0]
for index, row in sorted_df.iterrows():
    lag_date = row["DateLagged"]
    if((last['DateType']== lag_date) & (last['Weekly_Sales']>0)): 
        sorted_df.set_value(index, 'LaggedSales',last['Weekly_Sales'])
        sorted_df.set_value(index, 'LaggedAvailable',1)
    else:
        sorted_df.set_value(index, 'LaggedSales',row['The_Median'])
        sorted_df.set_value(index, 'LaggedAvailable',0)

    last = row 
    if(index%int(row_len/10)==0): 
        print(str(int(index*100/row_len))+'% loaded')
sorted_df[['Dept', 'Store','DateType','LaggedSales','Weekly_Sales','The_Median']].head()<jupyter_output><empty_output><jupyter_text>Take the risk of losing data,we use inner join here.The table created is the sales infromation in different dates for one department in one shop.<jupyter_code>df = df.merge(sorted_df[['Dept', 'Store','DateType','LaggedSales','LaggedAvailable']], how = 'inner', on = ['Dept', 'Store','DateType'])
df['Sales_dif'] = df['The_Median'] - df['LaggedSales']
df[['Dept', 'Store','DateType','LaggedSales','Weekly_Sales','The_Median']].head()
 df_backup = df
df['Disparity'] = df['The_Median'] - df['Weekly_Sales']<jupyter_output><empty_output><jupyter_text>We just add some new features and let's call the correlate funcation and take all features into it.<jupyter_code># This part of code is from https://www.kaggle.com/codemonkeyhq/forecasting-challenge/edit
sns.set(style="white")
corr = df.corr()
# Generate a mask for the upper triangle
mask = np.zeros_like(corr, dtype=np.bool)
mask[np.triu_indices_from(mask)] = True
f,ax = plt.subplots(figsize=(11, 9))
# Generate a custom diverging colormap
cmap = sns.diverging_palette(220, 10, as_cmap=True)
# Draw the heatmap with the mask and correct aspect ratio
sns.heatmap(corr, mask=mask, cmap=cmap, vmax=.3, center=0,square=True, linewidths=.5, cbar_kws={"shrink": .5})<jupyter_output><empty_output><jupyter_text>Obviously the new column we just added like median an lagged sales do play a role in sale change.There is another problem that features csv contains null and since we use join our dataframe there are many null value here.In order to make it up,md_present is added.<jupyter_code>df2=pd.merge(sales,features, on=['Store','Date', 'IsHoliday'], how='left')
df2=pd.merge(df2,stores, on=['Store'], how='left')
df2['Store'] = 'Store_' + df2['Store'].map(str)
df2['DateType'] = [datetime.strptime(date,'%d/%m/%Y').date() for date in df2['Date'].astype(str).values.tolist()]
df2=pd.merge(df,df2, on=['Store','DateType','Weekly_Sales'], how='left')
df2 = df2.assign(md1_present = df2.MarkDown1.notnull())
df2 = df2.assign(md2_present = df2.MarkDown2.notnull())
df2 = df2.assign(md3_present = df2.MarkDown3.notnull())
df2 = df2.assign(md4_present = df2.MarkDown4.notnull())
df2 = df2.assign(md5_present = df2.MarkDown5.notnull())
df=df2<jupyter_output><empty_output><jupyter_text>Our pre-process is done.Now we need to review what input features we gonna use and use and group them together.It will be easy to pick up all input features using the selector we make.<jupyter_code>selector = [
    'CPI',
    'Fuel_Price',
    'MarkDown1',
    'MarkDown2',
    'MarkDown3',
    'MarkDown4',
    'MarkDown5',
    'Size',
    'Temperature',
    'Unemployment',
    'md1_present',
    'md2_present',
    'md3_present',
    'md4_present',
    'md5_present',

    'IsHoliday_False',
    'IsHoliday_True',
    'Pre_christmas_no',
    'Pre_christmas_yes',
    'Black_Friday_no',
    'Black_Friday_yes',    
    'LaggedSales',
    'Sales_dif',
    'LaggedAvailable'
    ]
display(df2[selector].describe())
display(df2[selector].head())<jupyter_output><empty_output><jupyter_text>As we have merged the train and test dataset together,it's time to split it.<jupyter_code>train = df2.loc[df2['Divide']=='Value_Train']
test = df2.loc[df2['Divide']=='Value_test']
test<jupyter_output><empty_output><jupyter_text>Unluckly this test file can't provide sale infromation which means we can not "test" the output of out model.The solution here is to regard 20% of train data as development(test).<jupyter_code>np.random.seed(42)
X_train, X_dev, y_train, y_dev = train_test_split(train[selector], train['Disparity'], test_size=0.2, random_state=42)
X_train=X_train.fillna(0)
X_dev=X_dev.fillna(0)
y_train=y_train.fillna(0)
y_dev=y_dev.fillna(0)<jupyter_output><empty_output><jupyter_text>We choose to pick radomforest method here.We set the number of trees in the forest 20,"mse" means the function to measure the quality of a split,while None is set then nodes are expanded until all leaves are pure or until all leaves contain less than min_samples_split samples.min_samples_leaf shows The minimum number of samples required to be at a leaf node.max_leaf_nodesis the  grow trees with max_leaf_nodes in best-first fashion. Best nodes are defined as relative reduction in impurity.we set None here then we have unlimited number of leaf nodes.bootstrap is a boolean value:Whether bootstrap samples are used when building trees.oob_score is whether to use out-of-bag samples to estimate the R^2 on unseen data,meanwhile false is the default value.n_jobs is a integer number of jobs to run in parallel for both fit and predict. If -1, then the number of jobs is set to the number of cores.Random_state is also an integer shows RandomState instance or None, optional (default=None)If int, random_state is the seed used by the random number generator; If RandomState instance, random_state is the random number generator; If None, the random number generator is the RandomState instance used by np.random.verbose is used to controls the verbosity of the tree building process.This part is from radomForest documentation<jupyter_code>#A random forest is a meta estimator that fits a number of classifying decision trees on various sub-samples
#of the dataset and use averaging to improve the predictive accuracy and control over-fitting. The sub-sample 
#size is always the same as the original input sample size but the samples are drawn with replacement if 
#bootstrap=True (default).

regr = RandomForestRegressor(n_estimators=20, criterion='mse', max_depth=None, 
                      min_samples_split=2, min_samples_leaf=1, 
                      min_weight_fraction_leaf=0.0, max_features='auto', 
                      max_leaf_nodes=None, min_impurity_decrease=0.0, 
                      min_impurity_split=None, bootstrap=True, 
                      oob_score=False, n_jobs=1, random_state=None, 
                      verbose=2, warm_start=False)

#Train on data
regr.fit(X_train, y_train.ravel())
y_pred_random = regr.predict(X_dev)

y_dev = y_dev.to_frame()<jupyter_output>[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s
[Parallel(n_jobs=1)]: Done  20 out of  20 | elapsed:    1.1s finished<jupyter_text>What we do here is transfroming the trendcy into numbers:if price is going up then the direction is 1 else it is 0.Then just like correlation heatmap random forest confusion matrix is shown.Y is the catagory from the data(y developing value from train set) and X is the predicted value.<jupyter_code>def binary(input_data):
    direction = np.empty(input_data.shape[0])
    for i in range(input_data.shape[0]):
        if input_data[i] > 0:
            direction[i] = 1
        else:
            direction[i]= 0
    return direction

direction_true1= binary(y_dev.values)
direction_predict = binary(y_pred_random)

#shows the random forest confusion matrix 
cnf_matrix = confusion_matrix(direction_true1, direction_predict)
fig, ax = plt.subplots(1)
ax = sns.heatmap(cnf_matrix, ax=ax, cmap=plt.cm.Greens, annot=True)
plt.title('random forest predictions matrix ')
plt.ylabel('True category')
plt.xlabel('Predicted category')
plt.show();
y_dev['Predicted'] = y_pred_random
df_out = pd.merge(train,y_dev[['Predicted']],how = 'left',left_index = True, right_index = True,suffixes=['_True','_Pred'])
df_out = df_out[~pd.isnull(df_out['Predicted'])]
df_out.head()<jupyter_output><empty_output><jupyter_text>Plotting is the best way to show distribution of data."Predicted" is the column based on the direction we made and the prediction depends on both of median value of sale and "predictd".<jupyter_code>def plot_prediction(predicted,true,desciption):
    fig, ax = plt.subplots()
    ax.scatter(true, predicted, edgecolors=(0, 0, 0))
    ax.plot([true.min(), true.max()], [true.min(), true.max()], 'k--', lw=4)
    ax.set_xlabel('Measured')
    ax.set_ylabel('Predicted by '+desciption)
    ax.plot([-30,30], [0,0], 'k-')   
    ax.plot([0,0], [-30,30], 'k-')
    plt.show()
df_out['prediction'] = df_out['The_Median']-df_out['Predicted']
plot_prediction(df_out['Weekly_Sales'],df_out['prediction'],"Random Forest")
plot_prediction(y_pred_random,y_dev['Disparity'].values,"Random Forest")<jupyter_output><empty_output><jupyter_text>Belowing we should how the random forest model fitting the train(developing)data set and After debuging ,the function works well and we gonna feed it with test file.<jupyter_code>rf_model = RandomForestRegressor(n_estimators=80, criterion='mse', max_depth=None, 
                      min_samples_split=2, min_samples_leaf=1, 
                      min_weight_fraction_leaf=0.0, max_features='auto', 
                      max_leaf_nodes=None, min_impurity_decrease=0.0, 
                      min_impurity_split=None, bootstrap=True, 
                      oob_score=False, n_jobs=1, random_state=None, 
                      verbose=0, warm_start=False)


#Train on data
train=train.fillna(0)
rf_model.fit(train[selector], train['Disparity'])

test=test.fillna(0)
final_y_prediction = rf_model.predict(test[selector])
testfile = pd.concat([test.reset_index(drop=True), pd.DataFrame(final_y_prediction)], axis=1)
testfile['prediction'] = testfile['The_Median']-testfile[0]
testfile.head()<jupyter_output><empty_output><jupyter_text>Here is the head of prediction dataframe.<jupyter_code>predict_result = pd.DataFrame({'store_dept_date':pd.Series([''.join(list(filter(str.isdigit, x))) for x in testfile['Store']]).map(str) + '_' +
                           pd.Series([''.join(list(filter(str.isdigit, x))) for x in testfile['Dept_x']]).map(str)  + '_' +
                           testfile['Date_x'].map(str),
                          'Weekly_Sales':testfile['prediction']})
predict_result.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1203525822126794, 'fraction_numerical': 0.01356085433382303, 'mean_word_length': 5.63132259273136, 'pattern_counts': {'":': 1, '<': 91, '<?xml version=': 0, '>': 95, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '466111', 'license': 'MIT', 'n_tokens_mistral': 6056, 'n_tokens_neox': 5752, 'n_words': 1586, 'path': 'Part2/Part2.ipynb', 'repo_name': 'codemonkeyhhq/7390', 'size': '17698'}