<jupyter_start><jupyter_text>Initially, the dataset has 7 numerical variables and 1 categorical.  The Store and holiday flag is kind of a categorical feature   Count missing values<jupyter_code>def missing_values(data):
    return pd.concat([data.isna().sum(), 
        np.round(data.isna().sum()*100/data.shape[0])], 
        axis=1).rename({0:'count_missing', 1:'%_missing'}, axis=1)
missing_values(data)
data = data.dropna(subset=['Weekly_Sales', 'Date'], axis=0)
missing_values(data)
data
data['Date']=pd.to_datetime(data['Date']).dt.date
data.head()<jupyter_output>/home/carmina/miniconda3/envs/pez-jedha/lib/python3.7/site-packages/pandas/core/frame.py:3607: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  self._set_item(key, value)<jupyter_text>Some data cleaning<jupyter_code>data.iloc[: ,:-2].describe()
data.info()
profile = ProfileReport(data, title="Pandas Profiling Report")
profile.to_widgets()
#data['Date2']=pd.to_datetime(data['Date']).dt.date

data['Year']=pd.to_datetime(data['Date']).dt.year #Because of null values, year could be float, so casting is used to force it to int.
data['Month']=pd.to_datetime(data['Date']).dt.month.astype(np.int8)
data['dayOfWeek']=pd.to_datetime(data['Date']).dt.dayofweek.astype(np.int8)
data['weekday']=pd.to_datetime(data['Date']).dt.strftime("%A")
data['week']=pd.to_datetime(data['Date']).dt.isocalendar().week.astype(np.int8)
data['Store'] = data['Store'].astype('str')
data.sort_values(by=['Date','Month']).head(10)<jupyter_output>/home/carmina/miniconda3/envs/pez-jedha/lib/python3.7/site-packages/pandas/core/frame.py:3607: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  self._set_item(key, value)<jupyter_text>Check weeks with holidays<jupyter_code>data[data['Holiday_Flag']==1][['Store','Date','Year','Month', 'week']].sort_values('Date')
holiday_weeks = data[data['Holiday_Flag']==1].week.to_list()
del holiday_weeks[2]
holiday_weeks = np.unique(holiday_weeks)
print(f'Weeks with holidays: {holiday_weeks}')<jupyter_output>Weeks with holidays: [28 40 47 48 52]<jupyter_text>Sales per Store<jupyter_code>sales_per_store = data.groupby('Store').Weekly_Sales.sum().sort_values(ascending=False)
sales_per_store
plt.figure(figsize=(15, 5))
sns.barplot(x=sales_per_store.index, y=sales_per_store.values)
sales_per_store.index<jupyter_output><empty_output><jupyter_text>The stores 14, 13, 2, 1, 4 have the top sales<jupyter_code>def sales_per_month(data):
    sales_month = data.groupby('Month')['Weekly_Sales'].agg('sum')
    return sales_month<jupyter_output><empty_output><jupyter_text>data['Date2']=pd.to_datetime(data['Date']).dt.datedata['Year']=pd.to_datetime(data['Date']).dt.year.astype('Int16') Because of null values, year could be float, so casting is used to force it to int.data['Month']=pd.to_datetime(data['Date']).dt.month.astype('Int16')data['dayOfWeek']=pd.to_datetime(data['Date']).dt.dayofweek.astype('Int16')data['weekday']=pd.to_datetime(data['Date']).dt.strftime("%A")data['week']=pd.to_datetime(data['Date']).dt.isocalendar().weekdata.sort_values(by=['Date','Month']).head(10)  Sales per month<jupyter_code>sales_month = data.groupby('Month')['Weekly_Sales'].agg('sum')
sns.barplot(x=sales_month.index, y=sales_month.values)

filter_holiday = data['Holiday_Flag'] == 1
data[filter_holiday]
sns.barplot(x=sales_per_month(data[filter_holiday]).index, y=sales_per_month(data[filter_holiday]).values)<jupyter_output><empty_output><jupyter_text>It seems that there is a positive correlation between the holiday with the weekly sales  1. Checking for correlations<jupyter_code>sns.pairplot(data=data)<jupyter_output><empty_output><jupyter_text>At first glance to the heatmap and the graphs, there is no varible with strong correlation with Weekly_Sales  Visualize distributions<jupyter_code>data.hist(figsize=(20,15))
plt.show()<jupyter_output><empty_output><jupyter_text>2. Perform some cleaning<jupyter_code># First, split the date according
data_train, data_test = train_test_split(data, test_size=0.2)
data_train.hist(figsize=(20,15))
plt.show()<jupyter_output><empty_output><jupyter_text>After split, check the pourcentage of data per year<jupyter_code># Original data
data['Year'].value_counts()*100/data.shape[0]
print('Data instances per year')
data_train['Year'].value_counts()*100/data_train.shape[0]
sns.displot(x='Weekly_Sales', kind='kde', data=data, aspect=1.5 )
data.describe()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1595004095004095, 'fraction_numerical': 0.017813267813267815, 'mean_word_length': 8.304761904761905, 'pattern_counts': {'":': 0, '<': 38, '<?xml version=': 0, '>': 38, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '114536', 'license': 'MIT', 'n_tokens_mistral': 1849, 'n_tokens_neox': 1764, 'n_words': 285, 'path': 'projects/05-walmart/notebooks/01-EDA.ipynb', 'repo_name': 'pattypooh/pez-jedha', 'size': '4884'}