<jupyter_start><jupyter_text>Eploratory Analysis for the final projectThe purpose of this notebook is to make an exploratory analysis of the data for the final project, the idea is to find patterns and dicover information to complement a re-iterate over our defined questions in order to improve them and complement them.<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
snacks_df = pd.read_csv('./datasets/snacks.csv', dtype = { 'client_id': str })
snacks_df['date'] = pd.to_datetime(snacks_df['date'], format = '%d/%m/%Y')

snacks_df.head()<jupyter_output><empty_output><jupyter_text>**How many regions we have?**<jupyter_code>snacks_df['country_region'].unique()<jupyter_output><empty_output><jupyter_text>**Trim the country region data**<jupyter_code>def transform(x):
    if (isinstance(x, str)):
        return x.strip()
    else:
        return x

snacks_df['country_region'] = snacks_df['country_region'].map(transform)
snacks_df['country_region'].unique()<jupyter_output><empty_output><jupyter_text>**How many sales with nan are?****Why we don't have this data**<jupyter_code>snacks_df['country_region'].isna().sum()
snacks_df[snacks_df['country_region'].isna()]['units'].sum()<jupyter_output><empty_output><jupyter_text>**How many sales records are from every region?**<jupyter_code>snacks_df.groupby('country_region')['date'].count().reset_index(name = 'count')<jupyter_output><empty_output><jupyter_text>**How many sales unit records are from every region?**<jupyter_code>snacks_df.groupby('country_region')['units'].sum().reset_index()<jupyter_output><empty_output><jupyter_text>**How many cities and departments we have?**<jupyter_code>snacks_df['office_department_city'].unique()<jupyter_output><empty_output><jupyter_text>**How many warehouse cities we have?**<jupyter_code>snacks_df['warehouse_city'].unique()<jupyter_output><empty_output><jupyter_text>**How many categories we have?**<jupyter_code>snacks_df['category_name'].unique()<jupyter_output><empty_output><jupyter_text>**How many brands do we have?**<jupyter_code>snacks_df['brand_id'].unique()<jupyter_output><empty_output><jupyter_text>**How many clients we have?**<jupyter_code>snacks_df['client_name'].unique()<jupyter_output><empty_output><jupyter_text>**How many distributors do we have?**<jupyter_code>snacks_df['distributor'].unique()<jupyter_output><empty_output><jupyter_text>**How many sales per distributor type**<jupyter_code>snacks_df.groupby('distributor')['units'].sum().reset_index().sort_values('units')<jupyter_output><empty_output><jupyter_text>**How the sales quantity behave through time?**<jupyter_code>snacks_df['year'] = snacks_df['date'].map(lambda x: x.year)
snacks_df['month'] = snacks_df['date'].map(lambda x: x.month)
data = snacks_df.groupby(['year', 'month'])['units'].sum().reset_index()
plt.figure(figsize = (16, 10))
sns.barplot(x = 'month', y = 'units', data = data[data['year'] == 2017])<jupyter_output><empty_output><jupyter_text>**Distribution of sales units**<jupyter_code>plt.figure(figsize = (16, 10))
sns.distplot(snacks_df['units'])
x = snacks_df[snacks_df['country_region'].notna()][snacks_df['country_region'] != 'Ventas al Exterior']['units']

plt.figure(figsize = (16, 10))
sns.distplot(x)
plt.figure(figsize = (16, 10))
sns.boxplot(x = 'year', y = 'units', data = snacks_df)
test = snacks_df[snacks_df['country_region'].notna()][snacks_df['country_region'] == 'NI orte'][snacks_df['year'] == 2015]

plt.figure(figsize = (16, 10))
sns.boxplot(x = 'category_name', y = 'units', data = test)<jupyter_output>/Users/juanprq/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:1: UserWarning: Boolean Series key will be reindexed to match DataFrame index.
  """Entry point for launching an IPython kernel.<jupyter_text>Trying an approach that deals with outliers<jupyter_code># Snacks from the north in 2015
x = snacks_df[snacks_df['country_region'] == 'Norte'][snacks_df['year'] == 2015]

Q1 = x['units'].quantile(0.25)
Q3 = x['units'].quantile(0.75)
IQR = Q3 - Q1

filtered = x.query('(@Q1 - 1.5 * @IQR) <= units <= (@Q3 + 1.5 * @IQR)')
plt.figure(figsize = (16, 10))
sns.distplot(filtered['units'])<jupyter_output>/Users/juanprq/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:2: UserWarning: Boolean Series key will be reindexed to match DataFrame index.<jupyter_text>**There are any records with 0 of units?**<jupyter_code>snacks_df[snacks_df['units'] == 0].head()
snacks_df[(snacks_df['units'] > 0) & (snacks_df['devolution_units'] > 0)].head()<jupyter_output><empty_output><jupyter_text>Let's try a simple for all removing outliers<jupyter_code>x = snacks_df[
    (snacks_df['country_region'].notna()) &
    (snacks_df['country_region'] != 'Ventas al Exterior') &
    (snacks_df['units'] > 0)
]

Q1 = x['units'].quantile(0.25)
Q3 = x['units'].quantile(0.75)
IQR = Q3 - Q1

filtered = x.query('(@Q1 - 1.5 * @IQR) <= units <= (@Q3 + 1.5 * @IQR)')
plt.figure(figsize = (16, 10))
sns.boxplot(data = filtered, x = 'year', y = 'units', hue = 'distributor')
plt.legend(loc = 'upper left')<jupyter_output><empty_output><jupyter_text>How can we view a distributor distribution<jupyter_code>x = filtered[filtered['distributor'] == 'Cadenas Medianas']

plt.figure(figsize = (16, 10))
sns.distplot(x['units'], bins = 5) # test with least bins, and more<jupyter_output><empty_output><jupyter_text>I want to calculate the sum of units sales by month in the time by type of distributor<jupyter_code>x = snacks_df[
    (snacks_df['country_region'].notna()) &
    (snacks_df['country_region'] != 'Ventas al Exterior') &
    (snacks_df['units'] > 0) &
    (snacks_df['distributor'] == 'Cadenas Medianas')
]

x['yyyymm'] = x['date'].map(lambda x: x.strftime('%Y%m'))
grouping = x.groupby('yyyymm')['units'].sum().reset_index()
plt.figure(figsize = (16, 10))
sns.barplot(x = 'yyyymm', y = 'units', data = grouping)
plt.xticks(rotation = 90)<jupyter_output>/Users/juanprq/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:8: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy<jupyter_text>dates that stores bought a lot of product. Does this makes sense?<jupyter_code>x = snacks_df[snacks_df['country_region'] != 'Ventas al Exterior'][snacks_df['year'] == 2015][snacks_df['units'] > 0]

x['log'] = np. (x['units'])
plt.figure(figsize = (16, 10))
sns.distplot(x['log'])<jupyter_output>/Users/juanprq/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:1: UserWarning: Boolean Series key will be reindexed to match DataFrame index.
  """Entry point for launching an IPython kernel.<jupyter_text>What we identified is that there are some basic features that we should have into mind in order to segment the data and analyze it better    * country_region    * year    * month    * distributor    * category_name   Also i suspect that this are also features that help us to segment the date:    * price_type    * return? (if the return units it's more than 0, this doesn't exists right now)    * flavor Count by month for every year, sales units of cadenas medianas<jupyter_code>x = filtered[filtered['distributor'] == 'Cadenas Medianas']
x['yyyymm'] = x['date'].map(lambda x: x.strftime('%Y%m'))

fig = plt.figure(figsize = (16, 16))

for i, year in enumerate(x['year'].unique()):
    fig.add_subplot(3, 2, i + 1)
    aux = x[x['year'] == year]
    sns.barplot(data = aux, x = 'month', y = 'units').set_title(year)

# sw peonro agrupar bimestral o trimestral<jupyter_output>/Users/juanprq/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:2: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
{'fraction_non_alphanumeric': 0.15588454839512317, 'fraction_numerical': 0.01617317740731525, 'mean_word_length': 6.804854368932038, 'pattern_counts': {'":': 0, '<': 88, '<?xml version=': 0, '>': 89, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '22', 'dir': 'jupyter-structured-clean-dedup', 'id': '630104', 'license': 'MIT', 'n_tokens_mistral': 3036, 'n_tokens_neox': 2864, 'n_words': 611, 'path': 'exploration.ipynb', 'repo_name': 'juanprq/final-project', 'size': '8038'}