<jupyter_start><jupyter_text>Item categories analysis  Summary Columns:- item_category_name - category name- item_category_id - category id---  Insights- Most of categories names **~70%** consist of pattern `{Family name} - {Subfamily name}`- There are many categories contains `Цифра` word. It translates as electronic item.- `Доставка товара`, `Билеты (Цифра)`, `Чистые носители (шпиль)`, `Служебные`, `Элементы питания`, and `Чистые носители (штучные)` **(ID 8, 9, 79, 81,82,83)** do not have any additional category.- Family level are good separatable. It can be used as feature.- Unique subfamily names count very high. So It is not reason to use it as feature in feature generation.- Item count by count are separatable because of high deviation.  Import libraries and load datasets<jupyter_code>%%capture
%store -r item_cat
%store -r item
%store -r sub
%store -r shops
%store -r sales_test
%store -r sales_train
%store -r __ipy
%store -r __da
__ipy
__da
%%capture
import plotly.express as px
import seaborn as sns
from basic_text_preprocessing import BasicPreprocessText<jupyter_output><empty_output><jupyter_text>Glimpse - Most of categories names **~70%** consist of pattern `{Family name} - {Subfamily name}`- There are many categories contains `Цифра` word. It translates as electronic item.- `Доставка товара`, `Билеты (Цифра)`, `Чистые носители (шпиль)`, `Служебные`, `Элементы питания`, and `Чистые носители<jupyter_code>with pd.option_context('display.max_rows', 100):
    display(item_cat)<jupyter_output><empty_output><jupyter_text>All `item_category_id` are unique<jupyter_code>item_cat.describe().T<jupyter_output><empty_output><jupyter_text>Items count per category - Category **ID 40** contains the biggest items count. - Others categories contains less than **2 000** items.- **50%** of categories contains less than **41** items- **25%** of categories contains less than **8** items<jupyter_code>def set_plotly_layout(fig, title=""):
    fig.update_layout(
        font=dict(size=12, family='Helvetica'),
        title = title,
        yaxis = dict(
          scaleanchor = "x",
          scaleratio = 1,
        )
    )
    return fig


item_cat_grouped = (item.groupby('item_category_id', as_index=False)['item_id']
                    .count()
                    .sort_values('item_id', ascending=False).rename(columns={'item_id': "item_cnt"}))

item_cat_grouped = item_cat_grouped[item_cat_grouped['item_cnt'] < 5000]
fig = (px.bar(item_cat_grouped, y = 'item_cnt', x = 'item_category_id', width=1000, height=300)
.update_xaxes(type='category', title_text = "Category ID")
.update_yaxes(nticks=20, title_text = "Item count per category"))

set_plotly_layout(fig, "Items count per category").show()
pd.concat([item_cat_grouped[['item_cnt']].describe().T,
                      item_cat_grouped[['item_cnt']].median().rename('median'),
                      item_cat_grouped[['item_cnt']].skew().rename('skew'),
                      item_cat_grouped[['item_cnt']].kurt().rename('kurt')
                     ], axis=1)<jupyter_output><empty_output><jupyter_text>Family category  - Family category is first part of category name (before dash symbol)- Family category  maximum count of categories is 14 `'игры'`- Category names which consist only of one level contains 1 or 2 categories.<jupyter_code>item_category_names = pd.Series(BasicPreprocessText().vectorize_process_text(item_cat['item_category_name']))
item_category_names_category_1 = item_category_names.apply(lambda x: x.split()[0])
item_category_names_category_1 = (item_category_names_category_1.value_counts()
.to_frame().reset_index())

item_category_names_category_1.columns = ['name', 'item_cat_count']
fig = (px.bar(item_category_names_category_1, x='name', y='item_cat_count')
.update_xaxes(type='category', title_text = "Category name")
.update_yaxes(nticks=20, title_text = "Category count"))

set_plotly_layout(fig, "Category family").show()<jupyter_output><empty_output><jupyter_text>Subfamily category  - Subfamily category is first part of category name (before dash symbol)- Subfamily categories compraise 1 or 2 levels.<jupyter_code>item_category_names_category_2 = item_category_names.apply(lambda x: " ".join(x.split()[1:]))
item_category_names_category_2 = item_category_names_category_2.value_counts()\
.to_frame().reset_index()
item_category_names_category_2.columns = ['name', 'item_cat_count']

fig = (px.bar(item_category_names_category_2, x='name', y='item_cat_count')
.update_xaxes(type='category', title_text = "Category name")
.update_yaxes(nticks=20, title_text = "Category count"))

set_plotly_layout(fig, "Subcategory family").show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12481299422953622, 'fraction_numerical': 0.014746740756571917, 'mean_word_length': 5.481994459833795, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '649042', 'license': 'MIT', 'n_tokens_mistral': 1576, 'n_tokens_neox': 1536, 'n_words': 379, 'path': 'notebooks/data_exp/sales-prediction.item_cat.python.2.0.ipynb', 'repo_name': 'Saphirox/sales-prediction', 'size': '4679'}