<jupyter_start><jupyter_text>Load Bigmart Sales data<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
import warnings
warnings.filterwarnings("ignore")
bigmart = pd.read_csv('train_bm.csv')
bigmart.head()
bigmart.shape
bigmart.dtypes<jupyter_output><empty_output><jupyter_text>One hot Encoding<jupyter_code>bigmart['Outlet_Type'].value_counts()
pd.get_dummies(bigmart['Outlet_Type']).head()
bigmart_encoded = pd.get_dummies(bigmart)
bigmart_encoded.head()<jupyter_output><empty_output><jupyter_text>We have two problems here:  Problem 1:Look at the newly created variables *'Outlet_Size_High 	Outlet_Size_Medium 	Outlet_Size_Small'*, the order between these variables is destroyed. As a result we are missing out on some important information.<jupyter_code>bigmart_encoded[['Outlet_Size_High', 'Outlet_Size_Medium', 'Outlet_Size_Small']].head()<jupyter_output><empty_output><jupyter_text>Problem 2The number of features has increased from 12 to 1600, where maximum values are 0<jupyter_code>bigmart_encoded.shape, bigmart.shape
bigmart_encoded[['Item_Identifier_DRA12', 'Item_Identifier_DRA24',
       'Item_Identifier_DRA59', 'Item_Identifier_DRB01',
       'Item_Identifier_DRB13', 'Item_Identifier_DRB24',
       'Item_Identifier_DRB25', 'Item_Identifier_DRB48',
       'Item_Identifier_DRC01', 'Item_Identifier_DRC12',
       'Item_Identifier_DRC13', 'Item_Identifier_DRC24',
       'Item_Identifier_DRC25', 'Item_Identifier_DRC27',
       'Item_Identifier_DRC36', 'Item_Identifier_DRC49',
       'Item_Identifier_DRD01', 'Item_Identifier_DRD12',
       'Item_Identifier_DRD13', 'Item_Identifier_DRD15',
       'Item_Identifier_DRD24', 'Item_Identifier_DRD25',
       'Item_Identifier_DRD27', 'Item_Identifier_DRD37',
       'Item_Identifier_DRD49']].head()<jupyter_output><empty_output><jupyter_text>Problem 1 solution - Label Encoding<jupyter_code>from sklearn.preprocessing import LabelEncoder
bigmart['Outlet_Size'].value_counts()
le = LabelEncoder()
le.fit_transform(['Small', 'Medium', 'High'])<jupyter_output><empty_output><jupyter_text>Label encoder uses the alphabetical order.<jupyter_code>bigmart['Outlet_Size'] = bigmart['Outlet_Size'].map({'Small': 0,
                                                     'Medium': 1,
                                                     'High': 2})
bigmart['Outlet_Size'].head()<jupyter_output><empty_output><jupyter_text>So that solves the first challenge we encountered. Now we'll see how to deal with high cardinality.  Problem 2 solution<jupyter_code>bigmart.nunique()
bigmart['Item_Identifier'].value_counts()
temp= bigmart['Item_Identifier'].value_counts()
temp.head()
bigmart['Item_identifier_count'] = bigmart['Item_Identifier'].apply(lambda x: temp[x])
bigmart[['Item_Identifier','Item_identifier_count']].head()
for i in range(0, len(bigmart)):
    if bigmart['Item_identifier_count'][i] < 4:
        bigmart['Item_Identifier'][i] = 'other'
bigmart.head(7)
 bigmart['Item_Identifier'].value_counts()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11636363636363636, 'fraction_numerical': 0.022148760330578512, 'mean_word_length': 6.256594724220624, 'pattern_counts': {'":': 0, '<': 30, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '620549', 'license': 'MIT', 'n_tokens_mistral': 1043, 'n_tokens_neox': 992, 'n_words': 167, 'path': '12 Feature Engineering/7.  Combining Sparse classes/4. combining sparse classes.ipynb', 'repo_name': 'IamVaibhavsar/Machine_Learning_Files', 'size': '3025'}