<jupyter_start><jupyter_text>About the Data setThis data set includes 721 Pokemon, including their number, name, first and second type, and basic stats: HP, Attack, Defense, Special Attack, Special Defense, and Speed. Columns1. '' : PokeDex index number2. Name : Name of the Pokemon3. Type 1 : Type of pokemon4. Type 2 : Other Type of Pokemon5. Total : Sum of Attack, Sp. Atk, Defense, Sp. Def, Speed and HP6. HP : Hit Points7. Attack : Attack Strength8. Defense : Defensive Strength9. Sp. Atk : Special Attack Strength10. Sp. Def : Special Defensive Strength11. Speed : Speed12. Generation : Number of generation13. LegendaryTrue if Legendary Pokemon False if not (more revision on mythical vs legendary needed)<jupyter_code># Prints dimension of dataset (Number of rows, Number of columns)
df.shape
# Gives information about number of not null values and data type correspoding to a column
df.info()
# Prints the number of null values in each column
df.isnull().sum()<jupyter_output><empty_output><jupyter_text>On observing the above two cells we can see that there are 386 Null values are present in Type 2 column so first we have to remove these null values. Rest of the columns do not have null value.  Data Cleaning<jupyter_code># Replaces the Null Values with the mode of Type 2 column
df["Type 2"].fillna(df["Type 2"].mode()[0], inplace = True)
# Crosschecks the number of null values in each column
df.isnull().sum()<jupyter_output><empty_output><jupyter_text>Now we can see that all columns are free from Null Values, Cleaning part is done...!!!!!<jupyter_code># Drops the columns which are not important
df.drop(["#", "Name"], axis=1, inplace=True)

# Prints the first five rows with remaining columns
df.head(5)<jupyter_output><empty_output><jupyter_text>Transforming Nominal Attributes<jupyter_code># Prints the unique values present in Type 1 Column
df["Type 1"].unique()
# Prints the number of unique values present in Type 1 Column
len(df["Type 1"].unique())<jupyter_output><empty_output><jupyter_text>There are 18 distinct categories in column Type 1. We can now generate a label encoding scheme for mapping each category to a numeric value.<jupyter_code># Create the Labelencoder object
t1_le = LabelEncoder()

# Converts the categorical columns into numeric
df["Type 1_Label"] = t1_le.fit_transform(df["Type 1"])

# Prints the first five rows after modification
df.head(5)
# Shows the mapping of Numerical Labels to Categorical Labels
Type1_mappings = {index: label for index, label in enumerate(t1_le.classes_)}
Type1_mappings<jupyter_output><empty_output><jupyter_text>Similarly we have to do for column "Type 2" column<jupyter_code># Prints the unique values present in Type 2 Column
df["Type 2"].unique()
# Prints the number of unique values present in Type 2 Column
len(df["Type 2"].unique())<jupyter_output><empty_output><jupyter_text>There are 18 distinct categories in column Type 2. We can now generate a label encoding scheme for mapping each category to a numeric value.<jupyter_code># Create the Labelencoder object
t2_le = LabelEncoder()

# Converts the categorical columns into numeric
df["Type 2_Label"] = t2_le.fit_transform(df["Type 2"])

# Prints the first five rows after modification
df.head(5)
# Shows the mapping of Numerical Labels to Categorical Labels
Type2_mappings = {index: label for index, label in enumerate(t2_le.classes_)}
Type2_mappings<jupyter_output><empty_output><jupyter_text>Transforming Ordinal Attributes<jupyter_code># Prints the unique values present in Generation Column
df["Generation"].unique()
# Prints the unique values present in Legendary Column
df["Legendary"].unique()
# Create the Labelencoder object
lg_le = LabelEncoder()

# Converts the categorical columns into numeric
df["Legendary_Label"] = lg_le.fit_transform(df["Legendary"])

# Prints the first five rows after modification
df.head(5)
# Shows the mapping of Numerical Labels to Categorical Labels
Legendary_mappings = {index: label for index, label in enumerate(lg_le.classes_)}
Legendary_mappings<jupyter_output><empty_output><jupyter_text>Till Now, we just converted categories to numerical labels, why we need this now? The reason is quite simple. Considering column Type 1, if we directly fed the Type 1_Label attribute as a feature in a machine learning model, it would consider it to be a continuous numeric feature thinking value 10 (Ground) is greater than 6 (Fire) but that is meaningless because the Ground type is certainly not bigger or smaller than Fire, these are essentially different values or categories which cannot be compared directly. Hence we need an additional layer of encoding schemes where dummy features are created for each unique value or category out of all the distinct categories per attribute.  Encoding  One-hot Encoding of Nominal Attributes<jupyter_code># Encode Type 1 labels using one-hot encoding scheme

# Creating one hot encoder object
t1_ohe = OneHotEncoder()

# Returns a 2D array where number of column is number of unique labels and number of row is number of rows present
# in the dataset and it can only contain a value of 1 or 0, values as 1 (indicating it as active).
t1_feature_arr = t1_ohe.fit_transform(df[['Type 1_Label']]).toarray()

# Listing the different labels present in the column
t1_feature_labels = list(t1_le.classes_)

# Prints the labels
t1_feature_labels
# Attached a string "T1" with the labels of column Type 1

t1_labels = []
for label in t1_feature_labels:
    t1_labels.append("T1_"+label)
    
# Prints the labels after modification
t1_labels
# Makes a Data frame with 2D array and modified labels
t1_features = pd.DataFrame(t1_feature_arr, columns=t1_labels)

# Prints the first five rows of the data frame
t1_features.head(5)
# Encode Type 2 labels using one-hot encoding scheme

# Creating one hot encoder object
t2_ohe = OneHotEncoder()

# Returns a 2D array where number of column is number of unique labels and number of row is number of rows present
# in the dataset and it can only contain a value of 1 or 0, values as 1 (indicating it as active).
t2_feature_arr = t2_ohe.fit_transform(df[['Type 2_Label']]).toarray()

# Listing the different labels present in the column
t2_feature_labels = list(t2_le.classes_)

# Prints the labels
t2_feature_labels
# Attached a string "T2" with the labels of column Type 2

t2_labels = []
for label in t2_feature_labels:
    t2_labels.append("T2_"+label)
    
# Prints the labels after modification   
t2_labels
# Makes a Data frame with 2D array and modified labels
t2_features = pd.DataFrame(t2_feature_arr, columns=t2_labels)

# Prints the first five rows of the data frame
t2_features.head(5)<jupyter_output><empty_output><jupyter_text>One-hot Encoding of Ordinal Attributes<jupyter_code># Encode Legendary labels using one-hot encoding scheme

# Creating one hot encoder object
lg_ohe = OneHotEncoder()

# Returns a 2D array where number of column is number of unique labels and number of row is number of rows present
# in the dataset and it can only contain a value of 1 or 0, values as 1 (indicating it as active).
lg_feature_arr = lg_ohe.fit_transform(df[['Legendary_Label']]).toarray()

# Listing the different labels present in the column
lg_feature_labels = list(lg_le.classes_)

# Prints the labels
lg_feature_labels
# Attached a string "Lg" with the labels of column Legendary

lg_labels = []
for label in lg_feature_labels:
    lg_labels.append("Lg_"+str(label))
    
# Prints the labels after modification   
lg_labels
# Makes a Data frame with 2D array and modified labels
lg_features = pd.DataFrame(lg_feature_arr, columns=lg_labels)

# Prints the first five rows of the data frame
lg_features.head(5)
# Encode Generation labels using one-hot encoding scheme

# Creating one hot encoder object
gen_ohe = OneHotEncoder()

# Returns a 2D array where number of column is number of unique labels and number of row is number of rows present
# in the dataset and it can only contain a value of 1 or 0, values as 1 (indicating it as active).
gen_feature_arr = gen_ohe.fit_transform(df[['Generation']]).toarray()

# Listing the different labels present in the column
gen_feature_labels = list(df["Generation"].unique())

# Prints the labels
gen_feature_labels
# Attached a string "Gen" with the labels of column Generation

gen_labels = []
for label in gen_feature_labels:
    gen_labels.append("Gen_"+str(label))
    
# Prints the labels after modification
gen_labels
# Makes a Data frame with 2D array and modified labels
gen_features = pd.DataFrame(gen_feature_arr, columns=gen_labels)

# Prints the first five rows of the data frame
gen_features.head(5)
# Concatenates all the new data frame and the old data frame df 
df_ohe = pd.concat([t1_features, t2_features, df, gen_features, lg_features], axis=1)

# Prints the first five rows
df_ohe.head(5)
# Drops all the columns for which we did label encoding and one-hot encoding
df_ohe.drop(["Type 1", "Type 2", "Type 1_Label", "Type 2_Label", "Legendary", "Legendary_Label", "Generation"], axis=1, inplace=True)

# Prints the first five rows
df_ohe.head(5)
# Expands the output display to see more columns. In the above cell we are not able to see all columns
# but in the below cell we are able to see all columns
pd.set_option('display.max_columns', None)<jupyter_output><empty_output><jupyter_text>Final Data after applying Feature Engineering using Label Encoding and OneHotEncoding<jupyter_code># Prints first five rows
df_ohe.head(5)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0710970464135021, 'fraction_numerical': 0.014978902953586498, 'mean_word_length': 5.397435897435898, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 45, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '307039', 'license': 'MIT', 'n_tokens_mistral': 2895, 'n_tokens_neox': 2771, 'n_words': 1197, 'path': 'Feature Engineering on categorical data of Pokemon Dataset/Feature Engineering on categorical data of Pokemon Dataset.ipynb', 'repo_name': 'mohansah/Feature_Engineering', 'size': '9480'}