<jupyter_start><jupyter_text>Mushroom Kaggle's Mushroom Classification. Aim is to detect edible mushrooms. More info here.  Table of Contents1. [Imports](Imports)    2. [Data](Data)    2.1 [Import Data](Import_Data)     2.2 [Missing Values](Missing_Values)  3. [EDA](EDA)    3.1 [Target](Target)    3.2 [Features](Features)  4. [Preprocess](Preprocess)5. [Models](Models)    5.1 [Logistic Regression](Logistic_Regression)    5.2 [KNN](KNN)    5.3 [SVM](SVM)    5.4 [Gaussian Naive Bayes](Gaussian_Naive_Bayes)    5.5 [Decision Tree](Decision_Tree)    5.6 [Random Forest](Random_Forest)    5.7 [CatBoost](CatBoost)6. [Evaluation](Evaluation)    Imports<jupyter_code># main
import numpy as np
import pandas as pd
import scipy.stats as stats
import pickle

# datetimes
import datetime as dt
from datetime import datetime
import time

# dataviz
%matplotlib notebook
import matplotlib.pyplot as plt
import seaborn as sns
import graphviz

# plot style
%config InteractiveShellApp.pylab_import_all = False
%pylab inline
pylab.rcParams['figure.figsize'] = (10, 7)

plt.style.use('seaborn-colorblind')

# outline
from IPython.core.display import display, HTML
display(HTML("<style>.container { width:100% !important; }</style>"))

from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all"

%precision %.3f

# pandas display
pd.set_option('display.max_rows', 200)
pd.set_option('display.max_columns', 100)

# warnings
import warnings
warnings.filterwarnings('ignore')<jupyter_output>Populating the interactive namespace from numpy and matplotlib<jupyter_text>Data  Import_Data<jupyter_code>df = pd.read_csv('../data/mushrooms.csv')
# data available
len(df)
df.head().T<jupyter_output><empty_output><jupyter_text>Attribute Information: (classes: edible=e, poisonous=p)- cap-shape: bell=b,conical=c,convex=x,flat=f, knobbed=k,sunken=s  - cap-surface: fibrous=f,grooves=g,scaly=y,smooth=s  - cap-color: brown=n,buff=b,cinnamon=c,gray=g,green=r,pink=p,purple=u,red=e,white=w,yellow=y  - bruises: bruises=t,no=f  - odor: almond=a,anise=l,creosote=c,fishy=y,foul=f,musty=m,none=n,pungent=p,spicy=s  - gill-attachment: attached=a,descending=d,free=f,notched=n  - gill-spacing: close=c,crowded=w,distant=d  - gill-size: broad=b,narrow=n  - gill-color: black=k,brown=n,buff=b,chocolate=h,gray=g, green=r,orange=o,pink=p,purple=u,red=e,white=w,yellow=y  - stalk-shape: enlarging=e,tapering=t  - stalk-root: bulbous=b,club=c,cup=u,equal=e,rhizomorphs=z,rooted=r,missing=?  - stalk-surface-above-ring: fibrous=f,scaly=y,silky=k,smooth=s  - stalk-surface-below-ring: fibrous=f,scaly=y,silky=k,smooth=s  - stalk-color-above-ring: brown=n,buff=b,cinnamon=c,gray=g,orange=o,pink=p,red=e,white=w,yellow=y  - stalk-color-below-ring: brown=n,buff=b,cinnamon=c,gray=g,orange=o,pink=p,red=e,white=w,yellow=y  - veil-type: partial=p,universal=u  - veil-color: brown=n,orange=o,white=w,yellow=y  - ring-number: none=n,one=o,two=t  - ring-type: cobwebby=c,evanescent=e,flaring=f,large=l,none=n,pendant=p,sheathing=s,zone=z  - spore-print-color: black=k,brown=n,buff=b,chocolate=h,green=r,orange=o,purple=u,white=w,yellow=y  - population: abundant=a,clustered=c,numerous=n,scattered=s,several=v,solitary=y  - habitat: grasses=g,leaves=l,meadows=m,paths=p,urban=u,waste=w,woods=d    Missing_Values<jupyter_code>df.isna().mean()<jupyter_output><empty_output><jupyter_text>EDA<jupyter_code>class_map = {'e': 'edible', 'p': 'poisonous'}
shape_map = {'b': 'bell', 'c': 'conical', 'x': 'convex', 'f': 'flat', 'k': 'knobed', 's': 'sunken', 'e': 'enlarging', 't': 'tapering'}
surface_map = {'f': 'fibrous','g': 'grooves', 'y': 'scaly', 's': 'smooth', 'k': 'silky'}
color_map = {'n': 'brown', 'b': 'buff', 'c': 'cinnamon', 'g': 'gray', 'r': 'green', 'p': 'pink', 'u': 'purple', 'e': 'red',
             'w': 'white', 'y': 'yellow', 'h': 'chocolate', 'k': 'black', 'o': 'orange'}
odor_map = {'a': 'almond','l': 'anise','c': 'creosote','y': 'fishy','f': 'foul', 'm': 'musty','n': 'none','p': 'pungent','s': 'spicy'}

df['class'] = df['class'].map(class_map)
df['bruises'] = df['bruises'].map({'t': 'bruises', 'f': 'no'})
df['odor'] = df['odor'].map(odor_map)
df['gill-attachment'] = df['gill-attachment'].map({'a':'attached','d':'descending','f':'free','n':'notched'})
df['gill-spacing'] = df['gill-spacing'].map({'c':'close','w':'crowded','d':'distant'})
df['gill-size'] = df['gill-size'].map({'b':'broad','n':'narrow'})
df['stalk-root'] = df['stalk-root'].map({'b':'bulbous','c':'club','u':'cup','e':'equal','z':'rhizomorphs','r':'rooted','?':'missing'})
df['veil-type'] = df['veil-type'].map({'p':'partial','u':'universal'})
df['ring-number'] = df['ring-number'].map({'n':'none','o':'one','t':'two'})
df['ring-type'] = df['ring-type'].map({'c':'cobwebby','e':'evanescent','f':'flaring','l':'large','n':'none','p':'pendant','s':'sheathing','z':'zone'})
df['population'] = df['population'].map({'a':'abundant','c':'clustered','n':'numerous','s':'scattered','v':'several','y':'solitary'})
df['habitat'] = df['habitat'].map({'g':'grasses','l':'leaves','m':'meadows','p':'paths','u':'urban','w':'waste','d':'woods'})
    
for col in ['cap-shape','stalk-shape']:
    df[col] = df[col].map(shape_map)

for col in ['cap-surface','stalk-surface-above-ring','stalk-surface-below-ring']:
    df[col] = df[col].map(surface_map)

color_cols = [col for col in df.columns if 'color' in col]
for col in color_cols:
    df[col] = df[col].map(color_map)
    
df.head()
df.describe(include='all')<jupyter_output><empty_output><jupyter_text>Target<jupyter_code>df['class'].value_counts()<jupyter_output><empty_output><jupyter_text>Features<jupyter_code>from utils.plot_utils import *
# Cap

countplots_with_multiple_categories(df,'class','cap')
# Bruises

plt.figure(figsize=(10,7))
sns.countplot(x='bruises', hue='class', data=df)
# Odor

plt.figure(figsize=(10,7))
sns.countplot(x='odor', hue='class', data=df)
# Gill

countplots_with_multiple_categories(df,'class','gill')
plt.figure(figsize=(10,7))
sns.countplot(x='gill-color', hue='class', data=df)
# Stalk

countplots_with_multiple_categories(df,'class','stalk')
# Veil

countplots_with_multiple_categories(df,'class','veil')
# Ring

countplots_with_multiple_categories(df,'class','ring')
# Spore

plt.figure(figsize=(10,7))
sns.countplot(x='spore-print-color', hue='class', data=df)
# Population

plt.figure(figsize=(10,7))
sns.countplot(x='population', hue='class', data=df)
# Habitat

plt.figure(figsize=(10,7))
sns.countplot(x='habitat', hue='class', data=df)<jupyter_output><empty_output><jupyter_text>Preprocess<jupyter_code># separate target and features

X=df.drop('class',axis=1)
y=df['class']
# encode

from sklearn.preprocessing import LabelEncoder

Encoder_X = LabelEncoder() 

for col in X.columns:
    X[col] = Encoder_X.fit_transform(X[col])
    
Encoder_y=LabelEncoder()
y = Encoder_y.fit_transform(y)

X.head()
y
# get dummies

X=pd.get_dummies(X,columns=X.columns,drop_first=True)

X.head()
# train test split
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
# scale
from sklearn.preprocessing import StandardScaler

sc = StandardScaler()

X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)<jupyter_output><empty_output><jupyter_text>Models Try followings for this binary classification problem:- Logistic Regression- KNN- SVM- Gaussian Naive Bayes Classifier- Decision Tree- Random Forrest- CatBoost Classifier<jupyter_code>from sklearn.model_selection import GridSearchCV<jupyter_output><empty_output><jupyter_text>Logistic_Regression<jupyter_code>from sklearn.linear_model import LogisticRegression

# hyperparameters to tune
penalty = ['l1', 'l2']
C = [0.001,.009,0.01,.09,1,5,10,25]
class_weight = ['balanced', None]
solver = ['lbfgs', 'liblinear']

# hyperparameter options
hyperparameters = dict(C=C, penalty=penalty, class_weight=class_weight, solver=solver)

 
# cross-validation
lr = GridSearchCV(estimator=LogisticRegression(), param_grid=hyperparameters, cv=10)

# best model
lr.fit(X_train, y_train)

# best model params
lr.best_params_

# predict using best model
lr_preds = lr.predict(X_test)
score_lr = lr.score(X_train, y_train)
score_lr<jupyter_output><empty_output><jupyter_text>KNN<jupyter_code>from sklearn.neighbors import KNeighborsClassifier

# hyperparameters to tune
leaf_size = [10,15,30,50]
n_neighbors = [3,5,10,15,30]
weights = ['uniform','distance']
p = [1,2]
algorithm = ['auto', 'ball_tree', 'kd_tree', 'brute']

# hyperparameter options
hyperparameters = dict(leaf_size=leaf_size,n_neighbors=n_neighbors,weights=weights,p=p,algorithm=algorithm)

# cross-validation
knn = GridSearchCV(estimator=KNeighborsClassifier(), param_grid=hyperparameters, cv=10)

# best model
knn.fit(X_train, y_train)

# best model params
knn.best_params_

# predict using best model
knn_preds = knn.predict(X_test)
score_knn = knn.score(X_train, y_train)
score_knn<jupyter_output><empty_output><jupyter_text>SVM<jupyter_code>from sklearn.svm import SVC

# hyperparameters to tune
C = [0.5,1,2]
kernel = ['linear', 'rbf']
gamma = [0.02, 0.05]

# hyperparameter options
hyperparameters = dict(C=C,kernel=kernel,gamma=gamma)

# cross-validation
svc = GridSearchCV(estimator=SVC(), param_grid=hyperparameters, cv=10)

# best model
svc.fit(X_train, y_train)

# best model params
svc.best_params_

# predict using best model
svc_preds = svc.predict(X_test)
score_svc = svc.score(X_train, y_train)
score_svc<jupyter_output><empty_output><jupyter_text>Gaussian_Naive_Bayes<jupyter_code>from sklearn.naive_bayes import GaussianNB

# fit model
gs = GaussianNB()
gs.fit(X_train, y_train)
gs_preds = gs.predict(X_test)
score_gs = gs.score(X_train, y_train)
score_gs<jupyter_output><empty_output><jupyter_text>Decision_Tree<jupyter_code>from sklearn.tree import DecisionTreeClassifier

# hyperparameters to tune
criterion = ['gini','entropy']
splitter = ['best', 'random']
max_depth = [2,4,6,8,None]
min_samples_leaf = [1,3,5,7,10]

# hyperparameter options
hyperparameters = dict(criterion=criterion,splitter=splitter,max_depth=max_depth,min_samples_leaf=min_samples_leaf)

# cross-validation
dt = GridSearchCV(estimator=DecisionTreeClassifier(), param_grid=hyperparameters, cv=10)

# best model
dt.fit(X_train, y_train)

# best model params
dt.best_params_

# predict using best model
dt_preds = dt.predict(X_test)
score_dt = dt.score(X_train, y_train)
score_dt<jupyter_output><empty_output><jupyter_text>Random_Forest<jupyter_code>from sklearn.ensemble import RandomForestClassifier

# hyperparameters to tune
n_estimators = [100,300]
criterion = ['gini','entropy']
max_depth = [4,8]
max_features = [3,'auto']
min_samples_split = [3,7,10]

# hyperparameter options
hyperparameters = dict(n_estimators=n_estimators,criterion=criterion,max_depth=max_depth,max_features=max_features,min_samples_split=min_samples_split)

# cross-validation
rf = GridSearchCV(estimator=RandomForestClassifier(), param_grid=hyperparameters, cv=10)

# best model
rf.fit(X_train, y_train)

# best model params
rf.best_params_

# predict using best model
rf_preds = rf.predict(X_test)
score_rf = rf.score(X_train, y_train)
score_rf<jupyter_output><empty_output><jupyter_text>CatBoost<jupyter_code>from catboost import CatBoostClassifier

# hyperparameters to tune
learning_rate = [0.01,0.05,0.1,0.5]
loss_function = ['Logloss', 'CrossEntropy']
l2_leaf_reg = [2,10,25]

# hyperparameter options
hyperparameters = dict(learning_rate=learning_rate,loss_function=loss_function,l2_leaf_reg=l2_leaf_reg)

# cross-validation
cb = GridSearchCV(estimator=CatBoostClassifier(eval_metric='Accuracy',iterations=300), param_grid=hyperparameters, cv=10)

# best model
cb.fit(X_train, y_train, logging_level='Silent')

# best model params
cb.best_params_

# predict using best model
cb_preds = cb.predict(X_test)
score_cb = cb.score(X_train, y_train)
score_cb<jupyter_output><empty_output><jupyter_text>Evaluation<jupyter_code># evaluate classification
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score

# evaluate on

# accuracy = TP + TN / (TP + TN + FP + FN)
# precision = TP / (TP + FP)
# recall = TP / (TP + FN)  aka sensitivity, or true positive rate
# f1 = 2 * precision * recall / (precision + recall) 
models = pd.DataFrame({    
    'Model': [
        'Logistic Regression',
        'KNN', 
        'SVM',
        'Gaussian Naive Bayes',
        'Decision Tree',
        'Random Forest',
        'CatBoost'
              ],
    
    'Accuracy_Score':[
        accuracy_score(y_test, lr_preds),
        accuracy_score(y_test, knn_preds),
        accuracy_score(y_test, svc_preds),
        accuracy_score(y_test, gs_preds),
        accuracy_score(y_test, dt_preds),
        accuracy_score(y_test, rf_preds),
        accuracy_score(y_test, cb_preds)
                     ],
    
    'Precision': [
        precision_score(y_test, lr_preds),
        precision_score(y_test, knn_preds),
        precision_score(y_test, svc_preds),
        precision_score(y_test, gs_preds),
        precision_score(y_test, dt_preds),
        precision_score(y_test, rf_preds),
        precision_score(y_test, cb_preds)
                ],
    
    'Recall': [
        recall_score(y_test, lr_preds),
        recall_score(y_test, knn_preds),
        recall_score(y_test, svc_preds),
        recall_score(y_test, gs_preds),
        recall_score(y_test, dt_preds),
        recall_score(y_test, rf_preds),
        recall_score(y_test, cb_preds)
                ],
    
    'F1_Score': [
        f1_score(y_test, lr_preds),
        f1_score(y_test, knn_preds),
        f1_score(y_test, svc_preds),
        f1_score(y_test, gs_preds),
        f1_score(y_test, dt_preds),
        f1_score(y_test, rf_preds),
        f1_score(y_test, cb_preds)
                ],
})

models<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.16465660431862497, 'fraction_numerical': 0.012926987795190295, 'mean_word_length': 6.4371643394199785, 'pattern_counts': {'":': 0, '<': 66, '<?xml version=': 0, '>': 66, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '519936', 'license': 'Apache-2.0', 'n_tokens_mistral': 5559, 'n_tokens_neox': 5307, 'n_words': 832, 'path': 'notebooks/Mushroom.ipynb', 'repo_name': 'meren26/playground', 'size': '13847'}