<jupyter_start><jupyter_text>______  WELCOME! Welcome to "***Tree Types Prediction Project***". This is the second medium project of ***Machine Learning*** course.In this project, you must perform EDA processes for implementing predictive models. Handling outliers, domain knowledge and feature engineering (using ***sqlite3*** library) will be challenges. Also, this project aims to improve your ability to implement algorithms for ***Multi-Class Classification***. Thus, you will have the opportunity to implement many algorithms commonly used for Multi-Class Classification problems.Before diving into the project, please take a look at the determines and tasks.- ***NOTE:*** *This project assumes that you already know the basics of coding in Python. You should also be familiar with the theory behind classification models and scikit-learn module as well as Machine Learning before you begin.* ------  Determines Dataset contains tree observations from four areas of one national forest district. This dataset includes information on tree type, shadow coverage, distance to nearby landmarks, soil type, and local topography. The goal of the project is to build a model that predicts what types of trees grow in an area.***The Forest Dataset*** contains approximately 600 thousand lines, also you can easily find many information about it on the web (especially Kaggle).---To achieve high prediction success, you must understand the data well and develop different approaches that can affect the dependent variable.Firstly, try to understand the dataset column by column using pandas module. Do research within the scope of domain (forest, trees) knowledge on the internet to get to know the data set in the fastest way. You should implement cleaning, handling with outliers and missing values using Pandas, NumPy and other required modules for the best result in modeling. You should do Feature Engineering using SQLite local database. At this point, in order to improve your skills of using SQL with Python, you are asked to perform feature engineering operations using *sqlite3* library in Python.After that, your final dataset with the new variables you have created will be ready for model building. You will implement ***Support Vector Machine, XGBoost, Random Forest, Desicion Tree*** and ***K-Nearest Neighbors (KNN)*** algorithms. Also, evaluate the success of your models with appropriate performance metrics and visualize them using ***Yellowbrick, Seaborn*** or ***Matplotlib*** modules.At the end of the project, create a chart comparing the performance of all models and choose the most successful model.- ***NOTE:*** *Evaluate your models knowing that this is [imbalanced data](https://towardsdatascience.com/having-an-imbalanced-dataset-here-is-how-you-can-solve-it-1640568947eb). This is not the primary goal of the project, but you can study solve the class [imbalance problem](https://towardsdatascience.com/handling-imbalanced-datasets-in-machine-learning-7a0e84220f28) if you want.* ------  Tasks  1. Exploratory Data Analysis (EDA)- Import Libraries, Load Dataset, Exploring Data    *i. Import Libraries*        *ii. Load Dataset*        *iii. Explore Data* 2.  Data Cleaning- Detect Missing Values and Outliers     *i. Missing Value Detection*        *ii. Outlier Detection*    - Deal with Outliers        *i. Visualize Zscore Tresholds (how many times IQR) by Continuous Variables*        *ii. Drop Outliers* 3. Feature Engineering with sqlite3 Module 4. Prediction (Multi-class Classification)- Import libraries- Data Preprocessing- Implement KNN Classifer- Implement Decision Tree Classifier- Implement Random Forest Classifer- Implement XGBoost Classifer- Implement SVM Classifer- Compare The Models ------  1. Exploratory Data Analysis  Import Libraries, Load Dataset, Exploring Data  *i. Import Libraries* Besides Numpy and Pandas, you need to import the necessary modules for data visualization, data preprocessing, Model building and tuning.*Note: Check out the course materials.*<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import warnings
warnings.filterwarnings('ignore')
%matplotlib inline<jupyter_output><empty_output><jupyter_text>*ii. Load Dataset*<jupyter_code>df = pd.read_csv("covtype.csv")<jupyter_output><empty_output><jupyter_text>*iii. Explore Data*- Focus on numerical and categorical data- Detect Number of Unique values of each column- Focus on Target Variable (Cover_Type) - Detect relationships and correlations between independent variables and target variable. - It may be nice to visualize the class frequencies of the target variable.- Detect relationships and correlations between independent variables. (You can prefer to keep only one of the highly correlated continuous variables.)- Consider dropping features that contain little data or that you think will not contribute to the model.<jupyter_code>df
df.shape
df.info()
df.Soil_Type1.value_counts(dropna=False)<jupyter_output><empty_output><jupyter_text>******  2.  Data Cleaning  Detect Missing Values and Outliers  *i. Missing Value Detection*<jupyter_code>plt.figure(figsize=(20,7))
sns.heatmap(df.isnull(),yticklabels=False,cbar=False,cmap='viridis')<jupyter_output><empty_output><jupyter_text>*ii. Outlier Detection*The columns which have continuous value should be examined in terms of [outliers](https://datascience.foundation/sciencewhitepaper/knowing-all-about-outliers-in-machine-learning) (Watch out for columns that look like continuous but not continuous!). Some algorithms are [sensitive to outliers](https://arsrinevetha.medium.com/ml-algorithms-sensitivity-towards-outliers-f3862a13c94d), but some algorithms can tolerate them. You can decide to outlier detection according to the algorithm you will use.- You can check the outliers shape of continous features with respect to the target (Cover_Type) classes.- You can check how many outliers are there of each continuous variables.<jupyter_code>fig, axes = plt.subplots(nrows = 4,ncols = 3,figsize = (32,40))
i=0
y=0
for column in df.columns[:10]:
    sns.boxplot(column, data = df,ax=axes[i,y])
    if y != 2:
        y += 1
    else:
        i += 1
        y = 0<jupyter_output><empty_output><jupyter_text>Deal with Outliers  *i. Visualize Zscore Tresholds (how many times IQR) by Continuous Variables* There are many different methods for outliers. You can use IQR values used as standard to deal with outliers, or you can define two functions to help you understand the outliers and how you can deal with them.- Two functions given as extra for outlier detection in *EDA Project (Outo Scout)* are given below.<jupyter_code>def outlier_zscore(df, col, min_z=1, max_z = 5, step = 0.1, print_list = False):
    z_scores = stats.zscore(df[col].dropna())
    threshold_list = []
    for threshold in np.arange(min_z, max_z, step):
        threshold_list.append((threshold, len(np.where(z_scores > threshold)[0])))
        df_outlier = pd.DataFrame(threshold_list, columns = ['threshold', 'outlier_count'])
        df_outlier['pct'] = (df_outlier.outlier_count - df_outlier.outlier_count.shift(-1))/df_outlier.outlier_count*100
    plt.plot(df_outlier.threshold, df_outlier.outlier_count)
    best_treshold = round(df_outlier.iloc[df_outlier.pct.argmax(), 0],2)
    outlier_limit = int(df[col].dropna().mean() + (df[col].dropna().std()) * df_outlier.iloc[df_outlier.pct.argmax(), 0])
    percentile_threshold = stats.percentileofscore(df[col].dropna(), outlier_limit)
    plt.vlines(best_treshold, 0, df_outlier.outlier_count.max(), 
               colors="r", ls = ":"
              )
    plt.annotate("Zscore : {}\nValue : {}\nPercentile : {}".format(best_treshold, outlier_limit, 
                                                                   (np.round(percentile_threshold, 3), 
                                                                    np.round(100-percentile_threshold, 3))), 
                 (best_treshold, df_outlier.outlier_count.max()/2))
    #plt.show()
    if print_list:
        print(df_outlier)
    return (plt, df_outlier, best_treshold, outlier_limit, percentile_threshold)
def outlier_inspect(df, col, min_z=1, max_z = 5, step = 0.5, max_hist = None, bins = 50):
    fig = plt.figure(figsize=(20, 6))
    fig.suptitle(col, fontsize=16)
    plt.subplot(1,3,1)
    if max_hist == None:
        sns.distplot(df[col], kde=False, bins = 50)
    else :
        sns.distplot(df[df[col]<=max_hist][col], kde=False, bins = 50)
    plt.subplot(1,3,2)
    sns.boxplot(df[col])
    plt.subplot(1,3,3)
    z_score_inspect = outlier_zscore(df, col, min_z=min_z, max_z = max_z, step = step)
    plt.show()
from scipy import stats
for column in df.columns[:10]:
    outlier_inspect(df,column)
z_scores = stats.zscore(df.Elevation)
print(z_scores)
abs_z_scores = np.abs(z_scores)
print(abs_z_scores)
df.loc[abs_z_scores > 3, "Elevation"] = df.Elevation.mean()
df.groupby("Cover_Type")["Aspect"].mean()
z_scores = stats.zscore(df.Aspect)
abs_z_scores = np.abs(z_scores)

df.loc[abs_z_scores > 1.5, "Aspect"] = df.Aspect.mean()
z_scores = stats.zscore(df.Slope)
abs_z_scores = np.abs(z_scores)
df.loc[abs_z_scores > 3, "Slope"] = df.Slope.mean()

z_scores = stats.zscore(df.Horizontal_Distance_To_Hydrology)
abs_z_scores = np.abs(z_scores)
df.loc[abs_z_scores > 4, "Horizontal_Distance_To_Hydrology"] = df.Horizontal_Distance_To_Hydrology.mean()

z_scores = stats.zscore(df.Vertical_Distance_To_Hydrology)
abs_z_scores = np.abs(z_scores)
df.loc[abs_z_scores > 2.5, "Vertical_Distance_To_Hydrology"] = df.Vertical_Distance_To_Hydrology.mean()

z_scores = stats.zscore(df.Horizontal_Distance_To_Roadways)
abs_z_scores = np.abs(z_scores)
df.loc[abs_z_scores > 3, "Horizontal_Distance_To_Roadways"] = df.Horizontal_Distance_To_Roadways.mean()

z_scores = stats.zscore(df.Hillshade_9am)
abs_z_scores = np.abs(z_scores)
df.loc[abs_z_scores > 1.5, "Hillshade_9am"] = df.Hillshade_9am.mean()
z_scores = stats.zscore(df.Hillshade_Noon)
abs_z_scores = np.abs(z_scores)
df.loc[abs_z_scores > 1.5, "Hillshade_Noon"] = df.Hillshade_Noon.mean()
z_scores = stats.zscore(df["Hillshade_3pm"])
abs_z_scores = np.abs(z_scores)
df.loc[abs_z_scores > 2.5, "Hillshade_3pm"] = df.Hillshade_3pm.mean()


z_scores = stats.zscore(df.Horizontal_Distance_To_Fire_Points)
abs_z_scores = np.abs(z_scores)
df.loc[abs_z_scores > 3.5, "Horizontal_Distance_To_Fire_Points"] = df.Horizontal_Distance_To_Fire_Points.mean()

df.shape
df.isnull().sum()
fig, axes = plt.subplots(nrows = 4,ncols = 3,figsize = (32,40))
i=0
y=0
for column in df.columns[:10]:
    sns.boxplot(column, data = df,ax=axes[i,y],whis=3)
    if y != 2:
        y += 1
    else:
        i += 1
        y = 0
df.shape
df_cleaned = df<jupyter_output><empty_output><jupyter_text>*ii. Drop Outliers* You can define another function to detect outliers in accordance with the ``zscore`` (how many times IQR) value you choose according to the result from the previous functions. ******  3. Feature Engineering with *sqlite3* Library Feature engineering is an optional process to increase the predictive success of the model. The effort you put in should be worth increasing success. So you can develop your own feature engineering approach.Note that, after seeing the result of the models, there may be a possibility of making minor changes to the features in the modeling phase.You are requested to do feature engineering operations with SQL. There are two ways to do this:1. After moving the final version of your data set to ***SQLite Browser*** and performing feature engineering operations there, you can convert the resulting data set to dataframe format and use it again in python.2. In Python, you can create a database and table with your data set by using the functions of the sqlite3 library, after performing feature engineering with SQL, you can convert the resulting data set to a dataframe.In this case, we will illustrate the second method.Follow the steps below to do feature engineering with the [sqlite3](https://docs.python.org/3/library/sqlite3.html) library: 1. Import *sqlite3* library 2. Create a sqlite database (``"tree_database"``) and transferring dataframe(``tree1``) from python into database table (``covtype2``)  - You can use *connect(), to_sql() and read_sql_query()* functions. 3. Assign SQL codes for feature engineering to an object. (produce or transform new columns, get rid of unnecassary columns, make the dataset ready to model) 4. Transform final version of SQL table to dataframe. Finally, you can save the final version of your data as csv for use in your models and work on a different notebook. On the other hand, you can continue to work on this notebook with the last dataframe.<jupyter_code>df_cleaned

plt.figure(figsize=(10,20))
heatmap = df_cleaned.corr()
sns.heatmap(heatmap[["Cover_Type"]].sort_values(by=['Cover_Type'],ascending=False), annot=True,cmap="coolwarm")<jupyter_output><empty_output><jupyter_text>******  4. Prediction (Multi-class Classification) If you have done, use your data set resulting from Feature Engineering task. If you haven't done Feature Engineering, use the latest version of your data set.In this section, you have two main tasks that apply to each algorithm:1. Model Building and Prediction - XGBoost (Use ``XGBClassifier`` model from``xgboost`` module) - SVM (Use ``LinearSVC`` model from``sklearn.svm`` module) - Decision Tree (Use ``DecisionTreeClassifier`` model from ``sklearn.tree`` module) - KNN (Use ``KNeighborsClassifier`` model from ``sklearn.neighbors`` module) - Random Forest (Use ``RandomForestClassifier`` model from ``sklearn.ensemble`` module)  2. Visualizing the Result- Use [yellowbrick](https://www.scikit-yb.org/en/latest/), [seaborn](https://seaborn.pydata.org/tutorial/regression.html) or [matplotlib](https://matplotlib.org/) modules to visualize the model results.- Show three plots for the results: - Class Prediction Error Bar Plot - Confusion Matrix - Classification Report  Import Libraries<jupyter_code>from sklearn.preprocessing import scale, StandardScaler
from sklearn.model_selection import train_test_split, GridSearchCV, cross_val_score,RandomizedSearchCV
from sklearn.metrics import confusion_matrix, accuracy_score, mean_squared_error, r2_score, roc_auc_score, roc_curve, classification_report
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.metrics import plot_confusion_matrix
import warnings
warnings.filterwarnings("ignore")<jupyter_output><empty_output><jupyter_text>Data Preprocessing - Drop target variable- Train-Test Split*Note: You can use the train and test data generated here for all algorithms.*<jupyter_code>X = df_cleaned.drop("Cover_Type", axis=1)

y = df_cleaned["Cover_Type"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25,stratify=y, random_state=42)<jupyter_output><empty_output><jupyter_text>Implement KNeighborsClassiferThe first and most important step for the [KNN](https://www.analyticsvidhya.com/blog/2018/03/introduction-k-neighbours-algorithm-clustering/) algorithm is to determine the optimal k (number of neighbors). Build different models with k values in the range you specify. You can observe the change of train and test accuracy values according to different k values using a plot. The point at which train accuracy and test accuracy values begin to run parallel is the optimal k value. Then set up your final KNN model with the optimal k value you determined and calculate accuracy.- Import the modul- Fit the model - Predict the test set- Visualize the result- Evaluate the performance<jupyter_code>"""sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)"""
knn_model=KNeighborsClassifier().fit(X_train, y_train)
y_pred=knn_model.predict(X_test)
print(confusion_matrix(y_test, y_pred))
plot_confusion_matrix(knn_model,X_test,y_test)
print(classification_report(y_test, y_pred))<jupyter_output>precision    recall  f1-score   support

           1       0.97      0.96      0.96     52960
           2       0.97      0.97      0.97     70825
           3       0.95      0.96      0.95      8938
           4       0.88      0.72      0.80       687
           5       0.92      0.88      0.90      2373
           6       0.92      0.91      0.92      4342
           7       0.97      0.96      0.97      5128

    accuracy                           0.96    145253
   macro avg       0.94      0.91      0.92    145253
weighted avg       0.96      0.96      0.96    145253<jupyter_text>Implement Decision Tree Classifier- Import the modul - Fit the model - Predict the test set- Visualize and evaluate the result (use yellowbrick module)<jupyter_code>dt_mod = DecisionTreeClassifier()
X = df_cleaned.drop("Cover_Type", axis=1)

y = df_cleaned["Cover_Type"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25,stratify=y, random_state=42)
dt_mod.fit(X_train,y_train)
y_pred = dt_mod.predict(X_test)
print(confusion_matrix(y_test,y_pred))

plot_confusion_matrix(dt_mod,X_test,y_test)

print(classification_report(y_test,y_pred))<jupyter_output>[[49523  3123     6     0    37     7   264]
 [ 3176 66963   175     1   301   157    52]
 [    4   206  8266    90    20   352     0]
 [    0     8    81   567     0    31     0]
 [   53   359    23     0  1926    11     1]
 [   14   160   424    29     5  3710     0]
 [  264    39     1     0     1     0  4823]]
              precision    recall  f1-score   support

           1       0.93      0.94      0.93     52960
           2       0.95      0.95      0.95     70825
           3       0.92      0.92      0.92      8938
           4       0.83      0.83      0.83       687
           5       0.84      0.81      0.83      2373
           6       0.87      0.85      0.86      4342
           7       0.94      0.94      0.94      5128

    accuracy                           0.93    145253
   macro avg       0.90      0.89      0.89    145253
weighted avg       0.93      0.93      0.93    145253<jupyter_text>Implement Random Forest Classifier- Import the modul - Fit the model - Predict the test set- Visualize and evaluate the result (use yellowbrick module)<jupyter_code>rf_mod = RandomForestClassifier()
rf_mod.fit(X_train,y_train)
y_pred = rf_mod.predict(X_test)
print(confusion_matrix(y_test,y_pred))

plot_confusion_matrix(rf_mod,X_test,y_test)

print(classification_report(y_test,y_pred))<jupyter_output>[[49909  2927     1     0     9     2   112]
 [ 1671 68816   146     1    77    91    23]
 [    1   143  8566    32     8   188     0]
 [    0     1    81   574     0    31     0]
 [   33   495    22     0  1813    10     0]
 [    7   124   318    23     4  3866     0]
 [  257    32     0     0     0     0  4839]]
              precision    recall  f1-score   support

           1       0.96      0.94      0.95     52960
           2       0.95      0.97      0.96     70825
           3       0.94      0.96      0.95      8938
           4       0.91      0.84      0.87       687
           5       0.95      0.76      0.85      2373
           6       0.92      0.89      0.91      4342
           7       0.97      0.94      0.96      5128

    accuracy                           0.95    145253
   macro avg       0.94      0.90      0.92    145253
weighted avg       0.95      0.95      0.95    145253<jupyter_text>RT Tuned<jupyter_code>rf = RandomForestClassifier()
rf_params = {"n_estimators" :[100],
             "min_samples_split" :[4, 6, 8],
             "max_features" : [14, 16, 20],
             "max_depth":[20, 25, 30]}
# rf_cv_model = GridSearchCV(rf, rf_params, cv = 5, n_jobs = -1, verbose = 5).fit(X_train, y_train)
# rf_cv_model.best_params_
# rf_cv_model2 = RandomizedSearchCV(rf, rf_params, cv = 5, n_jobs = -1, verbose = 5, n_iter =10).fit(X_train, y_train)

# print(rf_cv_model2)
# rf_cv_model2.best_params_
rf_tuned = RandomForestClassifier(n_estimators = 100,
                                  min_samples_split = 6,
                                  max_features = 20,
                                  max_depth = 30).fit(X_train, y_train)
y_pred = rf_tuned.predict(X_test)
print(confusion_matrix(y_test,y_pred))

plot_confusion_matrix(rf_tuned,X_test,y_test)

print(classification_report(y_test,y_pred))<jupyter_output>[[50423  2416     2     0     7     2   110]
 [ 1368 69144   124     2    88    73    26]
 [    0   125  8605    39    10   159     0]
 [    0     0    69   594     0    24     0]
 [   26   441    21     0  1873    12     0]
 [    3   103   247    22     3  3964     0]
 [  209    32     0     0     1     0  4886]]
              precision    recall  f1-score   support

           1       0.97      0.95      0.96     52960
           2       0.96      0.98      0.97     70825
           3       0.95      0.96      0.96      8938
           4       0.90      0.86      0.88       687
           5       0.95      0.79      0.86      2373
           6       0.94      0.91      0.92      4342
           7       0.97      0.95      0.96      5128

    accuracy                           0.96    145253
   macro avg       0.95      0.92      0.93    145253
weighted avg       0.96      0.96      0.96    145253<jupyter_text>**from pandasgui import showgui = show(df)**  Implement XGBoost Classifer- Import the modul - Fit the model - Predict the test set- Visualize and evaluate the result (use yellowbrick module)<jupyter_code>from xgboost import XGBClassifier
xgb_classifier = XGBClassifier()
xgb_classifier.fit(X_train , y_train)
y_pred = xgb_classifier.predict(X_test)
xgb_accuracy = accuracy_score(y_test, y_pred)

xgb_accuracy
print(confusion_matrix(y_test,y_pred))

plot_confusion_matrix(rf_tuned,X_test,y_test)

print(classification_report(y_test,y_pred))<jupyter_output>[[50423  2416     2     0     7     2   110]
 [ 1368 69144   124     2    88    73    26]
 [    0   125  8605    39    10   159     0]
 [    0     0    69   594     0    24     0]
 [   26   441    21     0  1873    12     0]
 [    3   103   247    22     3  3964     0]
 [  209    32     0     0     1     0  4886]]
              precision    recall  f1-score   support

           1       0.97      0.95      0.96     52960
           2       0.96      0.98      0.97     70825
           3       0.95      0.96      0.96      8938
           4       0.90      0.86      0.88       687
           5       0.95      0.79      0.86      2373
           6       0.94      0.91      0.92      4342
           7       0.97      0.95      0.96      5128

    accuracy                           0.96    145253
   macro avg       0.95      0.92      0.93    145253
weighted avg       0.96      0.96      0.96    145253<jupyter_text>Implement Support Vector Machine- Import the modul - Fit the model - Predict the test set- Visualize and evaluate the result (use yellowbrick module)*Note: You probably won't get a successful result. You may need to make some changes to the model or data. This may be a topic worth investigating, you decide.*  Compare The ModelsSo far, you have created a multi-classifier model with 5 different algorithms and made predictions. You can observe the performance of the models together with a barplot of your choice.- Which algorithm did you achieve the highest prediction performance with? - What could be the factors that cause this? What are the principles of the most successful algorithm and its differences from other algorithms? In contrast;- Which algorithm did you achieve the lowest prediction performance with? - What could be the factors that cause this? What are the principles of the most successful algorithm and its differences from other algorithms? The answers you will look for to these questions will increase your gains from Machine Learning course.<jupyter_code>compare = pd.DataFrame({"Model": ["K-Nearest Neighbor", "Random Forest", "XGBoost", "Decision Tree"],
                        "Accuracy": [knn_accuracy, rf_accuracy, xgb_accuracy, tree_accuracy]})
compare = compare.sort_values(by="Accuracy", ascending=True)
plt.figure(figsize=(10,10))
ax = sns.barplot(x="Accuracy", y="Model", data=compare, palette="Blues_d")
plt.yticks(size = 14)
plt.xticks(size = 14)
plt.title("Accuracy Score of Different Models", size=14)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0850915930650965, 'fraction_numerical': 0.05503761858030749, 'mean_word_length': 3.0218714027298144, 'pattern_counts': {'":': 4, '<': 57, '<?xml version=': 0, '>': 67, 'https://': 9, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 42, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '429563', 'license': 'MIT', 'n_tokens_mistral': 8376, 'n_tokens_neox': 7462, 'n_words': 2493, 'path': '11-ML_TreeTypesPrediction_project/Tree_Types_Prediction_Orhan_solution.ipynb', 'repo_name': 'ssezerer/Projects', 'size': '24456'}