<jupyter_start><jupyter_text>OVERVIEW1. Announcements2. Gradient Boosting3. XGBoost4. Feature Selection    - GINI Impurity    - Drop column importances    - Permutational importances5. Project module overview Lambda School Data Science*Unit 2, Sprint 3, Module 3*---  Permutation & Boosting- Get **permutation importances** for model interpretation and feature selection- Use xgboost for **gradient boosting**  SetupRun the code cell below. You can work locally (follow the [local setup instructions](https://lambdaschool.github.io/ds/unit2/local/)) or on Colab.Libraries:- category_encoders- matplotlib- numpy- pandas- scikit-learn- [**xgboost**](https://xgboost.readthedocs.io/en/latest/)<jupyter_code>!pip install category_encoders
# data analysis and wrangling
import pandas as pd
import numpy as np

# visualization
import matplotlib.pyplot as plt
%matplotlib inline

from sklearn.model_selection import train_test_split

# encoders
from category_encoders import OneHotEncoder, OrdinalEncoder
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler

#metrics
from sklearn.metrics import accuracy_score, precision_score, recall_score, classification_report, roc_auc_score

#pipeline
from sklearn.pipeline import make_pipeline

# machine learning
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.tree import DecisionTreeClassifier
from xgboost import XGBClassifier
from xgboost import XGBRegressor

from IPython.display import Image
from IPython.core.display import HTML 
def wrangle(fm_path, tv_path=None):
    if tv_path:
        df = pd.merge(pd.read_csv(fm_path, parse_dates=['date_recorded'], na_values=[-2e-08, 0]), 
                         pd.read_csv(tv_path)).set_index('date_recorded')

    else:
        df = pd.read_csv(DATA_PATH+'waterpumps/test_features.csv', parse_dates=['date_recorded']).set_index('date_recorded')
    
    # When columns have zeros and shouldn't, they are like null values.
    # So we will replace the zeros with nulls, and impute missing values later.
    # Also create a "missing indicator" column, because the fact that
    # values are missing may be a predictive signal.
    cols_with_zeros = ['longitude', 'latitude', 'construction_year', 
                       'gps_height', 'population']
    for col in cols_with_zeros:
        df[col] = df[col].replace(0, np.nan)
        df[col+'_MISSING'] = df[col].isnull()
            
    # Drop duplicate columns
    duplicates = ['quantity_group', 'payment_type']
    df = df.drop(columns=duplicates)
    
    # Drop recorded_by (never varies) and id (always varies, random)
    unusable_variance = ['recorded_by', 'id']
    df = df.drop(columns=unusable_variance)
    
    
    # Edftract components from date_recorded, then drop the original column
    df['year_recorded'] = df.index.year
    df['month_recorded'] = df.index.month
    df['day_recorded'] = df.index.day

    # Engineer feature: how many years from construction_year to date_recorded
    df['years'] = df['year_recorded'] - df['construction_year']
    df['years_MISSING'] = df['years'].isnull()
    
    # return the wrangled dataframe
    return df
DATA_PATH = '../data/'
fm_path = 'train_features.csv'
tv_path = 'train_labels.csv'
df = wrangle(fm_path, tv_path)
df<jupyter_output><empty_output><jupyter_text>SPLIT DATA<jupyter_code>target = 'status_group'
X,y = df.drop(columns=target),df[target]
X_train,X_val,y_train,y_val = train_test_split(X,y,test_size = 0.20,random_state=42)<jupyter_output><empty_output><jupyter_text>ESTABLISH BASELINE<jupyter_code>y_train.value_counts(normalize=True).max()<jupyter_output><empty_output><jupyter_text>BUILD MODEL Decision TreesRandom Forests - bootstrapping to create data sets and use a subset of feature for each split : voted decision for classification and an average for regression - BAGGINGBOOSTING - Ensemble learning method where the trees aren't built in parallel.If I denote my models using fi(x) then you have the boosted model that yould be a weighted average across all the f(x)s.<jupyter_code>model_rf = make_pipeline(
    OrdinalEncoder(), 
    SimpleImputer(strategy='mean'), 
    RandomForestClassifier(random_state=42, n_jobs=-1)
)

# Fit on train, score on val
model_rf.fit(X_train, y_train)
# Tune parameters like learning rate, n_estimators, And the rest are like your RF models.
model_gbm = make_pipeline(
    OrdinalEncoder(), 
    SimpleImputer(strategy='mean'), 
    GradientBoostingClassifier(random_state=42)
)

model_gbm.fit(X_train, y_train)
# XGBOOST - Extreme gradient boosting is nothing but gradient boosting + regularization. It also does parallel computing.
model_xgb = make_pipeline(
    OrdinalEncoder(), 
    SimpleImputer(strategy='mean'), 
    XGBClassifier(random_state=42,n_jobs=-1)
)

# Fit on train, score on val
model_xgb.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>CHECK METRICS<jupyter_code>print('Training Accuracy', model_rf.score(X_train, y_train))
print('Validation Accuracy', model_rf.score(X_val, y_val))
print('Training Accuracy', model_gbm.score(X_train, y_train))
print('Validation Accuracy', model_gbm.score(X_val, y_val))
print('Training Accuracy', model_xgb.score(X_train, y_train))
print('Validation Accuracy', model_xgb.score(X_val, y_val))<jupyter_output>Training Accuracy 0.7521504669209522
Validation Accuracy 0.7423190235690236<jupyter_text>Decision trees -> Random Forest (Uses the technique of bootstrapping and bagging for prediction) -> Gradient boosting -> XGBoost  FEATURE SELECTION I Gini impurity<jupyter_code>importances =  model_rf.named_steps['randomforestclassifier'].feature_importances_
feature_names = X_train.columns
feat_imp =pd.Series(data=importances,index=feature_names).sort_values()
feat_imp.tail(10).plot(kind='barh')
importances =  model_xgb.named_steps['xgbclassifier'].feature_importances_
feature_names = X_train.columns
feat_imp =pd.Series(data=importances,index=feature_names).sort_values()
feat_imp.tail(10).plot(kind='barh')
import seaborn as sns
sns.scatterplot(x='latitude',y='longitude',data=df,hue='status_group')<jupyter_output><empty_output><jupyter_text>II drop column importances<jupyter_code>column = 'source'
# cols = X_train.columns
# for col in cols:
model_dci = make_pipeline(
    OrdinalEncoder(), 
    SimpleImputer(strategy='mean'), 
    XGBClassifier(random_state=42,n_jobs=-1))

model_dci.fit(X_train.drop(columns=column),y_train);
print('The validation accuracy by dropping quantity is', model_dci.score(X_val.drop(columns=column),y_val))<jupyter_output>The validation accuracy by dropping quantity is 0.73989898989899<jupyter_text>When we dropped 'quantity', our accuracy dropped from about 74% to 71%. Since it significantly affects our accuracy, we do not drop it.When we dropped 'payment', the accuracy dropped from 74.2% to about 74% RFE - Recursive feature elimination  III Permutation Importance We train our model first.Shuffle one column at a time (which is equivalent to corrupting or crating noise) and check if the accuracy drops. If it does, then it means the shuffled column is important and you would retain it in your model.<jupyter_code>Image(url= "https://i.imgur.com/h17tMUU.png")
model_dci = make_pipeline(
    OrdinalEncoder(), 
    SimpleImputer(strategy='mean'), 
    XGBClassifier(random_state=42,n_jobs=-1, n_estimators = 50))

model_dci.fit(X_train,y_train);
print('The validation accuracy for all features (nestiamtors of 25) is', model_dci.score(X_val,y_val))
X_val_perm = X_val.copy()
X_val_perm['latitude'] = np.random.permutation(X_val_perm['latitude'])
print('The validation accuracy by shuffling the quantity column is', model_dci.score(X_val_perm,y_val))
# Using sklearn permutation_importance library
from sklearn.inspection import permutation_importance
perm_imp = permutation_importance(model_dci,X_val,y_val,random_state=42)
data = {'imp_mean':perm_imp['importances_mean'],
        'imp_std':perm_imp['importances_std']}
df_perm = pd.DataFrame(data,index=X_val.columns).sort_values(by='imp_mean')
df_perm['imp_mean'].tail(10).plot(kind='barh')
df_perm<jupyter_output><empty_output><jupyter_text>Use xgboost for **gradient boosting**<jupyter_code>Image(url= "https://media.geeksforgeeks.org/wp-content/uploads/20200721214745/gradientboosting.PNG")
## The below diagram explains how gradient boosted trees are trained for regression problems.<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10017699115044248, 'fraction_numerical': 0.014041297935103244, 'mean_word_length': 5.863157894736842, 'pattern_counts': {'":': 0, '<': 35, '<?xml version=': 0, '>': 38, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '402680', 'license': 'MIT', 'n_tokens_mistral': 2876, 'n_tokens_neox': 2709, 'n_words': 668, 'path': 'DS29_GP_Unit2_Sprint3_Module3_Nivi.ipynb', 'repo_name': 'nicholashagemann/lambdalectures', 'size': '8475'}