<jupyter_start><jupyter_text>Lambda School Data Science*Unit 2, Sprint 2, Module 4*---  Overview1. Announcements2. Warmup3. A few pointers for yesterday's module4. Saving models5. Classification metrics6. Working locally vs cloud  Classification Metrics- get and interpret the **confusion matrix** for classification models- use classification metrics: **precision, recall**- understand the relationships between precision, recall, **thresholds, and predicted probabilities**, to help **make decisions and allocate budgets**<jupyter_code>%%capture
import sys

# If you're on Colab:
if 'google.colab' in sys.modules:
    DATA_PATH = 'https://raw.githubusercontent.com/LambdaSchool/DS-Unit-2-Kaggle-Challenge/master/data/'
    !pip install category_encoders==2.*

# If you're working locally:
else:
    DATA_PATH = '../data/'
from category_encoders import OneHotEncoder, OrdinalEncoder
from sklearn.ensemble import RandomForestClassifier
from sklearn.impute import SimpleImputer
from sklearn.metrics import plot_confusion_matrix, classification_report
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
import numpy as np
import pandas as pd
#Class 0 -  1000 samples - no attacks in your network
#Class 1 - 100 samples - network attacks
100/1100
#ASsume all 100 samples were misclassified as Class 0
1000/1100<jupyter_output><empty_output><jupyter_text>I. Wrangle Data<jupyter_code>train = pd.merge(
    pd.read_csv('train_features.csv', na_values=[0, -2.000000e-08], parse_dates=['date_recorded']), 
    pd.read_csv('train_labels.csv', na_values=[0, -2.000000e-08])
    )

def wrangle(df):

  # Set the index to 'id'
  df.set_index('id', inplace=True)

  # Drop Constant Column
  df.drop(columns='recorded_by', inplace=True)

  # Drop Duplicate Column
  df.drop(columns='quantity_group', inplace=True)

  # Create age feature
  df['pump_age'] = df['date_recorded'].dt.year - df['construction_year']
  df.drop(columns='date_recorded', inplace=True)

  # Drop High Cardinality Columns
  cols_to_drop = [col for col in df.select_dtypes('object') if df[col].nunique() > 100]
  df.drop(columns=cols_to_drop, inplace=True)

  # Drop columns with high proportion of null values
  df.drop(columns='num_private', inplace=True)

  return df

train = wrangle(train)

X_test=pd.read_csv('test_features.csv',na_values=[0, -2.000000e-08], parse_dates=['date_recorded'])
test=wrangle(X_test)
train.head()
pd.unique(train['status_group'])<jupyter_output><empty_output><jupyter_text>EDA- How can we transform our target so that this is a **binary classification** problem?<jupyter_code># Classify into pumps that need repair and pumps that do not need repairs.
#functional - 0
#non functional and functional needs repair - 1
train.loc[train['status_group'] == 'functional','needs_repair'] = 0
train.loc[train['status_group'] != 'functional','needs_repair'] = 1
train['needs_repair']=(train['status_group'] != 'functional').astype(int)
train.head()<jupyter_output><empty_output><jupyter_text>II. Split Data<jupyter_code>target='needs_repair'
X = train.drop(columns=target)
y = train[target]
X.drop(columns = ['status_group'],inplace=True)
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)<jupyter_output><empty_output><jupyter_text>Baseline<jupyter_code>print('Baseline Accuracy:', y_train.value_counts(normalize=True).max())<jupyter_output>Baseline Accuracy: 0.551217548802576<jupyter_text>Build Model- `OrdinalEncoder`- `SimpleImputer`- `RandomForestClassifier`<jupyter_code>model = make_pipeline(
    OrdinalEncoder(),
    SimpleImputer(),
    RandomForestClassifier(n_estimators=25,
                           n_jobs=-1,
                           random_state=42)
)

model.fit(X_train, y_train);<jupyter_output><empty_output><jupyter_text>Interlude: Beware or Leakage<jupyter_code>importances = model.named_steps['randomforestclassifier'].feature_importances_
features = X_train.columns
pd.Series(importances, index=features).sort_values().tail(10).plot(kind='barh');<jupyter_output><empty_output><jupyter_text>Check Metrics **Accuracy Score**<jupyter_code>print('Training Accuracy:', model.score(X_train, y_train))
print('Validation Accuracy:', model.score(X_val, y_val))<jupyter_output>Training Accuracy: 0.993660563973064
Validation Accuracy: 0.8176557239057239<jupyter_text>**Confusion Matrix**<jupyter_code>plot_confusion_matrix(model, X_val,y_val, values_format = '.0f',display_labels=['no repair needed','repair needed'])<jupyter_output><empty_output><jupyter_text>**Precision Score**Of all the pumps that you predicted as needing repair, what proportion actually needed repair?<jupyter_code>#precision = (tp/tp+fp)
3416/(3416+763)<jupyter_output><empty_output><jupyter_text>**Recall Score** Of those pumps that actually needed repair, what proportion did you correctly predict as needing repair?<jupyter_code># recall = (tp/tp+fn)
3416/(3416+970)<jupyter_output><empty_output><jupyter_text>**Classification Report**<jupyter_code>print(classification_report(y_val,model.predict(X_val),target_names = ['no repair needed','repair needed']))
classification_report
#f1 score = (2*precision*recall)/(precision+recall)<jupyter_output><empty_output><jupyter_text>Case StudyLet's say that is costs the Tanzanian government $100 to inspect a water pump, and there is only funding for 2000 pump inspections.<jupyter_code>n_inspections = 2000
repair_prob = y_val.value_counts(normalize=True).min()
print('The number of inpsections conducted is', n_inspections)<jupyter_output>The number of inpsections conducted is 2000<jupyter_text>Scenario 1: Choose pumps randomly<jupyter_code># if you randomly chose pumps, the number of pumps 
#you would correctly identify as needing repair = (n_observations)*repair_prob
print('The number of inpsections conducted is', n_inspections)
pumps_repaired = n_inspections * repair_prob
print('The number of pumps that we would identify as needing repair without a model is ',pumps_repaired)<jupyter_output>The number of inpsections conducted is 2000
The number of pumps that we would identify as needing repair without a model  922.979797979798<jupyter_text>Scenario 2: Using our model "out of the box"<jupyter_code>from sklearn.metrics import precision_score
model_precision = precision_score(y_val,model.predict(X_val))
print('The number of pumps that our model would identify as needing repair is ', model_precision*n_inspections)<jupyter_output>The number of pumps that our model would identify as needing repair is  1634.8408710217755<jupyter_text>Scenario 3: We empasize **precision** in our model, and only select pumps that our model is very certain (`>0.85`) need repair.<jupyter_code>data = {'y_val': y_val,
        'y_pred_prob': model.predict_proba(X_val)[:,-1]}
result = pd.DataFrame(data)
result
result.loc[result['y_pred_prob'] > 0.85,'y_pred_prob'] = 1
result.loc[result['y_pred_prob'] <= 0.85,'y_pred_prob'] = 0
model_precision = precision_score(result['y_val'],result['y_pred_prob'])
print('The number of pumps that our model would identify as needing repair is ', model_precision*n_inspections)<jupyter_output>The number of pumps that our model would identify as needing repair is  1908.5667215815486
{'fraction_non_alphanumeric': 0.11037709881640517, 'fraction_numerical': 0.03358106248279659, 'mean_word_length': 6.95947426067908, 'pattern_counts': {'":': 0, '<': 60, '<?xml version=': 0, '>': 62, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '427230', 'license': 'MIT', 'n_tokens_mistral': 2568, 'n_tokens_neox': 2315, 'n_words': 579, 'path': 'DS29_GS_Unit2_Sprint2_Module4_Nivi.ipynb', 'repo_name': 'nicholashagemann/lambdalectures', 'size': '7266'}