<jupyter_start><jupyter_text>_Lambda School Data Science, Unit 2_ Applied Modeling Sprint Challenge: Predict Chicago food inspections 🍕 For this Sprint Challenge, you'll use a dataset with information from inspections of restaurants and other food establishments in Chicago from January 2010 to March 2019. [See this PDF](https://data.cityofchicago.org/api/assets/BAD5301B-681A-4202-9D25-51B2CAE672FF) for descriptions of the data elements included in this dataset.According to [Chicago Department of Public Health — Food Protection Services](https://www.chicago.gov/city/en/depts/cdph/provdrs/healthy_restaurants/svcs/food-protection-services.html), "Chicago is home to 16,000 food establishments like restaurants, grocery stores, bakeries, wholesalers, lunchrooms, mobile food vendors and more. Our business is food safety and sanitation with one goal, to prevent the spread of food-borne disease. We do this by inspecting food businesses, responding to complaints and food recalls."   Your challenge: Predict whether inspections failedThe target is the `Fail` column.- When the food establishment failed the inspection, the target is `1`.- When the establishment passed, the target is `0`.  Run this cell to install packages in Colab:<jupyter_code>%%capture
import sys

if 'google.colab' in sys.modules:
    # Install packages in Colab
    !pip install category_encoders==2.*
    !pip install eli5
    !pip install pandas-profiling==2.*
    !pip install pdpbox
    !pip install shap<jupyter_output><empty_output><jupyter_text>Run this cell to load the data:<jupyter_code>import pandas as pd

train_url = 'https://drive.google.com/uc?export=download&id=13_tP9JpLcZHSPVpWcua4t2rY44K_s4H5'
test_url  = 'https://drive.google.com/uc?export=download&id=1GkDHjsiGrzOXoF_xcYjdzBTSjOIi3g5a'

train = pd.read_csv(train_url)
test  = pd.read_csv(test_url)

assert train.shape == (51916, 17)
assert test.shape  == (17306, 17)<jupyter_output><empty_output><jupyter_text>Part 1: PreprocessingYou may choose which features you want to use, and whether/how you will preprocess them. If you use categorical features, you may use any tools and techniques for encoding._To earn a score of 3 for this part, find and explain leakage. The dataset has a feature that will give you an ROC AUC score > 0.90 if you process and use the feature. Find the leakage and explain why the feature shouldn't be used in a real-world model to predict the results of future inspections._ Part 2: Modeling**Fit a model** with the train set. (You may use scikit-learn, xgboost, or any other library.) Use cross-validation or do a three-way split (train/validate/test) and **estimate your ROC AUC** validation score.Use your model to **predict probabilities** for the test set. **Get an ROC AUC test score >= 0.60.**_To earn a score of 3 for this part, get an ROC AUC test score >= 0.70 (without using the feature with leakage)._ Part 3: VisualizationMake visualizations for model interpretation. (You may use any libraries.) Choose two of these types:- Confusion Matrix- Permutation Importances- Partial Dependence Plot, 1 feature isolation- Partial Dependence Plot, 2 features interaction- Shapley Values_To earn a score of 3 for this part, make four of these visualization types._  Part 1: Preprocessing> You may choose which features you want to use, and whether/how you will preprocess them. If you use categorical features, you may use any tools and techniques for encoding.  Describe  Test, Train datasets<jupyter_code>train.describe()
test.describe()<jupyter_output><empty_output><jupyter_text>Peek at First 5 Rows of Test, Train datasets<jupyter_code>train.head()
test.head()<jupyter_output><empty_output><jupyter_text>Check for Nulls in Test, Train datasets<jupyter_code>train.isnull().sum()
test.isnull().sum()<jupyter_output><empty_output><jupyter_text>Check target 'Fail' Distribution<jupyter_code>(train['Fail'].map({0: 'Passed', 1: 'Failed'}).value_counts(normalize=True) * 100)\
  .plot.bar(title='Percentages of Inspection Results', figsize=(10, 5))<jupyter_output><empty_output><jupyter_text>The plot clearly shows this is an imbalanced class (binary) Classification problem.  Exploratory Data Analysis & Cleaning<jupyter_code># Drop 'AKA Name' column in train set since I will use the "DBA Name" column and the former has nulls while the latter does not,
# and both serve similar enough functions as a business identifier for my purposes
train = train.drop(columns=['AKA Name'])
train<jupyter_output><empty_output><jupyter_text>Get counts of businesses with 'serious' level 'Violations,' as 'Violations' may be a significantly predictive feature.  The more/more serious violations a store has, the more likely it seems to reason- it would or could 'fail' an inspection.<jupyter_code>import re # regex (regular expression) allows for string search/matching
train[train['Violations'].fillna('').str.contains('serious', flags=re.I)]['Fail'].value_counts()<jupyter_output><empty_output><jupyter_text>Arrange X features matrices and y target vectors<jupyter_code># Train dataset
X_train = train.drop(columns = 'Fail')
y_train = train['Fail']

# Test dataset
X_test = test.drop(columns = 'Fail')
y_test = test['Fail']

# check shapes of Train and Test
print(f' X_train.shape: {X_train.shape}\n X_test.shape:  {X_test.shape}\n \n y_train.shape: {y_train.shape} \n y_test.shape:  {y_test.shape}')
# check descriptive stats for Non-Numeric features; get KeyError: 'unique' if don't Transpose 
X_train.describe(exclude = 'number').T.sort_values(by = 'unique')<jupyter_output><empty_output><jupyter_text>**Calculate Baseline Accuracy**- for Classification problems, this is the Majority Class or most often occuring (= mode) class, aka Category/target/label.  From the barplot above, we see it is **'Passed' (0); occuring ~74% of the time**.<jupyter_code># get Baseline mode of target 'Fail'
y_train.value_counts(normalize = True)<jupyter_output><empty_output><jupyter_text>Convert dates to datetime format for accessibility/computation<jupyter_code>train['Inspection Date'] = pd.to_datetime(train['Inspection Date'])
train['Inspection Year'] = train['Inspection Date'].dt.year
train['Inspection Month'] = train['Inspection Date'].dt.month
train['Inspection Day'] = train['Inspection Date'].dt.day
test['Inspection Date'] = pd.to_datetime(test['Inspection Date'])
test['Inspection Year'] = test['Inspection Date'].dt.year
test['Inspection Month'] = test['Inspection Date'].dt.month
test['Inspection Day'] = test['Inspection Date'].dt.day
test<jupyter_output><empty_output><jupyter_text>Part 2: Modeling> **Fit a model** with the train set. (You may use scikit-learn, xgboost, or any other library.) Use cross-validation or do a three-way split (train/validate/test) and **estimate your ROC AUC** validation score.>> Use your model to **predict probabilities** for the test set. **Get an ROC AUC test score >= 0.60.**<jupyter_code>#PIPELINE 1 ("Fast 1st fit")

import category_encoders as ce
from sklearn.model_selection import train_test_split, cross_val_score, StratifiedKFold
from sklearn.pipeline import make_pipeline
from sklearn.impute import SimpleImputer
from sklearn.metrics import roc_auc_score
from sklearn.ensemble import RandomForestClassifier 

target = 'Fail'
features = ['Latitude', 'Longitude'] # perhaps a vendor's location is significantly predictive of Inspection Fail status

pipeline = make_pipeline(
    ce.OrdinalEncoder(),  # automatically imputes mode for string features/columns with Nulls(?)
    SimpleImputer(strategy='most_frequent'),
    RandomForestClassifier(n_estimators = 150,
                           class_weight = 'balanced',
                           max_depth = 50,
                           n_jobs = -1)
)

# Instantiate 
skf = StratifiedKFold(n_splits = 3) # set Cross-Val parameter

# (Instantiate? ie allocate memory?) Validation metric variables
acc_scores = []
roc_auc_scores = []

# Loop to split data into X features Matrices & y target vectors
for train_index, test_index in skf.split(X_train, y_train):
  X_train = train.drop(columns = target)
  X_train = train.loc[train_index, features]  #train_index? =? loc rows by index corresponding to indicated features/cols

  y_train = train.loc[train_index, target]    # Access a group of rows and columns by label(s) or a boolean array.  from https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.loc.html

  pipeline.fit(X_train, y_train)

  X_test = train.loc[test_index, features]    #test_index? (why not just test?) ^See above documentation
  y_test = train.loc[test_index, target]
  acc_scores.append(pipeline.score(X_test, y_test))

  roc_auc_scores.append(roc_auc_score(y_test, pipeline.predict(X_test)))

print("ROC AUC scores", roc_auc_scores)
print("Accuracy scores", acc_scores)
# PIPELINE 2: Same as Pipeline, EXCEPT HYPERPARAMETER TUNING to max_depth = 3, previously = 50; 
'''


# get ValueError: Found input variables with inconsistent numbers of samples: [18, 34611]
# Found input variables with inconsistent numbers of samplesdatascience.stackexchange.com › questions › train-test-s...
# Jul 6, 2017 - You are running into that error because your X and Y don't have the same length (which is what train_test_split requires), i.e., X.shape[0] !=


target = 'Fail'
features = ['Latitude', 'Longitude'] # perhaps a vendor's location is significantly predictive of Inspection Fail status

pipeline2 = make_pipeline(
    ce.OrdinalEncoder(),  # automatically imputes mode for string features/columns with Nulls(?)
    SimpleImputer(strategy='most_frequent'),
    RandomForestClassifier(max_depth=3)
)

# Instantiate 
skf = StratifiedKFold(n_splits = 3) # set Cross-Val parameter

# (Instantiate? ie allocate memory?) Validation metric variables
acc_scores = []
roc_auc_scores = []


for train_index, test_index in skf.split(X_train, y_train):
  X_train = train.drop(columns = target)
  X_train = train.loc[train_index, features]  #train_index? =? loc rows by index corresponding to indicated features/cols

  y_train = train.loc[train_index, target]

  pipeline.fit(X_train, y_train)

  X_test = train.loc[test_index, features]    #test_index? (why not just test?)
  y_test = train.loc[test_index, target]
  acc_scores.append(pipeline.score(X_test, y_test))

  roc_auc_scores.append(roc_auc_score(y_test, pipeline.predict(X_test)))

print("ROC AUC scores", roc_auc_scores)
print("Accuracy scores", acc_scores)
'''<jupyter_output><empty_output><jupyter_text>Alternative metrics and syntax for calculating and print Validation Scores:1) print('Validation Accuracy', pipeline.score(X_test, y_test)) 2) print('Mean cross val score', cross_val_score(pipeline, X_test, y_test).mean())3) print(cross_val_score(pipeline, X_train[features], y_train, cv = 5, scoring = 'roc_auc'))4) print('ROC AUC Score', roc_auc_score(y_test, pipeline.predict(X_test))) **For the following Pipeline 1:**pipeline = make_pipeline(    ce.OrdinalEncoder(),   automatically imputes mode for string features/columns with Nulls(?)    SimpleImputer(strategy='most_frequent'),    RandomForestClassifier(max_depth=3))**We get Output 1:**ROC AUC scores [0.5, 0.5, 0.5]Accuracy scores [0.7413613775569167, 0.7414042184339786, 0.7414042184339786]The low (< .60) ROC AUC scores indicate that together, 'Latitude' & 'Longitude' were not highly predictive features.I'll try hyperparameter tuning of the RandomForestClassifier model first. **For the following Pipeline 2:**pipeline = make_pipeline(    ce.OrdinalEncoder(),   automatically imputes mode for string features/columns with Nulls(?)    SimpleImputer(strategy='most_frequent'),    RandomForestClassifier(n_estimators = 100,                           class_weight = 'balanced',                           max_depth = 3,                           n_jobs = -1)**We get Output 2:**ROC AUC scores [0.5367240191971273, 0.5337981491529398, 0.5312787226061809]Accuracy scores [0.4839212966975817, 0.42992112334229005, 0.4578313253012048]Slightly better, closer to but still < .60 ROC AUC scores.I'll try hyperparameter tuning of the RandomForestClassifier model once more. **For the following Pipeline 3:**pipeline = make_pipeline(    ce.OrdinalEncoder(),   automatically imputes mode for string features/columns with Nulls(?)    SimpleImputer(strategy='most_frequent'),    RandomForestClassifier(n_estimators = 150,                           class_weight = 'balanced',                           max_depth = 50,                           n_jobs = -1))**We get Output 3:**ROC AUC scores [0.5160644506291542, 0.5186304400615169, 0.5232352021330752]Accuracy scores [0.6322152886115444, 0.6204654791314523, 0.6339877779222468]Slightly WORSE now, further from and still < .60 ROC AUC scores.I'll try (a) different feature(s). Consider 'Violations' again, for its weight or significance as an important (predictive) feature correlated with Inspection Failure. As previously noted: "The more/more serious violations a store has, the more likely it seems to reason- it would or could 'fail' an inspection."<jupyter_code>X_train
'''
 Reset X_train in order to then change features to include features/columns from X_train that we want to examine
 for its/their predictive relevance
 '''
X_train = train.drop(columns = target)
X_train
# Peruse metadata for 'Violations'
'''
The following notes seem to indicate OPEN/ONGOING inspection.. If so, this could be a source for "time travel" data leakage from Test dataset to Train (?)
"MUST PROVIDE" ... "REPAIR SAME"... "NO FOOD HANDLER TRAINING IN ILLINOIS CERTIFICATES AT THIS TIME OF INSPECTION..."
'''
X_train['Violations'].value_counts()
# PIPELINE 4: Drop 'Violations' and refit and re-score model: get SyntaxError: positional argument follows keyword argument for X_train = train.drop(columns = target, 'Violations')
X_train = train.columns.drop(['Fail', 'Violations'])
X_train
# PIPELINE 4: ValueError: Must train encoder before it can be used to transform data.
'''
target = 'Fail'
features = ['Latitude', 'Longitude'] # perhaps a vendor's location is significantly predictive of Inspection Fail status WITH 'Violations' REMOVED?

pipeline = make_pipeline(
    ce.OrdinalEncoder(),  # automatically imputes mode for string features/columns with Nulls(?)
    SimpleImputer(strategy='most_frequent'),
    RandomForestClassifier(n_estimators = 150,
                           class_weight = 'balanced',
                           max_depth = 50,
                           n_jobs = -1)
)

X_train = train.loc[train_index, features]  #train_index? =? loc rows by index corresponding to indicated features/cols

y_train = train.loc[train_index, target]

pipeline4.fit(X_train, y_train)

X_test = train.loc[test_index, features]
y_test = train.loc[test_index, target]

acc_scores.append(pipeline.score(X_test, y_test))
roc_auc_scores.append(roc_auc_score(y_test, pipeline.predict(X_test)))

print("ROC AUC scores", roc_auc_scores)
print("Accuracy scores", acc_scores)
'''
#PIPELINE 4: MODIFIED by Dropping 'Violations' (else SAME as "Fast 1st fit" Model)

import category_encoders as ce
from sklearn.model_selection import train_test_split, cross_val_score, StratifiedKFold
from sklearn.pipeline import make_pipeline
from sklearn.impute import SimpleImputer
from sklearn.metrics import roc_auc_score
from sklearn.ensemble import RandomForestClassifier 

# Drop 'Violations'
X_train = train.columns.drop(['Fail', 'Violations'])
X_train

#PIPELINE 4: MODIFIED by Dropping 'Violations' (else SAME as "Fast 1st fit" Model)

# get ValueError: Found input variables with inconsistent numbers of samples: [17, 34611]

target = 'Fail'
features = ['Latitude', 'Longitude'] # perhaps a vendor's location is significantly predictive of Inspection Fail status

pipeline = make_pipeline(
    ce.OrdinalEncoder(),  # automatically imputes mode for string features/columns with Nulls(?)
    SimpleImputer(strategy='most_frequent'),
    RandomForestClassifier(n_estimators = 150,
                           class_weight = 'balanced',
                           max_depth = 50,
                           n_jobs = -1)
)

# Instantiate 
skf = StratifiedKFold(n_splits = 3) # set Cross-Val parameter

# (Instantiate? ie allocate memory?) Validation metric variables
acc_scores = []
roc_auc_scores = []

# Loop to split data into X features Matrices & y target vectors
for train_index, test_index in skf.split(X_train, y_train):
  X_train = train.drop(columns = target)
  X_train = train.loc[train_index, features]  #train_index? =? loc rows by index corresponding to indicated features/cols

  y_train = train.loc[train_index, target]    # Access a group of rows and columns by label(s) or a boolean array.  from https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.loc.html

  pipeline.fit(X_train, y_train)

  X_test = train.loc[test_index, features]    #test_index? (why not just test?) ^See above documentation
  y_test = train.loc[test_index, target]
  acc_scores.append(pipeline.score(X_test, y_test))

  roc_auc_scores.append(roc_auc_score(y_test, pipeline.predict(X_test)))

print("ROC AUC scores", roc_auc_scores)
print("Accuracy scores", acc_scores)<jupyter_output><empty_output><jupyter_text>Vendor Location EVEN WITH 'Violations' dropped, is still too low: See commented out previous cell: ROC AUC scores [0.5160644506291542, 0.5186304400615169, 0.5232352021330752, 0.5225460073412261, 0.5236790367305396]Accuracy scores [0.6322152886115444, 0.6204654791314523, 0.6339877779222468, 0.633207645299701, 0.633207645299701]  Remove 'Violations' from X_train, and pick different features to refit model and score model performance with<jupyter_code>features4 = ['Inspection Type', 'Facility Type', 'Location'] # perhaps a vendor's location is significantly predictive of Inspection Fail status WITH 'Violations' REMOVED?

pipeline4 = make_pipeline(
    ce.OrdinalEncoder(),  # automatically imputes mode for string features/columns with Nulls(?)
    SimpleImputer(strategy='most_frequent'),
    RandomForestClassifier(n_estimators = 150,
                           class_weight = 'balanced',
                           max_depth = 50,
                           n_jobs = -1)
)

X_train = train.loc[train_index, features4]  #train_index? =? loc rows by index corresponding to indicated features/cols

y_train = train.loc[train_index, target]

pipeline4.fit(X_train, y_train)

X_test = train.loc[test_index, features4]
y_test = train.loc[test_index, target]
acc_scores.append(pipeline4.score(X_test, y_test))

roc_auc_scores.append(roc_auc_score(y_test, pipeline4.predict(X_test)))

print("ROC AUC scores", roc_auc_scores)
print("Accuracy scores", acc_scores)<jupyter_output>ROC AUC scores [0.5673645044566462]
Accuracy scores [0.6559375902918232]<jupyter_text>With features ='Inspection Type' and 'Violations' having been removed, get identical scores as with 'Latitude' & 'Longitude', SO SOMETHING IS REMISS...! TRY features = ['Inspection Type', 'Facility Type'].. still get same results, TRY XGBOOST  Part 3: Visualization> Make visualizations for model interpretation. (You may use any libraries.) Choose two of these types:>> - Permutation Importances> - Partial Dependence Plot, 1 feature isolation> - Partial Dependence Plot, 2 features interaction> - Shapley Values<jupyter_code># reset X_Train
X_train = train.columns.drop(['Fail'])

# Get feature importances
rf = pipeline.named_steps['randomforestclassifier']
importances = pd.Series(rf.feature_importances_, X_train.columns)

# Plot feature importances
%matplotlib inline
import matplotlib.pyplot as plt

n = 20
plt.figure(figsize=(10,n/2))
plt.title(f'Top {n} features')
importances.sort_values()[-n:].plot.barh(color='green');<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09726103968697597, 'fraction_numerical': 0.03399562985924082, 'mean_word_length': 5.049800184445128, 'pattern_counts': {'":': 0, '<': 63, '<?xml version=': 0, '>': 74, 'https://': 6, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '59635', 'license': 'MIT', 'n_tokens_mistral': 6604, 'n_tokens_neox': 5985, 'n_words': 1901, 'path': 'pt5_Sun_3_3_740pm_DS_Sprint_Challenge_7_ipynb.ipynb', 'repo_name': 'LambdaTheda/DS-Unit-2-Applied-Modeling', 'size': '19679'}