<jupyter_start><jupyter_text>Modeling Some Questions to Ask- Which model do we choose given the nature of our data?- How do we set the parameters?- How well can we predict the disease?- Which features are important for prediction purposes?- What is going on in the model?<jupyter_code>import eli5
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
from sklearn.tree import DecisionTreeClassifier, export_graphviz
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split, validation_curve, GridSearchCV

import util

%matplotlib inline
df = pd.read_csv(util.DATA_PATH_PREPROCESSED)
df.head()<jupyter_output><empty_output><jupyter_text>First Pass  Split Data into Train and Test<jupyter_code>X = df[util.FEATURES].values
y = df[util.TARGET].values
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)<jupyter_output><empty_output><jupyter_text>Train Model<jupyter_code>clf = DecisionTreeClassifier()
clf.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>Assess Result  Prediction PerformanceHow well can we predict the outcome?<jupyter_code>yp_test = clf.predict(X_test)
util.confusion_matrix_as_df(y_test, yp_test, labels=util.TARGET_VALUES)<jupyter_output><empty_output><jupyter_text>Feature ImportanceWhich features are important?<jupyter_code>feature_importances = pd.DataFrame({'DecisionTree': clf.feature_importances_}, 
                                   index=pd.Series(util.FEATURES, name='feature'))
(feature_importances.sort_values('DecisionTree', ascending=False)
                    .style.bar().format('{:.2f}'))<jupyter_output><empty_output><jupyter_text>Assess Model: Compare Train Test ScoresHow well do training and test scores match? Are we over- or underfitting?<jupyter_code>clf.score(X_train, y_train), clf.score(X_test, y_test)
eli5.show_weights(clf, feature_names=util.FEATURES, target_names=util.TARGET_VALUES, filled=True)<jupyter_output><empty_output><jupyter_text>Supplements- Explore Impact of Model Parameter Settings / Over- and Underfitting  Assess Model: Exemplify Effect of `min_samples_leaf` Parameter<jupyter_code>param_range = np.arange(1, 30, 2)
param_name = 'min_samples_leaf'

train_scores, valid_scores = validation_curve(
    DecisionTreeClassifier(), X, y, 
    param_name, param_range,
    cv=5,
)

fig, ax = plt.subplots()
ax.plot(param_range, train_scores.mean(axis=1), label='train')
ax.plot(param_range, valid_scores.mean(axis=1), label='CV test accuracy')
ax.set_xlim(left=param_range.min(), right=param_range.max())
ax.set_ylabel('accuracy')
ax.set_xlabel(param_name)
ax.legend()<jupyter_output><empty_output><jupyter_text>Perform Cross Validated Grid Search to Get Paremters Right<jupyter_code># this is a small search for demo purposes only
# consider performing stratified splits
param_grid = {'max_depth': [None, 5, 2, 1], 
              'min_samples_leaf': [1, 5, 10, 15]}

grid_search_cv = GridSearchCV(DecisionTreeClassifier(random_state=42), 
                              param_grid=param_grid, 
                              return_train_score=True, 
                              cv=5, refit=True)
grid_search_cv.fit(X, y)
util.cv_results_to_df(grid_search_cv.cv_results_).style.highlight_max(subset=['mean_test_score'], axis=0)
eli5.show_weights(grid_search_cv.best_estimator_, feature_names=util.FEATURES, target_names=util.TARGET_VALUES, filled=True)<jupyter_output><empty_output><jupyter_text>Second PassCan we do better? Ask More Experts aka Use a RandomForestWhy not ask several experts and aggregate their opinion?<jupyter_code>clf_rf = RandomForestClassifier(n_estimators=100, max_features=1, random_state=42)
clf_rf.fit(X_train, y_train)
yp_test_rf = clf_rf.predict(X_test)

display('Decision Tree')
display(util.confusion_matrix_as_df(y_test, yp_test))
display('Random Forest')
util.confusion_matrix_as_df(y_test, yp_test_rf)
clf_rf.score(X_test, y_test)
feature_importances['RandomForest'] = clf_rf.feature_importances_

(feature_importances.sort_values('DecisionTree', ascending=False)
                    .style.bar().format('{:.2f}'))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10119617224880383, 'fraction_numerical': 0.00861244019138756, 'mean_word_length': 6.183848797250859, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '372707', 'license': 'MIT', 'n_tokens_mistral': 1465, 'n_tokens_neox': 1413, 'n_words': 285, 'path': 'demo.ipynb', 'repo_name': 'caichinger/glt2019_ml_intro', 'size': '4180'}