<jupyter_start><jupyter_text>Data Dive Week 10: Fitting and Evaluating Classification ModelsToday we revisit our Kiva data to review model performance and tradeoffs between different classification methods. We'll use logistic regression, decision trees, random forests, and gradient boosting machines to predict whether loan requests posted on [Kiva.org](kiva.org) recevie funding. We'll be using `scikit-learn` for all of the models in today's exercise. *** ![alt text](https://designtoimprovelife.dk/wp-content/uploads/2009/08/KIVA.jpg)Kiva is an international nonprofit, founded in 2005 and based in San Francisco, with a mission to connect people through lending to alleviate poverty. Kiva seeks to celebrate and support people looking to create a better future for themselves, their families and their communities.*By lending as little as $25 on Kiva, anyone can help a borrower start or grow a business, go to school, access clean energy or realize their potential. For some, it’s a matter of survival, for others it’s the fuel for a life-long ambition.* Today's Modeling ObjectiveOur focus today will be determining whether microfinance projects on the site [Kiva.org](kiva.org) receive funding or not using a host of features made available by Kiva, along with some features we'll design ourselves. Today's data is a subsample of projects in Kenya, one of Kiva's most active countries for lending. We'll be working with 18,000 observations, 12,000 of which were funded. Documentation on the data is available on [Kiva's website](http://build.kiva.org/docs/data/basic_types).<jupyter_code>%matplotlib inline

import pandas as pd
import numpy as np
import warnings
import matplotlib.pyplot as plt
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.metrics import accuracy_score, recall_score, precision_score, confusion_matrix
from sklearn.metrics import roc_curve, roc_auc_score
from sklearn.model_selection import KFold, train_test_split

warnings.filterwarnings("ignore")
random_state = 20181112
df = pd.read_csv('https://grantmlong.com/data/kiva_kenya_sample.csv')
print(df.shape)
print(list(df))
df.head()<jupyter_output>(18000, 25)
['LOAN_ID', 'DESCRIPTION', 'FUNDED_AMOUNT', 'LOAN_AMOUNT', 'STATUS', 'ACTIVITY_NAME', 'SECTOR_NAME', 'LOAN_USE', 'TOWN_NAME', 'CURRENCY', 'PARTNER_ID', 'POSTED_TIME', 'PLANNED_EXPIRATION_TIME', 'DISBURSE_TIME', 'RAISED_TIME', 'LENDER_TERM', 'NUM_LENDERS_TOTAL', 'NUM_JOURNAL_ENTRIES', 'NUM_BULK_ENTRIES', 'TAGS', 'BORROWER_NAMES', 'BORROWER_GENDERS', 'BORROWER_PICTURED', 'REPAYMENT_INTERVAL', 'DISTRIBUTION_MODEL']<jupyter_text>Tidying Up Our DataThere are a couple of different things we'll want to off the bat:  1. Create a target variable 2. Generate a more usable version of the `POSTED_TIME` column. 3. Generate a variable with the amount of planned time before expiration for each project.  4. Generate boolean variables for each of the following categorical columns:   * `SECTOR_NAME`   * `ACTIVITY_NAME`   * `REPAYMENT_INTERVAL` 5. Create boolean variables for a handful of other free text and categorical columns, including:   * `BORROWER_PICTURED`   * `BORROWER_GENDERS`   * `DISTRIBUTION_MODEL`<jupyter_code>df.STATUS.value_counts()
df['success'] = (df.STATUS=='funded')*1

df.POSTED_TIME.head()
df['posted_year'] = pd.to_datetime(df.POSTED_TIME).dt.year
df['posted_duration'] = (pd.to_datetime(df.PLANNED_EXPIRATION_TIME)
                             - pd.to_datetime(df.POSTED_TIME)
                            ).dt.days
                            
sector_names = ['sector_' + name.lower() for name in df.SECTOR_NAME.unique()]
for name in df.SECTOR_NAME.unique():
    df['sector_' + name.lower()] = (df.SECTOR_NAME==name)*1

activity_names = ['activity_' + name.lower() for name in df.ACTIVITY_NAME.unique()]
for name in df.ACTIVITY_NAME.unique():
    df['activity_' + name.lower()] = (df.ACTIVITY_NAME==name)*1

repayment_types = ['repayment_' + interval.lower() for interval in df.REPAYMENT_INTERVAL.unique()]
for interval in df.REPAYMENT_INTERVAL.unique():
    df['repayment_' + interval.lower()] = (df.REPAYMENT_INTERVAL==interval)*1

other_names = ['has_borrower_pic', 'has_female_borrower', 'direct_distribution', 'currency_usd']

df['has_borrower_pic'] = df.BORROWER_PICTURED.str.contains('true')*1
df['has_borrower_pic'] = df['has_borrower_pic'].fillna(0)

df['has_female_borrower'] = df.BORROWER_GENDERS.str.contains('female')*1
df['has_female_borrower'] = df['has_female_borrower'].fillna(0)

df['direct_distribution'] = (df.DISTRIBUTION_MODEL=='direct')*1

df['currency_usd'] = (df.CURRENCY=='USD')*1<jupyter_output><empty_output><jupyter_text>Data Handling<jupyter_code>features = (['posted_year', 'posted_duration', 'LOAN_AMOUNT', 'LENDER_TERM', 'NUM_JOURNAL_ENTRIES'] 
            + repayment_types + sector_names + activity_names 
            + other_names
           )

model_df = df[(features + ['success'])].dropna().reset_index()

train_df, holdout_df, y_train, y_holdout = train_test_split(
    model_df[features], 
    model_df['success'], test_size=0.1,
    random_state=random_state)



train_df['success'] = y_train
holdout_df['success'] = y_holdout

train_df.reset_index(inplace=True)
holdout_df.reset_index(inplace=True)

print(train_df.shape[0], train_df.success.mean())
print(holdout_df.shape[0], holdout_df.success.mean())<jupyter_output>14297 0.621668881583549
1589 0.6280679672750157<jupyter_text>Model TrainingLet's take a quick look at all of our classification model options using cross validation. For the tree based models, we'll use the hyperparameter `max_depth=6` as a naive attempt at voiding overfitting before we dig deeper.  Let's fit and score the model, this time using cross validation:<jupyter_code>k_fold = KFold(n_splits=5, random_state=random_state)
def get_cv_results(classifier):
    
    results = []
    for train, test in k_fold.split(train_df):
        classifier.fit(train_df.loc[train, features], train_df.loc[train, 'success'])
        y_predicted = classifier.predict(train_df.loc[test, features])
        accuracy = accuracy_score(train_df.loc[test, 'success'], y_predicted)
        results.append(accuracy)
    
    return np.mean(results), np.std(results)<jupyter_output><empty_output><jupyter_text>First, let's look at the cv performance of Logistic Regression<jupyter_code>logreg = LogisticRegression(
    random_state=random_state, 
    solver='lbfgs'
)

get_cv_results(logreg)<jupyter_output><empty_output><jupyter_text>Next, let's take a naive attempt to beat Logistic Regression using a Tree<jupyter_code>dtree = DecisionTreeClassifier(
    random_state=random_state, 
    max_depth=6
)

get_cv_results(dtree)<jupyter_output><empty_output><jupyter_text>Looks like a non-linear method gives us a big boost. Let's try a Random Forest.<jupyter_code>rforest = RandomForestClassifier(
    random_state=random_state, 
    max_depth=6,
    n_estimators=100
)

get_cv_results(rforest)<jupyter_output><empty_output><jupyter_text>Finally, let's take a naive approach to Gradient Boosting.<jupyter_code>gbm = GradientBoostingClassifier(
    random_state=random_state, 
    max_depth=6,
    n_estimators=100
)

get_cv_results(gbm)<jupyter_output><empty_output><jupyter_text>Learning Curves, RevisitedNow that we're working with a more complex data set, we should be able to build more robust learning curves than we had experience with when we were using the<jupyter_code>hp_values = range(1,50, 2)
all_mu = []
all_sigma = []

for m in hp_values:

    dtree=DecisionTreeClassifier(
        criterion='entropy', 
        random_state=random_state, 
        max_depth=m,
    )

    mu, sigma = get_cv_results(dtree)
    all_mu.append(mu)
    all_sigma.append(sigma)
    
    print(m, mu, sigma)
plt.figure(figsize=(14, 5))
plt.plot(hp_values, all_mu)
plt.ylabel('Cross Validation Accuracy')
plt.xlabel('Max Depth')
plt.figure(figsize=(14, 5))
plt.plot(hp_values, all_sigma)
plt.ylabel('Cross Validation Std Dev.')
plt.xlabel('Max Depth')<jupyter_output><empty_output><jupyter_text>Evaluating Model PerformanceWe can use ROC curves to look at how our models perform across a variety of threshholds against our holdout data.<jupyter_code>def plot_roc(classifier, label, color):

    classifier.fit(train_df[features], train_df['success'])
    y_prob = classifier.predict_proba(holdout_df[features])
    
    fpr, tpr, thresh = roc_curve(holdout_df['success'], y_prob[:,1])
    plt.plot(fpr, tpr,
             label=label,
             color=color, linewidth=3)

    auc = roc_auc_score(holdout_df['success'], y_prob[:,1])
    
    print('AUC: %0.3f (%s)' % (auc, label))
 
f1 = plt.figure(figsize=(14,6))

logreg = LogisticRegression(
    random_state=random_state, 
    solver='lbfgs'
)
plot_roc(logreg, 'Logistic Regression', 'green')

dtree = DecisionTreeClassifier(
    random_state=random_state, 
    max_depth=6
)
plot_roc(dtree, 'Decision Tree', 'red')

rforest = RandomForestClassifier(
    random_state=random_state, 
    max_depth=6,
    n_estimators=100
)
plot_roc(rforest, 'Random Forest', 'blue')

gbm = GradientBoostingClassifier(
    random_state=random_state, 
    max_depth=6,
    n_estimators=100
)
plot_roc(gbm, 'GBM', 'lightblue')

plt.legend(loc='lower right')<jupyter_output>AUC: 0.871 (Logistic Regression)
AUC: 0.921 (Decision Tree)
AUC: 0.923 (Random Forest)
AUC: 0.953 (GBM)
{'fraction_non_alphanumeric': 0.10799661518933784, 'fraction_numerical': 0.01639517664480643, 'mean_word_length': 5.462747778537252, 'pattern_counts': {'":': 0, '<': 38, '<?xml version=': 0, '>': 38, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '81682', 'license': 'MIT', 'n_tokens_mistral': 3365, 'n_tokens_neox': 3156, 'n_words': 778, 'path': 'lecture-10/DataDive-Lecture10.ipynb', 'repo_name': 'grantmlong/itds2018', 'size': '9454'}