<jupyter_start><jupyter_text>Data Dive Week 6/7: Logistic Regression This week we take a look at *logistic regression*, the first classification model we'll be covering in class. We'll be using `scikit-learn` in today's exercise. ***As we discussed last week, logisitic regression is a *classification model* - meaning that it is designed to idenfity the likelihood that a given observed data point belongs to set class, or category. Today we'll be looking at a real world application of logistic regression using data from loans requests posted on [Kiva.org](kiva.org). *** ![alt text](https://www-kiva-org.global.ssl.fastly.net/cms/sites/default/files/kivablog/preview_logo_1.jpg)Kiva is an international nonprofit, founded in 2005 and based in San Francisco, with a mission to connect people through lending to alleviate poverty. Kiva seeks to celebrate and support people looking to create a better future for themselves, their families and their communities.*By lending as little as $25 on Kiva, anyone can help a borrower start or grow a business, go to school, access clean energy or realize their potential. For some, it’s a matter of survival, for others it’s the fuel for a life-long ambition.* Today's Modeling ObjectiveOur focus today will be determining whether microfinance projects on the site [Kiva.org](kiva.org) receive funding or not using a host of features made available by Kiva, along with some features we'll design ourselves. Today's data is a subsample of projects in Kenya, one of Kiva's most active countries for lending. We'll be working with 18,000 observation, 3,000 or which were funded. Documentation on the data is available on [Kiva's website](http://build.kiva.org/docs/data/basic_types).<jupyter_code>%matplotlib inline

import pandas as pd
import numpy as np
import statsmodels.api as sm
from sklearn.linear_model import LogisticRegression

df = pd.read_csv('https://grantmlong.com/data/kiva_kenya_sample.csv')
print(df.shape)
print(list(df))
df.head()<jupyter_output><empty_output><jupyter_text>Tidying Up Our DataThere are a couple of different things we'll want to off the bat:  1. Create a target variable 2. Generate a more usable version of the `POSTED_TIME` column. 3. Generate a variable with the amount of planned time before expiration for each project.      Based on a quick look at the data, what else might be interesting to try off the bat?<jupyter_code>df.STATUS.value_counts()
df['success'] = (df.STATUS=='funded')*1

df.POSTED_TIME.head()
df['posted_year'] = pd.to_datetime(df.POSTED_TIME).dt.year
df['posted_duration'] = (pd.to_datetime(df.PLANNED_EXPIRATION_TIME)
                             - pd.to_datetime(df.POSTED_TIME)
                            ).dt.days<jupyter_output><empty_output><jupyter_text>Data ExplorationWhich columns might be helpful in predicting successful funding? How do we know?<jupyter_code>var = 'ACTIVITY_NAME'
(df[[var, 'success']]
 .groupby(var)
 .agg(['count', 'mean'])
 .sort_values(by=('success', 'count'), ascending=False)
 .head(10)
)
df.loc[(df.success==1) & (df.LOAN_AMOUNT<=2000), 'LOAN_AMOUNT'].hist(bins=20, alpha=.6, color='blue')
df.loc[(df.success==0) & (df.LOAN_AMOUNT<=2000), 'LOAN_AMOUNT'].hist(bins=20, alpha=.6, color='red')
df.loc[df.posted_year>2010,['posted_year', 'posted_duration']].groupby('posted_year').median().plot.bar()
df.LENDER_TERM.hist()<jupyter_output><empty_output><jupyter_text>Training A Model in `statsmodels`As we saw the past two weeks, `statsmodels` can be helpful if we want to visualize the summary statistics of our output.  First, let's identify the columns we're interested in and extract the rows with valid results. We'll also want to add a constant column.<jupyter_code>df['constant'] = 1
model_columns = ['constant', 'LOAN_AMOUNT', 'posted_year', 'posted_duration', 'LENDER_TERM']
valids = df[model_columns].notna().all(axis=1)<jupyter_output><empty_output><jupyter_text>Next, let's use `statsmodels` to fit the model and print the result.<jupyter_code>logit = sm.Logit(df.loc[valids, 'success'], df.loc[valids, model_columns])
result = logit.fit()
print(result.summary())<jupyter_output><empty_output><jupyter_text>Training A Model in `sklearn`As we saw the past two weeks, `statsmodels` can be helpful if we want to visualize the summary statistics of our output. Documentation for the `LogisticRegression` object can be found [here](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html)<jupyter_code>model_columns = ['LOAN_AMOUNT', 'posted_year', 'posted_duration', 'LENDER_TERM']
valids = df[model_columns].notna().all(axis=1)
X = df.loc[valids, model_columns].values
y = df.loc[valids, 'success'].values<jupyter_output><empty_output><jupyter_text>Let's fit and score the model:<jupyter_code>clf = LogisticRegression(random_state=20181022).fit(X, y)
print('The accuracy of our model is %0.1f%%' % (clf.score(X, y)*100))<jupyter_output><empty_output><jupyter_text>Let's see where we went right and where we went wrong:<jupyter_code>df.loc[valids, 'likelihood'] = clf.predict_proba(X)[:,1]<jupyter_output><empty_output><jupyter_text>Let's take a look at the top values where we predicted incorrectly predicted a failure to fund:<jupyter_code>df.loc[(valids & (df.success==1)), ].sort_values(by='likelihood').head()<jupyter_output><empty_output><jupyter_text>Now let's take a look at where we predicted funding, but the projects were not funded.<jupyter_code>df.loc[(valids & (df.success==0)), ].sort_values(by='likelihood').tail()<jupyter_output><empty_output><jupyter_text>Finally, let's take a closer look at our summary metrics. How do they change as we change our cutoff value for our prediction?<jupyter_code>def calculate_metrics(df, threshold):
    
    df['predicted'] = (clf.predict_proba(X)[:,1]>=threshold)*1
    accuracy = sum(df['predicted']==y)/len(y)
    precision = df.loc[df.predicted==1, 'success'].mean()
    recall = df.loc[df.success==1, 'predicted'].mean()
    
    return accuracy, precision, recall, sum(df['predicted'])

for p in [.4, .5, .6, .9]:
    print(p, calculate_metrics(df.loc[valids,].copy(), p))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11871030776746458, 'fraction_numerical': 0.011561634912880639, 'mean_word_length': 5.862569832402235, 'pattern_counts': {'":': 0, '<': 47, '<?xml version=': 0, '>': 47, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '453055', 'license': 'MIT', 'n_tokens_mistral': 2026, 'n_tokens_neox': 1918, 'n_words': 606, 'path': 'lecture-6/DataDive-Lecture6.ipynb', 'repo_name': 'grantmlong/itds2018', 'size': '6141'}