<jupyter_start><jupyter_text>Website    Ghani, Rayid, <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, and <NAME>.  Inference---  Table of Contents- [Introduction](Introduction)    - [Learning Objectives](Learning-Objectives)    - [Methods](Methods)- [Python Setup](Python-Setup)- [Load the Data](Load-the-Data)  IntroductionDuring the lecture today we discussed different factors that can affect inference.  As a result of this notebook you will learn how a "grouping" variable can impact the fit of a model.  We'll also be reviewing some of the ML code you saw during the last session.The dataset used in this notebook is ada_class3.for_inference_example and was developed for this purpose.  It is a subset of household benefit spells reported as ending within 2013.  There are over 200k rows in the dataset.Our goal with this analysis is to predict individuals returning to benefits within 1 year after the end of a benefit spell.  Variables that will be used in this analysis are listed below.  The majority of the features will be what we will consider "individual variables," and we'll consider the *district* variable a group variable.  We will first run models with the individual features, and then see if the inclusion of the group variable changes our model output/prediction.We will first look at the more familiar Logistic Regression, then at ML models we learned in the last class session.  Inference Example  Important Variables  Python Setup- Back to [Table of Contents](Table-of-Contents)<jupyter_code>%pylab inline
import pandas as pd
import numpy as np
import psycopg2
import psycopg2.extras
import sklearn
import seaborn as sns
from sklearn.cross_validation import train_test_split
from sklearn.ensemble import (RandomForestClassifier, ExtraTreesClassifier,
                              GradientBoostingClassifier,
                              AdaBoostClassifier)
from sklearn.linear_model import LogisticRegression, SGDClassifier
from sklearn.metrics import precision_recall_curve, auc
from sklearn.metrics import accuracy_score, precision_score, recall_score
from sklearn.naive_bayes import GaussianNB
from sklearn.tree import DecisionTreeClassifier
import sqlalchemy
sns.set_style("white")<jupyter_output><empty_output><jupyter_text>Let's also load in some functions that we saw in the Machine Learning Class.<jupyter_code>def plot_precision_recall_n(y_true, y_prob, model_name):
    """
    y_true: ls 
        ls of ground truth labels
    y_prob: ls
        ls of predic proba from model
    model_name: str
        str of model name (e.g, LR_123)
    """
    from sklearn.metrics import precision_recall_curve
    y_score = y_prob
    precision_curve, recall_curve, pr_thresholds = precision_recall_curve(y_true, y_score)
    precision_curve = precision_curve[:-1]
    recall_curve = recall_curve[:-1]
    pct_above_per_thresh = []
    number_scored = len(y_score)
    for value in pr_thresholds:
        num_above_thresh = len(y_score[y_score>=value])
        pct_above_thresh = num_above_thresh / float(number_scored)
        pct_above_per_thresh.append(pct_above_thresh)
    pct_above_per_thresh = np.array(pct_above_per_thresh)
    plt.clf()
    fig, ax1 = plt.subplots()
    ax1.plot(pct_above_per_thresh, precision_curve, 'b')
    ax1.set_xlabel('percent of population')
    ax1.set_ylabel('precision', color='b')
    ax1.set_ylim(0,1.05)
    ax2 = ax1.twinx()
    ax2.plot(pct_above_per_thresh, recall_curve, 'r')
    ax2.set_ylabel('recall', color='r')
    ax2.set_ylim(0,1.05)
    
    name = model_name
    plt.title(name)
    plt.show()
    plt.clf()
def precision_at_k(y_true, y_scores,k):
    
    threshold = np.sort(y_scores)[::-1][int(k*len(y_scores))]
    y_pred = np.asarray([1 if i >= threshold else 0 for i in y_scores ])
    return precision_score(y_true, y_pred)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07821373257614868, 'fraction_numerical': 0.009550851832731027, 'mean_word_length': 4.243572395128552, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 22, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '519153', 'license': 'CC0-1.0', 'n_tokens_mistral': 1254, 'n_tokens_neox': 1178, 'n_words': 388, 'path': 'additional_content/8_2_Inference.ipynb', 'repo_name': 'Coleridge-Initiative/ada-2018-uchicago', 'size': '3967'}