<jupyter_start><jupyter_text>Lending Club releases data for all of the approved and declined loan applications periodically on their [website](https://www.lendingclub.com/info/download-data.action). You can select a few different year ranges to download the datasets (in CSV format) for both approved and declined loans.You'll also find a data dictionary (in XLS format) which contains information on the different column names towards the bottom of the page. We recommend downloading the data dictionary to so you can refer to it whenever you want to learn more about what a column represents in the datasets. Here's a link to the data dictionary file hosted on [Google Drive](https://docs.google.com/spreadsheets/d/191B2yJ4H1ZPXq0_ByhUgWMFZOYem5jFz0Y3by_7YBY4/edit).Before diving into the datasets themselves, let's get familiar with the data dictionary. The LoanStats sheet describes the approved loans datasets and the RejectStats describes the rejected loans datasets. Since rejected applications don't appear on the Lending Club marketplace and aren't available for investment, we'll be focusing on data on approved loans only.The approved loans datasets contain information on current loans, completed loans, and defaulted loans. Let's now define the problem statement for this machine learning project: - Can we build a machine learning model that can accurately predict if a borrower will pay off their loan on time or not?We'll focus on approved loans data from 2007 to 2011, since a good number of the loans have already finished. In the datasets for later years, many of the loans are current and still being paid off.We have reduced the size of the dataset to make it easier to work with, by: - removing the `desc` column:   - which contains a long text explanation for each loan - removing the `url` column:   - which contains a link to each loan on Lending Club which can only be accessed with an investor account - removing all columns containing more than 50% missing values:   - which allows us to move faster since we can spend less time trying to fill these valuesBefore we can start doing machine learning, we need to define what features we want to use and which column repesents the target column we want to predict. Let's start by reading in the dataset and exploring it.<jupyter_code>import pandas as pd
loans_2007 = pd.read_csv("loans_2007.csv")
loans_2007.drop_duplicates()
print(loans_2007.iloc[0])
print(loans_2007.shape[1])<jupyter_output>/Users/otto/opt/anaconda3/envs/datasci/lib/python3.7/site-packages/IPython/core/interactiveshell.py:3051: DtypeWarning: Columns (0) have mixed types. Specify dtype option on import or set low_memory=False.
  interactivity=interactivity, compiler=compiler, result=result)<jupyter_text>Clean the Data  First group of columnsThe Dataframe contains many columns and can be cumbersome to try to explore all at once. Let's break up the columns into 3 groups of 18 columns and use the data dictionary to become familiar with what each column represents. As you understand each feature, you want to pay attention to any features that: - leak information from the future (after the loan has already been funded) - don't affect a borrower's ability to pay back a loan (e.g. a randomly generated ID value by Lending Club) - formatted poorly and need to be cleaned up - require more data or a lot of processing to turn into a useful feature - contain redundant information After analyzing each column, we can conclude that the following features need to be removed: - `id`: randomly generated field by Lending Club for unique identification purposes only - `member_id`: also a randomly generated field by Lending Club for unique identification purposes only - `funded_amnt`: leaks data from the future (after the loan is already started to be funded) - `funded_amnt_inv`: also leaks data from the future (after the loan is already started to be funded) - `grade`: contains redundant information as the interest rate column (`int_rate)` - `sub_grade`: also contains redundant information as the interest rate column (`int_rate`) - `emp_title`: requires other data and a lot of processing to potentially be useful - `issue_d`: leaks data from the future (after the loan is already completed funded)Recall that Lending Club assigns a grade and a sub-grade based on the borrower's interest rate. While the grade and sub_grade values are categorical, the `int_rate` column contains continuous values, which are better suited for machine learning.Let's now drop these columns from the Dataframe before moving onto the next group of columns.<jupyter_code>loans_2007 = loans_2007.drop(["id", "member_id", "funded_amnt", "funded_amnt_inv", "grade", "sub_grade", "emp_title", "issue_d"], axis=1)<jupyter_output><empty_output><jupyter_text>2nd group of columnsWithin this group of columns, we need to drop the following columns: - `zip_code`: redundant with the `addr_state` column since only the first 3 digits of the 5 digit zip code are visible (which only can be used to identify the state the borrower lives in) - `out_prncp`: leaks data from the future, (after the loan already started to be paid off) - `out_prncp_inv`: also leaks data from the future, (after the loan already started to be paid off) - `total_pymnt`: also leaks data from the future, (after the loan already started to be paid off) - `total_pymnt_inv`: also leaks data from the future, (after the loan already started to be paid off) - `total_rec_prncp`: also leaks data from the future, (after the loan already started to be paid off)- The `out_prncp` and `out_prncp_inv` both describe the outstanding principal amount for a loan, which is the remaining amount the borrower still owes. These 2 columns as well as the `total_pymnt` column describe properties of the loan after it's fully funded and started to be paid off. This information isn't available to an investor before the loan is fully funded and we don't want to include it in our model.<jupyter_code>loans_2007 = loans_2007.drop(["zip_code", "out_prncp", "out_prncp_inv", "total_pymnt", "total_pymnt_inv", "total_rec_prncp"], axis=1)<jupyter_output><empty_output><jupyter_text>3rd group of columnsIn the last group of columns, we need to drop the following columns: - `total_rec_int`: leaks data from the future, (after the loan already started to be paid off), - `total_rec_late_fee`: also leaks data from the future, (after the loan already started to be paid off), - `recoveries`: also leaks data from the future, (after the loan already started to be paid off), - `collection_recovery_fee`: also leaks data from the future, (after the loan already started to be paid off), - `last_pymnt_d`: also leaks data from the future, (after the loan already started to be paid off), - `last_pymnt_amnt`: also leaks data from the future, (after the loan already started to be paid off).All of these columns leak data from the future, meaning that they're describing aspects of the loan after it's already been fully funded and started to be paid off by the borrower.<jupyter_code>loans_2007 = loans_2007.drop(["total_rec_int", "total_rec_late_fee", "recoveries", "collection_recovery_fee", "last_pymnt_d", "last_pymnt_amnt"], axis=1)
print(loans_2007.iloc[0])
print(loans_2007.shape[1])<jupyter_output>loan_amnt                            5000
term                            36 months
int_rate                           10.65%
installment                        162.87
emp_length                      10+ years
home_ownership                       RENT
annual_inc                          24000
verification_status              Verified
loan_status                    Fully Paid
pymnt_plan                              n
purpose                       credit_card
title                            Computer
addr_state                             AZ
dti                                 27.65
delinq_2yrs                             0
earliest_cr_line                 Jan-1985
inq_last_6mths                          1
open_acc                                3
pub_rec                                 0
revol_bal                           13648
revol_util                          83.7%
total_acc                               9
initial_list_status                     f
last_credit_pull_d               J[...]<jupyter_text>Target ColumnJust by becoming familiar with the columns in the dataset, we were able to reduce the number of columns from 52 to 32 columns. We now need to decide on a target column that we want to use for modeling.We should use the `loan_status` column, since it's the only column that directly describes if a loan was paid off on time, had delayed payments, or was defaulted on the borrower. Currently, this column contains text values and we need to convert it to a numerical one for training a model. Let's explore the different values in this column and come up with a strategy for converting the values in this column.<jupyter_code>print(loans_2007['loan_status'].value_counts())<jupyter_output>Fully Paid                                             33136
Charged Off                                             5634
Does not meet the credit policy. Status:Fully Paid      1988
Current                                                  961
Does not meet the credit policy. Status:Charged Off      761
Late (31-120 days)                                        24
In Grace Period                                           20
Late (16-30 days)                                          8
Default                                                    3
Name: loan_status, dtype: int64<jupyter_text>Binary classificationThere are 8 different possible values for the `loan_status` column. You can read about most of the different loan statuses on the Lending Clube [website](https://help.lendingclub.com/hc/en-us/articles/215488038-What-do-the-different-Note-statuses-mean-). The two values that start with "Does not meet the credit policy" aren't explained unfortunately. A quick Google search takes us to explanations from the lending community [here](https://forum.lendacademy.com/?topic=2427.msg20813msg20813).From the investor's perspective, we're interested in trying to predict which loans will be paid off on time and which ones won't be. Only the `Fully Paid` and `Charged Off` values describe the final outcome of the loan. The other values describe loans that are still on going and where the jury is still out on if the borrower will pay back the loan on time or not. While the `Default` status resembles the `Charged Off` status, in Lending Club's eyes, loans that are charged off have essentially no chance of being repaid while default ones have a small chance. You can read about the difference here.Since we're interested in being able to predict which of these 2 values a loan will fall under, we can treat the problem as a binary classification one. Let's remove all the loans that don't contain either `Fully Paid` and `Charged Off` as the loan's status and then transform the `Fully Paid` values to `1` for the positive case and the `Charged Off` values to `0` for the negative case. While there are a few different ways to transform all of the values in a column, we'll use the Dataframe method `replace`. We need to keep in mind is the class imbalance between the positive and negative cases. While there are 33,136 loans that have been fully paid off, there are only 5,634 that were charged off. This class imbalance is a common problem in binary classification and during training, the model ends up having a strong bias towards predicting the class with more observations in the training set and will rarely predict the class with less observations. The stronger the imbalance, the more biased the model becomes.<jupyter_code>loans_2007 = loans_2007[(loans_2007['loan_status'] == "Fully Paid") | (loans_2007['loan_status'] == "Charged Off")]

status_replace = {
    "loan_status" : {
        "Fully Paid": 1,
        "Charged Off": 0,
    }
}

loans_2007 = loans_2007.replace(status_replace)<jupyter_output><empty_output><jupyter_text>Removing Single Value ColumnsTo wrap up this mission, let's look for any columns that contain only one unique value and remove them. These columns won't be useful for the model since they don't add any information to each loan application. In addition, removing these columns will reduce the number of columns we'll need to explore further in the next mission.We'll need to compute the number of unique values in each column and drop the columns that contain only one unique value.<jupyter_code>orig_columns = loans_2007.columns
drop_columns = []
for col in orig_columns:
    col_series = loans_2007[col].dropna().unique()
    if len(col_series) == 1:
        drop_columns.append(col)
loans_2007 = loans_2007.drop(drop_columns, axis=1)
print(drop_columns)<jupyter_output>['pymnt_plan', 'initial_list_status', 'collections_12_mths_ex_med', 'policy_code', 'application_type', 'acc_now_delinq', 'chargeoff_within_12_mths', 'delinq_amnt', 'tax_liens']<jupyter_text>Preparing the FeaturesWe've exported the Dataframe from the end of the last mission to a CSV file named `filtered_loans_2007.csv` to differentiate the file with the loans_2007.csv we used in the last mission. In this mission, we'll prepare the data for machine learning by focusing on handling missing values, converting categorical columns to numeric columns, and removing any other extraneous columns we encounter throughout this process.This is because the mathematics underlying most machine learning models assumes that the data is numerical and contains no missing values. To reinforce this requirement, scikit-learn will return an error if you try to train a model using data that contain missing values or non-numeric values when working with models like linear regression and logistic regression.Let's start by computing the number of missing values and come up with a strategy for handling them. Then, we'll focus on the categorical columns.<jupyter_code>import pandas as pd
loans = pd.read_csv('filtered_loans_2007.csv')
null_counts = loans.isnull().sum()
print(null_counts[null_counts>0])
print(loans.pub_rec_bankruptcies.value_counts(normalize=True, dropna=False))<jupyter_output>0.0    0.939438
1.0    0.042456
NaN    0.017978
2.0    0.000129
Name: pub_rec_bankruptcies, dtype: float64<jupyter_text>Handling missing values We'll keep emp_length as it's often used in calculating risk on loans.<jupyter_code>loans = loans.drop("pub_rec_bankruptcies", axis=1)
loans = loans.dropna(axis=0)
print(loans.dtypes.value_counts())
## Text columns

object_columns_df = loans.select_dtypes(include=["object"])
print(object_columns_df.iloc[0])<jupyter_output>term                     36 months
int_rate                    10.65%
emp_length               10+ years
home_ownership                RENT
verification_status       Verified
purpose                credit_card
title                     Computer
addr_state                      AZ
earliest_cr_line          Jan-1985
revol_util                   83.7%
last_credit_pull_d        Jun-2016
Name: 0, dtype: object<jupyter_text>Converting Text ColumnsSome of the columns seem like they represent categorical values, but we should confirm by checking the number of unique values in those columns: - home_ownership: home ownership status, can only be 1 of 4 categorical values according to the data dictionary, - verification_status: indicates if income was verified by Lending Club, - emp_length: number of years the borrower was employed upon time of application, - term: number of payments on the loan, either 36 or 60, - addr_state: borrower's state of residence, - purpose: a category provided by the borrower for the loan request, - title: loan title provided the borrower,There are also some columns that represent numeric values, that need to be converted: - int_rate: interest rate of the loan in %, - revol_util: revolving line utilization rate or the amount of credit the borrower is using relative to all available credit, read more here.Based on the first row's values for purpose and title, it seems like these columns could reflect the same information. Let's explore the unique value counts separately to confirm if this is true.Lastly, some of the columns contain date values that would require a good amount of feature engineering for them to be potentially useful: - earliest_cr_line: The month the borrower's earliest reported credit line was opened, - last_credit_pull_d: The most recent month Lending Club pulled credit for this loan.Since these date features require some feature engineering for modeling purposes, let's remove these date columns from the Dataframe.  First five categorical columns<jupyter_code>cols = ['home_ownership', 'verification_status', 'emp_length', 'term', 'addr_state']
for c in cols:
    print(loans[c].value_counts())<jupyter_output>RENT        18112
MORTGAGE    16686
OWN          2778
OTHER          96
NONE            3
Name: home_ownership, dtype: int64
Not Verified       16281
Verified           11856
Source Verified     9538
Name: verification_status, dtype: int64
10+ years    8545
< 1 year     4513
2 years      4303
3 years      4022
4 years      3353
5 years      3202
1 year       3176
6 years      2177
7 years      1714
8 years      1442
9 years      1228
Name: emp_length, dtype: int64
 36 months    28234
 60 months     9441
Name: term, dtype: int64
CA    6776
NY    3614
FL    2704
TX    2613
NJ    1776
IL    1447
PA    1442
VA    1347
GA    1323
MA    1272
OH    1149
MD    1008
AZ     807
WA     788
CO     748
NC     729
CT     711
MI     678
MO     648
MN     581
NV     466
SC     454
WI     427
OR     422
AL     420
LA     420
KY     311
OK     285
KS     249
UT     249
AR     229
DC     209
RI     194
NM     180
WV     164
HI     162
NH     157
DE     110
MT      77
WY      76
AK      76
SD      60
VT  [...]<jupyter_text>The reason for the loanThe `home_ownership`, `verification_status`, `emp_length`, `term`, and `addr_state` columns all contain multiple discrete values. We should clean the `emp_length` column and treat it as a numerical one since the values have ordering (2 years of employment is less than 8 years).First, let's look at the unique value counts for the `purpose` and `title` columns to understand which column we want to keep.<jupyter_code>print(loans["title"].value_counts())
print(loans["purpose"].value_counts())<jupyter_output>Debt Consolidation                                                       2068
Debt Consolidation Loan                                                  1599
Personal Loan                                                             624
Consolidation                                                             488
debt consolidation                                                        466
                                                                         ... 
d.everett SOSHail Mary Time! Double-down & we can score the WIN-WIN!       1
GREAT BORROWER --DEBT CONSOLIDATION                                         1
Debt consolidation & Refinancing                                            1
2003 Mitsubishi Lancer                                                      1
wek refinance                                                               1
Name: title, Length: 18881, dtype: int64
debt_consolidation    17751
credit_card            4911
other                  3711
home_improvement [...]<jupyter_text>Categorical ColumnsThe home_ownership, verification_status, emp_length, and term columns each contain a few discrete categorical values. We should encode these columns as dummy variables and keep them.It seems like the purpose and title columns do contain overlapping information but we'll keep the purpose column since it contains a few discrete values. In addition, the title column has data quality issues since many of the values are repeated with slight modifications (e.g. Debt Consolidation and Debt Consolidation Loan and debt consolidation.We'll map the emp_length column to clean it.<jupyter_code>mapping_dict = {
    "emp_length": {
        "10+ years": 10,
        "9 years": 9,
        "8 years": 8,
        "7 years": 7,
        "6 years": 6,
        "5 years": 5,
        "4 years": 4,
        "3 years": 3,
        "2 years": 2,
        "1 year": 1,
        "< 1 year": 0,
        "n/a": 0
    }
}

loans = loans.drop(["last_credit_pull_d", "earliest_cr_line", "addr_state", "title"], axis=1)
loans["int_rate"] = loans["int_rate"].str.rstrip("%").astype("float")
loans["revol_util"] = loans["revol_util"].str.rstrip("%").astype("float")
loans = loans.replace(mapping_dict)<jupyter_output><empty_output><jupyter_text>Dummy VariablesLet's now encode the `home_ownership`, `verification_status`, `purpose`, and `term` columns as dummy variables so we can use them in our model. We first need to use the Pandas `get_dummies` method to return a new Dataframe containing a new column for each dummy variable.<jupyter_code>cat_columns = ["home_ownership", "verification_status", "purpose", "term"]
dummy_df = pd.get_dummies(loans[cat_columns])
loans = pd.concat([loans, dummy_df], axis=1)
loans = loans.drop(cat_columns, axis=1)<jupyter_output><empty_output><jupyter_text>Making PredictionsOur eventual goal is to generate features from the data, which can feed into a machine learning algorithm. The algorithm will make predictions about whether or not a loan will be paid off on time, which is contained in the `loan_status` column of the clean dataset.We noticed that there's a class imbalance in our target column. There are about 6 times as many loans that were paid off on time (positive case, label of 1) than those that weren't (negative case, label of 0). Imbalances can cause issues with many machine learning algorithms, where they appear to have high accuracy, but actually aren't learning from the training data. Because of its potential to cause issues, we need to keep the class imbalance in mind as we build machine learning models.After all of our data cleaning in the past two missions, we ended up with the csv file called `cleaned_loans_2007.csv`:<jupyter_code>import pandas as pd
loans = pd.read_csv("cleaned_loans_2007.csv")
print(loans.info())<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 37675 entries, 0 to 37674
Data columns (total 38 columns):
loan_amnt                              37675 non-null float64
int_rate                               37675 non-null float64
installment                            37675 non-null float64
emp_length                             37675 non-null int64
annual_inc                             37675 non-null float64
loan_status                            37675 non-null int64
dti                                    37675 non-null float64
delinq_2yrs                            37675 non-null float64
inq_last_6mths                         37675 non-null float64
open_acc                               37675 non-null float64
pub_rec                                37675 non-null float64
revol_bal                              37675 non-null float64
revol_util                             37675 non-null float64
total_acc                              37675 non-null float64
home_ownership_MORTGAGE    [...]<jupyter_text>Picking an error metricCan we build a machine learning model that can accurately predict if a borrower will pay off their loan on time or not?We established that this is a binary classification problem in the first mission of this course, and we converted the `loan_status` column to `0`s and `1`s as a result. Before diving in and selecting an algorithm to apply to the data, we should select an error metric.We've generated some predictions automatically, and they are stored in a NumPy array called `predictions`<jupyter_code>import numpy as np

# Predict that all loans will be paid off on time.
predictions = pd.Series(np.ones(loans.shape[0]))

# False positives.
fp_filter = (predictions == 1) & (loans["loan_status"] == 0)
fp = len(predictions[fp_filter])

# True positives.
tp_filter = (predictions == 1) & (loans["loan_status"] == 1)
tp = len(predictions[tp_filter])

# False negatives.
fn_filter = (predictions == 0) & (loans["loan_status"] == 1)
fn = len(predictions[fn_filter])

# True negatives
tn_filter = (predictions == 0) & (loans["loan_status"] == 0)
tn = len(predictions[tn_filter])<jupyter_output><empty_output><jupyter_text>Class imbalanceoptimize for: - high recall (true positive rate) - low fall-out (false positive rate)<jupyter_code>import pandas as pd
import numpy

# Predict that all loans will be paid off on time.
predictions = pd.Series(numpy.ones(loans.shape[0]))

# False positives.
fp_filter = (predictions == 1) & (loans["loan_status"] == 0)
fp = len(predictions[fp_filter])

# True positives.
tp_filter = (predictions == 1) & (loans["loan_status"] == 1)
tp = len(predictions[tp_filter])

# False negatives.
fn_filter = (predictions == 0) & (loans["loan_status"] == 1)
fn = len(predictions[fn_filter])

# True negatives
tn_filter = (predictions == 0) & (loans["loan_status"] == 0)
tn = len(predictions[tn_filter])

# Rates
tpr = tp / (tp + fn)
fpr = fp / (fp + tn)

print(tpr)
print(fpr)<jupyter_output>1.0
1.0<jupyter_text>Logistic RegressionA good first algorithm to apply to binary classification problems is logistic regression, for the following reasons: - it's quick to train and we can iterate more quickly, - it's less prone to overfitting than more complex models like decision trees, - it's easy to interpret.<jupyter_code>from sklearn.linear_model import LogisticRegression
lr = LogisticRegression()
cols = loans.columns
train_cols = cols.drop("loan_status")
features = loans[train_cols]
target = loans["loan_status"]
lr.fit(features, target)
predictions = lr.predict(features)<jupyter_output>/Users/otto/opt/anaconda3/envs/datasci/lib/python3.7/site-packages/sklearn/linear_model/_logistic.py:939: ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.

Increase the number of iterations (max_iter) or scale the data as shown in:
    https://scikit-learn.org/stable/modules/preprocessing.html.
Please also refer to the documentation for alternative solver options:
    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)<jupyter_text>Cross ValidationWhile we generated predictions in the last screen, those predictions were overfit. They were overfit because we generated predictions using the same data that we trained our model on.In order to get a realistic depiction of the accuracy of the model, let's perform `k-fold` cross validation. We can use the `cross_val_predict()` function from the `sklearn.model_selection` package.<jupyter_code>from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_predict
lr = LogisticRegression()

predictions = cross_val_predict(lr, features, target, cv=3)
predictions = pd.Series(predictions)
# False positives.
fp_filter = (predictions == 1) & (loans["loan_status"] == 0)
fp = len(predictions[fp_filter])

# True positives.
tp_filter = (predictions == 1) & (loans["loan_status"] == 1)
tp = len(predictions[tp_filter])

# False negatives.
fn_filter = (predictions == 0) & (loans["loan_status"] == 1)
fn = len(predictions[fn_filter])

# True negatives
tn_filter = (predictions == 0) & (loans["loan_status"] == 0)
tn = len(predictions[tn_filter])
# Rates
tpr = tp  / (tp + fn)
fpr = fp  / (fp + tn)
print(tpr)
print(fpr)<jupyter_output>/Users/otto/opt/anaconda3/envs/datasci/lib/python3.7/site-packages/sklearn/linear_model/_logistic.py:939: ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.

Increase the number of iterations (max_iter) or scale the data as shown in:
    https://scikit-learn.org/stable/modules/preprocessing.html.
Please also refer to the documentation for alternative solver options:
    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)<jupyter_text>Penalizing the classifierEven through we're not using accuracy as an error metric, the classifier is, and it isn't accounting for the imbalance in the classes. There are a few ways to get a classifier to correct for imbalanced classes. The two main ways are: - Use oversampling and undersampling to ensure that the classifier gets input that has a balanced number of each class. - Tell the classifier to penalize misclassifications of the less prevalent class more than the other class.<jupyter_code>lr = LogisticRegression(class_weight="balanced")
predictions = cross_val_predict(lr, features, target, cv=3)
predictions = pd.Series(predictions)

# False positives.
fp_filter = (predictions == 1) & (loans["loan_status"] == 0)
fp = len(predictions[fp_filter])

# True positives.
tp_filter = (predictions == 1) & (loans["loan_status"] == 1)
tp = len(predictions[tp_filter])

# False negatives.
fn_filter = (predictions == 0) & (loans["loan_status"] == 1)
fn = len(predictions[fn_filter])

# True negatives
tn_filter = (predictions == 0) & (loans["loan_status"] == 0)
tn = len(predictions[tn_filter])

# Rates
tpr = tp / (tp + fn)
fpr = fp / (fp + tn)

print(tpr)
print(fpr)<jupyter_output>/Users/otto/opt/anaconda3/envs/datasci/lib/python3.7/site-packages/sklearn/linear_model/_logistic.py:939: ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.

Increase the number of iterations (max_iter) or scale the data as shown in:
    https://scikit-learn.org/stable/modules/preprocessing.html.
Please also refer to the documentation for alternative solver options:
    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)
/Users/otto/opt/anaconda3/envs/datasci/lib/python3.7/site-packages/sklearn/linear_model/_logistic.py:939: ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.

Increase the number of iterations (max_iter) or scale the data as shown in:
    https://scikit-learn.org/stable/modules/preprocessing.html.
Please also refer to the documentation for alternative solver options:
    https://scikit-learn[...]<jupyter_text>Manual penalties<jupyter_code>penalty = {
    0: 10,
    1: 1
}

lr = LogisticRegression(class_weight=penalty)
predictions = cross_val_predict(lr, features, target, cv=3)
predictions = pd.Series(predictions)

# False positives.
fp_filter = (predictions == 1) & (loans["loan_status"] == 0)
fp = len(predictions[fp_filter])

# True positives.
tp_filter = (predictions == 1) & (loans["loan_status"] == 1)
tp = len(predictions[tp_filter])

# False negatives.
fn_filter = (predictions == 0) & (loans["loan_status"] == 1)
fn = len(predictions[fn_filter])

# True negatives
tn_filter = (predictions == 0) & (loans["loan_status"] == 0)
tn = len(predictions[tn_filter])

# Rates
tpr = tp / (tp + fn)
fpr = fp / (fp + tn)

print(tpr)
print(fpr)<jupyter_output>/Users/otto/opt/anaconda3/envs/datasci/lib/python3.7/site-packages/sklearn/linear_model/_logistic.py:939: ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.

Increase the number of iterations (max_iter) or scale the data as shown in:
    https://scikit-learn.org/stable/modules/preprocessing.html.
Please also refer to the documentation for alternative solver options:
    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)<jupyter_text>Random Forests<jupyter_code>from sklearn.ensemble import RandomForestClassifier
# from sklearn.cross_validation import cross_val_predict
rf = RandomForestClassifier(class_weight="balanced", random_state=1)
predictions = cross_val_predict(rf, features, target, cv=3)
predictions = pd.Series(predictions)

# False positives.
fp_filter = (predictions == 1) & (loans["loan_status"] == 0)
fp = len(predictions[fp_filter])

# True positives.`
tp_filter = (predictions == 1) & (loans["loan_status"] == 1)
tp = len(predictions[tp_filter])

# False negatives.
fn_filter = (predictions == 0) & (loans["loan_status"] == 1)
fn = len(predictions[fn_filter])

# True negatives
tn_filter = (predictions == 0) & (loans["loan_status"] == 0)
tn = len(predictions[tn_filter])

# Rates
tpr = tp / (tp + fn)
fpr = fp / (fp + tn)

print(tpr)
print(fpr)<jupyter_output>0.9977079848850895
0.9918352198923733
{'fraction_non_alphanumeric': 0.07262208819275626, 'fraction_numerical': 0.028464165923008277, 'mean_word_length': 3.552815914822079, 'pattern_counts': {'":': 15, '<': 73, '<?xml version=': 0, '>': 72, 'https://': 14, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 37, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '364344', 'license': 'Apache-2.0', 'n_tokens_mistral': 9783, 'n_tokens_neox': 8870, 'n_words': 3680, 'path': '7.2 Machine Learning Project/Machine Learning Project.ipynb', 'repo_name': 'orspain/Dataquest', 'size': '32497'}