<jupyter_start><jupyter_text>Cleaning and Dummification NotebookThis notebook contains all the steps I took to clean my data and make it viable for all types of classificaton algo's.At the end of this notebook, I load the dataframe into a PostgreSQL database. There isnt really a need for this, but it's a matter of demonstrating the skillset.<jupyter_code>%matplotlib inline
import pickle
%run helper_functions.py
pd.options.display.max_columns = 1000
plt.rcParams["figure.figsize"] = (15,10)
from datetime import datetime
# from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
df = unpickle_object("non_current_df.pkl") #loans that are 'complete'
df.shape<jupyter_output><empty_output><jupyter_text>I will manipulate my dataset in order for it to be compatible with both GLM's and classification algorithms.As such, I will create dummies and scale all of data. Scaling is incredibly important for KNN and will improve model performance for Logisitc Regression.I am not particularly concerned with coeff interpretability as the purpose is the assign a class.By manipulating my data in this way, I will be ready to be used by any ML model.I will use 3 in particular:- Dummy Classifier. This will the global baseline I have to beat.- KNN (My most constrained model)- Logistic Regression- Random ForestsNote that Multi-collinearity does **NOT** matter for models like DT's and RF's - however, it will matter for Logistic regression. I will first throw all of my data at LGR, and remove variables (multi-collinear) accordingly (this creates a baselinne model for lgr). My project will be concerned with classifying whether an individual will re-pay their loan on time. I will change the 'loan status' feature in this dataset into a binary form of "Fully Paid" or "Late"<jupyter_code>df['loan_status'].unique()
mask = df['loan_status'] != "Fully Paid"
rows_to_change = df[mask]
rows_to_change.loc[:, 'loan_status'] = 'Late'
df.update(rows_to_change)
df['loan_status'].unique() #sweet!
df.shape # no dimensionality lost
plot_corr_matrix(df)<jupyter_output><empty_output><jupyter_text>Let's have a quick look at all of our columns, their descriptions and associated datatype.Perhaps we can reduce the dimension of our dataset off the bat by dropping columns that are not pertinent<jupyter_code>no_desc = []
for column in df.columns:
    try:
        print(column+":",lookup_description(column)," DataType:", df[column].dtype)
        print()
    except KeyError:
        no_desc.append(column)
columns_to_drop = ["id", "member_id", "emp_title","desc","title","out_prncp","out_prncp_inv","total_pymnt","total_pymnt_inv", "total_rec_prncp", "total_rec_int", "total_rec_late_fee", "recoveries", "collection_recovery_fee","last_pymnt_d", "last_pymnt_amnt","next_pymnt_d", "last_credit_pull_d", "collections_12_mths_ex_med","mths_since_last_major_derog", "all_util", ]
# df.loc[:, ["loan_amnt","funded_amnt","out_prncp","out_prncp_inv","total_pymnt","total_pymnt_inv","total_rec_prncp","last_credit_pull_d"]]
no_desc
df['verification_status_joint'].unique()
df['total_rev_hi_lim'].unique()
df['verification_status_joint'].dtype
df['total_rev_hi_lim'].dtype<jupyter_output><empty_output><jupyter_text>After going through the list, I have decided to drop 5 columns!These will not be relevant to the task at hand. Although, I could use some natural language processig via NLTK to parse job descriptions and loan descriptions. I will leave this for another day.It is also important to note that I will be dropping variables that hint (i.e. information leakage) at what the final result will be.<jupyter_code>df.drop(columns_to_drop, axis=1, inplace=True)
df.shape #just what we expected<jupyter_output><empty_output><jupyter_text>After reviewing the above, the following columns need to be changed to categorical datatypes from float64.- policy_codeI will first make it an object datatype as later I will write a function that changed all object datatypes into categorical datatypes.<jupyter_code>df["policy_code"] = df["policy_code"].astype('object')<jupyter_output><empty_output><jupyter_text>I will have to transform the following columns as they are currently in percentages. I will take the natural log of these columns before proceeding:- `pct_tl_nvr_dlq`- `percent_bc_gt_75`This will ensure better model performance for logistic regression as % may not follow a linear relationship.<jupyter_code>df['pct_tl_nvr_dlq'] = df['pct_tl_nvr_dlq'].apply(lambda x: x/100)
df['percent_bc_gt_75'] = df['percent_bc_gt_75'].apply(lambda x: x/100)<jupyter_output><empty_output><jupyter_text>My categorical features (those of type Object) have `np.nan` values, I will change these to something more meaningful like "Missing Data".I will then create dummies for all of my categorical features. This will lead to an explosion in the number of columns - this will be more computationally expensive, however, this is NOT an explosion in the 'feature space' as our dataframe contains the same amount of information.<jupyter_code>object_columns = df.select_dtypes(include=['object']).columns

for c in object_columns:
    df.loc[df[df[c].isnull()].index, c] = "missing"<jupyter_output><empty_output><jupyter_text>So, our dataset is comprised of features which are categorical and features that are numeric. We need to ensure that the object datatypes are converted to categorical datatypes.Also, whether we use a GLM or classifier, we need to ensure that these datatypes stay consistent.NOTE: changing columns to categorical datatypes will NOT change how a machine learning model interprets the data. i.e. The algorithm will still think that 5 > 4. As such, one hot encoding (i.e. making dummies) is the only way to ensure that a Machine Learning Model can detect the presence of a particular attribute.I will be changing the object datatypes to categorical purely for data consistency within the dataframe.<jupyter_code>obj_df = df.select_dtypes(include=['object'])

obj_df_cols = obj_df.columns

for col in obj_df_cols:
    df[col] = df[col].astype("category")
    
df.dtypes.unique() #This is what we wanted!
df.shape
df.head()
unique_val_dict = {}
for col in df.columns:
    if col not in unique_val_dict:
        unique_val_dict[col] = df[col].unique()
unique_val_dict #will use this later when making flask app.
category_columns = df.select_dtypes(include=['category']).columns
df = pd.get_dummies(df, columns=category_columns, drop_first=True)
df.shape<jupyter_output><empty_output><jupyter_text>Let's ensure that all of our missing values in float columns be nan values via the numpy library. I am doing this because Numpy is a highly optimized library.<jupyter_code>float_columns = df.select_dtypes(include=['float64']).columns

for c in float_columns:
    df.loc[df[df[c].isnull()].index, c] = np.nan
pickle_object(unique_val_dict, "unique_values_for_columns")
pickle_object(df, "dummied_dataset")
df = unpickle_object("dummied_dataset.pkl")
df.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08582834331337326, 'fraction_numerical': 0.0044197319646421445, 'mean_word_length': 6.18014329580348, 'pattern_counts': {'":': 1, '<': 37, '<?xml version=': 0, '>': 38, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '575939', 'license': 'MIT', 'n_tokens_mistral': 2151, 'n_tokens_neox': 2087, 'n_words': 768, 'path': '03-Project-McNulty/Cleaning_and_dummies.ipynb', 'repo_name': 'igabr/Metis_Projects_Chicago_2017', 'size': '7014'}