<jupyter_start><jupyter_text>Process train and test files  About dataapplication_train/application_test: This is main training and testing data with information about each loan application at Home Credit. Every loan has its own row and is identified by the feature SK_ID_CURR. The training application data comes with the TARGET indicating 0: the loan was repaid or 1: the loan was not repaid. Feature descriptions are as below. These have been taken from file 'HomeCredit_columns_description.csv'>/i>. For a more accurate description, please refer to the file.   Feature descriptionsSK_ID_CURR: 	ID of loan in our sampleTARGET: 	Target variable (1 - client with payment difficulties: he/she had late payment more than X days on at least one of the first Y installments of the loan in our sample, 0 - all other cases)NAME_CONTRACT_TYPE: 	Identification if loan is cash or revolvingCODE_GENDER: 	Gender of the clientFLAG_OWN_CAR: 	Flag if the client owns a carFLAG_OWN_REALTY: 	Flag if client owns a house or flatCNT_CHILDREN: 	Number of children the client hasAMT_INCOME_TOTAL: 	Income of the clientAMT_CREDIT: 	Credit amount of the loanAMT_ANNUITY: 	Loan annuityAMT_GOODS_PRICE: 	For consumer loans it is the price of the goods for which the loan is givenNAME_TYPE_SUITE: 	Who was accompanying client when he was applying for the loanNAME_INCOME_TYPE: 	Clients income type (businessman, working, maternity leave,…)NAME_EDUCATION_TYPE: 	Level of highest education the client achievedNAME_FAMILY_STATUS: 	Family status of the clientNAME_HOUSING_TYPE: 	What is the housing situation of the client (renting, living with parents, ...)REGION_POPULATION_RELATIVE: 	Normalized population of region where client lives (higher number means the client lives in more populated region)DAYS_BIRTH: 	Client's age in days at the time of applicationDAYS_EMPLOYED: 	How many days before the application the person started current employmentDAYS_REGISTRATION: 	How many days before the application did client change his registrationDAYS_ID_PUBLISH: 	How many days before the application did client change the identity document with which he applied for the loanOWN_CAR_AGE: 	Age of client's carFLAG_MOBIL: 	Did client provide mobile phone (1=YES, 0=NO)FLAG_EMP_PHONE: 	Did client provide work phone (1=YES, 0=NO)FLAG_WORK_PHONE: 	Did client provide home phone (1=YES, 0=NO)FLAG_CONT_MOBILE: 	Was mobile phone reachable (1=YES, 0=NO)FLAG_PHONE: 	Did client provide home phone (1=YES, 0=NO)FLAG_EMAIL: 	Did client provide email (1=YES, 0=NO)OCCUPATION_TYPE: 	What kind of occupation does the client haveCNT_FAM_MEMBERS: 	How many family members does client haveREGION_RATING_CLIENT: 	Our rating of the region where client lives (1,2,3)REGION_RATING_CLIENT_W_CITY: 	Our rating of the region where client lives with taking city into account (1,2,3)WEEKDAY_APPR_PROCESS_START: 	On which day of the week did the client apply for the loanHOUR_APPR_PROCESS_START: 	Approximately at what hour did the client apply for the loanREG_REGION_NOT_LIVE_REGION: 	Flag if client's permanent address does not match contact address (1=different, 0=same, at region level)REG_REGION_NOT_WORK_REGION: 	Flag if client's permanent address does not match work address (1=different, 0=same, at region level)LIVE_REGION_NOT_WORK_REGION: 	Flag if client's contact address does not match work address (1=different, 0=same, at region level)REG_CITY_NOT_LIVE_CITY: 	Flag if client's permanent address does not match contact address (1=different, 0=same, at city level)REG_CITY_NOT_WORK_CITY: 	Flag if client's permanent address does not match work address (1=different, 0=same, at city level)LIVE_CITY_NOT_WORK_CITY: 	Flag if client's contact address does not match work address (1=different, 0=same, at city level)ORGANIZATION_TYPE: 	Type of organization where client worksEXT_SOURCE_1: 	Normalized score from external data sourceEXT_SOURCE_2: 	Normalized score from external data sourceEXT_SOURCE_3: 	Normalized score from external data sourceAPARTMENTS_AVG: 	Normalized information about building where the client lives.BASEMENTAREA_AVG: 	Normalized information about building where the client lives.YEARS_BEGINEXPLUATATION_AVG: 	Normalized information about building where the client lives.YEARS_BUILD_AVG: 	Normalized information about building where the client lives.COMMONAREA_AVG: 	Normalized information about building where the client lives.ELEVATORS_AVG: 	Normalized information about building where the client lives.ENTRANCES_AVG: 	Normalized information about building where the client lives.FLOORSMAX_AVG: 	Normalized information about building where the client lives.FLOORSMIN_AVG: 	Normalized information about building where the client lives.LANDAREA_AVG: 	Normalized information about building where the client lives.LIVINGAPARTMENTS_AVG: 	Normalized information about building where the client lives.LIVINGAREA_AVG: 	Normalized information about building where the client lives.NONLIVINGAPARTMENTS_AVG: 	Normalized information about building where the client lives.NONLIVINGAREA_AVG: 	Normalized information about building where the client lives.APARTMENTS_MODE: 	Normalized information about building where the client lives.BASEMENTAREA_MODE: 	Normalized information about building where the client lives.YEARS_BEGINEXPLUATATION_MODE: 	Normalized information about building where the client lives.YEARS_BUILD_MODE: 	Normalized information about building where the client livesCOMMONAREA_MODE: 	Normalized information about building where the client livesELEVATORS_MODE: 	Normalized information about building where the client livesENTRANCES_MODE: 	Normalized information about building where the client livesFLOORSMAX_MODE: 	Normalized information about building where the client livesFLOORSMIN_MODE: 	Normalized information about building where the client livesLANDAREA_MODE: 	Normalized information about building where the client livesLIVINGAPARTMENTS_MODE: 	Normalized information about building where the client livesLIVINGAREA_MODE: 	Normalized information about building where the client livesNONLIVINGAPARTMENTS_MODE: 	Normalized information about building where the client livesNONLIVINGAREA_MODE: 	Normalized information about building where the client livesAPARTMENTS_MEDI: 	Normalized information about building where the client livesBASEMENTAREA_MEDI: 	Normalized information about building where the client livesYEARS_BEGINEXPLUATATION_MEDI: 	Normalized information about building where the client livesYEARS_BUILD_MEDI: 	Normalized information about building where the client livesCOMMONAREA_MEDI: 	Normalized information about building where the client livesELEVATORS_MEDI: 	Normalized information about building where the client livesENTRANCES_MEDI: 	Normalized information about building where the client livesFLOORSMAX_MEDI: 	Normalized information about building where the client livesFLOORSMIN_MEDI: 	Normalized information about building where the client livesLANDAREA_MEDI: 	Normalized information about building where the client livesLIVINGAPARTMENTS_MEDI: 	Normalized information about building where the client livesLIVINGAREA_MEDI: 	Normalized information about building where the client livesNONLIVINGAPARTMENTS_MEDI: 	Normalized information about building where the client livesNONLIVINGAREA_MEDI: 	Normalized information about building where the client livesFONDKAPREMONT_MODE: 	Normalized information about building where the client livesHOUSETYPE_MODE: 	Normalized information about building where the client livesTOTALAREA_MODE: 	Normalized information about building where the client livesWALLSMATERIAL_MODE: 	Normalized information about building where the client livesEMERGENCYSTATE_MODE: 	Normalized information about building where the client livesOBS_30_CNT_SOCIAL_CIRCLE: 	How many observation of client's social surroundings with observable 30 DPD (days past due) defaultDEF_30_CNT_SOCIAL_CIRCLE: 	How many observation of client's social surroundings defaulted on 30 DPD (days past due) OBS_60_CNT_SOCIAL_CIRCLE: 	How many observation of client's social surroundings with observable 60 DPD (days past due) defaultDEF_60_CNT_SOCIAL_CIRCLE: 	How many observation of client's social surroundings defaulted on 60 (days past due) DPDDAYS_LAST_PHONE_CHANGE: 	How many days before application did client change phoneFLAG_DOCUMENT_2: 	Did client provide document 2FLAG_DOCUMENT_3: 	Did client provide document 3FLAG_DOCUMENT_4: 	Did client provide document 4FLAG_DOCUMENT_5: 	Did client provide document 5FLAG_DOCUMENT_6: 	Did client provide document 6FLAG_DOCUMENT_7: 	Did client provide document 7FLAG_DOCUMENT_8: 	Did client provide document 8FLAG_DOCUMENT_9: 	Did client provide document 9FLAG_DOCUMENT_10: 	Did client provide document 10FLAG_DOCUMENT_11: 	Did client provide document 11FLAG_DOCUMENT_12: 	Did client provide document 12FLAG_DOCUMENT_13: 	Did client provide document 13FLAG_DOCUMENT_14: 	Did client provide document 14FLAG_DOCUMENT_15: 	Did client provide document 15FLAG_DOCUMENT_16: 	Did client provide document 16FLAG_DOCUMENT_17: 	Did client provide document 17FLAG_DOCUMENT_18: 	Did client provide document 18FLAG_DOCUMENT_19: 	Did client provide document 19FLAG_DOCUMENT_20: 	Did client provide document 20FLAG_DOCUMENT_21: 	Did client provide document 21AMT_REQ_CREDIT_BUREAU_HOUR: 	Number of enquiries to Credit Bureau about the client one hour before applicationAMT_REQ_CREDIT_BUREAU_DAY: 	Number of enquiries to Credit Bureau about the client one day before applicationAMT_REQ_CREDIT_BUREAU_WEEK: 	Number of enquiries to Credit Bureau about the client one week before applicationion)AMT_REQ_CREDIT_BUREAU_MON: 	Number of enquiries to Credit Bureau about the client one month before application AMT_REQ_CREDIT_BUREAU_QRT: 	Number of enquiries to Credit Bureau about the client 3 month before application AMT_REQ_CREDIT_BUREAU_YEAR: 	Number of enquiries to Credit Bureau about the client one day year<jupyter_code># Last amended: 24rd October, 2020
# Myfolder: C:\Users\Administrator\OneDrive\Documents\home_credit_default_risk
# Objective: 
#           Solving Kaggle problem: Home Credit Default Risk
#           Processing application train/test datasets
#
# Data Source: https://www.kaggle.com/c/home-credit-default-risk/data
# Ref: https://www.kaggle.com/jsaguiar/lightgbm-with-simple-features
# 1.0 Libraries

import numpy as np
import pandas as pd

# 1.1 Reduce read data size
#     There is a file reducing.py
#      in this folder. A class
#       in it is used to reduce
#        dataframe size
#     (Code modified to
#      exclude 'category' dtype)
#     Refer: https://wkirgsn.github.io/2018/02/10/auto-downsizing-dtypes/
import reducing

# 1.2 Misc
import warnings
import os
warnings.simplefilter(action='ignore', category=FutureWarning)

# 1.3 In view of large dataset, some useful options
pd.set_option('display.max_colwidth', -1)
pd.set_option('display.max_columns', 100)
pd.set_option('display.max_rows', 100)
# 1.4 Display outputs from multiple commands from a jupyter cell
from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all"
# 2.0 Onehot encoding (OHE) function. Uses pd.get_dummies()
#     i) To transform 'object' columns to dummies. 
#    ii) Treat NaN as one of the categories
#   iii) Returns transformed-data and new-columns created

def one_hot_encoder(df, nan_as_category = True):
    original_columns = list(df.columns)
    categorical_columns = [col for col in df.columns if df[col].dtype == 'object']
    df = pd.get_dummies(df,
                        columns= categorical_columns,
                        dummy_na= nan_as_category       # Treat NaNs as category
                       )
    new_columns = [c for c in df.columns if c not in original_columns]
    return df, new_columns
# 2.1
pathToFolder = "C:\\Users\\Administrator\\OneDrive\\Documents\\home_credit_default_risk"
os.chdir(pathToFolder)
# 2.2 Some constants
num_rows=None                # Implies read all rows
nan_as_category = True       # While transforming 
                             #   'object' columns to dummies
# 3.0 Read previous application data first
df = pd.read_csv(
                   'application_train.csv.zip',
                   nrows = num_rows
                   )

# 3.0.1 Reduce memory usage by appropriately
#       changing data-types per feature:

df = reducing.Reducer().reduce(df)
# 3.0 Read previous application data first
test_df = pd.read_csv(
                      'application_test.csv.zip',
                       nrows = num_rows
                   )

# 3.0.1 Reduce memory usage by appropriately
#       changing data-types per feature:

test_df = reducing.Reducer().reduce(test_df)
# 3.1
df.shape   # (307511, 122)
df.head()
# 3.1.1 There are 16 object types
df.dtypes.value_counts()
# 3.2
test_df.shape   # (48744, 121)
test_df.head()
# 3.3 There are 16 object types
test_df.dtypes.value_counts()
# 3.4 Append test_df to train
df = df.append(test_df).reset_index()
# 3.5 Examine merged data
df.shape     # (356255, 123)
df.head()
# 3.6 This gender is rare. So such
#     rows can be dropped
df[df['CODE_GENDER'] == 'XNA']
# 3.7 Optional: Remove 4 applications with XNA CODE_GENDER (train set)
df = df[df['CODE_GENDER'] != 'XNA']
# 3.8 Categorical features with Binary encode (0 or 1; two categories)
for bin_feature in ['CODE_GENDER', 'FLAG_OWN_CAR', 'FLAG_OWN_REALTY']:
    df[bin_feature], uniques = pd.factorize(df[bin_feature])
# 3.8.1
df.head()
uniques
# 3.8.2
df.dtypes
df.dtypes.value_counts()
df.dtypes['CODE_GENDER']
df.dtypes['FLAG_OWN_CAR']
df.dtypes['FLAG_OWN_REALTY']
# 4.0 Categorical features with One-Hot encode
df, cat_cols = one_hot_encoder(df, nan_as_category)
# 4.1
len(cat_cols)   # 146
cat_cols
# 4.2 NaN values for DAYS_EMPLOYED: 365.243 -> nan
df['DAYS_EMPLOYED'].replace(365243, np.nan, inplace= True)
# 4.3 Some simple new features (percentages)
df['DAYS_EMPLOYED_PERC'] = df['DAYS_EMPLOYED'] / df['DAYS_BIRTH']
df['INCOME_CREDIT_PERC'] = df['AMT_INCOME_TOTAL'] / df['AMT_CREDIT']
df['INCOME_PER_PERSON'] = df['AMT_INCOME_TOTAL'] / df['CNT_FAM_MEMBERS']
df['ANNUITY_INCOME_PERC'] = df['AMT_ANNUITY'] / df['AMT_INCOME_TOTAL']
df['PAYMENT_RATE'] = df['AMT_ANNUITY'] / df['AMT_CREDIT']
# 5.0 Save the results for subsequent use:
df.to_csv("processed_df.csv.zip", compression = "zip")
#################<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.05569849106459912, 'fraction_numerical': 0.01724497601001321, 'mean_word_length': 5.2557633753806, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 8, 'https://': 3, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '492198', 'license': 'Apache-2.0', 'n_tokens_mistral': 4534, 'n_tokens_neox': 4235, 'n_words': 1523, 'path': 'application_train_test.ipynb', 'repo_name': 'harnalashok/credit_risk', 'size': '14381'}