<jupyter_start><jupyter_text>Credit Card balance data About the Datasetcredit_card_balance: It is monthly data about previous credit cards clients have had with Home Credit. Each row is one month of a credit card balance, and a single credit card can have many rows.  Feature ExplanationsSK_ID_PREV : 	ID of previous credit in Home credit related to loan in our sample. (One loan in our sample can have 0,1,2 or more previous loans in Home Credit)		SK_ID_CURR: 	ID of loan in our sample		MONTHS_BALANCE: 	Month of balance relative to application date (-1 means the freshest balance date)	AMT_BALANCE: 	Balance during the month of previous credit		AMT_CREDIT_LIMIT_ACTUAL: 	Credit card limit during the month of the previous credit		AMT_DRAWINGS_ATM_CURRENT: 	Amount drawing at ATM during the month of the previous credit		AMT_DRAWINGS_CURRENT: 	Amount drawing during the month of the previous credit		AMT_DRAWINGS_OTHER_CURRENT: 	Amount of other drawings during the month of the previous credit		AMT_DRAWINGS_POS_CURRENT: 	Amount drawing or buying goods during the month of the previous credit		AMT_INST_MIN_REGULARITY: 	Minimal installment for this month of the previous credit		AMT_PAYMENT_CURRENT: 	How much did the client pay during the month on the previous credit		AMT_PAYMENT_TOTAL_CURRENT: 	How much did the client pay during the month in total on the previous credit		AMT_RECEIVABLE_PRINCIPAL: 	Amount receivable for principal on the previous credit		AMT_RECIVABLE: 	Amount receivable on the previous credit		AMT_TOTAL_RECEIVABLE: 	Total amount receivable on the previous credit		CNT_DRAWINGS_ATM_CURRENT: 	Number of drawings at ATM during this month on the previous credit		CNT_DRAWINGS_CURRENT: 	Number of drawings during this month on the previous credit		CNT_DRAWINGS_OTHER_CURRENT: 	Number of other drawings during this month on the previous credit		CNT_DRAWINGS_POS_CURRENT: 	Number of drawings for goods during this month on the previous credit		CNT_INSTALMENT_MATURE_CUM: 	Number of paid installments on the previous credit		NAME_CONTRACT_STATUS: 	Contract status (active signed,...) on the previous credit		SK_DPD: 	DPD (Days past due) during the month on the previous credit		SK_DPD_DEF: 	DPD (Days past due) during the month with tolerance (debts with low loan amounts are ignored) of the previous credit<jupyter_code># Last amended: 23rd October, 2020
# Myfolder: C:\Users\Administrator\OneDrive\Documents\home_credit_default_risk
# Objective: 
#           Solving Kaggle problem: Home Credit Default Risk
#           Processing credit_card_balance dataset
#
# Data Source: https://www.kaggle.com/c/home-credit-default-risk/data
# Ref: https://www.kaggle.com/jsaguiar/lightgbm-with-simple-features
# 1.0 Libraries
#     (Some of these may not be needed here.)
%reset -f
import numpy as np
import pandas as pd
import gc

# 1.1 Reduce read data size
#     There is a file reducing.py
#      in this folder. A class
#       in it is used to reduce
#        dataframe size
#     (Code modified by me to
#      exclude 'category' dtype)
import reducing

# 1.2 Misc
import warnings
import os
warnings.simplefilter(action='ignore', category=FutureWarning)
# 1.3
pd.set_option('display.max_colwidth', -1)
pd.set_option('display.max_columns', 100)
pd.set_option('display.max_rows', 100)
# 1.4 Display multiple commands outputs from a cell
from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all"
# 2.0 Onehot encoding (OHE) function. Uses pd.get_dummies()
#     i) To transform 'object' columns to dummies. 
#    ii) Treat NaN as one of the categories
#   iii) Returns transformed-data and new-columns created

def one_hot_encoder(df, nan_as_category = True):
    original_columns = list(df.columns)
    categorical_columns = [col for col in df.columns if df[col].dtype == 'object']
    df = pd.get_dummies(df,
                        columns= categorical_columns,
                        dummy_na= nan_as_category       # Treat NaNs as category
                       )
    new_columns = [c for c in df.columns if c not in original_columns]
    return df, new_columns
# 2.1
pathToFolder = "C:\\Users\\Administrator\\OneDrive\\Documents\\home_credit_default_risk"
os.chdir(pathToFolder)
# 2.2 Some constants
num_rows=None                # Implies read all rows
nan_as_category = True       # While transforming 
                             #   'object' columns to dummies<jupyter_output><empty_output><jupyter_text>About the Datasetcredit_card_balance: It is monthly data about previous credit cards clients have had with Home Credit. Each row is one month of a credit card balance, and a single credit card can have many rows.<jupyter_code># 2.3 Read the data
cc = pd.read_csv(
                'credit_card_balance.csv.zip',
                 nrows = num_rows
                )
# 2.4
cc.shape      # (rows = 38,40,312, columns = 23)
cc.head()<jupyter_output><empty_output><jupyter_text>Feature explanations:MONTHS_BALANCE: Month of balance relative to application date (-1 means the freshest balance date)AMT_BALANCE: 	    Balance during the month of previous creditAMT_CREDIT_LIMIT_ACTUAL: 	Credit card limit during the month of the previous creditAMT_DRAWINGS_ATM_CURRENT: 	Amount drawing at ATM during the month of the previous creditAMT_DRAWINGS_CURRENT: 	Amount drawing during the month of the previous creditAMT_DRAWINGS_OTHER_CURRENT: 	Amount of other drawings during the month of the previous creditAMT_DRAWINGS_POS_CURRENT: 	Amount drawing or buying goods during the month of the previous creditAMT_INST_MIN_REGULARITY: 	Minimal installment for this month of the previous creditAMT_PAYMENT_CURRENT: 	How much did the client pay during the month on the previous creditAMT_PAYMENT_TOTAL_CURRENT: 	How much did the client pay during the month in total on the previous creditAMT_RECEIVABLE_PRINCIPAL: 	Amount receivable for principal on the previous creditAMT_RECIVABLE: 	Amount receivable on the previous creditAMT_TOTAL_RECEIVABLE: 	Total amount receivable on the previous creditCNT_DRAWINGS_ATM_CURRENT: 	Number of drawings at ATM during this month on the previous creditCNT_DRAWINGS_CURRENT: 	Number of drawings during this month on the previous creditCNT_DRAWINGS_OTHER_CURRENT: 	Number of other drawings during this month on the previous creditCNT_DRAWINGS_POS_CURRENT: 	Number of drawings for goods during this month on the previous creditCNT_INSTALMENT_MATURE_CUM: 	Number of paid installments on the previous creditNAME_CONTRACT_STATUS: 	Contract status (active signed,...) on the previous creditSK_DPD: 	DPD (Days past due) during the month on the previous creditSK_DPD_DEF: 	DPD (Days past due) during the month with tolerance (debts with low loan amounts are ignored) of the previous credit<jupyter_code># 2.5 There is one 'object' feature
cc.dtypes.value_counts()
# 2.6 Transform the 'object' feature to OHE
cc, cat_cols = one_hot_encoder(cc, nan_as_category= True)
# 2.7
cc.shape   # (3840312, 30)
cat_cols   # Even NaN is a feature
# 2.8 Drop this unique ID. We do not need it
cc.drop(['SK_ID_PREV'], axis= 1, inplace = True)
# 3.0 Aggregate all features over SK_ID_CURR.
cc_agg = cc.groupby('SK_ID_CURR').agg(['min', 'max', 'mean', 'sum', 'var'])
# 3.1
cc_agg.shape     # (103558, 140)
cc_agg.head()    # It has multi-index feature
# 3.2 Change column names
cc_agg.columns = pd.Index(['CC_' + e[0] + "_" + e[1].upper() for e in cc_agg.columns.tolist()])
# 3.3 Create another feature
#     For each client, how many observations
#     exist in this dataset

cc_agg['CC_COUNT'] = cc.groupby('SK_ID_CURR').size()

# 3.3.1
cc_agg['CC_COUNT'].head()
# 4.0 Save the results for subsequent use:
cc_agg.to_csv("processed_creditCard_agg.csv.zip", compression = "zip")
################<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06316600333804083, 'fraction_numerical': 0.011426370522531775, 'mean_word_length': 4.365013774104683, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '522239', 'license': 'Apache-2.0', 'n_tokens_mistral': 2473, 'n_tokens_neox': 2338, 'n_words': 896, 'path': 'credit_card_balance.ipynb', 'repo_name': 'harnalashok/credit_risk', 'size': '7789'}