<jupyter_start><jupyter_text>1. Loading and Exploring the given csv file<jupyter_code># Untouched data
orig_data = pd.read_csv('Loan payments data.csv') 
# all the alterations will be done in this
data = orig_data.copy() 
data.head() 
# printing the features in a list format for better understanding
print(colored("FEATURES OF THE LOAN DATASET:",'red',attrs=['bold','underline']))
for i in range(len(data.columns)):
               print(colored(i+1, attrs=['bold']), data.columns[i])
data.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 454 entries, 0 to 453
Data columns (total 11 columns):
Loan_ID           454 non-null object
loan_status       454 non-null object
Principal         454 non-null int64
terms             454 non-null int64
effective_date    454 non-null object
due_date          454 non-null object
paid_off_time     369 non-null object
past_due_days     170 non-null float64
age               454 non-null int64
education         454 non-null object
Gender            454 non-null object
dtypes: float64(1), int64(3), object(7)
memory usage: 39.1+ KB<jupyter_text>Take Away:From the above information it is clear that out of 454 total enteries, 2 features including paid_off_time and past_due_days have missing values.<jupyter_code>## Finding null values if exists in any column of the dataset
print(colored("FEATURE_NAME,  TOTAL_NULL_VALUES  and  MISSING_VALUE_PERCENTAGE\n", 'blue',attrs=['bold']))  

for i in range(len(data.columns)):
    current_feature = data[ data.columns[i] ]    
    if  (current_feature.isnull().values.any()):        
        sum_of_null_values = current_feature.isnull().sum()      
        null_value_percentage = (sum_of_null_values/len(current_feature))*100      
        
        result = ([sum_of_null_values,  float("{0:.2f}".format(null_value_percentage))])
        print(data.columns[i], result)<jupyter_output>[1m[34mFEATURE_NAME,  TOTAL_NULL_VALUES  and  MISSING_VALUE_PERCENTAGE
[0m
paid_off_time [85, 18.72]
past_due_days [284, 62.56]<jupyter_text>Note:Since, the percentage of missing values is quite considerable especially for past_due_days feature, we should test and try how its removal might affect the accuracy of the intended prediction.Making a test_try dataframe of the same dataset for understading how necessary these columns are will be discussed further.<jupyter_code>### NOT USED
## make all values None for the null values only in data 
# data = data.fillna( 'NAN' )
## LOOKING FOR UNIQUE VALUES OF COLUMNS OF INTEREST
## coi = [1,2,3,4,5,6,,9,10]

# Unique values for each attribute/feature
for i in range(1,11):
    print(colored(i, "red", attrs=['bold']),
          colored(data.columns[i], "red", attrs=['bold'] ), 
          data[data.columns[i]].unique(),"\n")<jupyter_output>[1m[31m1[0m [1m[31mloan_status[0m ['PAIDOFF' 'COLLECTION' 'COLLECTION_PAIDOFF'] 

[1m[31m2[0m [1m[31mPrincipal[0m [1000  300  800  900  700  500] 

[1m[31m3[0m [1m[31mterms[0m [30 15  7] 

[1m[31m4[0m [1m[31meffective_date[0m ['09/08/16' '09/09/16' '09/10/16' '09/11/16' '09/12/16' '9/13/2016'
 '9/14/2016'] 

[1m[31m5[0m [1m[31mdue_date[0m ['10/07/16' '9/22/2016' '10/08/16' '9/15/2016' '9/24/2016' '9/16/2016'
 '10/09/16' '10/10/16' '9/25/2016' '9/17/2016' '11/09/16' '10/25/2016'
 '9/18/2016' '9/26/2016' '10/26/2016' '10/11/16' '11/10/16' '9/19/2016'
 '10/12/16' '9/27/2016' '9/28/2016' '10/13/2016' '11/12/16' '9/23/2016'
 '11/07/16'] 

[1m[31m6[0m [1m[31mpaid_off_time[0m ['9/14/2016 19:31' '10/07/16 9:00' '9/25/2016 16:58' '9/22/2016 20:00'
 '9/23/2016 21:36' '09/09/16 13:45' '10/07/16 23:07' '10/05/16 20:33'
 '10/08/16 16:00' '9/24/2016 13:00' '09/11/16 19:11' '10/09/16 16:00'
 '10/07/16 23:32' '9/13/2016 21:57' '9/15/2016 14:27' '9/24/2016 16:00'
 '9[...]<jupyter_text>Take Away:* 'paid_off_time' attribute is a date-time stamp. Though it has 18.72% of missing values, extracting just the date and onserving how it affects the prediction of 'loan_status' will be a wise thing to do.* For this a new column is attched to the copy of original dataframe by the name 'paid_off_date'. With the introduction of this column 'paid_off_time' can be dropped off from this copy dataframe.<jupyter_code># PAID OFF TIME is a date-time stamp 
# Label encoder won't work on it and give it a unique integer
# Since time is of no such important but date is, 
# it is wise to replace this column with a new paid_off_date column
# which will only have date

## Method 1: the method below gives warning but the dates produced seemed to be the same as done by method 2
data['paid_off_date'] = None
count = 0

for i in range(len(data.paid_off_time)):
    
     if not pd.isnull(data['paid_off_time'][i]):
        count = count + 1
        
        data['paid_off_date'][i] = pd.to_datetime(data['paid_off_time'][i]).date().strftime('%m/%d/%Y')

print('MISSING VALUES ARE:', 454-count)
print("Hence rechecked!")

### Method 2
# data['paid_off_date'] = pd.DatetimeIndex(data.paid_off_time).normalize()
    
# Finding the unique values of the new column

print(colored("THE UNIQUE VALUED OF 'paid_off_date'\n", "red", attrs=['bold']))
print(data['paid_off_date'].unique())

data.head(5)
# # rechecking the null or missing values

# count = 0
# for i in range(len(data.paid_off_date)):
    
#     if data['paid_off_date'][i] == 'nan':
#         count = count + 1
# print("MISSING VALUE COUNT:", count)

# With the introduction of paid_off_date, paid_off_time can be deleted

data = data.drop(['paid_off_time'], axis = 1)

data.head(2)
print(colored("THE UNIQUE VALUED OF 'past_due_days'\n", "red", attrs=['bold']))
print(data['paid_off_date'].unique())<jupyter_output><empty_output><jupyter_text>VISUALIZING DATA<jupyter_code>data.head()
# Before being able to visualize each and every feature it is must that all 
# strings are in integer form
# and null values are treated as a category
# For this we make use of label encoder

le=preprocessing.LabelEncoder()

#String values
data['loan_status'] = le.fit_transform(data['loan_status'])
data['Gender'] = le.fit_transform(data['Gender'])
data['education'] = le.fit_transform(data['education'])
data['effective_date'] = le.fit_transform(data['effective_date'])
data['due_date'] = le.fit_transform(data['due_date'])
data['Loan_ID'] = le.fit_transform(data['Loan_ID'])


#nan values - treated as a category 
data['past_due_days'] = le.fit_transform(data['past_due_days'])

##won't work on date-time format (no need too)
# data['paid_off_date'] = le.fit_transform(data['paid_off_date'])                       
                                              

## Can't apply the same for Nan valued paid_off_time 
# Now seeing the changed dataframe
data.head()<jupyter_output><empty_output><jupyter_text>HEATMAP<jupyter_code># HEATMAP VISUALIZARION (does multivarite data analysis)
fig, ax = plt.subplots(figsize=(7,7))

corr = data.corr()
heatmap = sns.heatmap(corr, annot=True, linewidths=.5,  ax=ax)

loc, labels = plt.xticks()
heatmap.set_xticklabels(labels, rotation=90)


print(heatmap)<jupyter_output><empty_output><jupyter_text>HELPER FUNCTION TO GET HEIGT OF EACH BAR IN BAR PLOT<jupyter_code>def get_height(ax):
    for box in ax.patches:
        ax.annotate(box.get_height(), (box.get_x(), box.get_height()*1.01))<jupyter_output><empty_output><jupyter_text>GRAPHICAL<jupyter_code>fig = plt.figure(figsize=(7,7))

ax = sns.countplot(data['loan_status'])
ax.set_title("Analysing of Loan Status")
get_height(ax)
pd.DataFrame(data.groupby('loan_status')['Loan_ID'].count())<jupyter_output><empty_output><jupyter_text>Principal amount and loan status<jupyter_code>plt.figure(figsize=(12, 10))

ax = sns.countplot(x='Principal', hue = "loan_status", data=data)
plt.legend(loc='upper left', fontsize='12')

get_height(ax)
pd.DataFrame(data.groupby(['Principal','loan_status'])['Loan_ID'].count())
pd.crosstab(data.loan_status, data.Principal, margins=True)<jupyter_output><empty_output><jupyter_text>TERMS<jupyter_code>plt.figure(figsize=(12, 10))

ax = sns.countplot(hue='terms', x = "loan_status", data=data)
plt.legend(loc='upper left', fontsize='12')

get_height(ax)
terms = pd.DataFrame(data.groupby('terms')['Loan_ID'].count())
fig, ax = plt.subplots(figsize=(6,6))
pos = np.arange(len(data.terms.unique()))
# Terms can be labelled as weekly (7), biweekly (14), monthly (30)
mapTerms={7:'weekly', 15:'bi-weekly', 30: 'monthly'}
pie = ax.pie(terms.Loan_ID.values, labels=[mapTerms[l] for l in list(terms.index)])<jupyter_output><empty_output><jupyter_text>EFFECTIVE DATE<jupyter_code>plt.figure(figsize=(12, 10))

ax = sns.countplot(x='effective_date', hue='loan_status', data= data)
plt.legend(loc='upper left', fontsize='12')

get_height(ax)
data.groupby('Gender')['Gender'].agg(['count'])
plt.figure(figsize=(12,8))
ax = sns.countplot( x = "loan_status",hue = "Gender", data=data)
plt.legend(loc='upper right', fontsize='12')
get_height(ax)
pd.crosstab(data.loan_status, data.Principal, margins=True)
x = data.drop(['loan_status'], axis = 1)
y = data.loan_status
# cdata.drop('loan_status', axis=1, inplace=True)
# label = cdata.pop('loan_status')
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn import svm<jupyter_output><empty_output><jupyter_text>PREDICTING LOAN_STATUS<jupyter_code>from sklearn.model_selection import train_test_split
data_train, data_test, label_train, label_test = train_test_split(x, y, test_size = 0.2, random_state = 42) 
from sklearn.linear_model import LogisticRegression
logis = LogisticRegression()
logis.fit(data_train, label_train)
logis_score_train = logis.score(data_train, label_train)
print("Training score: ",logis_score_train)
logis_score_test = logis.score(data_test, label_test)
print("Testing score: ",logis_score_test)
from sklearn.ensemble import RandomForestClassifier
dt = RandomForestClassifier()
dt.fit(data_train, label_train)
dt_score_train = dt.score(data_train, label_train)
print("Training score: ",dt_score_train)
dt_score_test = dt.score(data_test, label_test)
print("Testing score: ",dt_score_test)

from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn import svm

### Create function to evaluate the score of each classification model
def eval_model_classifier(model, data, target, split_ratio):
    trainX, testX, trainY, testY = train_test_split(data, target, train_size=split_ratio, random_state=0)
    model.fit(trainX, trainY)    
    return model.score(testX,testY)
num_estimators_array = np.array([1,5,10,50,100,200,500]) 
num_smpl = 5 # Test run the model according to samples_number
num_grid = len(num_estimators_array)
score_array_mu = np.zeros(num_grid) # Keep mean
score_array_sigma = np.zeros(num_grid) # Keep Standard deviation 
j=0

# print("{}: RandomForestClassification Starts!".format(now()))
for n_estimators in num_estimators_array:
    score_array = np.zeros(num_smpl) # Initialize
    for i in range(0,num_smpl):
        rf_class = RandomForestClassifier(n_estimators = n_estimators, n_jobs=1, criterion="gini")
        score_array[i] = eval_model_classifier(rf_class, data, label, 0.8)
#         print("{}: Try {} with n_estimators = {} and score = {}".format(now(), i, n_estimators, score_array[i]))
    score_array_mu[j], score_array_sigma[j] = np.mean(score_array), np.std(score_array)
    j=j+1

# print("{}: RandomForestClassification Done!".format(now()))
fig = plt.figure(figsize=(7,3))
plt.errorbar(num_estimators_array, score_array_mu, yerr=score_array_sigma, fmt='k.-')
plt.xscale("log")
plt.xlabel("number of estimators",size = 16)
plt.ylabel("accuracy",size = 16)
plt.xlim(0.9,600)
plt.ylim(0.3,0.8)
plt.title("Random Forest Classifier", size = 18)
plt.grid(which="both")
plt.show();<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14296603709663758, 'fraction_numerical': 0.056407216058270516, 'mean_word_length': 5.2976, 'pattern_counts': {'":': 0, '<': 47, '<?xml version=': 0, '>': 47, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '621899', 'license': 'MIT', 'n_tokens_mistral': 4799, 'n_tokens_neox': 4266, 'n_words': 991, 'path': 'Try1.ipynb', 'repo_name': 'Invince11/Loan-Prediction_with-Visualizations', 'size': '11807'}