<jupyter_start><jupyter_text>Problem Description:Prepare a Machine Learning Model to predict the Persistency 13M Payment Behaviour at the New Business stage. Objective:Using Machine Learning techniques, provide scores for each policy at the New Business stage the likelihood to pay the 13M premium.Identify the segments where maximum non payers are captured. Dataset:“Training” & “Test” Dataset with the raw input attributes and the 13M actual paid/not paid flag.“Out of Time” Datasets would be provided with just the raw input attributes. Expected Steps: Conduct appropriate Data Treatments for e.g. Missing Value Imputation, Outlier treatment etc.  Conduct required Feature Engineering for e.g. Binning, Ratio, Interaction, Polynomial  etc.  Use any machine learning algorithm or combination of machine learning algorithms you deem fit.  Prepare your model on the Train Data and you can evaluate the generalization capability of your model by using K-Fold Cross Validation, Leave One Out.  Cross Validation or any other validation technique that you see appropriate.  Score the Test and Out of Time Data and share it back to us along with the scored Train Data for evaluation. Also share all the Model Codes.  1. Import the Dependencies<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import pickle
import warnings
from patsy import dmatrices                
import sklearn
import statsmodels.api as sm
from scipy import stats
from scipy.special import factorial
from sklearn import preprocessing
from sklearn.preprocessing import StandardScaler 
%matplotlib inline
sns.set(style='darkgrid')
warnings.filterwarnings('ignore')<jupyter_output><empty_output><jupyter_text>2. Load the datasetBy giving a parameter value for the file name (e.g. pd. read csv("filename. csv")), we may import tabular data from CSV files into pandas dataframes using the pandas package's read csv() method. Remember that we gave pandas an alias (pd), so will'll be calling pandas functions with pd.<jupyter_code>df_train = pd.read_csv('E:/PIP_Devmallya/Week_1/Problem_2/Final_data.csv')
df_train.head()
df_train.columns
df_train.info()
df_train.std()<jupyter_output><empty_output><jupyter_text>3. Label Encoding* Label encoding is the process of translating labels into a numeric format so that they may be read by machines. Machine learning algorithms can then make better decisions about how those labels should be used. In supervised learning, it is a crucial pre-processing step for the structured dataset.* Sklearn is an extremely efficient technique for encoding categorical feature levels into numeric values. LabelEncoder encodes labels with values ranging from 0 to n classes-1, where n is the number of different labels. If a label is repeated, it is allocated the same value as before.* one hot encoding = A one-hot encoding provides for more expressive categorical data representation. Many machine learning algorithms are unable to operate directly with categorical data. The categories must be numerically transformed. This is necessary for categorical input and output variables.<jupyter_code>label_encoder = preprocessing.LabelEncoder()
df_train['Auto Debit of Premium Opted Flag(Yes/No)']= label_encoder.fit_transform(df_train['Auto Debit of Premium Opted Flag(Yes/No)']) 
df_train['Auto Debit of Premium Opted Flag(Yes/No)'].unique() 
df_train['Application  Sourcing Channel']= label_encoder.fit_transform(df_train['Application  Sourcing Channel']) 
df_train['Application  Sourcing Channel'].unique()
df_train['Application  Life Assured City']= label_encoder.fit_transform(df_train['Application  Life Assured City']) 
df_train['Application  Life Assured City'].unique()
df_train['Application  Life Assured City Tier']= label_encoder.fit_transform(df_train['Application  Life Assured City Tier']) 
df_train['Application  Life Assured City Tier'].unique() 
df_train['Application  Life Assured Education']= label_encoder.fit_transform(df_train['Application  Life Assured Education']) 
df_train['Application  Life Assured Education'].unique() 
df_train['Application  Life Assured Gender']= label_encoder.fit_transform(df_train['Application  Life Assured Gender']) 
df_train['Application  Life Assured Gender'].unique()
df_train['Application  Life Assured Industry']= label_encoder.fit_transform(df_train['Application  Life Assured Industry']) 
df_train['Application  Life Assured Industry'].unique()
df_train['Application  Life Assured Marital Status']= label_encoder.fit_transform(df_train['Application  Life Assured Marital Status']) 
df_train['Application  Life Assured Marital Status'].unique() 
df_train['Application  Life Assured Occupation']= label_encoder.fit_transform(df_train['Application  Life Assured Occupation']) 
df_train['Application  Life Assured Occupation'].unique() 
df_train["Applicant's Policy PAR/NON PAR/ULIP Tag"]= label_encoder.fit_transform(df_train["Applicant's Policy PAR/NON PAR/ULIP Tag"]) 
df_train["Applicant's Policy PAR/NON PAR/ULIP Tag"].unique() 
df_train['Application Specific Sourcing Partner']= label_encoder.fit_transform(df_train['Application Specific Sourcing Partner']) 
df_train['Application Specific Sourcing Partner'].unique() 
df_train['First Premium Payment Type']= label_encoder.fit_transform(df_train['First Premium Payment Type']) 
df_train['First Premium Payment Type'].unique() 
df_train["Applicant's Policy Product Category"]= label_encoder.fit_transform(df_train["Applicant's Policy Product Category"]) 
df_train["Applicant's Policy Product Category"].unique() 
df_train["Applicant's Policy Premium Payment Frequency"]= label_encoder.fit_transform(df_train["Applicant's Policy Premium Payment Frequency"]) 
df_train["Applicant's Policy Premium Payment Frequency"].unique() 
df_train["Applicant's Policy Product Name"]= label_encoder.fit_transform(df_train["Applicant's Policy Product Name"]) 
df_train["Applicant's Policy Product Name"].unique()
df_train['Application Life Assured Residential Status']= label_encoder.fit_transform(df_train['Application Life Assured Residential Status']) 
df_train['Application Life Assured Residential Status'].unique()
df_train['Application Life Assured State']= label_encoder.fit_transform(df_train['Application Life Assured State']) 
df_train['Application Life Assured State'].unique() 
df_train['Application Sourcing Sub Channel']= label_encoder.fit_transform(df_train['Application Sourcing Sub Channel']) 
df_train['Application Sourcing Sub Channel'].unique() 
df_train['HDFC Life Operational Zone']= label_encoder.fit_transform(df_train['HDFC Life Operational Zone']) 
df_train['HDFC Life Operational Zone'].unique() 
df_train['Mapped Agent city']= label_encoder.fit_transform(df_train['Mapped Agent city']) 
df_train['Mapped Agent city'].unique() 
df_train["Applicant's Policy Contract city"]= label_encoder.fit_transform(df_train["Applicant's Policy Contract city"]) 
df_train["Applicant's Policy Contract city"].unique() 
df_train["Application  Life Assured Age"]= label_encoder.fit_transform(df_train["Application  Life Assured Age"]) 
df_train["Application  Life Assured Age"].unique()
df_train["smoker_drinker"]= label_encoder.fit_transform(df_train["smoker_drinker"]) 
df_train["smoker_drinker"].unique()
df_train = df_train.drop('Perf+vintage', axis = 1)
df_train_1 = df_train.copy()
scaler = StandardScaler()
scaler.fit(df_train_1)
df_train_1 = scaler.transform(df_train_1)<jupyter_output><empty_output><jupyter_text>4. Principal Component Analysis Model* Principal component analysis (PCA) is a technique used to reduce the dimensionality of such datasets, boosting interpretability while minimising information loss. It accomplishes this by generating new uncorrelated variables that gradually optimise variance.* Principal Component Analysis, or PCA, is a dimensionality-reduction approach that is frequently used to decrease the dimensionality of big data sets by reducing a large collection of variables into a smaller set that retains the majority of the information in the large set.* PCA is based on mathematics.  * Take the entire dataset with d+1 dimensions and remove the labels, resulting in a d dimensional dataset.  * Calculate the mean for each dimension of the whole dataset.  * Calculate the covariance matrix for the entire dataset.  * Compute the eigenvectors and eigenvalues.* Principal components are formed in the order in which they capture the most variance: PC1 catches the most variation, PC2 captures the second most, and so on. Each of them provides some data information, and there are as many principle components as there are features in a PCA.<jupyter_code>from sklearn.decomposition import PCA
pca = PCA(n_components=4)
pca.fit(df_train_1)
x_pca = pca.transform(df_train_1)<jupyter_output><empty_output><jupyter_text>The transform technique transforms all of the characteristics based on their mean and variance. Now, we want scaling to be applied to our test data as well, but we don't want our model to be biassed. We want our test data to be an entirely fresh and unexpected collection for our model.<jupyter_code>plt.figure(figsize=(15,8))
plt.scatter(x_pca[:,0],x_pca[:,1],c=df_train['Target'],cmap='plasma')
plt.xlabel('First principal component')
plt.ylabel('Second Principal Component')
df_train_comp = pd.DataFrame(pca.components_, columns = df_train.columns)
df_train_comp.head()
plt.figure(figsize=(15,8))
sns.heatmap(df_train_comp, cmap='plasma',)
df_train_2 = df_train_comp.T
df_train_2.columns = ['PC1', 'PC2', 'PC3', 'PC4']
df_train_2<jupyter_output><empty_output><jupyter_text>We are observing four PCA components<jupyter_code>pca.explained_variance_
pca.explained_variance_ratio_*100
df_train_2.head()
df_train_3 = df_train_2.T
df_train_3
print('Index which gives maximum value from a target function:', df_train_3['Mapped Agent 13M Persistency'].argmax())<jupyter_output>Index which gives maximum value from a target function: 2<jupyter_text>In index 2 we can see that PC3 gives maximum value from a target function<jupyter_code>plt.figure(figsize=(15,8))
plt.plot([1,2,3,4], pca.explained_variance_ratio_, '-o')
plt.ylabel('Proportion of Variance Explained')
plt.xlabel('Principal Component')
plt.xlim(0.75,4.25)
plt.ylim(0,0.15)
plt.xticks([1,2,3,4])<jupyter_output><empty_output><jupyter_text>4.1 KMeansThe K-means clustering technique is used to detect groupings in data that have not been explicitly categorised. This may be used to validate business assumptions about the sorts of groups that exist or to find unknown groups in large data sets.<jupyter_code>from sklearn.cluster import KMeans
k = 3
clusters = KMeans(k, random_state = 42 )
clusters.fit(df_train_2)
df_train_2['Cluster'] = clusters.labels_
df_train_2
df_train_2[df_train_2.Cluster == 0]
df_train_2[df_train_2.Cluster == 1]
df_train_2[df_train_2.Cluster == 2]
k = 5
clusters = KMeans( k, random_state = 42 )
clusters.fit(df_train_1)
y = pd.DataFrame(clusters.labels_)
y.column = 'Cluster'
df_2 = df_train.copy()
df_2['Cluster'] = y
Cluster_1 = df_2[df_2.Cluster == 0]
Cluster_2 = df_2[df_2.Cluster == 1]
Cluster_3 = df_2[df_2.Cluster == 2]
Cluster_4 = df_2[df_2.Cluster == 3]
Cluster_5 = df_2[df_2.Cluster == 4]
Cluster_1['Target'].value_counts()
Cluster_2['Target'].value_counts()
Cluster_3['Target'].value_counts()
Cluster_4['Target'].value_counts()
Cluster_5['Target'].value_counts()<jupyter_output><empty_output><jupyter_text>4.2 KModesKModes clustering is an unsupervised Machine Learning technique for grouping categorical data. KMeans clusters continuous data using mathematical metrics (distance). The closer our data points are together, the shorter the distance. Means is responsible for keeping centroids up to date.<jupyter_code>from kmodes.kmodes import KModes
km_cao = KModes(n_clusters=5, init = "Cao", n_init = 1, verbose=1)
fitClusters_cao = km_cao.fit_predict(df_train)
fitClusters_cao
clusterCentroidsDf = pd.DataFrame(km_cao.cluster_centroids_)
clusterCentroidsDf.columns = df_train.columns<jupyter_output><empty_output><jupyter_text>The centre of the clusters may be calculated by dividing the total by the number of cluster members. In the above example, 283 divided by four equals 70.75, and 213 divided by four equals 53.25, indicating that the cluster's centroid is (70.75, 53.25)<jupyter_code>clusterCentroidsDf
km_huang = KModes(n_clusters=5, init = "Huang", n_init = 1, verbose=1)
fitClusters_huang = km_huang.fit_predict(df_train)
fitClusters_huang
clusterCentroidsDf = pd.DataFrame(km_huang.cluster_centroids_)
clusterCentroidsDf.columns = df_train.columns
clusterCentroidsDf<jupyter_output><empty_output><jupyter_text>Seperating out the columns on the basis of paid and not paid. Which column is more convinient on paid or not paid<jupyter_code>NP = df_train[df_train['Target'] == 0]
P = df_train[df_train['Target'] == 1]
NP = NP.drop('Target',axis = 1)
P = P.drop('Target',axis = 1)
df_train.columns
a = []
b = []
for i in range(len(P.columns)):
        a.append(NP.iloc[:,i].mean())
for j in range(len(P.columns)):
        b.append(P.iloc[:,j].mean())
dx = pd.DataFrame()
len(a)
dx['NP'] = a
dx['P'] = b
dx.index = P.columns
dx
dx['diff'] = dx['P'] - dx['NP']
dx['diff'] = (dx['diff'] - dx['diff'].min())/(dx['diff'].max() - dx['diff'].min())
dx['diff']
dx['diff'].plot(kind = 'bar', figsize = (15,8) )
plt.xticks(fontsize=20)
plt.yticks(fontsize=20)
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09498582301149082, 'fraction_numerical': 0.012162363826294582, 'mean_word_length': 6.729527104959631, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 44, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '607746', 'license': 'MIT', 'n_tokens_mistral': 4136, 'n_tokens_neox': 3923, 'n_words': 1308, 'path': '13M Payment Behaviour Principal Component Analysis on Preprocessed Data (1).ipynb', 'repo_name': 'DevmallyaK/13M-Payment-Behaviour', 'size': '13402'}