<jupyter_start><jupyter_text>Capstone Project: Create a Customer Segmentation Report for Arvato Financial ServicesIn this project, you will analyze demographics data for customers of a mail-order sales company in Germany, comparing it against demographics information for the general population. You'll use unsupervised learning techniques to perform customer segmentation, identifying the parts of the population that best describe the core customer base of the company. Then, you'll apply what you've learned on a third dataset with demographics information for targets of a marketing campaign for the company, and use a model to predict which individuals are most likely to convert into becoming customers for the company. The data that you will use has been provided by our partners at Bertelsmann Arvato Analytics, and represents a real-life data science task.If you completed the first term of this program, you will be familiar with the first part of this project, from the unsupervised learning project. The versions of those two datasets used in this project will include many more features and has not been pre-cleaned. You are also free to choose whatever approach you'd like to analyzing the data rather than follow pre-determined steps. In your work on this project, make sure that you carefully document your steps and decisions, since your main deliverable for this project will be a blog post reporting your findings.<jupyter_code>!pip install scikit-plot
# import libraries here; add more as necessary


import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import math
from time import time
from sklearn.preprocessing import Imputer
import copy
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA


from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor, GradientBoostingClassifier, RandomForestClassifier
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import AdaBoostRegressor, AdaBoostClassifier
from sklearn.linear_model import  BayesianRidge
from sklearn.model_selection import GridSearchCV
from sklearn.linear_model import LogisticRegression
from sklearn import datasets
from sklearn.metrics import r2_score 
from sklearn.ensemble import BaggingClassifier
from sklearn.neighbors import KNeighborsClassifier
from scikitplot.metrics import plot_roc_curve
from sklearn.tree import DecisionTreeClassifier
from joblib import dump, load
from sklearn.metrics import accuracy_score
from sklearn.tree import DecisionTreeClassifier
from sklearn.cluster import KMeans

# import scikitplot as skplt
# skplt.metrics

# magic word for producing visualizations in notebook
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Part 0: Get to Know the DataThere are four data files associated with this project:- `Udacity_AZDIAS_052018.csv`: Demographics data for the general population of Germany; 891 211 persons (rows) x 366 features (columns).- `Udacity_CUSTOMERS_052018.csv`: Demographics data for customers of a mail-order company; 191 652 persons (rows) x 369 features (columns).- `Udacity_MAILOUT_052018_TRAIN.csv`: Demographics data for individuals who were targets of a marketing campaign; 42 982 persons (rows) x 367 (columns).- `Udacity_MAILOUT_052018_TEST.csv`: Demographics data for individuals who were targets of a marketing campaign; 42 833 persons (rows) x 366 (columns).Each row of the demographics files represents a single person, but also includes information outside of individuals, including information about their household, building, and neighborhood. Use the information from the first two files to figure out how customers ("CUSTOMERS") are similar to or differ from the general population at large ("AZDIAS"), then use your analysis to make predictions on the other two files ("MAILOUT"), predicting which recipients are most likely to become a customer for the mail-order company.The "CUSTOMERS" file contains three extra columns ('CUSTOMER_GROUP', 'ONLINE_PURCHASE', and 'PRODUCT_GROUP'), which provide broad information about the customers depicted in the file. The original "MAILOUT" file included one additional column, "RESPONSE", which indicated whether or not each recipient became a customer of the company. For the "TRAIN" subset, this column has been retained, but in the "TEST" subset it has been removed; it is against that withheld column that your final predictions will be assessed in the Kaggle competition.Otherwise, all of the remaining columns are the same between the three data files. For more information about the columns depicted in the files, you can refer to two Excel spreadsheets provided in the workspace. [One of them](./DIAS Information Levels - Attributes 2017.xlsx) is a top-level list of attributes and descriptions, organized by informational category. [The other](./DIAS Attributes - Values 2017.xlsx) is a detailed mapping of data values for each feature in alphabetical order.In the below cell, we've provided some initial code to load in the first two datasets. Note for all of the `.csv` data files in this project that they're semicolon (`;`) delimited, so an additional argument in the [`read_csv()`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html) call has been included to read in the data properly. Also, considering the size of the datasets, it may take some time for them to load completely.You'll notice when the data is loaded in that a warning message will immediately pop up. Before you really start digging into the modeling and analysis, you're going to need to perform some cleaning. Take some time to browse the structure of the data and look over the informational spreadsheets to understand the data values. Make some decisions on which features to keep, which features to drop, and if any revisions need to be made on data formats. It'll be a good idea to create a function with pre-processing steps, since you'll need to clean all of the datasets before you work with them.<jupyter_code>%%time
# load in the data
azdias = pd.read_csv('../../data/Term2/capstone/arvato_data/Udacity_AZDIAS_052018.csv', sep=';')
customers = pd.read_csv('../../data/Term2/capstone/arvato_data/Udacity_CUSTOMERS_052018.csv', sep=';')
# Load file description files 
azdisas_attrs = pd.read_excel('DIAS Information Levels - Attributes 2017.xlsx')
azdisas_attrs_values = pd.read_excel('DIAS Attributes - Values 2017.xlsx') 
# Print the head of the azdias dataset
azdias.head()
customers.head()<jupyter_output><empty_output><jupyter_text>1 Data Wrangling and Data CleaningHere will take deep dive into the data to see insights about the data set. Main Steps here will include. 1.1 Null Columns and Rows - Identify Null columns  - Create a dataframe showing propostion of null with percentages alongside field description - Plot a visualisation of the propotions of null in each column. - Select and drop all null rows and coloumns  - Drop all null columns that pass a threshold value of 65% of nulls<jupyter_code># Describe our data
azdias.describe()
# Identify Null coloumns
azdias.isnull().sum()<jupyter_output><empty_output><jupyter_text>From the output above we see there are alot of columns with null.Below we will get the same results but this time we put results into a Data Frame with other fields  > | Column Name |  Non null rows | No Value Preset | Percentage Null | Coloumn Desciption<jupyter_code>def get_nullpercentage(df, desc_df):
    # Iterate through column and create details
    col_details = []
    n_rows = df.shape[0]

    for index, col in enumerate(df.columns):
        null_count = df[col].isnull().sum()
        percentage_nulls = math.floor(((null_count/n_rows) * 100) * 10)/10
        non_nulls = n_rows - null_count

        col_details.append((col, non_nulls, null_count, percentage_nulls))

    # Create Dataframe and from list

    col_details_df = pd.DataFrame(col_details, columns=['Column Name', 'Non Null Rows', 'Null Rows', 'Percentage Null'])

    # Use the **azdisa_attrs** Data frame in this function 'desc_df' code cell 3 to get the column Information and concat to col_details_df
    
    col_description = desc_df['Description']
    col_details_df = pd.concat([col_details_df, pd.Series(col_description)], axis=1)
    col_details_df = col_details_df.sort_values(by='Percentage Null', ascending=False)

    return col_details_df

def plot_nulls(df, x, y,title, threshold=False, limit=None):
    '''
    Description: Plot a bar chart from the given points
    
    Parameters:
        df : Dataframe
        x : label or position
        y : label, position or list of label, positions, default None
        title: Title of the Plot, default to 'x agains y'
        threshold: boolean, defauls none
        limit: Number of points
    '''
   
    df = df[:limit]
    
    df.plot(kind='bar', 
                x=x, 
                y=y,
                title = title,
                figsize=(20,10), 
                fontsize=14,
                sort_columns=True,
                mark_right=True);

col_details_df = get_nullpercentage(azdias, azdisas_attrs)
col_details_df.head()<jupyter_output><empty_output><jupyter_text>Plot the visualization of the nulls in a histogram<jupyter_code>nullcol_df = col_details_df[col_details_df['Percentage Null'] > 0]
plot_nulls(nullcol_df, x='Column Name', y='Percentage Null',title='Column Null Percentage',threshold=True, limit=40 )<jupyter_output><empty_output><jupyter_text>From the the visualization it can be seen that 6 coloumns have null values more that 60%. We have to look into this coloumns an see if they are important to our anysis other wise they lead to un nucessary Dimmentionality and could caouse overfiting, adding more features that extend our model into more outiers.  Next Step - We look into the 6 coloumns and validate if we drop them or not - Drop Approriate null coloumns  - Drop Null rows<jupyter_code>nullcols = nullcol_df[nullcol_df['Percentage Null'] > 60.0]['Column Name']
nullcol_df[nullcol_df['Column Name'].isin(nullcols)]<jupyter_output><empty_output><jupyter_text>We look into the customer DataFrame to see if this columns still maintain as the coloumns with the must nulls, if the columns still have above 60% null values then we conclude this is not so important for cutomer convertion<jupyter_code>cusotmer_null_df = get_nullpercentage(customers, azdisa_attrs)

cusotmer_null_df[cusotmer_null_df['Column Name'].isin(nullcols)]<jupyter_output><empty_output><jupyter_text>We See this fields still have alot of null value, in the customer dataset, so we conclude this fields can be removed as they may not be so import and also increase dimmensionlity. NextRemove All null columns with more that 60% nulls from azdisa dataframe and visualise new dataframe's null and non null count for coloumns<jupyter_code>%%time
azdias_new = azdias.drop(azdias.columns[azdias.isnull().sum()/azdias.shape[0] > .6], axis=1)
newdf_detail = get_nullpercentage(azdias_new, azdisa_attrs)
plot_nulls(newdf_detail, x='Column Name', y=['Non Null Rows', 'Null Rows'],title='Column Null Percentage',threshold=True, limit=40 )
print('Number of Columns reduced from {} in oraginal Dataset to {} after removing null columns beyond treshol'.format(azdias.shape[1], azdias_new.shape[1]))<jupyter_output><empty_output><jupyter_text>We See the new Data Frame has far less more less null coloumns than the values present and number of columns has reduced to 360 from 366 in oraginal dataframe. NextDrop Null Rows<jupyter_code>%%time
# Find Null Rows  .hist(bins=20)
azdias_new.isnull().sum(axis=1).plot(kind='hist')<jupyter_output>CPU times: user 7.58 s, sys: 4.22 s, total: 11.8 s
Wall time: 1min 26s<jupyter_text>From the histogram we find rows missing up to 250 columns, we will consider columns with more than 30% of missen values as outliers and remove them from the data set as we assume them not to have a high weightage.Consideration where influenced from https://analyticsindiamag.com/5-ways-handle-missing-values-machine-learning-datasets/  Next Step    - Remove rows with up to 30% missen column values.<jupyter_code>%%time
# Threshold her is 70% meaning we require atleast 70 of the fields coloumns for a row to be null to delete.
azdias_rnullrow = azdias_new.dropna(thresh=math.floor(70/100 * azdias_new.shape[1]), axis=0)

print('Number of rows before and after deleting rows')
print('Before: {}'.format(azdias_new.shape[0]))
print('After: {}'.format(azdias_rnullrow.shape[0]))
%%time
# Find Null Rows  .hist(bins=20)
azdias_rnullrow.isnull().sum(axis=1).plot(kind='hist')
azdias_rnullrow.head()
# azdias_rnullrow['type'].value_counts()
# Drop all rows having all fields as null
# azdias_remove_nrow = azdias_new.dropna(how='all', axis=0)
azdias_rnullrow.shape, azdias_new.shape<jupyter_output><empty_output><jupyter_text>1.2 Handing Corrolated Columns Having corolated columns increases dimmentionality and hence makes producing a sparce matrix, probably increasing over feeting, which can be diffiecult to analyse. Steps her include.        - Generate a corrolation Matrix our data    - User Sns to plot a heat map showing corrolation     - Find and remove corrolated Columns<jupyter_code># azdias_df = azdias_remove_nrow
%%time
# Generate Corrolation Matrix
azdias_corrmatix = azdias_rnullrow.corr()
%%time
#plot the Collolatino matrix with sns
# sns.heatmap(azdias_corrmatix, annot = True, vmin=-1, vmax=1, center= 0)
# Removing Highly corrolated Matric take the absolute value
azdias_corrmatix = azdias_corrmatix.abs()

# Create a True/False mask and apply it
mask = np.triu(np.ones_like(azdias_corrmatix, dtype=bool))

tri_df = azdias_corrmatix.mask(mask)

# List column names of highly correlated features (r > 0.97)
to_drop = [c for c in tri_df.columns if any(tri_df[c] >  0.97)]

# Drop the features in the to_drop list
reduced_azdf = azdias_rnullrow.drop(to_drop, axis=1)

print("Data Frame Columns Before and after removing corrolated columns")
print("Before: {} ".format(azdias_rnullrow.shape[1]))
print("After: {} ".format(reduced_azdf.shape[1]))
# Write reduced dataset to csv file 
reduced_azdf.to_csv('reduced_azd.csv')<jupyter_output><empty_output><jupyter_text>Data Imputaion After the above data cleaning procedurs we end upwith 6 categoric variable, which will be handles Nan's now. Steps     - Create new dataframe with categorical variable to handle Nan's    - Delete Categorical variables from oraginal dataframe    - Perform back and forward filling respectively on category dataframes     - Handle Types for fields descibed below     - Extract month and year from EINGEFUEGT_AM and create 2 new dataframes     - Use sklearn's Imputer class on oraginal Dataframe to do imputation of missind Nan Values for numric colums using the Column mean.    - Concat Numeric and Cagegorical columns.Remove Categorical variable handle nans and merge concat back to oraginal df.CAMEO_DEU_2015  - Feature describing German Topology for specific groups  Action:   - For missing values Nan's in this coloumn will be replaced by XX to signify unknown   - This field will be kept the same.CAMEO_DEUG_2015- Ordinal Feature describing German Topology for the uppergroup- After doing a value count on this variable we notice some variable numericaly will be same but string representation will not, like 8.0 and 8 which definitly refer to the same category Action:  - For missing values Nan's in this column will be replaced with -1 for unknown - Replace X with -1, they are both equivalent to unknown - Convert field to string represention of int, converting all floats to int  CAMEO_INTL_2015 - An Ordinal Feature which defined German Topology in an international scale.  - After doing a value count on this variable we notice some variable numericaly will be same but string representation will not,  like 50.0 and 50 which definitly refer to the same categoryAction: - For missing values Nan's in this coloumn will be replaced with -1 for unknown. - Replace XX with -1, they are both equivalent to unknown - Convert field to string represention of int, converting all floats to int . EINGEFUEGT_AM - A Date feature - Notice the time is already in timestamp Action:   - Extract month and year and create 2 new columns for year and month.D19_LETZTER_KAUF_BRANCHE    After Alot of diging there was not much mata data about this feature from both attribute information and atribute value excel files about this field.  Action: - This field contain over 175157 out or 785421 rows, to handle this nan I choose to use foward and back filling respectivery after carefully observing the data set as opposed to taking just the mean and avoiding bias the mean after this back and foward filling  moves from 23.9% to 30.9% wich accounts for a majority null filling process s<jupyter_code>%%time
# Seleect all Categorical Variables from data set.
categorical_cols = reduced_azdf.select_dtypes(include=['object'])
categorical_cols.head(20)
category_names = categorical_cols.columns
%%time
# For all categorical variable print the count the possibilities in each varianble
print("Column | Categories")
for col in categorical_cols.columns:
    number_categories = categorical_cols[col].value_counts().size
    print("{} | {}".format(col, number_categories))
%%time
# Drope Categorical columns from oraginal dataframe 
reduced_df = reduced_azdf.drop(columns=category_names, axis=1)
%%time
# Fill Nan variable with  XX for CAMEO_DEU_2015 and -1 for CAMEO_DEUG_2015 and CAMEO_INTL_2015 columns 
values = {'CAMEO_DEU_2015':'XX', 'CAMEO_DEUG_2015': '-1', 'CAMEO_INTL_2015': '-1'}
categorical_cols = categorical_cols.fillna(value=values)
%%time
# Fill X values of variable CAMEO_DEUG_2015 with -1 with represents the unknows values
categorical_cols.loc[categorical_cols['CAMEO_DEUG_2015'] == 'X'] = '-1'



# Fill XX values of variable CAMEO_INTL_2015 with -1 with represents the unknows values
categorical_cols.loc[categorical_cols['CAMEO_INTL_2015'] == 'XX'] = '-1'
%%time
# Perform forward and back filling on column D19_LETZTER_KAUF_BRANCHE
categorical_cols['D19_LETZTER_KAUF_BRANCHE'] = categorical_cols['D19_LETZTER_KAUF_BRANCHE'].fillna(method='ffill').fillna(method='bfill')
# Check for any XX values in CAMEO_DEUG_2015 categorical fields

categorical_cols[categorical_cols['CAMEO_DEUG_2015'] == 'X']
# Check for any XX values in CAMEO_INTL_2015 categorical fields
categorical_cols[categorical_cols['CAMEO_INTL_2015'] == 'XX']
# Check for any null categorical fields
# categorical_cols.CAMEO_INTL_2015.value_counts()
cat = copy.deepcopy(categorical_cols)
# cat['EINGEFUEGT_AM'] = pd.Series(categorical_cols['EINGEFUEGT_AM'].values.astype('datetime64[D]'))
y = categorical_cols['EINGEFUEGT_AM'].value_counts().reset_index()
cat.isnull().sum()<jupyter_output><empty_output><jupyter_text>Above we have replace all Nan's from  categorical variables Next - Convert field to string represention of int, converting all floats to int  - Extract month and year and create 2 new columns for year and month<jupyter_code>def topology_float_to_int_type_con(string):
    converted_float =  str(round(int(float(string)))) 
    return converted_float
# Convert all string variable with numeric value af float to int for 'CAMEO_DEUG_2015' and 'CAMEO_INTL_2015'
col_list = ['CAMEO_DEUG_2015', 'CAMEO_INTL_2015']
for col in col_list:
    print('Doing String conversion for {}'.format(col))
    categorical_cols[col] = categorical_cols[col].apply(topology_float_to_int_type_con)
print('All Float and Int equvalent whole numbers converted to string ')
# categorical_cols['CAMEO_INTL_2015'].value_counts()
%%time
# categorical_cols['EINGEFUEGT_AM'].apply(lambda st: pd.to_datetime(st))

# Encounterd a problem of having strings of different formats long, short and missen string encoded as -1 solution from here
# https://stackoverflow.com/questions/55440145/pd-to-datetime-valueerror-given-date-string-not-likely-a-datetime

date_long = pd.to_datetime(categorical_cols['EINGEFUEGT_AM'],format='%Y-%m-%d %H:%M:%S', errors='coerce')
date_short = pd.to_datetime(categorical_cols['EINGEFUEGT_AM'],format='%Y-%m-%d', errors='coerce')

# Combine the short dates to long dates
categorical_cols['EINGEFUEGT_AM']= date_long.combine_first(date_short)


categorical_cols['EINGEFUEGT_AM_Year'] = categorical_cols['EINGEFUEGT_AM'].dt.year
categorical_cols['EINGEFUEGT_AM_Month'] = categorical_cols['EINGEFUEGT_AM'].dt.month

# Fill Nan variable with  XX for CAMEO_DEU_2015 and -1 for CAMEO_DEUG_2015 and CAMEO_INTL_2015 columns 
values = {'EINGEFUEGT_AM':'-1', 'EINGEFUEGT_AM_Year': '-1', 'EINGEFUEGT_AM_Month': '-1'}
categorical_cols = categorical_cols.fillna(value=values)
categorical_cols.isnull().sum()<jupyter_output><empty_output><jupyter_text>Above we handled nan categorical variable and seprated EINGEFUEGT_AM columns into EINGEFUEGT_AM_Year       EINGEFUEGT_AM_Month columns  Next     - Handle Nan's for Numeric variables, use sklearn's Imputer class for this.<jupyter_code>reduced_df.shape
%%time
# Use sklearn's Imputer class on oraginal Dataframe to do imputation of missind Nan Values for numric colums using the Column mean
imp = Imputer(missing_values=np.nan, strategy='mean')
imp.fit(reduced_df)
reduced_dfnew = imp.transform(reduced_df)
%%time
reduced_azdf = pd.DataFrame(reduced_dfnew, columns=reduced_df.columns)
reduced_azdf.shape
categorical_cols.shape
# Concat two data frame together, reduced_dfnew and categorical_cols
#  pd.concat([reduced_azdf.loc[:500].reset_index(), categorical_cols[:500].reset_index()], axis=1)
azdias_filldf.head(50)<jupyter_output><empty_output><jupyter_text>Next    - Perform hot encoding to pivot categorical variables    - Save new DataFrame as reduced data frame<jupyter_code>%% time
# Fill dummies
azdias_df = pd.get_dummies(azdias_filldf) 
%% time
# Write reduced dataset to csv file 
azdias_df.to_csv('azdias_df.csv')
def topology_float_to_int_type_con(string):
    '''
    Description: Convert float to interget and then to string
    
    Parameters: 
        string: String to convert
    '''
    
    converted_float =  str(round(int(float(string)))) 
    return converted_float

def clean_data(df, droprows=True, scale=True):
    '''
    Desciption:
        Takes a data frame cleans it and produced a cleaned dataframe
        
    Parameters:
        df : Dataframe
        droprows: Drop null rows or not, defauls to True
        scale: Scale resulting dataframe or not, defaults to True
    
    Output:
        cleaned_df: Cleand Data Frame
    '''
    print('Cleaning DataFrame')
    print('Drop any column with more than 60% Missen Values')
    df = df.drop(df.columns[df.isnull().sum()/df.shape[0] > .6], axis=1)
    print('Done')

    if droprows:
        # Threshold her is 70% meaning we require atleast 70 of the fields coloumns for a row to be null to delete.
        print('Drop any row with more than 70% Missen Values')
        df = df.dropna(thresh=math.floor(70/100 * df.shape[1]), axis=0)

    # Generate Corrolation Matrix
    df_corrmatix = df.corr()
    print('Calculating Corrolation')

    print('Removing Corolation matrix')
    # Removing Highly corrolated Matric take the absolute value
    azdias_corrmatix = df_corrmatix.abs()

    # Create a True/False mask and apply it
    mask = np.triu(np.ones_like(azdias_corrmatix, dtype=bool))

    tri_df = azdias_corrmatix.mask(mask)

    print('Drop Columns with corolation > .97')
    # List column names of highly correlated features (r > 0.97)
    to_drop = [c for c in tri_df.columns if any(tri_df[c] >  0.97)]

    # Drop the features in the to_drop list
    reduced_df = df.drop(to_drop, axis=1)

    print("Data Frame Columns Before and after removing corrolated columns")
    print("Before: {} ".format(df.shape[1]))
    print("After: {} ".format(reduced_df.shape[1]))


    # Data Imputation 
    
    categorical_cols = reduced_df.select_dtypes(include=['object'])
    category_names = categorical_cols.columns
    
    # Drope Categorical columns from oraginal dataframe 
    reduced_df = reduced_df.drop(columns=category_names, axis=1)
    
    # Fill Nan variable with  XX for CAMEO_DEU_2015 and -1 for CAMEO_DEUG_2015 and CAMEO_INTL_2015 columns 
    values = {'CAMEO_DEU_2015':'XX', 'CAMEO_DEUG_2015': '-1', 'CAMEO_INTL_2015': '-1'}
    categorical_cols = categorical_cols.fillna(value=values)

    # Fill X values of variable CAMEO_DEUG_2015 with -1 with represents the unknows values
    categorical_cols.loc[categorical_cols['CAMEO_DEUG_2015'] == 'X'] = '-1'


    # Fill XX values of variable CAMEO_INTL_2015 with -1 with represents the unknows values
    categorical_cols.loc[categorical_cols['CAMEO_INTL_2015'] == 'XX'] = '-1'

    # Perform forward and back filling on column D19_LETZTER_KAUF_BRANCHE
    categorical_cols['D19_LETZTER_KAUF_BRANCHE'] = categorical_cols['D19_LETZTER_KAUF_BRANCHE'].fillna(method='ffill').fillna(method='bfill')

    # Check for any XX values in CAMEO_DEUG_2015 categorical fields
    categorical_cols[categorical_cols['CAMEO_DEUG_2015'] == 'X']

    # Convert all string variable with numeric value of float to int for 'CAMEO_DEUG_2015' and 'CAMEO_INTL_2015'
    col_list = ['CAMEO_DEUG_2015', 'CAMEO_INTL_2015']

    for col in col_list:
        print('Doing String conversion for {}'.format(col))
        categorical_cols[col] = categorical_cols[col].apply(topology_float_to_int_type_con)
    print('All Float and Int equvalent whole numbers converted to string ')

    # Encounterd a problem of having strings of different formats long, short and missen string encoded as -1 solution from here
    # https://stackoverflow.com/questions/55440145/pd-to-datetime-valueerror-given-date-string-not-likely-a-datetime

    ## I will drop this coloumns 
#     date_long = pd.to_datetime(categorical_cols['EINGEFUEGT_AM'],format='%Y-%m-%d %H:%M:%S', errors='coerce')
#     date_short = pd.to_datetime(categorical_cols['EINGEFUEGT_AM'],format='%Y-%m-%d', errors='coerce')

#     # Combine the short dates to long dates
#     categorical_cols['EINGEFUEGT_AM']= date_long.combine_first(date_short)


#     categorical_cols['EINGEFUEGT_AM_Year'] = categorical_cols['EINGEFUEGT_AM'].dt.year
#     categorical_cols['EINGEFUEGT_AM_Month'] = categorical_cols['EINGEFUEGT_AM'].dt.month
#     categorical_cols['EINGEFUEGT_AM_Day'] = categorical_cols['EINGEFUEGT_AM'].dt.day
    
#     # Drope EINGEFUEGT_AM column as its values have already been captured above into year, month and day
    
    categorical_cols = categorical_cols.drop(columns='EINGEFUEGT_AM', axis=1)
    
    # Fill Nan variable with  XX for CAMEO_DEU_2015 and -1 for CAMEO_DEUG_2015 and CAMEO_INTL_2015 columns 
    
#     values = {'EINGEFUEGT_AM_Year':'-1', 'EINGEFUEGT_AM_Month': '-1', 'EINGEFUEGT_AM_Day': '-1'}
#     categorical_cols = categorical_cols.fillna(value=values)
    
#     categorical_cols = categorical_cols.drop(columns=['EINGEFUEGT_AM_Year', 'EINGEFUEGT_AM_Month', 'EINGEFUEGT_AM_Day'], axis=1)
   
#     categorical_cols = categorical_cols.drop(columns=['EINGEFUEGT_AM_Year'], axis=1)

    print("Replacing Nan with mean for numeirc variables")
    # Use sklearn's Imputer class on oraginal Dataframe to do imputation of missind Nan Values for numric colums using the Column mean
    imp = Imputer(missing_values=np.nan, strategy='mean')
    reduced_matrix = imp.fit_transform(reduced_df)
    reduced_df = pd.DataFrame(reduced_matrix, columns=reduced_df.columns)
    
    categorical_cols = pd.get_dummies(categorical_cols) 
    
    reduced_df = reduced_df.reset_index(drop=True)
#     categorical_cols = categorical_cols.reset_index(drop=True)
    
    for col in categorical_cols.columns:
        reduced_df[col] = categorical_cols[col].reset_index(drop=True)
    
#     cleaned_df = pd.concat([reduced_df, categorical_cols], axis=1)

    if scale:
        print('Scaling DataFrame')
        scaler = StandardScaler()
        scaled_df = scaler.fit_transform(reduced_df.astype('float'))
        azdias_scaled = pd.DataFrame(scaled_df, index=reduced_df.index, columns=reduced_df.columns)
        reduced_df = azdias_scaled
        
        print('Scaling Complete')
    
    print('Cleaning Complete')
    
    reduced_df = reduced_df.drop(columns=['LNR'], axis=1)
    
    return reduced_df


# %%time
# cleaned_azdias =  clean_data(mailout_test[:10], scale=False)
# cleaned_azdias<jupyter_output><empty_output><jupyter_text>Part 1: Customer Segmentation ReportThe main bulk of your analysis will come in this part of the project. Here, you should use unsupervised learning techniques to describe the relationship between the demographics of the company's existing customers and the general population of Germany. By the end of this part, you should be able to describe parts of the general population that are more likely to be part of the mail-order company's main customer base, and which parts of the general population are less so.  Preprocessing Stage Scaling (Standard Scaling)Due to the variabiliy in our dataset it become deffficult for unsuperided learning algorithns to perfomr appropriated as they are highly sensitive to scaling and outliers to over come this we scale the dataset to a unit scale, which scaled values to a set of values that have a mean of zero and standard diviation of 1 and - 1. This scaling allows comparisin scores between different types varianbles and values contribute to a given scale when added together. For this scaling we us sklearns SVS package to perfom this scaling.<jupyter_code>%%time
cleaned_azdias =  clean_data(azdias,droprows=True, scale=True)
clean_customer = clean_data(customers,droprows=True, scale=True)

# cleaned_azdias.to_csv('cleaned_azdias.csv')
# clean_customer.tocsv('clean_customer.csv')
%%time
cleaned_azdias = pd.read_csv('cleaned_azdias.csv')
%%time
def scale_data(df):
    scaler = StandardScaler()
    scaled_data = scaler.fit_transform(df)
    scaled_df = pd.DataFrame(scaled_data)

    print('Check the that the new scaled matrix has a mean of zero and standard diviation of 1')

    # print('Mean : {}, Standard Diviation: {}'.format(np.mean(azdias_scaled ),np.std(azdias_scaled)))

#     print('Print Shape before and after scaling')
#     print('Before: {}'.format(df.shape))
#     print('After: {}'.format(df.shape))
    
    return  scaled_df

azdias_scaled = scale_data(cleaned_azdias.astype('float'))
customer_scaled = scale_data(clean_customer.astype('float'))<jupyter_output>Check the that the new scaled matrix has a mean of zero and standard diviation of 1
Print Shape before and after scaling
Before: (140371, 519)
After: (140371, 519)<jupyter_text>Reducing Dimmensionality Here we seek to reduce the variance among features of the dataset by reducing the dimensions of our dataset, this avoids the curse of dimmensionality, when the number of dimmensions in the dataset is huge or greatly inceased it leads to data sparcity. Also this also reduces the possiblity of overfitting due to adding more feartures that extend our model in new data. https://en.wikipedia.org/wiki/Principal_component_analysisFor dimmensional reducation we use the PCA in sklearn which uses an orthogonal transformation to convert a set of observations of possibly correlated variables,yielding an orthogonal basis in which different individual dimensions of the data are uncorrelated The standardisation above helps make this process efficient as PCA is very sensitive to scale.These basis vectors are called principal componentsThis resulting vector will be a vector of maximum variance Step:Determine the best n_component to use for PCAI decided to use the scaler within the clean function above to avoid running into datetype issues in scalling<jupyter_code>%%time
scaled_df, n = cleaned_azdias, 200

# def plot_dimension_bestn(scaled_df, n):
pca_model = PCA(n_components=n)
pca_model.fit_transform(scaled_df)

# print(my_model.explained_variance_)

# Print parameter returning a vector of the variance explained by each dimension
# print(my_model.explained_variance_ratio_)

# Use checksum to for monitoring change detection
# print(my_model.explained_variance_ratio_.cumsum())

# Plot the checksum to see where to get the best variance and the best component respectively

y = pca_model.explained_variance_ratio_.cumsum() * 100
# y = [0.32047581, 0.59549787, 0.80178824, 0.932976 , 1]
x = [x for x in range(len(y))]
plt.figure(figsize=(12,8))
plt.plot(x, y, color='blue', marker='3',  linewidth=2, markersize=12)
plt.xlabel("Principal Component")
plt.ylabel("Variance Explained (%)")
plt.title("Variance for n Components")


# clean_customer = clean_data(customer)
# clean_azdias = clean(azdias)

# plot_dimension_bestn(azdias_scaled, 20)<jupyter_output>CPU times: user 1min 5s, sys: 4.89 s, total: 1min 10s
Wall time: 1min 16s<jupyter_text>From the plot yet at 200 components we still hace not captured the maximum variance, SO i decided to take the component number that captures above 65% of the total variance.I will then plot a stem plot to capture the points beter<jupyter_code># Stem plot
plt.stem(x, y, use_line_collection=True)
plt.plot(x, y, color='blue', marker='3',  linewidth=2, markersize=12)
plt.xlabel('Component Number')
plt.ylabel('Explained Variance %')
plt.title('Variance For Components')
plt.show()<jupyter_output><empty_output><jupyter_text>Capturing a variance of atlest 65% places us between 75 - 150 components. I notice at 95% i was able to capture 66.5% of the total variance.  NextReduce the dimmensionlity of both clean_customer and clean_azdias then we move on to clustersing<jupyter_code>%%time
def reduce_dimensions(scaled_df, n):
    pca_model = PCA(n_components=n)
    reduced_dim = pca_model.fit_transform(scaled_df)
    
    return reduced_dim

reduce_azdias_df = reduce_dimensions(cleaned_azdias, 95)
reduce_customer_df = reduce_dimensions(clean_customer, 95)<jupyter_output>CPU times: user 39.3 s, sys: 3.46 s, total: 42.8 s
Wall time: 55.5 s<jupyter_text>K Mean Clustering. Presently our data has alot of dimmension which is difficult to see relationships between rows, we user an Unsupervised learning process here to cluster similar data points based on the feature correlation between themset into clusters with smaller data items.We user KMean will be used for clusteringFirst we user the elbow point to determin an approprate number of clusters.Finall We compare clusters for Azdias and those of customers to see pottential customersSteps;    - Determine optimal clusters by try for 25 diffent cluster numbers (Elbo Method)    - Plot Number of clusters agains score    - Determin number of clusters and generat final clusters     - Plot a scatter map to show differnt clusters<jupyter_code>%%time
def plot_kmean_distortion(df, n): 
    # We can easily run K-Means for a range of clusters using a for loop and collecting the distortions into a list

    distortions = []
    K = range(1, n)
    for k in K:
        print('Fitting for cluster {}'.format(k))
        kmeanModel = KMeans(n_clusters=k)
        kmeanModel.fit(df)
        distortions.append(kmeanModel.inertia_)

    plt.figure(figsize=(16,8))
    plt.plot(K, distortions, 'bx-')
    plt.xlabel('k')
    plt.ylabel('Distortion')
    plt.title('The Elbow Method showing the optimal k')
    plt.show()
    
plot_kmean_distortion(reduce_azdias_df, 25)<jupyter_output>Fitting for cluster 1
Fitting for cluster 2
Fitting for cluster 3
Fitting for cluster 4
Fitting for cluster 5
Fitting for cluster 6
Fitting for cluster 7
Fitting for cluster 8
Fitting for cluster 9
Fitting for cluster 10
Fitting for cluster 11
Fitting for cluster 12
Fitting for cluster 13
Fitting for cluster 14
Fitting for cluster 15
Fitting for cluster 16
Fitting for cluster 17
Fitting for cluster 18
Fitting for cluster 19
Fitting for cluster 20
Fitting for cluster 21
Fitting for cluster 22
Fitting for cluster 23
Fitting for cluster 24<jupyter_text>From the distortion above we can see that the Elbow point is betwenn 10 and 15  clusters. NextWe use cluster size as 11 to create new clusters<jupyter_code>population_dim = reduce_azdias_df
customers_dim = reduce_customer_df
n = 11
cluster_lable1 = 'Population'
cluster_lable2 = 'Customers'
disp_center = True
%%time

    
kmeanModel = KMeans(n_clusters=n)
kmeanModel.fit(population_dim)

print('Population Fit Complete')

kmeanModel2 = KMeans(n_clusters=n)
kmeanModel2.fit(customers_dim)

y_kmeans = kmeanModel.predict(population_dim)

y_kmeans2 = kmeanModel.predict(customers_dim)

# cluster_data(reduce_azdias_df,reduce_customer_df, 2, cluster_lable1='Population', cluster_lable2='Customers')<jupyter_output>Population Fit Complete
CPU times: user 2min 32s, sys: 30.8 ms, total: 2min 32s
Wall time: 2min 33s<jupyter_text>Comparing customer data to general demographic data.We have taken the customer and general demographic data through data cleaning, normalization, reduced dimmensionlity to a displaying clusters. are mulit dimmentional data frame has been recuded 15 features and now displaying on 10 not the original 11 clusters which we can use to compare the data sets.  Steps     - Plot customer and demographic data on a histogram to perform visual comparism.<jupyter_code>plt.figure(figsize=(16,8))
plt.hist([y_kmeans, y_kmeans2], label=['Population', 'Customer'])
plt.ylabel('Propotion')
plt.xlabel('Clusters')
plt.xticks([x + 1 for x in range(10)])
plt.legend(loc="upper left")
plt.show()<jupyter_output><empty_output><jupyter_text>From the Histogram plot Above we can see that both the population and customers have a high presence in some clusters compared to other clusters For use to be able to conclude on this Analysis we need to look for clusters with over all high customer presence compared to the population which give a much higher chance of getting potential customers from those clusters We notice Cluster 8 and 9 have an almost even destribution of Customers and Population and cluster 9 over all has a higher proportion of both meaning there is a possibility of having potential customers fron that cluster.But ove all cluster 5 has the highest proportion of customers compared to Population henxe this cluster will be the best in terms of a campaign for potential customers clusters 1,2, 7 nad 10 have an over all higher population presence compared  to the customer presence they may not be the best fit for potential customers except cluster 1 with a little customer presence as well relatively. cluster 4 has almost a zero customer presence meaning this cluster will not be at all any good for us in terms of customer segmentation for a campaign.<jupyter_code># %%time
# def cluster_data(population_dim, customers_dim,  n, cluster_lable1, cluster_lable2, disp_center=False): 
    
#     kmeanModel = KMeans(n_clusters=n)
#     kmeanModel.fit(population_dim)
    
#     kmeanModel2 = KMeans(n_clusters=n)
#     kmeanModel2.fit(customers_dim)

#     y_kmeans = kmeanModel.predict(population_dim)

#     y_kmeans2 = kmeanModel.predict(customers_dim)
    
    
#     fig, axes = plt.subplots(1, 3, figsize=(20,8))
#     axes[0].scatter(population_dim[:,0], population_dim[:,1], c=y_kmeans)
#     axes[1].scatter(customers_dim[:,0], customers_dim[:, 1], c=y_kmeans, cmap=plt.cm.Set1)
    
#     axes[2].hist([y_kmeans, y_kmeans2])

#     axes[0].set_title(cluster_lable1, fontsize=18)
#     axes[1].set_title(cluster_lable2, fontsize=18)

# #     plt.scatter(X[:,0], X[:, 3], c=y_kmeans, s=50, cmap='viridis')
    
#     if disp_center:
#         centers = kmeanModel.cluster_centers_
#         plt.scatter(centers[:, 0], centers[:, 0], c='black', s=200, alpha=0.5);
    
# cluster_data(reduce_azdias_df,reduce_customer_df, 2, cluster_lable1='Population', cluster_lable2='Customers')<jupyter_output><empty_output><jupyter_text>Part 2: Supervised Learning ModelNow that you've found which parts of the population are more likely to be customers of the mail-order company, it's time to build a prediction model. Each of the rows in the "MAILOUT" data files represents an individual that was targeted for a mailout campaign. Ideally, we should be able to use the demographic information from each individual to decide whether or not it will be worth it to include that person in the campaign.The "MAILOUT" data has been split into two approximately equal parts, each with almost 43 000 data rows. In this part, you can verify your model with the "TRAIN" partition, which includes a column, "RESPONSE", that states whether or not a person became a customer of the company following the campaign. In the next part, you'll need to create predictions on the "TEST" partition, where the "RESPONSE" column has been withheld.<jupyter_code>mailout_train = pd.read_csv('../../data/Term2/capstone/arvato_data/Udacity_MAILOUT_052018_TRAIN.csv', sep=';')
mailout_train.head()<jupyter_output><empty_output><jupyter_text>Step    - Look into the data set information about the new Rsponse columns     - Clean the data set and standardize  The Response column is free of nulls,  Next    - Drop RESPONSE column     - Clean the new dataset, I decided not to remove the rows <70% of information to avoid loss of information as  the dataset is really large and the margin of error is small.    Note: Cleaning process will not affect the response column since it has no nulls and no rows are being droped.<jupyter_code>response_out_df = mailout_train.drop(columns='RESPONSE', axis=1)
cleaned_mailout = clean_data(response_out_df, droprows=False, scale=True)
cleaned_mailout.head()
cleaned_mailout.shape, mailout_train["RESPONSE"].shape
mailout_train["RESPONSE"].plot(kind='hist', xticks=[0,1], fontsize=16, title="Plot of Responses ")<jupyter_output><empty_output><jupyter_text>A plot of number of response for the dataset   Running Machine Leaninf AlgorithmsFrom the figure above you will notice that this dataset is highly unstable since we have more 0's than 1's which our algorithms can work on to make strong predictions. Hence ROC and AUC is preferably used to evaluate the performance as opposed to scoring. A Receiver Operator Characteristic (ROC) curve is a graphical plot used to show the diagnostic ability of binary classifiers. 0.5 suggests no discrimination (i.e., the ability of not producing false positives ), 0.7 to 0.8 is considered acceptable, 0.8 to 0.9 is considered excellent, and more than 0.9 is considered outstandingSince the dataset responses are categorical and not continues I decided to use Classifiers for my supervisory learning models.    - RandomForestClassifier    - DecisionTreeClassifier    - GradientBoostingClassifier    - BaggingClassifier    - AdaBoostClassifier     Next    - Build a model basic models withouut any paramters and with no grid search    - Optimize the Existing models to user grid search    - Plot Visualisations    - Conclusion<jupyter_code>def train_predict_score(estemators, X_train, X_test, y_train, y_test, with_grid_cv=False, param_grid ={}):
    '''
    Description:
        Use Estemator to train a model run predictions, display fscore and other details
        
    Parameters:
        estemators: An Estimator
        X_train: Data to Fit to Model
        y_train: Target Value
        X_test: Data test
        y_test: Test Data
        with_grid_cv: If to apply grid search or not, Defaults(False)
        param_grid: Parameters to Grid Search defaults to an exmpty object
        
    Result:
        mode_df: DataFrame of Details of the training process
        regr_model: Built Models in a dictionry
    '''
        
    model_res = []
    mode_df = []
    regr = []
    regr_model = dict()
    
    for estem in estemators:
        print('{} Algorithm starting'.format(estem.__class__.__name__))
        
        if with_grid_cv:  
            regr = GridSearchCV(estimator=estem, param_grid=param_grid , scoring='roc_auc', cv=5)
            regr.fit(X_train, y_train)
            y_pred =  regr.predict(X_test)
            
            regr_rscore = accuracy_score(y_test, y_pred) 
            
            square_error = mean_squared_error(y_test, y_pred) #The mean squared error tells you how close a regression line is to a set of points

            print(regr.best_estimator_)
            
        else :     
            regr = copy.deepcopy(estem)
            regr.fit(X_train, y_train)
            y_pred = regr.predict(X_test)
            
            regr_rscore = accuracy_score(y_test, y_pred)
            square_error = mean_squared_error(y_test, y_pred) #The mean squared error tells you how close a regre
        
        # Put Informinto a dictionary
        regr_dic = dict()
        model_name = estem.__class__.__name__
        
        regr_dic['Algorithm'] = model_name
        regr_dic['Accuracy'] = regr_rscore 
        regr_dic['Percentage Accuracy'] = round(regr_rscore * 100, 2)
        regr_dic['Square Error'] = round(square_error, 4 )

        if with_grid_cv:
            regr_dic['best_score'] = regr.best_score_
            regr_model[model_name] = regr.best_estimator_
        else:
            regr_model[model_name] = regr
            
        model_res.append(regr_dic)
        
    
    model_res = sorted(model_res, key=lambda k: k['Accuracy'], reverse=True ) #Sort list in rscore values 
    
    mode_res_df = pd.DataFrame.from_dict(model_res)

    
    # Return the model alongside the data frame.
        
    return mode_res_df , regr_model<jupyter_output><empty_output><jupyter_text>Below we will run Initialize Models, create a list of models and pass into train_predict_score function.<jupyter_code># Grid Search Parameterd
param_grid = {
    'min_samples_leaf': [3, 4, 5],
    'min_samples_split': [8, 10],
    'n_estimators': [50,100,200,500],
    'random_state': [42, 45, 50]
}


gradient_booster_classifer = GradientBoostingClassifier(random_state=30)

# DecisionTreeReggresion:
decision_tree_class = DecisionTreeClassifier()

randomf_clasifier = RandomForestClassifier( n_estimators=100)

bagging_classifier = BaggingClassifier(max_samples=0.5, max_features=0.5)

adaBoost_classifier = AdaBoostClassifier(n_estimators=50)

regr_list = [decision_tree_class, randomf_clasifier, gradient_booster_classifer, bagging_classifier, adaBoost_classifier]
def plot_roc(model, X_test, y_test, title):
    '''
    Description: Plot Receiver Operator Characteristic (ROC) curves for 'model'
    
    Parameters:
        - model: Classifer Model
        - X_test: Test Date 
        - y_test: Target
        - title: title of the plot
    '''
    y_probas = model.predict_proba(X_test)
    plot_roc_curve(y_test, y_probas)
    plt.title(title)
    plt.show()
#Split dataset into train and test model
X_train, X_test, y_train, y_test = train_test_split(cleaned_mailout, mailout_train['RESPONSE'], test_size=0.33, random_state=42)
# Run Cell to train models specified in regr_list, result is a datrame with details and dictioinary if models 
# Run Model withough gridsearch

mode_res_df , regr_model = train_predict_score(regr_list, X_train, X_test, y_train, y_test, with_grid_cv=False,param_grid = {})
mode_res_df
regr = RandomForestClassifier( n_estimators=100)
regr.fit(X_train, y_train)
y_pred = regr.predict(X_test)
            
regr_rscore = accuracy_score(y_test, y_pred)
regr_rscore<jupyter_output><empty_output><jupyter_text>From the above analysis after and looking at the dataframe you can see that the resuls it terms of accuracy appear to be good because we have almost 100% in all classifiers nut remember ebove the true way to determine if our classifiers did well is to Compare thier ROC curves as dicused aboveWe plot the ROC curve for all 4 clasiffiers<jupyter_code>plot_roc(regr_model['RandomForestClassifier'], X_test, y_test, 'RandomForestClassifier  ROC Curves')
plot_roc(regr_model['BaggingClassifier'], X_test, y_test, 'BaggingClassifier  ROC Curves')
plot_roc(regr_model['AdaBoostClassifier'], X_test, y_test, 'AdaBoostClassifier	  ROC Curves')
plot_roc(regr_model['GradientBoostingClassifier'], X_test, y_test, 'GradientBoostingClassifier  ROC Curves')
plot_roc(regr_model['DecisionTreeClassifier'], X_test, y_test, 'DecisionTreeClassifier  ROC Curves')<jupyter_output>/opt/conda/lib/python3.6/site-packages/sklearn/utils/deprecation.py:77: DeprecationWarning: Function plot_roc_curve is deprecated; This will be removed in v0.5.0. Please use scikitplot.metrics.plot_roc instead.
  warnings.warn(msg, category=DeprecationWarning)<jupyter_text>From the above we can see that although from the sorted  data frame return `RandomForestClassifier` and `BaggingClassifier` appear to be at the top but a look at thier ROC curves the `GradientBoostingClassifier` and `AdaBoostClassifier` appear to perform beter in case of ROC have the higher probabilities of not producing false positives, as they follow the left-hand border and then the top border of the ROC space and curves are far from the  45-degree diagonal of the ROC space more compared to the other classifiers.Below i decided to optimize the`GradientBoostingClassifier` and compare thier outputs and roc curves Parameter Optimization.They are a could of options available for parameter optimization.- GridSearch- RandomSearch- Bayesian optimizationFor the case I choose to use GridSearch.**Note: Under Normal Situations I would have used pipelines plus gridsearch to perform this step but I chosed to do it seperately for my eventual Analytic purposes.** The above train_predict_score funtion was run again, this time allowing for gridsearch optimization.<jupyter_code>%%time
# Run Cell to train models specified in regr_list, result is a datrame with details and dictioinary if models
param_grid = {
    'n_estimators': [50,100],
    'random_state': [30, 45],
    'learning_rate': [0.01, 0.1, 1.0],
}

regr_list = [gradient_booster_classifer]
gmode_res_df , gregr_model = train_predict_score(regr_list, X_train, X_test,y_train,y_test,True, param_grid)<jupyter_output>GradientBoostingClassifier Algorithm starting
GradientBoostingClassifier(criterion='friedman_mse', init=None,
              learning_rate=0.01, loss='deviance', max_depth=3,
              max_features=None, max_leaf_nodes=None,
              min_impurity_decrease=0.0, min_impurity_split=None,
              min_samples_leaf=1, min_samples_split=2,
              min_weight_fraction_leaf=0.0, n_estimators=50,
              presort='auto', random_state=30, subsample=1.0, verbose=0,
              warm_start=False)
CPU times: user 22min 21s, sys: 67 ms, total: 22min 21s
Wall time: 22min 24s<jupyter_text><jupyter_code>gmode_res_df , gregr_model
gmode_res_df<jupyter_output><empty_output><jupyter_text>We take a look at the results and we see its really on conclusize once small as we have above 98%As above we will plot the `GradientBoostingClassifier` roc curve and do some somparison againTo have beter understanding or an approximaiton of the solution will do a ROC plot for  `GradientBoostingClassifier` and do some comparism<jupyter_code>plot_roc(m, X_test, y_test, 'GradientBoostingClassifier  ROC Curves')
#Model to be used for Kaggle predictions
kaggle_model = gregr_model['GradientBoostingClassifier']
for key, value in gregr_model.items():
    dump(value, '{}.joblib'.format(key.lower()))
    
# cross_val_score(regr_model['DecisionTreeClassifier'], X_test, y_test, cv=10)<jupyter_output><empty_output><jupyter_text>ObservationFrom the above 2 curves i was able to make some observationsIn the Second plot `GradientBoostingClassifier` plot  we can see the curves follow the left-hand border and then the top border of the ROC space and curves are far from the  45-degree diagonal of the ROC space more compared to to the other alorithms and with the GridSearchCV optimization the perfomace was slightlly increade. Conclustion Base on the Accuracy percentage from the above result dataframe and the ROC plot which captures the probability of a not producing false positives, we can see that the `GradientBoostingClassifier` model peformed slightly beter compared to the others.This Model will be used for the Kaggle competition. Below I will plot the relative importance in feautes identifiying how often in our model they where used for conclustions to be taken or for model decission making.<jupyter_code>def plot_feature_importance(n, fi, labels):
    '''
    Description: Plot of feature importance 
    parameters: 
        n: Number of features to display
        fi: Feature Importance
        labels: List of columns names
    '''
    feature_importance = fi[:n]
    sorted_idx = np.argsort(feature_importance)
    pos = np.arange(sorted_idx.shape[0]) + .5
    fig = plt.figure(figsize=(12, 6))
    plt.subplot(1, 2, 1)
    plt.barh(pos, feature_importance[sorted_idx], align='center')
    plt.yticks(pos, np.array(labels[:n])[sorted_idx])
    plt.title('Feature Importance (MDI)')
    fig.tight_layout()
    plt.show()
    
plot_feature_importance(10,gregr_model['GradientBoostingClassifier'].feature_importances_, cleaned_mailout.columns )<jupyter_output><empty_output><jupyter_text>From the above plot we can see the features that account are the fields that account for must of the decision of the model. We take a look into two the top two of the fields with the highest importance.<jupyter_code>## Do a merge of azdisas_attrs and azdisas_attrs_values to get additional information about the improtans atributes
merge_df = azdisas_attrs.merge(azdisas_attrs_values, how='inner', on='Attribute')
merge_df[merge_df['Attribute'].isin(['ANZ_TITEL' , 'ARBEIT', 'ANZ_KINDER', 'ANZ_PERSONEN'])]
mailout_train['ANZ_PERSONEN'].plot(kind='hist')<jupyter_output><empty_output><jupyter_text>This plot information related to nummber of persons know in house hold  Part 3: Kaggle CompetitionNow that you've created a model to predict which individuals are most likely to respond to a mailout campaign, it's time to test that model in competition through Kaggle. If you click on the link [here](http://www.kaggle.com/t/21e6d45d4c574c7fa2d868f0e8c83140), you'll be taken to the competition page where, if you have a Kaggle account, you can enter. If you're one of the top performers, you may have the chance to be contacted by a hiring manager from Arvato or Bertelsmann for an interview!Your entry to the competition should be a CSV file with two columns. The first column should be a copy of "LNR", which acts as an ID number for each individual in the "TEST" partition. The second column, "RESPONSE", should be some measure of how likely each individual became a customer – this might not be a straightforward probability. As you should have found in Part 2, there is a large output class imbalance, where most individuals did not respond to the mailout. Thus, predicting individual classes and using accuracy does not seem to be an appropriate performance evaluation method. Instead, the competition will be using AUC to evaluate performance. The exact values of the "RESPONSE" column do not matter as much: only that the higher values try to capture as many of the actual customers as possible, early in the ROC curve sweep.<jupyter_code>mailout_test = pd.read_csv('../../data/Term2/capstone/arvato_data/Udacity_MAILOUT_052018_TEST.csv', sep=';')
# response_out_df = mailout_train.drop(columns='RESPONSE', axis=1)
cleaned_mailout_test = clean_data(mailout_test, droprows=False, scale=True)

y_probas = kaggle_model.predict_proba(cleaned_mailout_test)[:,1]

submission_df = pd.DataFrame({'LNR':mailout_test['LNR'], 'RESPONSE': y_probas})

submission_df.to_csv('kanggle_submission.csv', index = False)

submission_df.head()<jupyter_output>Cleaning DataFrame
Drop any column with more than 60% Missen Values
Done
Calculating Corrolation
Removing Corolation matrix
Drop Columns with corolation > .97
Data Frame Columns Before and after removing corrolated columns
Before: 362 
After: 355 
Doing String conversion for CAMEO_DEUG_2015
Doing String conversion for CAMEO_INTL_2015
All Float and Int equvalent whole numbers converted to string 
Replacing Nan with mean for numeirc variables
Scaling DataFrame
Scaling Complete
Cleaning Complete
{'fraction_non_alphanumeric': 0.07425380457083727, 'fraction_numerical': 0.017501917896201674, 'mean_word_length': 4.842401500938086, 'pattern_counts': {'":': 0, '<': 129, '<?xml version=': 0, '>': 139, 'https://': 5, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '34', 'dir': 'jupyter-structured-clean-dedup', 'id': '135152', 'license': 'FTL', 'n_tokens_mistral': 17849, 'n_tokens_neox': 16660, 'n_words': 6003, 'path': '.ipynb_checkpoints/Arvato Customer Segmentation-checkpoint.ipynb', 'repo_name': 'austin047/udacity-datascience-arvato-customer-segmentation-and-prediction', 'size': '56051'}