<jupyter_start><jupyter_text>Capstone Project: Create a Customer Segmentation Report for Arvato Financial ServicesIn this project, you will analyze demographics data for customers of a mail-order sales company in Germany, comparing it against demographics information for the general population. You'll use unsupervised learning techniques to perform customer segmentation, identifying the parts of the population that best describe the core customer base of the company. Then, you'll apply what you've learned on a third dataset with demographics information for targets of a marketing campaign for the company, and use a model to predict which individuals are most likely to convert into becoming customers for the company. The data that you will use has been provided by our partners at Bertelsmann Arvato Analytics, and represents a real-life data science task.If you completed the first term of this program, you will be familiar with the first part of this project, from the unsupervised learning project. The versions of those two datasets used in this project will include many more features and has not been pre-cleaned. You are also free to choose whatever approach you'd like to analyzing the data rather than follow pre-determined steps. In your work on this project, make sure that you carefully document your steps and decisions, since your main deliverable for this project will be a blog post reporting your findings.<jupyter_code># import libraries here; add more as necessary
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib
import seaborn as sns

from sklearn.preprocessing import Imputer
from sklearn.preprocessing import MinMaxScaler
from sklearn.decomposition import PCA
from sklearn.cluster import KMeans
from sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier, GradientBoostingClassifier
from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.model_selection import KFold
from sklearn.metrics import roc_auc_score

from sklearn import metrics

%matplotlib inline<jupyter_output><empty_output><jupyter_text>Part 0: Get to Know the DataThere are four data files associated with this project:- `Udacity_AZDIAS_052018.csv`: Demographics data for the general population of Germany; 891 211 persons (rows) x 366 features (columns).- `Udacity_CUSTOMERS_052018.csv`: Demographics data for customers of a mail-order company; 191 652 persons (rows) x 369 features (columns).- `Udacity_MAILOUT_052018_TRAIN.csv`: Demographics data for individuals who were targets of a marketing campaign; 42 982 persons (rows) x 367 (columns).- `Udacity_MAILOUT_052018_TEST.csv`: Demographics data for individuals who were targets of a marketing campaign; 42 833 persons (rows) x 366 (columns).Each row of the demographics files represents a single person, but also includes information outside of individuals, including information about their household, building, and neighborhood. Use the information from the first two files to figure out how customers ("CUSTOMERS") are similar to or differ from the general population at large ("AZDIAS"), then use your analysis to make predictions on the other two files ("MAILOUT"), predicting which recipients are most likely to become a customer for the mail-order company.The "CUSTOMERS" file contains three extra columns ('CUSTOMER_GROUP', 'ONLINE_PURCHASE', and 'PRODUCT_GROUP'), which provide broad information about the customers depicted in the file. The original "MAILOUT" file included one additional column, "RESPONSE", which indicated whether or not each recipient became a customer of the company. For the "TRAIN" subset, this column has been retained, but in the "TEST" subset it has been removed; it is against that withheld column that your final predictions will be assessed in the Kaggle competition.Otherwise, all of the remaining columns are the same between the three data files. For more information about the columns depicted in the files, you can refer to two Excel spreadsheets provided in the workspace. [One of them](./DIAS Information Levels - Attributes 2017.xlsx) is a top-level list of attributes and descriptions, organized by informational category. [The other](./DIAS Attributes - Values 2017.xlsx) is a detailed mapping of data values for each feature in alphabetical order.In the below cell, we've provided some initial code to load in the first two datasets. Note for all of the `.csv` data files in this project that they're semicolon (`;`) delimited, so an additional argument in the [`read_csv()`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html) call has been included to read in the data properly. Also, considering the size of the datasets, it may take some time for them to load completely.You'll notice when the data is loaded in that a warning message will immediately pop up. Before you really start digging into the modeling and analysis, you're going to need to perform some cleaning. Take some time to browse the structure of the data and look over the informational spreadsheets to understand the data values. Make some decisions on which features to keep, which features to drop, and if any revisions need to be made on data formats. It'll be a good idea to create a function with pre-processing steps, since you'll need to clean all of the datasets before you work with them.<jupyter_code># load in the data
azdias = pd.read_csv('../../data/Term2/capstone/arvato_data/Udacity_AZDIAS_052018.csv', sep=';')
customers = pd.read_csv('../../data/Term2/capstone/arvato_data/Udacity_CUSTOMERS_052018.csv', sep=';')
print(azdias.shape)
azdias.head()
azdias.info()
print(customers.shape)
customers.head()
customers.info()
azdias_1 = azdias.loc[:100000]
azdias_2 = azdias.loc[100001:200000]
azdias_3 = azdias.loc[200001:300000]
azdias_4 = azdias.loc[300001:400000]
azdias_5 = azdias.loc[400001:500000]
azdias_6 = azdias.loc[500001:600000]
azdias_7 = azdias.loc[600001:700000]
azdias_8 = azdias.loc[700001:800000]
azdias_9 = azdias.loc[800001:]<jupyter_output><empty_output><jupyter_text>Part 1: Customer Segmentation ReportThe main bulk of your analysis will come in this part of the project. Here, you should use unsupervised learning techniques to describe the relationship between the demographics of the company's existing customers and the general population of Germany. By the end of this part, you should be able to describe parts of the general population that are more likely to be part of the mail-order company's main customer base, and which parts of the general population are less so. **Assessing missing data:**<jupyter_code># Load in the feature summary file
feat_info = pd.read_csv('AZDIAS_Feature_Summary.csv', sep=';')
feat_info.head()
feat_info.drop(['information_level'], axis=1, inplace=True)
missing_or_unknown = []
for value in feat_info['missing_or_unknown']:
    value = str(value)
    list_value = value[1:-1].split(',')
    missing_or_unknown.append(list_value)
feat_info['missing_or_unknown'] = missing_or_unknown
def missing_to_nan(df1, df2):
    for row in df2['attribute']:
        if row in df1.columns:
            na_map = df2.loc[df2['attribute'] == row, 'missing_or_unknown'].iloc[0]
            na_idx = df1.loc[:, row].isin(na_map)
            df1.loc[na_idx, row] = np.nan
        else:
            continue
    return df1
azdias_1 = missing_to_nan(azdias_1, feat_info)
azdias_2 = missing_to_nan(azdias_2, feat_info)
azdias_3 = missing_to_nan(azdias_3, feat_info)
azdias_4 = missing_to_nan(azdias_4, feat_info)
azdias_5 = missing_to_nan(azdias_5, feat_info)
azdias_6 = missing_to_nan(azdias_6, feat_info)
azdias_7 = missing_to_nan(azdias_7, feat_info)
azdias_8 = missing_to_nan(azdias_8, feat_info)
azdias_9 = missing_to_nan(azdias_9, feat_info)
azdias = pd.concat([azdias_1, azdias_2, azdias_3, azdias_4, azdias_5, azdias_6, azdias_7, azdias_8, azdias_9])<jupyter_output><empty_output><jupyter_text>**Load of the DIAS Attributes dataset, useful to get the remaining unknown attributes or without transactions:**<jupyter_code># Laod DIAS Attributes
dias_attributes = pd.read_excel('DIAS Attributes - Values 2017.xlsx', skiprows=[0])
dias_attributes.head()
dias_attributes.drop(['Unnamed: 0'], axis=1, inplace=True)
dias_attributes = dias_attributes[dias_attributes['Attribute'].isnull() == False]
dias_attributes.head()
# We take the attributes that are not already in the feat_info dataframe 
dias_attributes = dias_attributes[dias_attributes['Attribute'].isin(feat_info['attribute']) == False]
dias_attributes.head()
dias_attributes['Meaning'].value_counts()<jupyter_output><empty_output><jupyter_text>**DIAS with unknown meaning or no transactions:**<jupyter_code>missing_or_unknown_dias = dias_attributes[dias_attributes['Meaning'].isin(['unknown', 
                                                                           'no transaction known', 
                                                                           'no transactions known'])]
missing_or_unknown_dias = missing_or_unknown_dias[missing_or_unknown_dias['Attribute'].isin(azdias.columns)]
missing_or_unknown_dias.head()
missing_or_unknown_dias['Value'].value_counts()
azdias[azdias.columns[azdias.columns.isin(missing_or_unknown_dias['Attribute'])]].dtypes.value_counts()
missing_or_unknown_dias['type'] = 'ordinal'
# Make a dataframe of features proprieties
missing_or_unknown_dias['missing_or_unknown'] = missing_or_unknown_dias['Value'].apply(lambda x: [x])
missing_or_unknown_dias.drop(['Description', 'Value', 'Meaning'], axis=1, inplace=True)
missing_or_unknown_dias.columns = ['attribute', 'type', 'missing_or_unknown']
missing_or_unknown_dias.head()
missing_or_unknown = []
for value in missing_or_unknown_dias['missing_or_unknown']:
    value = str(value)
    list_value = value[1:-1].split(',')
    missing_or_unknown.append(list_value)
missing_or_unknown_dias['missing_or_unknown'] = missing_or_unknown
azdias_1 = missing_to_nan(azdias_1, missing_or_unknown_dias)
azdias_2 = missing_to_nan(azdias_2, missing_or_unknown_dias)
azdias_3 = missing_to_nan(azdias_3, missing_or_unknown_dias)
azdias_4 = missing_to_nan(azdias_4, missing_or_unknown_dias)
azdias_5 = missing_to_nan(azdias_5, missing_or_unknown_dias)
azdias_6 = missing_to_nan(azdias_6, missing_or_unknown_dias)
azdias_7 = missing_to_nan(azdias_7, missing_or_unknown_dias)
azdias_8 = missing_to_nan(azdias_8, missing_or_unknown_dias)
azdias_9 = missing_to_nan(azdias_9, missing_or_unknown_dias)
azdias = pd.concat([azdias_1, azdias_2, azdias_3, azdias_4, azdias_5, azdias_6, azdias_7, azdias_8, azdias_9])<jupyter_output><empty_output><jupyter_text>**Complete feat_info with the remaining variables (which do not have missing or unknown values):**<jupyter_code># Investigate the rest of attributes 
remaining_dias = dias_attributes[dias_attributes['Meaning'].isin(['unknown',
                                                                  'no transaction known',
                                                                  'no transactions known']) == False]
remaining_dias = remaining_dias[remaining_dias['Attribute'].isin(azdias.columns)]
remaining_dias.head()
azdias[azdias.columns[azdias.columns.isin(remaining_dias['Attribute'])]].dtypes.value_counts()
remaining_dias['type'] = 'ordinal'
remaining_dias['missing_or_unknown'] = remaining_dias['Value'].apply(lambda x: list())
remaining_dias.drop(['Description', 'Value', 'Meaning'], axis=1, inplace=True)
remaining_dias.columns = ['attribute', 'type', 'missing_or_unknown']
remaining_dias.head()
new_feat_info = pd.concat([feat_info, missing_or_unknown_dias, remaining_dias])
# azdias columns that are not in the df of feat info
remaining_cols = azdias.columns[azdias.columns.isin(new_feat_info['attribute']) == False]
azdias[remaining_cols].dtypes.value_counts()
categoric_cols = azdias[remaining_cols].select_dtypes(include='object').columns.tolist()
numeric_cols = azdias[remaining_cols].select_dtypes(exclude='object').columns.tolist()
remaining_attrib = pd.DataFrame({'attribute': remaining_cols})
len(categoric_cols)
len(numeric_cols)
types = []
for attribute in remaining_attrib['attribute']:
    if attribute in numeric_cols:
        att_type = 'numeric'
    elif attribute in categoric_cols:
        att_type = 'categorical'
    types.append(att_type)
    
remaining_attrib['type'] = types
remaining_attrib['type'].value_counts()
remaining_attrib['missing_or_unknown'] = remaining_attrib['type'].apply(lambda x: list())
remaining_attrib.head()
# Concat feat info
new_feat_info = pd.concat([new_feat_info, remaining_attrib], ignore_index=True)
print(new_feat_info.shape)
new_feat_info.head()<jupyter_output>(366, 3)<jupyter_text>**Create copy of main dfs:** We'll work with those now in order to keep the new_feat_info dataset original for later use.<jupyter_code>copy_feat_info = new_feat_info.copy()<jupyter_output><empty_output><jupyter_text>**Convert missing data to NaN:**<jupyter_code>def is_int(value):
    try:
        int(value)
        return True
    except ValueError:
        return False
copy_feat_info['missing_or_unknown'] = copy_feat_info.apply(lambda x: [int(i) if is_int(i) else i for i in x['missing_or_unknown']], axis=1)
for idx in copy_feat_info.index:
    column = idx
    if column in azdias.columns:
        azdias[column] = azdias[column].replace(copy_feat_info.loc[column]['missing_or_unknown'], np.nan)<jupyter_output><empty_output><jupyter_text>**Assess missing data in each column:**<jupyter_code>count_nan = azdias.isnull().sum()/len(azdias)
plt.figure()
sns.distplot(count_nan.values, kde=False)
plt.ylabel('Count of columns')
plt.xlabel('% of NaN')
plt.title('Distribution of missing values per columns')
plt.show()
# Perform a more exhaustive assessment of how much missing data there is in each column of the dataset

count_list = []
nan_list = []
for column in azdias.columns:
    nan_count = azdias[column].isnull().sum()
    count_list.append(nan_count)
    nan_list.append((column, nan_count))

nan_data = pd.DataFrame(nan_list, columns=['Column', 'Count'])
nan_data.sort_values('Count', inplace=True, ascending=False)
# Investigate patterns in the amount of missing data in each column.
nan_group = []
nan_prop = []
nan_percent = []
dropping = []

print("From the graph, we set a threshold of 30% for the variables to drop.")
print("Thus, we will be dropping:")

for i, count in enumerate(nan_data['Count']):
    col = nan_data['Column'].iloc[i]
    x_nan = count/len(azdias)
    perc_nan = round(x_nan * 100, 2)

    nan_prop.append(x_nan)
    nan_percent.append(x_nan * 100)

    if x_nan == 0:
        nan_group.append(1)
    elif x_nan > 0 and x_nan <= 0.05:
        nan_group.append(2)
    elif x_nan > 0.05 and x_nan <= 0.1:
        nan_group.append(3)
    elif x_nan > 0.1 and x_nan <= 0.15:
        nan_group.append(4)
    elif x_nan > 0.15 and x_nan <= 0.2:
        nan_group.append(5)
    elif x_nan > 0.2 and x_nan <= 0.25:
        nan_group.append(6)
    elif x_nan > 0.25 and x_nan <= 0.3:
        nan_group.append(7)
    else:
        print("{}, {}% of NaN values".format(col, perc_nan))
        dropping.append(col)
        
nan_data['Proportion'] = nan_prop
nan_data['Percentage'] = nan_percent
dropping
# Remove the outlier columns from the dataset. (You'll perform other data
# engineering tasks such as re-encoding and imputation later.)
azdias = azdias.drop(dropping, axis=1)
azdias.shape
print("Let's get some insights!")
print("No missing values:")
print("{} attributes without missing or unknown values.".format(nan_group.count(1)))
print("Missing values with less than 20% of missing values:")
print("{} with a proportion lower or equal to 5%, {} with a proportion between 10% and 15% and {} between 15% and 20%.".format(nan_group.count(2), nan_group.count(3), nan_group.count(4)))
print("Missing values with a proportion higher than 20% but lower than our threshold, 30%:")
print("Finaly, there are {} between 20% and 25%, and {} between 25% and 30%.".format(nan_group.count(5), nan_group.count(6)))
# Remove from feat_info the rows correspondent to the variables that were deleted from azdias 
# This is necessary to keep consistency between both datasets
for variable in dropping:
    copy_feat_info = copy_feat_info[copy_feat_info.attribute != variable]<jupyter_output><empty_output><jupyter_text>Discussion 1.1.2: Assess Missing Data in Each ColumnThere are 21 columns that standout considerably because of their amount of missing values. These are: ALTER_KIND4, TITEL_KZ, ALTER_KIND3, ALTER_KIND2, D19_TELKO_ANZ_12, D19_BANKEN_ANZ_12, D19_TELKO_ANZ_24, D19_VERSI_ANZ_12, ALTER_KIND1, D19_BANKEN_ANZ_24, D19_VERSI_ANZ_24, AGER_TYP, EXTSEL992, D19_VERSAND_ANZ_12, D19_GESAMT_ANZ_12, KK_KUNDENTYP, D19_VERSAND_ANZ_24, D19_GESAMT_ANZ_24, KBA05_BAUMAX, GEBURTSJAHR and ALTER_HH. All of them have more than 30% of their values missing, which was the threshold used to decide which columns to drop. It is relevant noting that there are similarities in the amounts of missing or unknown values for different groups of the attributes. We can see above that there is a considerable number of variables without any missing value, and then most of the rest fall between 0 and 20%, with special emphasis in the interval 15-20%. **Assess missing data by rows:**<jupyter_code># Amount of data missing in each row of the azdias dataset
missing_values_by_row = azdias.isnull().sum(axis=1)
fig = plt.figure(figsize=(8, 5))
sns.distplot(missing_values_by_row.values/343, kde=False)
plt.ylabel('count of rows')
plt.xlabel('% of NaN')
plt.title('Distribution of missing value per row')
plt.show()
# Getting some information
missing_values_by_row.describe()
rows = len(azdias)
rows_with_missing_values = missing_values_by_row[missing_values_by_row != 0].count()
rows_without_missing_values = missing_values_by_row[missing_values_by_row == 0].count()

percent_missing = rows_with_missing_values / rows * 100
percent_no_missing = 100 - percent_missing
print("Number of rows without any missing values:")
print("{}, {}% of total number of rows".format(rows_without_missing_values, round(percent_no_missing, 2)))
missing_values_by_row.describe()
# Write code to divide the data into two subsets based on the number of missing
# values in each row.

# With describe() we could see that the mean value of missing values by row is 49.56
# Thus, the threshold will be set as 50

# Reminder: missing_values_by_row = azdias.isnull().sum(axis=1)

threshold = 50
azdias_c_low = azdias[missing_values_by_row < threshold].reset_index(drop=True)
azdias_c_high = azdias[missing_values_by_row >= threshold].reset_index(drop=True)<jupyter_output><empty_output><jupyter_text>Discussion 1.1.3: Assess Missing Data in Each RowWhen assessing the missing data by rows the thereshold was set as 50 missing values by row since the mean of missing values was 49.56. Thus, our final dataset is one where all rows have less than 50 missing values. It can be concluded from the analysis that many of the variables present a different distribrution between the rows with many missing values and those with just a few, which means that data is qualitatively different between them.  **Re-encode categorical features:**<jupyter_code>copy_feat_info.type.value_counts(dropna=False)
categ_var = copy_feat_info[copy_feat_info.type == 'categorical'].attribute.tolist()
azdias_c_low[categ_var].head()
binary_num_var = []
binary_str_var = []
multi_level_var = []

for variable in categ_var:
    if len(azdias_c_low[variable].value_counts()) > 2:
        multi_level_var.append(variable)
    else:
        if azdias_c_low[variable].dtype == 'object':
            binary_str_var.append(variable)
        else:
            binary_num_var.append(variable)
binary_str_var
azdias_c_low[binary_str_var].dtypes
azdias_c_low[binary_str_var].head()
# Re-encode OST_WEST_KZ as dummy variable
azdias_c_low['OST_WEST_KZ'] = azdias_c_low['OST_WEST_KZ'].map({'W': 1, 'O': 2})
binary_num_var
azdias_c_low[binary_num_var].dtypes
multi_level_var
azdias_c_low.drop(['D19_LETZTER_KAUF_BRANCHE', 'EINGEFUEGT_AM'], axis=1, inplace=True)
copy_feat_info = copy_feat_info[copy_feat_info.attribute != 'D19_LETZTER_KAUF_BRANCHE']
copy_feat_info = copy_feat_info[copy_feat_info.attribute != 'EINGEFUEGT_AM']
multi_level_var = multi_level_var[:-2]
multi_level_var
# Convert categorical variables to dummies 
def encode_cat_dummies(df, variables_list):
    for variable in variables_list:
        dummies_df = pd.get_dummies(df[variable], prefix=variable)
        df.drop(variable, axis=1, inplace=True)
        df = pd.concat([df, dummies_df], axis=1)
    return df
new_azdias = encode_cat_dummies(azdias_c_low, multi_level_var)
# Engineer Mixed-Type Features
mixed_var = copy_feat_info.attribute[copy_feat_info.type == 'mixed'].tolist()
mixed_var
life_stage = {1: 'younger_age',    2: 'middle_age',      3: 'younger_age',     4: 'middle_age',    5: 'advanced_age', 
              6: 'retirement_age', 7: 'advanced_age',    8: 'retirement_age',  9: 'middle_age',   10: 'middle_age', 
             11: 'advanced_age',  12: 'retirement_age', 13: 'advanced_age',   14: 'younger_age',  15: 'advanced_age',
             16: 'advanced_age',  17: 'middle_age',     18: 'younger_age',    19: 'advanced_age', 20: 'advanced_age', 
             21: 'middle_age',    22: 'middle_age',     23: 'middle_age',     24: 'middle_age',   25: 'middle_age', 
             26: 'middle_age',    27: 'middle_age',     28: 'middle_age',     29: 'younger_age',  30: 'younger_age',
             31: 'advanced_age',  32: 'advanced_age',   33: 'younger_age',    34: 'younger_age',  35: 'younger_age', 
             36: 'advanced_age',  37: 'advanced_age',   38: 'retirement_age', 39: 'middle_age',   40: 'retirement_age'}

fine_scale = {1: 'low',       2: 'low',      3: 'average',  4: 'average',   5: 'low',      6: 'low',      7: 'average',   
              8: 'average',   9: 'average', 10: 'wealthy', 11: 'average',  12: 'average', 13: 'top',     14: 'average', 
             15: 'low',      16: 'average', 17: 'average', 18: 'wealthy',  19: 'wealthy', 20: 'top',     21: 'low',
             22: 'average',  23: 'wealthy', 24: 'low',     25: 'average',  26: 'average', 27: 'average', 28: 'top',     
             29: 'low',      30: 'average', 31: 'low',     32: 'average',  33: 'average', 34: 'average', 35: 'top', 
             36: 'average',  37: 'average', 38: 'average', 39: 'top',      40: 'top'}

new_azdias['LP_life_stage'] = new_azdias['LP_LEBENSPHASE_FEIN'].map(life_stage)
new_azdias['LP_fine_scale'] = new_azdias['LP_LEBENSPHASE_FEIN'].map(fine_scale)

# Drop LP_LEBENSPHASE_FEIN
new_azdias.drop('LP_LEBENSPHASE_FEIN', axis=1, inplace=True)
# R-encode LP_life_stage and LP_fine_scale as ordinal
dic_life = {'younger_age': 1, 'middle_age': 2, 'advanced_age': 3, 'retirement_age': 4}
dic_scale = {'low': 1, 'average': 2, 'wealthy': 3, 'top': 4}

new_azdias['LP_life_stage'] = new_azdias['LP_life_stage'].map(dic_life)
new_azdias['LP_fine_scale'] = new_azdias['LP_fine_scale'].map(dic_scale)

new_azdias.drop('LP_LEBENSPHASE_GROB', axis=1, inplace=True)
# Investigate "PRAEGENDE_JUGENDJAHRE" and engineer two new variables.

new_azdias['PRAEGENDE_JUGENDJAHRE'].describe()

decade = {1:40, 2:40, 3:50, 4:50, 5:60, 6:60, 7:60, 8:70, 9:70, 10:80, 11:80, 12:80, 13:80, 14:90, 15:90, 
          np.nan: np.nan, -1: np.nan, 0: np.nan}
avantgarde = {1:0, 2:1, 3:0, 4:1, 5:0, 6:1, 7:1, 8:0, 9:1, 10:0, 11:1, 12:0, 13:1, 14:0, 15:1,
              np.nan: np.nan, -1: np.nan, 0: np.nan}

new_azdias['YOUTH'] = new_azdias['PRAEGENDE_JUGENDJAHRE'].map(decade)
new_azdias['AVANTGARDE'] = new_azdias['PRAEGENDE_JUGENDJAHRE'].map(avantgarde)

new_azdias = new_azdias.drop('PRAEGENDE_JUGENDJAHRE', axis=1)
# replace 0 by nan
new_azdias['WOHNLAGE'] = new_azdias['WOHNLAGE'].replace(0, np.nan)
dummies_list = ['WOHNLAGE', 'PLZ8_BAUMAX']
new_azdias = encode_cat_dummies(new_azdias, dummies_list)
def encode_CAMEO_INTL_2015(value):
    if np.isnan(value):
        return [np.nan, np.nan]
    else:
        value = str(value).split('.')[0]
        return [int(value[0]), int(value[1])]


new_azdias["CAMEO_INTL_2015"] = new_azdias["CAMEO_INTL_2015"].apply(lambda x: float(x))
new_azdias["CAMEO_INTL_2015"] = new_azdias["CAMEO_INTL_2015"].apply(encode_CAMEO_INTL_2015)

# CAMEO_INTL_2015_WEALTH and CAMEO_INTL_2015_LIFE_STAGE
new_azdias["CAMEO_INTL_2015_WEALTH"] = new_azdias["CAMEO_INTL_2015"].apply(lambda x: x[0])
new_azdias["CAMEO_INTL_2015_LIFE_STAGE"] = new_azdias["CAMEO_INTL_2015"].apply(lambda x: x[1])

# drop CAMEO_INTL_2015
new_azdias.drop('CAMEO_INTL_2015', axis=1, inplace=True)<jupyter_output><empty_output><jupyter_text>**Cleaning function:**<jupyter_code>def is_int(value):
    try:
        int(value)
        return True
    except ValueError:
        return False
def encode_cat_dummies(df, variables_list):
    for variable in variables_list:
        dummies_df = pd.get_dummies(df[variable], prefix=variable)
        df.drop(variable, axis=1, inplace=True)
        df = pd.concat([df, dummies_df], axis=1)
    return df
def encode_CAMEO_INTL_2015(value):
    if np.isnan(value):
        return [np.nan, np.nan]
    else:
        value = str(value).split('.')[0]
        return [int(value[0]), int(value[1])]
dropping = ['ALTER_KIND4', 'TITEL_KZ', 'ALTER_KIND3', 'ALTER_KIND2', 'D19_TELKO_ANZ_12', 'D19_BANKEN_ANZ_12',
            'D19_TELKO_ANZ_24', 'D19_VERSI_ANZ_12', 'ALTER_KIND1', 'D19_BANKEN_ANZ_24', 'D19_VERSI_ANZ_24',
            'AGER_TYP', 'EXTSEL992', 'D19_VERSAND_ANZ_12', 'D19_GESAMT_ANZ_12', 'KK_KUNDENTYP', 'D19_VERSAND_ANZ_24',
            'D19_GESAMT_ANZ_24', 'KBA05_BAUMAX', 'GEBURTSJAHR', 'ALTER_HH', 'D19_LETZTER_KAUF_BRANCHE', 'EINGEFUEGT_AM']

categ_var = ['ANREDE_KZ', 'CJT_GESAMTTYP', 'FINANZTYP', 'GFK_URLAUBERTYP', 'GREEN_AVANTGARDE', 'LP_FAMILIE_FEIN',
             'LP_FAMILIE_GROB', 'LP_STATUS_FEIN', 'LP_STATUS_GROB', 'NATIONALITAET_KZ', 'SHOPPER_TYP', 'SOHO_KZ',
             'VERS_TYP', 'ZABEOTYP', 'GEBAEUDETYP', 'OST_WEST_KZ', 'CAMEO_DEUG_2015', 'CAMEO_DEU_2015',
             'D19_LETZTER_KAUF_BRANCHE', 'EINGEFUEGT_AM']

binary_str_var = ['OST_WEST_KZ']

binary_num_var = ['ANREDE_KZ', 'GREEN_AVANTGARDE', 'SOHO_KZ', 'VERS_TYP']

multi_level_var = ['CJT_GESAMTTYP', 'FINANZTYP', 'GFK_URLAUBERTYP', 'LP_FAMILIE_FEIN', 'LP_FAMILIE_GROB', 
                   'LP_STATUS_FEIN', 'LP_STATUS_GROB', 'NATIONALITAET_KZ','SHOPPER_TYP', 'ZABEOTYP', 
                   'GEBAEUDETYP', 'CAMEO_DEUG_2015', 'CAMEO_DEU_2015']

mixed_var = ['LP_LEBENSPHASE_FEIN', 'LP_LEBENSPHASE_GROB', 'PRAEGENDE_JUGENDJAHRE', 'WOHNLAGE', 'CAMEO_INTL_2015', 
             'PLZ8_BAUMAX']

life_stage = {1: 'younger_age',    2: 'middle_age',      3: 'younger_age',     4: 'middle_age',    5: 'advanced_age', 
              6: 'retirement_age', 7: 'advanced_age',    8: 'retirement_age',  9: 'middle_age',   10: 'middle_age', 
              11: 'advanced_age', 12: 'retirement_age', 13: 'advanced_age',   14: 'younger_age',  15: 'advanced_age',
              16: 'advanced_age', 17: 'middle_age',     18: 'younger_age',    19: 'advanced_age', 20: 'advanced_age', 
              21: 'middle_age',   22: 'middle_age',     23: 'middle_age',     24: 'middle_age',   25: 'middle_age', 
              26: 'middle_age',   27: 'middle_age',     28: 'middle_age',     29: 'younger_age',  30: 'younger_age',
              31: 'advanced_age', 32: 'advanced_age',   33: 'younger_age',    34: 'younger_age',  35: 'younger_age', 
              36: 'advanced_age', 37: 'advanced_age',   38: 'retirement_age', 39: 'middle_age',   40: 'retirement_age'}

fine_scale = {1: 'low',       2: 'low',      3: 'average',  4: 'average',   5: 'low',      6: 'low',      7: 'average',   
              8: 'average',   9: 'average', 10: 'wealthy', 11: 'average',  12: 'average', 13: 'top',     14: 'average', 
              15: 'low',     16: 'average', 17: 'average', 18: 'wealthy',  19: 'wealthy', 20: 'top',     21: 'low',
              22: 'average', 23: 'wealthy', 24: 'low',     25: 'average',  26: 'average', 27: 'average', 28: 'top',     
              29: 'low',     30: 'average', 31: 'low',     32: 'average',  33: 'average', 34: 'average', 35: 'top', 
              36: 'average', 37: 'average', 38: 'average', 39: 'top',      40: 'top'}

dic_life = {'younger_age': 1, 'middle_age': 2, 'advanced_age': 3, 'retirement_age': 4}
dic_scale = {'low': 1, 'average': 2, 'wealthy': 3, 'top': 4}

decade = {1:40, 2:40, 3:50, 4:50, 5:60, 6:60, 7:60, 8:70, 9:70, 10:80, 11:80, 12:80, 13:80, 14:90, 15:90, 
          np.nan: np.nan, -1: np.nan, 0: np.nan}
avantgarde = {1:0, 2:1, 3:0, 4:1, 5:0, 6:1, 7:1, 8:0, 9:1, 10:0, 11:1, 12:0, 13:1, 14:0, 15:1,
              np.nan: np.nan, -1: np.nan, 0: np.nan}

dummies_list = ['WOHNLAGE', 'PLZ8_BAUMAX']
def clean_data(df):
    
    df_c = df.copy()
    feat_info_c = new_feat_info.copy()
    
    feat_info_c['missing_or_unknown'] = feat_info_c.apply(lambda x: [int(i) if is_int(i) 
                                                                     else i for i in x['missing_or_unknown']], axis=1)
    
    for idx in feat_info_c.index:
        column = idx
        if column in df_c.columns:
            df_c[column] = df_c[column].replace(feat_info_c.loc[column]['missing_or_unknown'], np.nan)
            
    df_c = df_c.drop(dropping, axis=1)
    
    for variable in dropping:
        feat_info_c = feat_info_c[feat_info_c.attribute != variable]
    
    missing_values_by_row = df_c.isnull().sum(axis=1)
    
    threshold = 50
    df_c = df_c[missing_values_by_row < threshold].reset_index(drop=True)
    
    df_c['OST_WEST_KZ'] = df_c['OST_WEST_KZ'].map({'W': 1, 'O': 2})
    
    df_c = encode_cat_dummies(df_c, multi_level_var)
    
    df_c['LP_life_stage'] = df_c['LP_LEBENSPHASE_FEIN'].map(life_stage)
    df_c['LP_fine_scale'] = df_c['LP_LEBENSPHASE_FEIN'].map(fine_scale)
    df_c.drop('LP_LEBENSPHASE_FEIN', axis=1, inplace=True)
    
    df_c['LP_life_stage'] = df_c['LP_life_stage'].map(dic_life)
    df_c['LP_fine_scale'] = df_c['LP_fine_scale'].map(dic_scale)
    df_c.drop('LP_LEBENSPHASE_GROB', axis=1, inplace=True)
    
    df_c['YOUTH'] = df_c['PRAEGENDE_JUGENDJAHRE'].map(decade)
    df_c['AVANTGARDE'] = df_c['PRAEGENDE_JUGENDJAHRE'].map(avantgarde)

    df_c = df_c.drop('PRAEGENDE_JUGENDJAHRE', axis=1)
    
    df_c['WOHNLAGE'] = df_c['WOHNLAGE'].replace(0, np.nan)
    
    df_c = encode_cat_dummies(df_c, dummies_list)
    
    df_c["CAMEO_INTL_2015"] = df_c["CAMEO_INTL_2015"].replace('XX', np.nan)
    df_c["CAMEO_INTL_2015"] = df_c["CAMEO_INTL_2015"].apply(lambda x: float(x))
    df_c["CAMEO_INTL_2015"] = df_c["CAMEO_INTL_2015"].apply(encode_CAMEO_INTL_2015)

    df_c["CAMEO_INTL_2015_WEALTH"] = df_c["CAMEO_INTL_2015"].apply(lambda x: x[0])
    df_c["CAMEO_INTL_2015_LIFE_STAGE"] = df_c["CAMEO_INTL_2015"].apply(lambda x: x[1])
    df_c.drop('CAMEO_INTL_2015', axis=1, inplace=True)
    
    return df_c
azdias_clean = clean_data(azdias)<jupyter_output><empty_output><jupyter_text>**Feature Scaling:**<jupyter_code># Initialize the imputer and scaler
imputer = Imputer(missing_values='NaN', strategy='mean', axis=0)
scaler = MinMaxScaler()
azdias_imputed = pd.DataFrame(imputer.fit_transform(new_azdias), columns=new_azdias.columns)
azdias_ss = pd.DataFrame(scaler.fit_transform(azdias_imputed), columns=azdias_imputed.columns)<jupyter_output><empty_output><jupyter_text>Discussion 2.1: Apply Feature ScalingIn order to replace NaN values (needed for the unsupervised method to work properly) I decided to use Imputer and substitute them with the mean value of the variable, since I thought it was the simplest way and the mean a good measure to apply to the data for the prediction. Then, data was scaled to obtain a minimum value of 0 and maximum of 1. **Dimensionality Reduction:**<jupyter_code># Apply PCA to the data.
pca = PCA()
pca_fit = pca.fit(azdias_ss)
# Investigate the variance accounted for by each principal component.

def scree_plot(pca): # Function from PCA lesson
    '''
    Creates a scree plot associated with the principal components 
    
    INPUT: pca - the result of instantian of PCA in scikit learn
            
    RETURN: None
    '''
    num_components= len(pca.explained_variance_ratio_)
    idx = np.arange(num_components)
    vals = pca.explained_variance_ratio_
 
    plt.figure(figsize=(15, 10))
    ax = plt.subplot(111)
    cumvals = np.cumsum(vals)
    ax.bar(idx, vals)
    ax.plot(idx, cumvals)
    for i in range(num_components):
        if(i%20 == 0 or i<6):
            ax.annotate(r"%s%%" % ((str(vals[i]*100)[:4])), (idx[i]+0.2, vals[i]), va="bottom", ha="center", fontsize=8)
 
    ax.xaxis.set_tick_params(width=0, gridOn=True)
    ax.yaxis.set_tick_params(width=2, length=12, gridOn=True)
 
    ax.set_xlabel("Principal Component")
    ax.set_ylabel("Variance Explained (%)")
    plt.title('Explained Variance Per Principal Component')
scree_plot(pca_fit)<jupyter_output><empty_output><jupyter_text>Discussion 2.2: Perform Dimensionality ReductionI chose to retain 200 components since it provides more than 90% of variance, which is a pretty high value. It is interesting mentioning that just around 5 principal components would provide about 20% of variance, and retaining just 20 would provide almost 50%. However, 200 were chosen since we are looking to keep the highest variance possible in order not to lose much precision in the results.<jupyter_code># Re-apply PCA to the data while selecting for number of components to retain.
pca = PCA(n_components=200)
azdias_pca = pca.fit_transform(azdias_ss)
# CLUSTER
centers = []
scores = []

# Over a number of different cluster counts...
for cluster in range(2, 15, 1):
    # run k-means clustering on the data and...
    kmean = KMeans(cluster)
    model = kmean.fit(azdias_pca)
    
    # compute the average within-cluster distances.
    centers.append(cluster)
    scores.append(np.abs(model.score(azdias_pca)))    
# Investigate the change in within-cluster distance across number of clusters

plt.plot(centers, scores, linestyle='--', marker='o', color='b');
plt.xlabel('K');
plt.ylabel('SSE');
plt.title('SSE vs. K');
# Re-fit the k-means model with the selected number of clusters and obtain
# cluster predictions for the general population demographics data.

kmeans = KMeans(7).fit(azdias_pca)
azdias_pred = kmeans.predict(azdias_pca)<jupyter_output><empty_output><jupyter_text>Discussion 3.1: Apply Clustering to General PopulationIt was considered that 7 clusters will be enough for the prediction, being there where the slope changes a little bit closer to the horizontal axis in the graph.  **Customers:**<jupyter_code># Apply preprocessing, feature transformation, and clustering from the general
# demographics onto the customer data, obtaining cluster predictions for the
# customer demographics data.
customers_clean = clean_data(customers)
customers_clean.select_dtypes(include='object').head()
customers_clean['PRODUCT_GROUP'].value_counts()
customers_clean['CUSTOMER_GROUP'].value_counts()
dummies = ['PRODUCT_GROUP', 'CUSTOMER_GROUP']
customers_clean = encode_cat_dummies(customers_clean, dummies)
customers_clean = customers_clean.replace(np.nan, 0)
customers_clean.drop('LNR', axis=1, inplace=True)
customers_imputed = pd.DataFrame(imputer.fit_transform(customers_clean), columns=customers_clean.columns)
customers_ss = pd.DataFrame(scaler.fit_transform(customers_imputed), columns=customers_imputed.columns)
# Re-apply PCA to the data while selecting for number of components to retain
customers_pca = pca.fit_transform(customers_ss)
customers_pred = kmeans.predict(customers_pca)<jupyter_output><empty_output><jupyter_text>**Comparison:**<jupyter_code>counts_cluster_gen = pd.Series(azdias_pred).value_counts().sort_index()
counts_cluster_gen.index = [1,2,3,4,5,6,7]

prop_cluster_gen = []
for value in counts_cluster_gen:
    prop = value / len(azdias_pred)
    prop_cluster_gen.append(prop)

prop_cluster_general = pd.Series(prop_cluster_gen)
prop_cluster_general.index = [1,2,3,4,5,6,7]
counts_cluster_cust = pd.Series(customers_pred).value_counts().sort_index()
counts_cluster_cust.index = [1,2,3,4,5,6,7]

prop_cluster_cust = []
for value in counts_cluster_cust:
    prop = value / len(customers_pred)
    prop_cluster_cust.append(prop)

prop_cluster_customers = pd.Series(prop_cluster_cust)
prop_cluster_customers.index = [1,2,3,4,5,6,7]
# Compare the proportion of data in each cluster for the customer data to the
# proportion of data in each cluster for the general population.

prop = pd.DataFrame({'General Population Prop': prop_cluster_general,
                     'Customers Prop': prop_cluster_customers})
ax = prop.plot.bar(rot=0, figsize=(20, 10))
# What kinds of people are part of a cluster that is overrepresented in the
# customer data compared to the general population?

over = scaler.inverse_transform(pca.inverse_transform(customers_pca[np.where(customers_pred==3)])).round()
df_under = pd.DataFrame(data=over, columns=customers_clean.columns)
df_under.head()
# What kinds of people are part of a cluster that is underrepresented in the
# customer data compared to the general population?

under = scaler.inverse_transform(pca.inverse_transform(customers_pca[np.where(customers_pred==1)])).round()
df_under = pd.DataFrame(data=under, columns=customers_clean.columns)
df_under.head()<jupyter_output><empty_output><jupyter_text>Part 2: Supervised Learning ModelNow that you've found which parts of the population are more likely to be customers of the mail-order company, it's time to build a prediction model. Each of the rows in the "MAILOUT" data files represents an individual that was targeted for a mailout campaign. Ideally, we should be able to use the demographic information from each individual to decide whether or not it will be worth it to include that person in the campaign.The "MAILOUT" data has been split into two approximately equal parts, each with almost 43 000 data rows. In this part, you can verify your model with the "TRAIN" partition, which includes a column, "RESPONSE", that states whether or not a person became a customer of the company following the campaign. In the next part, you'll need to create predictions on the "TEST" partition, where the "RESPONSE" column has been withheld.<jupyter_code>mailout_train = pd.read_csv('../../data/Term2/capstone/arvato_data/Udacity_MAILOUT_052018_TRAIN.csv', sep=';')
print(mailout_train.shape)
mailout_train.head()
# As indicated within the project, our target is the variable RESPONSE
responses_count = mailout_train['RESPONSE'].value_counts(dropna=False)
responses_count
perc_no_resp = responses_count.values[0] * 100 / len(mailout_train)
perc_resp = 100 - perc_no_resp
print('{}% of users did not respond and {}% did'.format(np.round(perc_no_resp,2), np.round(perc_resp,2)))<jupyter_output>98.76% of users did not respond and 1.24% did<jupyter_text>**Preparing the data:**<jupyter_code># Cleaning data
mailout_clean = clean_data(mailout_train)
# Response values
responses_count = mailout_clean['RESPONSE'].value_counts(dropna=False)
responses_count
perc_no_resp = responses_count.values[0] * 100 / len(mailout_clean)
perc_resp = 100 - perc_no_resp
print('{}% no response'.format(np.round(perc_no_resp,2)))
print('{}% response'.format(np.round(perc_resp,2)))
# Cheking number of registers with missing value
mailout_clean.isnull().sum().sum()
# Checking number of object data (in case there is any, we need to remove it) 
mailout_clean.select_dtypes(include='object').sum().sum()
# Drop LNR, the ID variable (that will not provide information to the model)
mailout_clean.drop('LNR', axis=1, inplace=True)
# split data to features and labels
target = mailout_clean['RESPONSE']

# drop RESPONSE variable
features = mailout_clean.drop('RESPONSE', axis=1)
# Imputer
imputed_mailout = pd.DataFrame(imputer.fit_transform(features.values), columns=features.columns)
# feature scaling
scaled_mailout = pd.DataFrame(scaler.fit_transform(imputed_mailout.values), columns=imputed_mailout.columns)<jupyter_output><empty_output><jupyter_text>**Modeling the data:**<jupyter_code># Initialize the three models
model_A = RandomForestClassifier(random_state=42)
model_B = AdaBoostClassifier(random_state=42)
model_C = GradientBoostingClassifier(random_state=42)
model_D = RandomForestRegressor(random_state=42)
model_E = LogisticRegression(random_state=42)
model_F = GradientBoostingRegressor(random_state=42)<jupyter_output><empty_output><jupyter_text>**Note:** In order to measure the models performance and choose the most suitable one for the purposes of our problem, the KFold cross validation model will be used. The mailout dataset will be divided in k (5 will be determined in our case) different subsets through KFold, and then this method will provide train and test indices to split data in train and test sets. Each fold will then be used once as a validation while the k-1 remaining folds form the training set.<jupyter_code>subsets = KFold(n_splits=5, random_state=42)
subsets.get_n_splits(features, target)
print(subsets)
for i, model in enumerate([model_A, model_B, model_C, model_D, model_E, model_F]):
    
    j = 0
    for train_idx, valid_idx in subsets.split(features, target):
        
        X_train, X_valid = features.iloc[train_idx], features.iloc[valid_idx]
        y_train, y_valid = target.iloc[train_idx], target.iloc[valid_idx]

        model.fit(X_train, y_train)

        y_pred = model.predict(X_valid)
        
        score = roc_auc_score(y_valid, y_pred)
        
        j += 1

        print('Classifier {}: Fold {} Score: {}'.format(i+1, j, score))<jupyter_output>Classifier 1: Fold 1 Score: 0.5
Classifier 1: Fold 2 Score: 0.5
Classifier 1: Fold 3 Score: 0.5
Classifier 1: Fold 4 Score: 0.5
Classifier 1: Fold 5 Score: 0.5
Classifier 2: Fold 1 Score: 0.49992603550295855
Classifier 2: Fold 2 Score: 0.5
Classifier 2: Fold 3 Score: 0.5
Classifier 2: Fold 4 Score: 0.49992617746936363
Classifier 2: Fold 5 Score: 0.5
Classifier 3: Fold 1 Score: 0.4996301775147929
Classifier 3: Fold 2 Score: 0.49977774485108906
Classifier 3: Fold 3 Score: 0.5050059737156511
Classifier 3: Fold 4 Score: 0.49992617746936363
Classifier 3: Fold 5 Score: 0.49963006806747556
Classifier 4: Fold 1 Score: 0.5964336636486776
Classifier 4: Fold 2 Score: 0.5922540139910405
Classifier 4: Fold 3 Score: 0.5853428912783751
Classifier 4: Fold 4 Score: 0.5584917002383414
Classifier 4: Fold 5 Score: 0.5427597274482447
Classifier 5: Fold 1 Score: 0.5
Classifier 5: Fold 2 Score: 0.5
Classifier 5: Fold 3 Score: 0.5
Classifier 5: Fold 4 Score: 0.5
Classifier 5: Fold 5 Score: 0.5
Classifier 6: F[...]<jupyter_text>As the results show, the 6h classifier (model_F, GradientBoostingRegressor) is the one providing best results with the AUC ROC metric, so we will continue working with this model from now on.<jupyter_code># Inorder to evaluate models, separate the mailout_train data into train and validation data
X_train, X_test, y_train, y_test = train_test_split(features, target, test_size=0.3, stratify=target)
parameters = {'n_estimators': [50, 100, 200],
              'min_samples_split': [2, 3, 4]}

gb_model = GridSearchCV(estimator=model_F, param_grid=parameters, scoring='roc_auc')
gb_model.fit(X_train, y_train)

gb_model.best_params_
model = GradientBoostingRegressor(min_samples_split=2, n_estimators=50, random_state=42)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
fpr, tpr, _ = metrics.roc_curve(y_test,  y_pred)
auc = roc_auc_score(y_test, y_pred)
plt.plot(fpr, tpr, label="data 1, auc="+str(auc))
plt.legend(loc=4)
plt.show()
print(auc)<jupyter_output><empty_output><jupyter_text>**Discusion about AUC ROC metric:**When working with classification problems, AUC (Area Under The Curve) ROC (Receiver Operating Characteristics) curves are one of the most important evaluation metrics for checking the model’s performance.This curve is a measurement of the performance for a classification problem at various thresholds settings. ROC is a probability curve and AUC represents degree or measure of separability, and informs of how much the model is capable of distinguishing between classes. Higher the AUC, better the model is at predicting, thus is a model has an AUC near to 1 it means it has a good measure of separability and it is an excellent model. It is also worth mentioning that, when AUC is 0.5, it means model has no class separation capacity at all. On the other hand, the ROC curve is plotted with TPR (True Positive Rate) against the FPR (False Positive Rate) where TPR is on y-axis and FPR is on the x-axis.As it can be observed in the previous graph, our curve provides, for the chosen model, values of TPR higher than FPR at any stage of the curve and an area of 0.7925, which means there is a good separability degree, our model is performing pretty well and no overfitting is being produced.  **Dicusion about possible improvements on the model:**Even though our model showed quite good results on its performance, some aspects could be improved in order to optimize the metric thus the outcomes. For example, we could perform more hyper parameters tuning including more values for the parameters specified on the Grid Search method applied. We could also have added more parameters to the method, not just the n_stimators and the min_samples_split. We could have also tested more models at the beginning of the process in order to see if its AUC value was higher than in the case of Gradient Boosting Regression. As well, optimizing the pre-processing of data could also improve the model further.  Part 3: Kaggle CompetitionNow that you've created a model to predict which individuals are most likely to respond to a mailout campaign, it's time to test that model in competition through Kaggle. If you click on the link [here](http://www.kaggle.com/t/21e6d45d4c574c7fa2d868f0e8c83140), you'll be taken to the competition page where, if you have a Kaggle account, you can enter. If you're one of the top performers, you may have the chance to be contacted by a hiring manager from Arvato or Bertelsmann for an interview!Your entry to the competition should be a CSV file with two columns. The first column should be a copy of "LNR", which acts as an ID number for each individual in the "TEST" partition. The second column, "RESPONSE", should be some measure of how likely each individual became a customer – this might not be a straightforward probability. As you should have found in Part 2, there is a large output class imbalance, where most individuals did not respond to the mailout. Thus, predicting individual classes and using accuracy does not seem to be an appropriate performance evaluation method. Instead, the competition will be using AUC to evaluate performance. The exact values of the "RESPONSE" column do not matter as much: only that the higher values try to capture as many of the actual customers as possible, early in the ROC curve sweep.<jupyter_code>mailout_test = pd.read_csv('../../data/Term2/capstone/arvato_data/Udacity_MAILOUT_052018_TEST.csv', sep=';')
# Clean dat
clean_test = clean_data(mailout_test)
LNR_test = clean_test['LNR']
clean_test.drop('LNR', axis=1, inplace=True)
# Cheking number of registers with missing value
clean_test.isnull().sum().sum()
# Checking number of object data (in case there is any, we need to remove it) 
clean_test.select_dtypes(include='object').sum().sum()
# Imputer
imputed_test = pd.DataFrame(imputer.transform(clean_test.values), columns=clean_test.columns)
# feature scaling
scaled_test = pd.DataFrame(scaler.transform(imputed_test.values), columns=imputed_test.columns)
# Model A
pred = model.predict(scaled_test)
submission = pd.DataFrame(LNR_test)
submission['RESPONSE'] = pred
# Save file to submit
submission.to_csv('Submission.csv', index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10612415232133542, 'fraction_numerical': 0.03440792905581638, 'mean_word_length': 5.000500813822462, 'pattern_counts': {'":': 0, '<': 99, '<?xml version=': 0, '>': 98, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 17, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '221334', 'license': 'MIT', 'n_tokens_mistral': 17460, 'n_tokens_neox': 16003, 'n_words': 4597, 'path': 'Arvato Project.ipynb', 'repo_name': 'kushaldev75/Bertelsmann-Arvato-Customer-Segmentation', 'size': '47925'}