<jupyter_start><jupyter_text>Identify Customer SegmentsIn this project, I applied unsupervised learning techniques to identify segments of the population that form the core customer base for a mail-order sales company in Germany. These segments can then be used to direct marketing campaigns towards audiences that will have the highest expected rate of returns.<jupyter_code># import libraries here; add more as necessary
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.preprocessing import Imputer, StandardScaler
from sklearn.decomposition import PCA
from sklearn.cluster import KMeans
# magic word for producing visualizations in notebook
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Step 0: Load the DataThere are four files associated with this project:- `Udacity_AZDIAS_Subset.csv`: Demographics data for the general population of Germany; 891211 persons (rows) x 85 features (columns).- `Udacity_CUSTOMERS_Subset.csv`: Demographics data for customers of a mail-order company; 191652 persons (rows) x 85 features (columns).- `Data_Dictionary.md`: Detailed information file about the features in the provided datasets.- `AZDIAS_Feature_Summary.csv`: Summary of feature attributes for demographics data; 85 features (rows) x 4 columnsEach row of the demographics files represents a single person, but also includes information outside of individuals, including information about their household, building, and neighborhood. I used this information to cluster the general population into groups with similar demographic properties. Then, I examined how the people in the customers dataset fit into those created clusters. The hypothesis was that certain clusters were over-represented in the customers data, as compared to the general population; those over-represented clusters would be assumed to be part of the core userbase. This information can then be used for further applications, such as targeting for a marketing campaign.<jupyter_code># Load in the general demographics data.
azdias = pd.read_csv('Udacity_AZDIAS_Subset.csv', sep = ';')
original_columns = set(azdias.columns)
azdias.head()
# Load in the feature summary file.
feat_info = pd.read_csv('AZDIAS_Feature_Summary.csv', sep = ';')
feat_info.head()<jupyter_output><empty_output><jupyter_text>Step 1: Preprocessing Step 1.1: Assess Missing DataThe feature summary file contains a summary of properties for each demographics data column. I used this file to make cleaning decisions. First of all, I assessed the demographics data in terms of missing data.  Step 1.1.1: Convert Missing Value Codes to NaNsThe fourth column of the feature attributes summary (loaded in above as `feat_info`) documents the codes from the data dictionary that indicate missing or unknown data. I converted data that matches a 'missing' or 'unknown' value code into a numpy NaN value.<jupyter_code># Identify missing or unknown data values and convert them to NaNs.
non_numerical = []
for c in azdias.columns:
    if azdias[c].dtypes == 'object':
        non_numerical.append(c)
        
print(non_numerical)       
for c in non_numerical:
    azdias[c] = azdias[c].replace(['XX', 'X'], np.nan)
numerical = [ c for c in azdias.columns if c not in non_numerical]
for c in numerical:
    missing_or_unknown = feat_info[feat_info.attribute == c].missing_or_unknown.values[0]
    if missing_or_unknown != '[]': 
        nan = [int(i) for i in missing_or_unknown.strip('[]').split(',')]
        azdias[c] = azdias[c].replace(nan, np.nan)<jupyter_output><empty_output><jupyter_text>Step 1.1.2: Assess Missing Data in Each ColumnThere are a few columns that are outliers in terms of the proportion of values that are missing. I used matplotlib's [`hist()`](https://matplotlib.org/api/_as_gen/matplotlib.pyplot.hist.html) function to visualize the distribution of missing value counts to find these columns.<jupyter_code># Perform an assessment of how much missing data there is in each column of the
# dataset.

perc_missing = azdias.isnull().mean().sort_values(ascending = False)
perc_missing.hist(); 
perc_missing.head(10)
azdias.notnull().all().sum()
# Investigate patterns in the amount of missing data in each column.
columns_to_drop = perc_missing.index[:6]
for c in columns_to_drop:
    display(feat_info[feat_info.attribute == c])
# Remove the outlier columns from the dataset. 
# azdias =  azdias.drop(columns_to_drop, axis = 1)<jupyter_output><empty_output><jupyter_text>Discussion 1.1.2: Assess Missing Data in Each ColumnThere are 6 columns ('TITEL_KZ', 'AGER_TYP', 'KK_KUNDENTYP', 'KBA05_BAUMAX', 'GEBURTSJAHR','ALTER_HH') with more than 30% of missing information. There are 24 columns without any missing information. The rest of the columns have less than 20% of missing information. I ended up keeping those columns because the customers dataset has a different distribution of missing information. In order to keep the columns consistent between the two datasets, I did not removed those columns.   Step 1.1.3: Assess Missing Data in Each RowI performed a similar assessment for the rows of the dataset. I divided the data into two subsets: one for data points that are above some threshold for missing values, and a second subset for points below that threshold.<jupyter_code># How much data is missing in each row of the dataset?
perc_missing = azdias.isnull().mean(axis = 1).sort_values(ascending = False)
perc_missing.hist();
# Divide the data into two subsets based on the number of missing
# values in each row.

high_perc_missing = perc_missing[perc_missing > 0.35].index
low_perc_missing = perc_missing[perc_missing <= 0.35].index
len(high_perc_missing)/len(perc_missing)
high_perc_missing = azdias.iloc[high_perc_missing]
low_perc_missing = azdias.iloc[low_perc_missing]
# Identify columns where there are no or few missing values
high_perc_missing.notnull().mean().sort_values(ascending = False).index[:30]
columns_low_perc_missing = ['ZABEOTYP', 'SEMIO_TRADV', 'GREEN_AVANTGARDE', 'FINANZTYP', 'ANREDE_KZ', 'ALTERSKATEGORIE_GROB', 'ONLINE_AFFINITAET', 'RETOURTYP_BK_S',
       'CJT_GESAMTTYP', 'GFK_URLAUBERTYP', 'LP_STATUS_GROB']
def plot_dist(c):
    plt.subplot(121)
    sns.countplot(high_perc_missing[c])
    plt.title('high_perc_missing')
    plt.subplot(122)
    sns.countplot(low_perc_missing[c])
    plt.title('low_perc_missing')
    plt.show()
# Compare the distribution of values for columns where there are
# no or few missing values, between the two subsets.

for c in columns_low_perc_missing:
    plot_dist(c)<jupyter_output><empty_output><jupyter_text>Discussion 1.1.3: Assess Missing Data in Each Row The data with many missing values looks very different from the data with few or no missing values. As a result, I made a note on those data as special, and will revisit these data later on.   Step 1.2: Select and Re-Encode FeaturesIn this step, I made a few encoding changes or additional assumptions to be able to make progress. While almost all of the values in the dataset are encoded using numbers, not all of them represent numeric values. The third column of the feature summary (`feat_info`) shows a summary of types of measurement.- For numeric and interval data, I kept these features without changes.- Most of the variables in the dataset are ordinal in nature. While ordinal values may technically be non-linear in spacing, I made the simplifying assumption that the ordinal variables could be treated as being interval in nature (that is, kept without any changes).- The remaining two variable types: categorical, and 'mixed' needed special handling .<jupyter_code># How many features are there of each data type?

feat_info.type.value_counts()<jupyter_output><empty_output><jupyter_text>Step 1.2.1: Re-Encode Categorical FeaturesFor categorical data, I re-encoded the levels as dummy variables.<jupyter_code># Assess categorical variables: which are binary, which are multi-level, and
# which one needs to be re-encoded?

categorical = [c for c in feat_info[feat_info.type == 'categorical'].attribute.values] # if c not in columns_to_drop]
levels = [len(azdias[c].value_counts()) for c in categorical]        

df = pd.DataFrame(np.array([categorical, levels]).T, columns = ['categorical', 'levels'])
df = df.set_index('categorical')
df['levels'] = df['levels'].astype(int)
df = df.sort_values('levels')

binary = df[df.levels == 2].index
for c in binary:
    print(azdias[c].value_counts())

# Re-encode the binary categories
azdias['OST_WEST_KZ'] = azdias['OST_WEST_KZ'].apply(lambda x: 1 if x == 'W' else 0)
azdias['ANREDE_KZ'] = azdias['ANREDE_KZ'].apply(lambda x: 1 if x == 2 else 0)
azdias['VERS_TYP'] = azdias['VERS_TYP'].apply(lambda x: 1 if x == 2 else 0)
multi = df[(df.levels >2) & (df.levels <44)].index
columns_to_drop = df[df.levels >2].index

# Re-encode categorical variable(s) to be kept in the analysis.
azdias[multi] = azdias[multi].astype('str')
ohc = pd.get_dummies(azdias[multi], drop_first = True)
azdias = azdias.drop(columns_to_drop, axis = 1)
azdias = pd.concat([azdias, ohc], axis = 1)<jupyter_output><empty_output><jupyter_text>Discussion 1.2.1: Re-Encode Categorical FeaturesI re-encoded the binary variables using 0 and 1, and re-encoded the multi-level variables with one-hot-encoder. I dropped the multi-level variables with 44 levels.   Step 1.2.2: Engineer Mixed-Type FeaturesThere are a two features that are marked as "mixed" in the feature summary that required special treatment in order to be included in the analysis. - "PRAEGENDE_JUGENDJAHRE" combines information on three dimensions: generation by decade, movement (mainstream vs. avantgarde), and nation (east vs. west). I created two new variables to capture the other two dimensions: a numeric variable for decade, and a binary variable for movement.- "CAMEO_INTL_2015" combines information on two axes: wealth and life stage. I broke up the two-digit codes by their 'tens'-place and 'ones'-place digits into two new ordinal variables (which, for the purposes of this project, is equivalent to just treating them as their raw numeric values).<jupyter_code># Investigate "PRAEGENDE_JUGENDJAHRE" and engineer two new variables.
"""### 1.18. PRAEGENDE_JUGENDJAHRE
Dominating movement of person's youth (avantgarde vs. mainstream; east vs. west)
- -1: unknown
-  0: unknown
-  1: 40s - war years (Mainstream, E+W)
-  2: 40s - reconstruction years (Avantgarde, E+W)
-  3: 50s - economic miracle (Mainstream, E+W)
-  4: 50s - milk bar / Individualisation (Avantgarde, E+W)
-  5: 60s - economic miracle (Mainstream, E+W)
-  6: 60s - generation 68 / student protestors (Avantgarde, W)
-  7: 60s - opponents to the building of the Wall (Avantgarde, E)
-  8: 70s - family orientation (Mainstream, E+W)
-  9: 70s - peace movement (Avantgarde, E+W)
- 10: 80s - Generation Golf (Mainstream, W)
- 11: 80s - ecological awareness (Avantgarde, W)
- 12: 80s - FDJ / communist party youth organisation (Mainstream, E)
- 13: 80s - Swords into ploughshares (Avantgarde, E)
- 14: 90s - digital media kids (Mainstream, E+W)
- 15: 90s - ecological awareness (Avantgarde, E+W)"""
# azdias["PRAEGENDE_JUGENDJAHRE"] = azdias["PRAEGENDE_JUGENDJAHRE"].astype('float')

decade_map = {1: 40, 2: 40, 3: 50, 4: 50, 5: 60, 6: 60, 7: 60, 8: 70, 9: 70, 10: 80, 11: 80, 12: 80, 13: 80, 14: 90, 15: 90}
azdias["MOVEMENT"] = azdias["PRAEGENDE_JUGENDJAHRE"].apply(lambda x: 1 if x in [1, 3, 5, 8, 10, 12, 14] else 0)
azdias['DECADE'] = azdias["PRAEGENDE_JUGENDJAHRE"].map(decade_map)
azdias[["MOVEMENT",'DECADE']].head()
# Investigate "CAMEO_INTL_2015" and engineer two new variables.
"""### 4.3. CAMEO_INTL_2015
German CAMEO: Wealth / Life Stage Typology, mapped to international code
- -1: unknown
- 11: Wealthy Households - Pre-Family Couples & Singles
- 12: Wealthy Households - Young Couples With Children
- 13: Wealthy Households - Families With School Age Children
- 14: Wealthy Households - Older Families &  Mature Couples
- 15: Wealthy Households - Elders In Retirement
- 21: Prosperous Households - Pre-Family Couples & Singles
- 22: Prosperous Households - Young Couples With Children
- 23: Prosperous Households - Families With School Age Children
- 24: Prosperous Households - Older Families & Mature Couples
- 25: Prosperous Households - Elders In Retirement
- 31: Comfortable Households - Pre-Family Couples & Singles
- 32: Comfortable Households - Young Couples With Children
- 33: Comfortable Households - Families With School Age Children
- 34: Comfortable Households - Older Families & Mature Couples
- 35: Comfortable Households - Elders In Retirement
- 41: Less Affluent Households - Pre-Family Couples & Singles
- 42: Less Affluent Households - Young Couples With Children
- 43: Less Affluent Households - Families With School Age Children
- 44: Less Affluent Households - Older Families & Mature Couples
- 45: Less Affluent Households - Elders In Retirement
- 51: Poorer Households - Pre-Family Couples & Singles
- 52: Poorer Households - Young Couples With Children
- 53: Poorer Households - Families With School Age Children
- 54: Poorer Households - Older Families & Mature Couples
- 55: Poorer Households - Elders In Retirement
- XX: unknown"""
azdias["CAMEO_INTL_2015"] = azdias["CAMEO_INTL_2015"].astype(float)
azdias["WEALTH"] = azdias["CAMEO_INTL_2015"].apply(lambda x: x//10)
azdias["LIEF_STAGE"] = azdias["CAMEO_INTL_2015"].apply(lambda x: x%10)
azdias[["WEALTH", "LIEF_STAGE"]].head()
azdias = azdias.drop(["PRAEGENDE_JUGENDJAHRE", "CAMEO_INTL_2015"], axis =1)<jupyter_output><empty_output><jupyter_text>Discussion 1.2.2: Engineer Mixed-Type FeaturesI splited the "PRAEGENDE_JUGENDJAHRE" column into two columns: "MOVEMENT",'DECADE'. I splited the "CAMEO_INTL_2015" into two columns: "WEALTH", "LIEF_STAGE". Then I dropped "PRAEGENDE_JUGENDJAHRE" and "CAMEO_INTL_2015".   Step 1.2.3: Complete Feature Selection<jupyter_code>azdias.shape
new_columns = set(azdias.columns)

new_columns.difference(original_columns)<jupyter_output><empty_output><jupyter_text>Step 1.3: Create a Cleaning FunctionI created the function below to execute the main feature selection, encoding, and re-engineering steps performed above. So that I can just run this function on the DataFrame to get the trimmed dataset in a single step.<jupyter_code>def clean_data(df):
    """
    Perform feature trimming, re-encoding, and engineering for demographics
    data
    
    INPUT: Demographics DataFrame
    OUTPUT: Trimmed and cleaned demographics DataFrame
    """
    feat_info = pd.read_csv('AZDIAS_Feature_Summary.csv', sep = ';')
    
    # Identify missing or unknown data values and convert them to NaNs.
    non_numerical = [c for c in df.columns if df[c].dtypes == 'object']
    for c in non_numerical:
        df[c] = df[c].replace(['XX', 'X'], np.nan)  

    numerical = [ c for c in df.columns if c not in non_numerical]
    for c in numerical:
        missing_or_unknown = feat_info[feat_info.attribute == c].missing_or_unknown.values[0]
        if missing_or_unknown != '[]': 
            nan = [int(i) for i in missing_or_unknown.strip('[]').split(',')]
            df[c] = df[c].replace(nan, np.nan)

    # Drop columns with more than 20% of data missing
#     perc_missing = df.isnull().mean().sort_values(ascending = False)

#     columns_to_drop = perc_missing[perc_missing > 0.2].index

#     df =  df.drop(columns_to_drop, axis = 1)

    # Assess categorical variables: which are binary, which are multi-level, and
    # which one needs to be re-encoded?
    categorical = [c for c in feat_info[feat_info.type == 'categorical'].attribute.values] # if c not in columns_to_drop]
    levels = [len(df[c].value_counts()) for c in categorical]        

    categorical_level_df = pd.DataFrame(np.array([categorical, levels]).T, columns = ['categorical', 'levels'])
    categorical_level_df = categorical_level_df.set_index('categorical')
    categorical_level_df['levels'] = categorical_level_df['levels'].astype(int)
    categorical_level_df = categorical_level_df.sort_values('levels')

#     binary = categorical_level_df[categorical_level_df.levels == 2].index

    # Re-encode the binary categories
    df['OST_WEST_KZ'] = df['OST_WEST_KZ'].apply(lambda x: 1 if x == 'W' else 0)
    df['ANREDE_KZ'] = df['ANREDE_KZ'].apply(lambda x: 1 if x == 2 else 0)
    df['VERS_TYP'] = df['VERS_TYP'].apply(lambda x: 1 if x == 2 else 0)

    multi = categorical_level_df[(categorical_level_df.levels >2) & (categorical_level_df.levels <44)].index
    columns_to_drop = categorical_level_df[categorical_level_df.levels >2].index

    # Re-encode multi-level categorical variable(s) to be kept in the analysis.
    df[multi] = df[multi].astype('str')
    ohc = pd.get_dummies(df[multi], drop_first = True)
    df = df.drop(columns_to_drop, axis = 1)
    df = pd.concat([df, ohc], axis = 1)

    # Investigate "PRAEGENDE_JUGENDJAHRE" and engineer two new variables.
    decade_map = {1: 40, 2: 40, 3: 50, 4: 50, 5: 60, 6: 60, 7: 60, 8: 70, 9: 70, 10: 80, 11: 80, 12: 80, 13: 80, 14: 90, 15: 90}
    df["MOVEMENT"] = df["PRAEGENDE_JUGENDJAHRE"].apply(lambda x: 1 if x in [1, 3, 5, 8, 10, 12, 14] else 0)
    df['DECADE'] = df["PRAEGENDE_JUGENDJAHRE"].map(decade_map)

    # Investigate "CAMEO_INTL_2015" and engineer two new variables.
    df["CAMEO_INTL_2015"] = df["CAMEO_INTL_2015"].astype(float)
    df["WEALTH"] = df["CAMEO_INTL_2015"].apply(lambda x: x//10)
    df["LIEF_STAGE"] = df["CAMEO_INTL_2015"].apply(lambda x: x%10)
    return df<jupyter_output><empty_output><jupyter_text>Step 2: Feature Transformation Step 2.1: Apply Feature ScalingBefore applying dimensionality reduction techniques to the data, i performed feature scaling so that the principal component vectors are not influenced by the natural differences in scale for features. - sklearn requires that data not have missing values in order for its estimators to work properly. So, before applying the scaler, I removed all data points with missing data by applying an [Imputer](http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.Imputer.html).- For the actual scaling function, I scaled each feature to mean 0 and standard deviation 1 by applying a [StandardScaler](http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html).<jupyter_code># Apply Imputer

imputer = Imputer()
imputed_azdias = imputer.fit_transform(azdias)
# Apply feature scaling to the general population demographics data.

scaler = StandardScaler()
scaled_azdias = scaler.fit_transform(imputed_azdias)<jupyter_output><empty_output><jupyter_text>Discussion 2.1: Apply Feature ScalingI used the Imputer to replace the missing values with the mean of the column, and then used the Standard Scaler to scale the data.   Step 2.2: Perform Dimensionality ReductionOn the scaled data, I applied dimensionality reduction techniques.- I used sklearn's [PCA](http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html) class to apply principal component analysis on the data, thus finding the vectors of maximal variance in the data.- I examined the ratio of variance explained by each principal component as well as the cumulative variance explained. I also plotted the cumulative or sequential values using matplotlib's [`plot()`](https://matplotlib.org/api/_as_gen/matplotlib.pyplot.plot.html) function. Based on the plot, I selected the number of transformed features to be retained for the clustering.- I re-fit a PCA instance to perform the decided-on transformation.<jupyter_code># Apply PCA to the data.
pca = PCA()

pca.fit_transform(scaled_azdias)
pca.explained_variance_ratio_.cumsum()[:66]
# Investigate the variance accounted for by each principal component.

plt.plot(pca.explained_variance_ratio_.cumsum())
# Re-apply PCA to the data while selecting for number of components to retain.
pca = PCA(n_components = 66)
pc = pca.fit_transform(scaled_azdias)<jupyter_output><empty_output><jupyter_text>Discussion 2.2: Perform Dimensionality ReductionI retained 66 principal componentsare for the next step of the analysis. They contributed to 80% of the variability.   Step 2.3: Interpret Principal ComponentsAfter obtaining the transformed principal components, I examined the weight of each variable on the first few components to see if they can be interpreted in some fashion.Each principal component is a unit vector that points in the direction of highest variance (after accounting for the variance captured by earlier principal components). The further a weight is from zero, the more the principal component is in the direction of the corresponding feature. If two features have large weights of the same sign (both positive or both negative), then increases in one tend expect to be associated with increases in the other. To contrast, features with different signs can be expected to show a negative correlation: increases in one variable should result in a decrease in the other.To investigate the features, I mapped each weight to their corresponding feature name, then sort the features according to weight. The most interesting features for each principal component are those at the beginning and end of the sorted list. I used the data dictionary document to understand these most prominent features, their relationships, and what a positive or negative value on the principal component might indicate.<jupyter_code># Map weights for the first principal component to corresponding feature names
# and then print the linked values, sorted by weight.

pca_df = pd.DataFrame(pca.components_, columns = azdias.columns)
pca_df.head()
def feature_and_weight(pc):
    return pca_df.iloc[pc-1].sort_values(ascending = False)
feature_and_weight(1)
# Map weights for the second principal component to corresponding feature names
# and then print the linked values, sorted by weight.

feature_and_weight(2)
# Map weights for the third principal component to corresponding feature names
# and then print the linked values, sorted by weight.

feature_and_weight(3)<jupyter_output><empty_output><jupyter_text>Discussion 2.3: Interpret Principal ComponentsEach principal component has one dimension. Within that dimention, the sign (positive or negative) of each variable's weight shows the direction that varialbe is going.  For principal component 1, the variables with important roles in explaining the variation on the dimention include those with the highest positive weights (PLZ8_ANTG3, PLZ8_ANTG4, PLZ8_BAUMAX, WEALTH, KBA05_BAUMAX) and those with the lowest negative weights (PLZ8_ANTG1, KBA05_ANTG1 FINANZ_MINIMALIST, MOBI_REGIO). The variables in the middle have smaller roles in explaining the variation. As the weights getting closer to zero, the variables do not play any role in explaining the variation. PLZ8_ANTG3 and PLZ8_ANTG1 have almost perfect contrast, one with the weight close to 0.18 and the other with the weight close to -0.18. Finally, the principle component scores derived from this principle component will show that individuals with negative principle component scores will tend to have greater values of PLZ8_ANTG1, KBA05_ANTG1 FINANZ_MINIMALIST, and MOBI_REGIO, and lower values for the remaining Vars. While individuals with principle component scores greater than 0 will tend to have greater values of PLZ8_ANTG3, PLZ8_ANTG4, PLZ8_BAUMAX, WEALTH, and KBA05_BAUMAX, and lower of the remaining.  Step 3: Clustering Step 3.1: Apply Clustering to General PopulationIn this substep, I applied k-means clustering to the dataset and used the average within-cluster distances from each point to their assigned cluster's centroid to decide on a number of clusters to keep.- I used sklearn's [KMeans](http://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.htmlsklearn.cluster.KMeans) class to perform k-means clustering on the PCA-transformed data.- I then computed the average difference from each point to its assigned cluster's center for a number of different cluster counts. - After selecting a final number of clusters to use, I re-fit a KMeans instance to perform the clustering operation.<jupyter_code># Investigate the change in within-cluster distance across number of clusters.
ks = range(1, 11)
inertias = []
for k in ks:
    # Initialize the KMeans object using the current number of clusters (k)
    km = KMeans(n_clusters=k, random_state=0)
    # Fit the scaled features to the KMeans object
    km.fit(pc)
    # Append the inertia for `km` to the list of inertias
    inertias.append(km.inertia_)
    
# Plot the results in a line plot
plt.plot(ks, inertias, marker='o')
# Re-fit the k-means model with the selected number of clusters and obtain
# cluster predictions for the general population demographics data.

km = KMeans(random_state = 0, n_clusters = 7)<jupyter_output><empty_output><jupyter_text>Discussion 3.1: Apply Clustering to General PopulationI decided to segment the population into 7 clusters based on the elbow plot.   Step 3.2: Apply All Steps to the Customer DataAfter obtaining the clusters and cluster centers for the general population, I examined how the customer data maps on to those clusters.<jupyter_code># Load in the the general population data 
azdias = pd.read_csv('Udacity_AZDIAS_Subset.csv', sep = ';')

# Load in the customer demographics data.
customers = pd.read_csv('Udacity_CUSTOMERS_Subset.csv', sep = ';')
# Clean the general population data
azdias_clean = clean_data(azdias)

# Clean the customer demographics data
customers_clean = clean_data(customers)
# Compare the columns of the general population and the customer demographics data
old = set(azdias_clean.columns)
new = set(customers_clean.columns)
columns_to_drop = list(old.difference(new))

# Drop the columns that are not consistent between the two
azdias_clean = azdias_clean.drop(columns_to_drop, axis = 1)
columns_to_drop
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import Imputer, StandardScaler
from sklearn.decomposition import PCA
from sklearn.cluster import KMeans

# Build the preprocessing and training pipeline
pipe = Pipeline([
        ('imputer', Imputer()), 
        ('scaler', StandardScaler())]
)

# Preprocssing the general population data
azdias_transformed = pipe.fit_transform(azdias_clean)

# Preprocessing the customer demographics data
customer_transformed = pipe.transform(customers_clean)
# Instantiate PCA
pca = PCA(n_components = 66, random_state = 0)

# Dimentionality reduction for the general population data
azdias_pc = pca.fit_transform(azdias_transformed)

# Dimentionality reduction for the customer demographics data
customer_pc = pca.transform(customer_transformed)
# Instantiate KMeans
km = KMeans(n_clusters = 7, random_state=0)

# Clustering for the general population data
azdias_km = km.fit_transform(azdias_transformed)
demographic_clusters = km.labels_

# Predict the clustering for the customer demographics data
customer_clusters_prediction = km.predict(customer_transformed)<jupyter_output><empty_output><jupyter_text>Step 3.3: Compare Customer Data to Demographics DataI clustered data based on demographics of the general population of Germany, and seen how the customer data for a mail-order sales company maps onto those demographic clusters in the previous steps. In this final substep, I compared the two cluster distributions to see where the strongest customer base for the company is.I considered the proportion of persons in each cluster for the general population, and the proportions for the customers. If the company's customer base to be universal, then the cluster assignment proportions should be fairly similar between the two. If there are only particular segments of the population that are interested in the company's products, there would be a mismatch from one to the other. If there is a higher proportion of persons in a cluster for the customer data compared to the general population (e.g. 5% of persons are assigned to a cluster for the general population, but 15% of the customer data is closest to that cluster's centroid), it suggests the people in that cluster to be a target audience for the company. On the other hand, the proportion of the data in a cluster being larger in the general population than the customer data (e.g. only 2% of customers closest to a population centroid that captures 6% of the data) suggests that group of persons to be outside of the target demographics.- I computed the proportion of data points in each cluster for the general population and the customer data, and visualized it. - I then performed an investigation for the overrepresented and underrepresented clusters.<jupyter_code># Create a function to plot the proportion of data in each cluster 

def plot_percent(data, label):
    df = pd.DataFrame(dict(cluster=data))
    ax = sns.barplot(x="cluster", y="cluster", data=df, estimator=lambda x: len(x) / len(df) * 100)
    ax.set(ylabel="Percent")
    plt.title(f'The number of individuals in each cluster from the {label} data')
# plot the proportion of data in each cluster for the general population
plot_percent(demographic_clusters, 'general demographic')
# plot the proportion of data in each cluster for the customer data
plot_percent(customer_clusters_prediction, 'customer')
# What kinds of people are part of a cluster that is overrepresented in the
# customer data compared to the general population?
index = np.where(demographic_clusters == 6)
cluster_6 = azdias.iloc[index]
# Age group
"""Estimated age based on given name analysis
- -1: unknown (missing)
-  0: unknown (cannot be determined)
-  1: < 30 years old
-  2: 30 - 45 years old
-  3: 46 - 60 years old
-  4: > 60 years old
-  9: uniformly distributed
"""
print('Age group of cluster 6 \n', \
      cluster_6['ALTERSKATEGORIE_GROB'].value_counts(normalize = True))
print(' ')
print('Age group of all the clusters \n', \
      azdias['ALTERSKATEGORIE_GROB'].value_counts(normalize = True))
# Preferred information and buying channels group
"""Customer-Journey-Typology: preferred information and buying channels for
consumer
- 0: unknown
- 1: Advertising- and Consumptionminimalist
- 2: Advertising- and Consumptiontraditionalist
- 3: advertisinginterested Store-shopper
- 4: advertisinginterested Online-shopper
- 5: Advertising- and Cross-Channel-Enthusiast
- 6: Advertising-Enthusiast with restricted Cross-Channel-Behaviour
"""
print('Preferred information and buying channels group of cluster 6 \n', \
      cluster_6['CJT_GESAMTTYP'].value_counts(normalize = True))
print(' ')
print('Preferred information and buying channels group of all the clusters \n', \
      azdias['CJT_GESAMTTYP'].value_counts(normalize = True))
# Preferred information and buying channels group
"""Most descriptive financial type for individual
- -1: unknown
-  1: low financial interest (MINIMALIST)
-  2: money-saver (SPARER)
-  3: home ownership (HAUSBAUER)
-  4: be prepared (VORSORGER)
-  5: investor (ANLEGER)
-  6: inconspicuous (UNAUFFAELLIGER)
"""
print('Financial type of cluster 6 \n', \
      cluster_6['FINANZTYP'].value_counts(normalize = True))
print(' ')
print('Financial type of all the clusters \n', \
      azdias['FINANZTYP'].value_counts(normalize = True))

# What kinds of people are part of a cluster that is underrepresented in the
# customer data compared to the general population?

index = np.where(demographic_clusters == 1)
cluster_1 = azdias.iloc[index]
# Age group
"""Estimated age based on given name analysis
- -1: unknown (missing)
-  0: unknown (cannot be determined)
-  1: < 30 years old
-  2: 30 - 45 years old
-  3: 46 - 60 years old
-  4: > 60 years old
-  9: uniformly distributed
"""
print('Age group of cluster 1 \n', \
      cluster_1['ALTERSKATEGORIE_GROB'].value_counts(normalize = True))
print(' ')
print('Age group of all the clusters \n', \
      azdias['ALTERSKATEGORIE_GROB'].value_counts(normalize = True))
# Preferred information and buying channels group
"""Customer-Journey-Typology: preferred information and buying channels for
consumer
- 0: unknown
- 1: Advertising- and Consumptionminimalist
- 2: Advertising- and Consumptiontraditionalist
- 3: advertisinginterested Store-shopper
- 4: advertisinginterested Online-shopper
- 5: Advertising- and Cross-Channel-Enthusiast
- 6: Advertising-Enthusiast with restricted Cross-Channel-Behaviour
"""
print('Preferred information and buying channels group of cluster 1 \n', \
      cluster_1['CJT_GESAMTTYP'].value_counts(normalize = True))
print(' ')
print('Preferred information and buying channels group of all the clusters \n', \
      azdias['CJT_GESAMTTYP'].value_counts(normalize = True))
# Preferred information and buying channels group
"""Most descriptive financial type for individual
- -1: unknown
-  1: low financial interest (MINIMALIST)
-  2: money-saver (SPARER)
-  3: home ownership (HAUSBAUER)
-  4: be prepared (VORSORGER)
-  5: investor (ANLEGER)
-  6: inconspicuous (UNAUFFAELLIGER)
"""
print('Financial type of cluster 1 \n', \
      cluster_1['FINANZTYP'].value_counts(normalize = True))
print(' ')
print('Financial type of all the clusters \n', \
      azdias['FINANZTYP'].value_counts(normalize = True))<jupyter_output>Financial type of cluster 1 
 1    0.376827
3    0.216698
4    0.206146
6    0.179066
5    0.011191
2    0.010072
Name: FINANZTYP, dtype: float64
 
Financial type of all the clusters 
 6    0.325808
1    0.223931
4    0.146569
2    0.124399
5    0.119427
3    0.059866
Name: FINANZTYP, dtype: float64
{'fraction_non_alphanumeric': 0.08377166766288105, 'fraction_numerical': 0.021159593544530782, 'mean_word_length': 4.916018387553041, 'pattern_counts': {'":': 0, '<': 69, '<?xml version=': 0, '>': 72, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '573999', 'license': 'MIT', 'n_tokens_mistral': 10673, 'n_tokens_neox': 9820, 'n_words': 3866, 'path': 'Identify_Customer_Segments.ipynb', 'repo_name': 'iDataist/Create-Customer-Segments', 'size': '33460'}