<jupyter_start><jupyter_text>There are four data files associated with this project:- `Udacity_AZDIAS_052018.csv`: Demographics data for the general population of Germany; 891 211 persons (rows) x 366 features (columns).- `Udacity_CUSTOMERS_052018.csv`: Demographics data for customers of a mail-order company; 191 652 persons (rows) x 369 features (columns).- `Udacity_MAILOUT_052018_TRAIN.csv`: Demographics data for individuals who were targets of a marketing campaign; 42 982 persons (rows) x 367 (columns).- `Udacity_MAILOUT_052018_TEST.csv`: Demographics data for individuals who were targets of a marketing campaign; 42 833 persons (rows) x 366 (columns).Each row of the demographics files represents a single person, but also includes information outside of individuals, including information about their household, building, and neighborhood. Use the information from the first two files to figure out how customers ("CUSTOMERS") are similar to or differ from the general population at large ("AZDIAS"), then use your analysis to make predictions on the other two files ("MAILOUT"), predicting which recipients are most likely to become a customer for the mail-order company.The "CUSTOMERS" file contains three extra columns ('CUSTOMER_GROUP', 'ONLINE_PURCHASE', and 'PRODUCT_GROUP'), which provide broad information about the customers depicted in the file. The original "MAILOUT" file included one additional column, "RESPONSE", which indicated whether or not each recipient became a customer of the company. For the "TRAIN" subset, this column has been retained, but in the "TEST" subset it has been removed; it is against that withheld column that your final predictions will be assessed in the Kaggle competition.Otherwise, all of the remaining columns are the same between the three data files. For more information about the columns depicted in the files, you can refer to two Excel spreadsheets provided in the workspace. [One of them](./DIAS Information Levels - Attributes 2017.xlsx) is a top-level list of attributes and descriptions, organized by informational category. [The other](./DIAS Attributes - Values 2017.xlsx) is a detailed mapping of data values for each feature in alphabetical order. As the first step I used the feature summary file (DIAS Attributes - Values 2017.xlsx) to understand the data, mainly:- what kind of information is provided in the dataset - how the values assigned to each attributes- data types of the features/attributes - missing values encoding<jupyter_code>df_features = pd.read_excel(
    'DIAS Attributes - Values 2017.xlsx', header=[1], usecols=[1, 3, 4], index_col=[0, 1])
df_features = df_features.reset_index()
df_features.head(5)
# "CAMEO_INTL_2015" in df_features.Attribute
dtype={'CAMEO_DEUG_2015': 'object', 'CAMEO_INTL_2015': 'object'}
azdias = pd.read_csv('./data/Udacity_AZDIAS_052018.csv',
                     sep=';', dtype = dtype, nrows=5000 )
del azdias['LNR'] 

attributes = list(set(np.concatenate([df_features.Attribute.unique(), azdias.columns])))

fea_ = np.setdiff1d(azdias.columns, df_features.Attribute.unique())
print('{:} features not explained in feature summary file:'.format(len(fea_)))
# print(fea_)
fea__ = np.setdiff1d( df_features.Attribute.unique(), azdias.columns)
print('{:} features not in azdias file:'.format(len(fea__)))
# print(fea__)<jupyter_output>42 features not in azdias file:<jupyter_text>From the comparison above, is noticed that there are 93 features not explained in the metadata file. We will need to figure out ourslves how these features are desribed in the general population dataset.<jupyter_code># remove the ones not used in azdias file 
attributes = list(set(attributes) - set(fea__))
customers = pd.read_csv(
    './data/Udacity_CUSTOMERS_052018.csv', sep=';',  dtype= dtype)
print('features in the customer, but not in the attribute list: {:}'.format(np.setdiff1d(customers.columns, attributes)))

mailout_train = pd.read_csv('./data/Udacity_MAILOUT_052018_TRAIN.csv', sep=';', dtype = dtype)
np.setdiff1d(mailout_train.columns, attributes)
print('features in the mailout, but not in the combined attribute list: {:}'.format(np.setdiff1d(mailout_train.columns, attributes)))

compare_dataframe_columns(df1 = customers, df2 = mailout_train, df1_name='customer', df2_name='mailout_train')
# Determine if the mailout_train is a subset of the customer
df1 = customers.drop(columns=['CUSTOMER_GROUP', 'ONLINE_PURCHASE','PRODUCT_GROUP']).copy()
df2 = mailout_train.drop(columns=['RESPONSE']).copy()

df = pd.concat([df1, df2])
df.duplicated().sum()
## Summary section
print('total number of features: {:}'.format(len(attributes))) 
print("total number of features in azdias: {:}".format(np.shape(azdias)[1]))
print("total number of features in customers: {:}".format(np.shape(customers)[1]))
print("total number of features in training data: {:}".format(np.shape(mailout_train)[1]))<jupyter_output>total number of features: 365
total number of features in azdias: 365
total number of features in customers: 369
total number of features in training data: 367<jupyter_text>feature summary In this section, I will utilize information contained in 'DIAS Attributes - Values 2017.xlsx' to provide a summary on attributes values. A quick scan show that data in file is mostly ordinal. For example,"D19_BANKEN_ANZ_12" describes the frequency of transaction activity of the client. the values range from least frequent (0 = very low activity) to the most frequent (6 = very high activity). Since the values used to describe the feature is ranking based, i will treat it as ordinal. As most of features are ordinal, i will pre-assign feature type as 'ordinal'. If further analysis shows otherwise, I will update it  at a later time.<jupyter_code>df_features[df_features["Attribute"] == 'D19_BANKEN_ANZ_12']
# crate a dataframe for feature summary
df_feat_summary = pd.DataFrame()
df_feat_summary['Attribute'] = attributes
df_feat_summary['type'] = 'ordinal' 

# create an empty list to contain cat/num features that need to be ajusted in the summary df
cat_features = []
num_features = []
df_feat_summary['Attribute'].head()<jupyter_output><empty_output><jupyter_text>Names of attributes in 'DIAS Attributes - Values 2017.xlsx' mostly compose of several parts. For example,  "D19_NAHRUNGSERGAENZUNG_RZ", the first part "D19" presents data source and the rest describe the attributes (NAHRUNGSERGAENZUN = FOOD SUPPLEMENT). As data from the same source tend to be encoded in a similar manner, i decided to split the attribute names and use the firt part (prefix) to investigate feature encoding in groups.   split attribute names<jupyter_code>df_features['attribute_prefix'] = df_features.Attribute.apply(attribute_group)
# Count the sources of features, or the types of features.
top_count_features = df_features.Attribute.drop_duplicates().apply(attribute_group).value_counts().head(10)
top_count_features
feat_counts = df_features.Attribute.drop_duplicates().count()
print('features count, total: {:}'.format(feat_counts))

feat_anlyzed = top_count_features.sum()
print('features count with a specified pre-fix: {:}'.format(feat_anlyzed))

print('features count without a specified pre-fix: {:}'.format(feat_counts - feat_anlyzed))<jupyter_output>features count, total: 314
features count with a specified pre-fix: 276
features count without a specified pre-fix: 38<jupyter_text>D19<jupyter_code># D19 provides informat about trasaction activities, cat data
# columns kept: D19_KONSUMTYP
d19_attribute_meanings = attribute_meanings(df_features,'D19')<jupyter_output>['no transactions known' 'very low activity' 'low activity'
 'slightly increased activity' 'increased activity' 'high activity'
 'very high activity' 'highest activity within the last 12 months'
 'very high activity within the last 12 months'
 'high activity within the last 12 months'
 'increased activity within the last 12 months'
 'slightly increased activity within the last 12 months'
 'activity elder than 1 year' 'activity elder than 1,5 years'
 'activity elder than 2 years' 'activity elder than 3 years'
 'no transaction known' 'Multibuyer 0-12 months' 'Doublebuyer 0-12 months'
 'Singlebuyer 0-12 months' 'Multi-/Doublebuyer 13-24 months'
 'Singlebuyer 13-24 months' 'Buyer > 24 months' 'Prospects > 24 months'
 'activity elder than 12 months' 'activity elder than 18 months'
 'activity elder than 24 months' 'activity elder than 36 months'
 'no Online-transactions within the last 12 months'
 '10% Online-transactions within the last 12 months'
 '20% Online-transactions within the last 1[...]<jupyter_text>D19 data describes the trascation activity. All features, except for "D19_KONSUMTYP" are ornidal data. "D19_KONSUMTYP" is catgorical, therefore it will be appended to the cat feature list. Missing vlaues or unknows are descrbied either as "no transactions known", or "no transaction known", or 'unknown', or unspecified in the information file.<jupyter_code>cat_features.extend(['D19_KONSUMTYP'])<jupyter_output><empty_output><jupyter_text>KBA05<jupyter_code># Family and cars, ordnial information, ordinal data
kba_05_meanings  = attribute_meanings(df_features, 'KBA05')<jupyter_output>['unknown' 'none' 'low' 'average' 'high' 'very high' 'very low' 'some'
 'some more' 'very many' 'no 1-2 family homes'
 'lower share of 1-2 family homes' 'average share of 1-2 family homes'
 'high share of 1-2 family homes' 'very high share of 1-2 family homes'
 'no 3-5 family homes' 'lower share of 3-5 family homes'
 'average share of 3-5 family homes' 'high share of 3-5 family homes'
 'very high share of 3-5 family homes' 'no 6-10 family homes'
 'lower share of 6-10 family homes' 'average share of 6-10 family homes'
 'high share of 6-10 family homes' 'no >10 family homes'
 'lower share of >10 family homes' 'high share of >10 family homes'
 'very low car quote' 'low car quote' 'average car quote' 'high car quote'
 'very high car quote' 'mainly 1-2 family homes in the microcell'
 'mainly 3-5 family homes in the microcell'
 'mainly 6-10 family homes in the microcell'
 'mainly>10 family homes in the microcell'
 'mainly business buildings in the microcell' '1-2 buildings'
 '3-4 buildings' [...]<jupyter_text>KBA05 data describes house and car the person owns.<jupyter_code>for att in ["KBA05_BAUMAX", "KBA05_HERSTTEMP", \
            "KBA05_MAXHERST","KBA05_MAXSEG", "KBA05_MAXVORB", "KBA05_MOTOR"]:
    display(df_features[df_features.Attribute == att])
cat_features.extend(["KBA05_BAUMAX", "KBA05_HERSTTEMP", \
            "KBA05_MAXHERST","KBA05_MAXSEG", "KBA05_MAXVORB", "KBA05_MOTOR"])
print(cat_features)<jupyter_output>['D19_KONSUMTYP', 'KBA05_BAUMAX', 'KBA05_HERSTTEMP', 'KBA05_MAXHERST', 'KBA05_MAXSEG', 'KBA05_MAXVORB', 'KBA05_MOTOR']<jupyter_text>KBA13<jupyter_code># get_features(df_features, 'KBA13')
# Familiy and cars of the neighbourhood, Ordinal information
kba_05_meanings  = attribute_meanings(df_features, 'KBA13')
display(df_features[df_features.Attribute == "KBA13_MOTOR"])
cat_features.extend(["KBA13_MOTOR"])<jupyter_output><empty_output><jupyter_text>LP<jupyter_code># Describes the family status of each person, categorical, to reduce the dimensions: keep only '_GROB'
lp_meanings  = attribute_meanings(df_features, 'LP')<jupyter_output>['single ' 'couple' 'young single parent ' 'single parent with teenager'
 'single parent with child of full age' 'young family '
 'family with teenager ' 'family with child of full age' 'shared flat'
 'two-generational household' 'multi-generational household'
 'single parent' nan 'family' 'multiperson household'
 'single low-income earners of younger age'
 'single low-income earners of middle age'
 'single average earners of younger age '
 'single average earners of middle age'
 'single low-income earners of advanced age'
 'single low-income earners at retirement age '
 'single average earners of advanced age'
 'single average earners at retirement age' 'single independant persons'
 'wealthy single homeowners' 'single homeowners of advanced age'
 'single homeowners at retirement age' 'single top earners of higher age '
 'low-income- and average earner-couples of younger age'
 'low-income earner-couples of higher age'
 'average earner-couples of higher age ' 'independant couples'
 'wea[...]<jupyter_text>all the data is nominal, and provided in both 'fein' (fine) and 'groub' (rough).<jupyter_code>def get_features(df_features, prefix):
    """Get all attributes with the same prefix
    """
    feats = df_features.loc[df_features['attribute_prefix'] == prefix, 'Attribute'].unique()
    return feats
get_features(df_features, 'LP')
cat_features.extend( get_features(df_features, 'LP'))
# print(cat_features)<jupyter_output><empty_output><jupyter_text>FINANZ Features that contains 'FINANZ' in the names, describes the financial typology a savor or an investor). All the features, except for "FINANZTYP" (TYP- type) will be treated as ordinal data with 'unknown' encoded as [-1]. FINANZTYP best describes the financial type for the person. it will be treated as catogrical, with unkonwn treated as [-1].<jupyter_code># Describes the financial typology 
# mostly ordinal, except: FINANZTYP
# Unknown is encoded as 'unknown'
df_features.loc[df_features.Attribute.str.contains('FINANZ'), "Meaning"].unique()
cat_features.extend(['FINANZTYP'])<jupyter_output><empty_output><jupyter_text>ANZ<jupyter_code>## ANZ: all are numerical vlaues 
df_features[df_features['attribute_prefix'] == 'ANZ']
num_features.extend(get_features(df_features, 'ANZ'))

# print(list(set(num_features)))<jupyter_output><empty_output><jupyter_text>CAMEO Features<jupyter_code>## CAMEO: all are numerical vlaues 
df_features.loc[df_features['attribute_prefix'] == 'CAMEO', 'Meaning'].unique()
cat_mixed = ['CAMEO_DEUG_2015', 'CAMEO_DEUINTL_2015'] 
get_features(df_features, prefix='CAMEO')
cat_features.extend(['CAMEO_DEUG_2015', 'CAMEO_DEU_2015', 'CAMEO_DEUINTL_2015', "CAMEO_INTL_2015"])<jupyter_output><empty_output><jupyter_text>- 'CAMEO_DEU_2015' includes 44 categories in total, requires further analyssi to decide how to treat it- 'CAMEO_DEUINTL_2015' provides information about the housing and family status. therefore it is decided to split the data into two, one for housing and the other for family statu  GEBAEUDETYP<jupyter_code>GEBAEUDETYP_feats = get_features(df_features, prefix='GEBAEUDETYP')
# GEBAEUDETYP_feats
# cat gorical data, keep both
df_features.loc[df_features['attribute_prefix'] == 'GEBAEUDETYP', 'Meaning'].unique()
cat_features.extend(['GEBAEUDETYP', 'GEBAEUDETYP_RASTER'])<jupyter_output><empty_output><jupyter_text>Other features<jupyter_code>search_for = '|'.join(top_count_features.index.values)
feat_to_analyze = df_feat_summary.loc[~df_feat_summary['Attribute'].str.contains(search_for, na=False), 'Attribute'].unique()
# Include features not explained in the file
feat_to_analyze = np.concatenate([feat_to_analyze, fea_])
print(feat_to_analyze)<jupyter_output>['CJT_GESAMTTYP' 'RELAT_AB' 'CJT_TYP_5' 'CJT_TYP_2' 'CJT_TYP_6' 'DSL_FLAG'
 'KOMBIALTER' 'GREEN_AVANTGARDE' 'CJT_TYP_1' 'UMFELD_ALT' 'VERS_TYP'
 'MOBI_REGIO' 'WOHNDAUER_2008' 'VHN' 'KK_KUNDENTYP' 'WOHNLAGE'
 'ALTERSKATEGORIE_GROB' 'KKK' 'FIRMENDICHTE' 'ALTER_KIND2' 'HEALTH_TYP'
 'GEMEINDETYP' 'EXTSEL992' 'TITEL_KZ' 'EINGEZOGENAM_HH_JAHR'
 'CJT_KATALOGNUTZER' 'PRAEGENDE_JUGENDJAHRE' 'KONSUMZELLE'
 'RETOURTYP_BK_S' 'RT_UEBERGROESSE' 'AKT_DAT_KL' 'VK_ZG11' 'ANREDE_KZ'
 'ZABEOTYP' 'UNGLEICHENN_FLAG' 'ALTER_KIND1' 'W_KEIT_KIND_HH'
 'RT_KEIN_ANREIZ' 'MIN_GEBAEUDEJAHR' 'INNENSTADT' 'ALTERSKATEGORIE_FEIN'
 'SOHO_KZ' 'ORTSGR_KLS9' 'EINGEFUEGT_AM' 'KONSUMNAEHE' 'ARBEIT' 'VK_DHT4A'
 'VHA' 'HH_DELTA_FLAG' 'STRUKTURTYP' 'EWDICHTE' 'NATIONALITAET_KZ'
 'ALTER_KIND4' 'REGIOTYP' 'ALTER_KIND3' 'RT_SCHNAEPPCHEN' 'AGER_TYP'
 'CJT_TYP_4' 'MOBI_RASTER' 'UMFELD_JUNG' 'BALLRAUM' 'SHOPPER_TYP'
 'OST_WEST_KZ' 'GEBURTSJAHR' 'HH_EINKOMMEN_SCORE' 'VERDICHTUNGSRAUM'
 'ALTER_HH' 'ONLINE_AFFINITAET' 'GFK_URLAUBERTYP'[...]<jupyter_text>categorical data contains 'TYP' in the names<jupyter_code># I assume all the attributes with TYP in the name can be treated as cat data
typ_ = [feat for feat in feat_to_analyze if 'TYP' in feat]
cat_features.extend(typ_)<jupyter_output><empty_output><jupyter_text>D19<jupyter_code>feat_ = []

d19_ = [feat for feat in feat_to_analyze if 'D19' in feat and 'TYP' not in feat]
print(d19_)

for feat in d19_:
    if feat not in df_features['Attribute'].values:
#         print("{:} contains {:.2f}% na's".format(feat, 100*azdias[feat].isna().mean()))
        fig, ax = plt.subplots()
        sns.countplot(azdias[feat])
        feat_.append(feat)<jupyter_output>['D19_BANKEN_DIREKT', 'D19_BANKEN_GROSS', 'D19_BANKEN_LOKAL', 'D19_BANKEN_REST', 'D19_BEKLEIDUNG_GEH', 'D19_BEKLEIDUNG_REST', 'D19_BILDUNG', 'D19_BIO_OEKO', 'D19_BUCH_CD', 'D19_DIGIT_SERV', 'D19_DROGERIEARTIKEL', 'D19_ENERGIE', 'D19_FREIZEIT', 'D19_GARTEN', 'D19_HANDWERK', 'D19_HAUS_DEKO', 'D19_KINDERARTIKEL', 'D19_KOSMETIK', 'D19_LEBENSMITTEL', 'D19_LETZTER_KAUF_BRANCHE', 'D19_LOTTO', 'D19_NAHRUNGSERGAENZUNG', 'D19_RATGEBER', 'D19_REISEN', 'D19_SAMMELARTIKEL', 'D19_SCHUHE', 'D19_SONSTIGE', 'D19_SOZIALES', 'D19_TECHNIK', 'D19_TELKO_MOBILE', 'D19_TELKO_ONLINE_QUOTE_12', 'D19_TELKO_REST', 'D19_TIERARTIKEL', 'D19_VERSAND_REST', 'D19_VERSICHERUNGEN', 'D19_VERSI_DATUM', 'D19_VERSI_OFFLINE_DATUM', 'D19_VERSI_ONLINE_DATUM', 'D19_VERSI_ONLINE_QUOTE_12', 'D19_VOLLSORTIMENT', 'D19_WEIN_FEINKOST']<jupyter_text>without much information available, i will assume all the above listed features are ordinal data<jupyter_code>cat_features.extend(['D19_LETZTER_KAUF_BRANCHE'])  # last purchase sector? the informatoin doesn't seem to be so relavent
# azdias['D19_LETZTER_KAUF_BRANCHE'].unique()<jupyter_output><empty_output><jupyter_text>KBA05<jupyter_code>kba05_ = [feat for feat in feat_to_analyze if 'KBA05' in feat and 'TYP' not in feat]
print(kba05_)<jupyter_output>[]<jupyter_text>KBA13<jupyter_code>kba13_ = [feat for feat in feat_to_analyze if 'KBA13' in feat and 'TYP' not in feat]
print(kba13_)
for feat in kba13_:
    if feat not in df_features['Attribute'].values:
#         print("{:} contains {:.2f}% na's".format(feat, 100*azdias[feat].isna().mean()))
        fig, ax = plt.subplots()
        sns.countplot(azdias[feat])
        feat_.append(feat)<jupyter_output><empty_output><jupyter_text>KBA13 data is mostly ordinal, i will assume it to be ordinal  LP<jupyter_code>lp_ = [feat for feat in feat_to_analyze if 'LP' in feat and 'TYP' not in feat]
print(lp_)<jupyter_output>[]<jupyter_text>FINANZ<jupyter_code>finanz_ = [feat for feat in feat_to_analyze if 'FINANZ' in feat and 'TYP' not in feat]
print(finanz_)<jupyter_output>[]<jupyter_text>ANZ<jupyter_code>anz_ = [feat for feat in feat_to_analyze if 'ANZ' in feat and 'TYP' not in feat]
print(anz_)
for feat in anz_:
    if feat not in df_features['Attribute'].values:
#         print("{:} contains {:.2f}% na's".format(feat, 100*azdias[feat].isna().mean()))
        fig, ax = plt.subplots()
        sns.countplot(azdias[feat])
        feat_.append(feat)
num_features.extend(anz_)<jupyter_output><empty_output><jupyter_text>CAMEO<jupyter_code>cameo_ = [feat for feat in feat_to_analyze if 'CAMEO' in feat and 'TYP' not in feat]
print(cameo_)
cat_features.extend(cameo_)<jupyter_output><empty_output><jupyter_text>GEBAEUDETYP<jupyter_code>GEBAEUDETYP_ = [feat for feat in feat_to_analyze if 'GEBAEUDETYP' in feat and 'TYP' not in feat]
print(GEBAEUDETYP_)<jupyter_output>[]<jupyter_text>analyze unexplored features<jupyter_code>remaining_feats = feat_to_analyze
for f in [d19_, kba05_, kba13_, lp_, finanz_, anz_, cameo_, GEBAEUDETYP_, typ_]:
#     print(f)
    remaining_feats = list (set(remaining_feats) - set(f))
    
print( 'there are in total {:} features remain to be examined manually/individually'.format(len(remaining_feats)))
print(remaining_feats)
for feat in remaining_feats:
    if feat in df_features['Attribute'].values:
        print(feat)
        display(df_features.loc[df_features.Attribute == feat])
num_features.extend(['MIN_GEBAEUDEJAHR', 'GEBURTSJAHR', ])
cat_features.extend(['GREEN_AVANTGARDE', 'OST_WEST_KZ',
                     'TITEL_KZ', 'ALTERSKATEGORIE_GROB', 'NATIONALITAET_KZ', 
                     'ANREDE_KZ',  'PRAEGENDE_JUGENDJAHRE', "WOHNLAGE", "ALTER", "ALTER_HH", 
                     'KONSUMNAEHE', 'ORTSGR_KLS9', 'PRAEGENDE_JUGENDJAHRE', 
                    ])
# remaining_feats
feat_ = []

for feat in remaining_feats:
    if feat not in df_features['Attribute'].values:
        print("{:} contains {:.2f}% na's".format(feat, 100*azdias[feat].isna().mean()))
        
        fig, ax = plt.subplots()
        sns.countplot(azdias[feat])
        feat_.append(feat)
# I will drop theses columns either because they do not provides too much info (contains too many cats, or too many nan's)
# to_drop = ['ALTER_KIND3', 'AKT_DAT_KL', 'ALTER_KIND1', 'VHA', 
#            'ALTER_KIND4','VERDICHTUNGSRAUM', 'EXTSEL992','ALTER_KIND2'] 

num_features.extend(['EINGEFUEGT_AM', "ALTER_KIND3 ",
                     'EINGEZOGENAM_HH_JAHR',
                    'EXTSEL992',
                    'VERDICHTUNGSRAUM',
                    'ALTER_KIND2',
                    'ALTER_KIND1',
                    'ALTER_KIND4'])
cat_features.extend(list(set(feat_) - set(['EINGEFUEGT_AM', "ALTER_KIND3 ",
                     'EINGEZOGENAM_HH_JAHR',
                    'EXTSEL992',
                    'VERDICHTUNGSRAUM',
                    'ALTER_KIND2',
                    'ALTER_KIND1',
                    'ALTER_KIND4'])))<jupyter_output><empty_output><jupyter_text>update feature summary table update using data type<jupyter_code>for feat in cat_features:
    df_feat_summary.loc[df_feat_summary['Attribute'] == feat, 'type'] = 'categorical'
for feat in num_features:
    df_feat_summary.loc[df_feat_summary['Attribute'] == feat, 'type'] = 'numerical'
df_feat_summary['type'].value_counts()<jupyter_output><empty_output><jupyter_text>update feature summary table include missing values encoding It is noticed from the analysis the some of values presents unknowns or missing data, Those valuse are normally described as 'no transactions', 'unknown', 'no transaction' in the informatoin file ('DIAS Attributes - Values 2017.xlsx'). I will serach for these key words in the df_features to get to know the values assigned to unknowns or missing values.<jupyter_code>df_features_nan = unknown_encoding(
    df_features, search_for='|'.join(['no transactions', 'unknown', 'no transaction']))
df_features_nan['Meaning'].unique()
df_feat_summary = df_feat_summary.merge( df_features_nan,left_on='Attribute', right_on='Attribute', how = 'left')
df_feat_summary.loc[df_feat_summary.type == 'numerical', "Value"] = [0]  # Numerical vlaues normally start from 1, if there is a zero, it means hte 
df_feat_summary
df_feat_summary.drop(['Meaning', 'attribute_prefix'], axis=1, inplace=True)
#  ['ANREDE_KZ','ALTERSKATEGORIE_GROB']
df_feat_summary.to_csv('feat_summary.csv')
df_feat_summary.to_pickle('feat_summary.pkl')
# cat_features<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10823468193053207, 'fraction_numerical': 0.025410866831447033, 'mean_word_length': 5.851455733808675, 'pattern_counts': {'":': 0, '<': 104, '<?xml version=': 0, '>': 110, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '29', 'dir': 'jupyter-structured-clean-dedup', 'id': '112218', 'license': 'MIT', 'n_tokens_mistral': 8398, 'n_tokens_neox': 7835, 'n_words': 2213, 'path': 'Part_I_Feature_Summary.ipynb', 'repo_name': 'jinchao-chen/capstone_arvato', 'size': '23061'}