<jupyter_start><jupyter_text>Libraries and setup variables<jupyter_code>from IPython.core.display import display, HTML
display(HTML("<style>.container { width:100% !important; }</style>"))

%matplotlib inline

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import missingno as msno
import seaborn as sns
from utils import summarise_column<jupyter_output><empty_output><jupyter_text>Loading the processed datasetHere we'll load the data into a dataframe, and run a series of initial exploratory analysis.<jupyter_code>df = pd.read_csv('../data/interim/census_income_learn.csv')<jupyter_output><empty_output><jupyter_text>By using a sample of 250 rows we can quicly inspect which rows might be missing some values.In the original dataset, the term 'Not in universe' seems to be used in place of a missing value.From this small sample, we can already see that certain variables are not as complete as others. For example 'class of worker' or 'major occupation code', among others appear incomplete<jupyter_code>df_na = df.replace(" Not in universe", np.nan)
msno.matrix(df_na.sample(250),figsize=(15,5))<jupyter_output><empty_output><jupyter_text>Columns exploration The columns in this dataset are of two types:- Continuous (7 columns)- Nominal (33 columns)For the continuous columns we can build histograms and understand the data in this way. In addition we can also consider filling the missing values if this is a possibility  1) AgeA summary for the age column shows that over 75% of the population is 50 years old or younger.<jupyter_code>summarise_column(df, 'age')<jupyter_output>Summary for column: age
                 age
count  199523.000000
mean       34.494199
std        22.310895
min         0.000000
25%        15.000000
50%        33.000000
75%        50.000000
99%        85.000000
max        90.000000<jupyter_text>2) Class of workerThe categorical columns are sometimes a bit harder to interpret, as the missing values or 'Not in universe' are not very descriptive.<jupyter_code>summarise_column(df, 'class of worker')<jupyter_output>Summary for column: class of worker
         class of worker
count             199523
unique                 9
top      Not in universe
freq              100245<jupyter_text>3) Detailed industry recodeIn this case as well, the industry codes are not provided, just some numeric codes without a context<jupyter_code>summarise_column(df, 'detailed industry recode')<jupyter_output>Summary for column: detailed industry recode
       detailed industry recode
count             199523.000000
mean                  15.352320
std                   18.067129
min                    0.000000
25%                    0.000000
50%                    0.000000
75%                   33.000000
99%                   49.000000
max                   51.000000<jupyter_text>4) EducationIn the case of education we can observe that most of people are either high-school graduates or children (not in education)<jupyter_code>summarise_column(df, 'education')<jupyter_output>Summary for column: education
                    education
count                  199523
unique                     17
top      High school graduate
freq                    48407<jupyter_text>5) Wage per hourThis should have a strong correlation with our target column (should be explored later). But it is also noticeable that since a large proportion of our population are people under 15 yo (25% of the total), they're not going to have a wage at all.<jupyter_code>summarise_column(df, 'wage per hour')<jupyter_output>Summary for column: wage per hour
       wage per hour
count  199523.000000
mean       55.426908
std       274.896454
min         0.000000
25%         0.000000
50%         0.000000
75%         0.000000
99%      1400.000000
max      9999.000000<jupyter_text>6) Enroll in edu inst last wkUnclear what this column might indicate. A lot fo the values are missing.<jupyter_code>summarise_column(df, 'enroll in edu inst last wk')<jupyter_output>Summary for column: enroll in edu inst last wk
       enroll in edu inst last wk
count                      199523
unique                          3
top               Not in universe
freq                       186943<jupyter_text>7) Marital statusHere as well, children will take a big chunk of the Never married category.<jupyter_code>summarise_column(df, 'marital stat')<jupyter_output>Summary for column: marital stat
          marital stat
count           199523
unique               7
top      Never married
freq             86485<jupyter_text>8) Major industry code<jupyter_code>summarise_column(df, 'major industry code')<jupyter_output>Summary for column: major industry code
                 major industry code
count                         199523
unique                            24
top      Not in universe or children
freq                          100684<jupyter_text>9) Major occupation code<jupyter_code>summarise_column(df, 'major occupation code')<jupyter_output>Summary for column: major occupation code
       major occupation code
count                 199523
unique                    15
top          Not in universe
freq                  100684<jupyter_text>10) RaceThis attribute might as well show a strong correlation to salary. Should be explored later. It is noticeable the overwhelming White population, since according to the official census statistics, While-alone is around 60%. But here the percentage is over 83%.Census official page - https://www.census.gov/quickfacts/fact/table/US/PST045217qf-headnote-a<jupyter_code>summarise_column(df, 'race')<jupyter_output>Summary for column: race
          race
count   199523
unique       5
top      White
freq    167365<jupyter_text>11) Sex<jupyter_code>summarise_column(df, 'sex')<jupyter_output>Summary for column: sex
            sex
count    199523
unique        2
top      Female
freq     103984<jupyter_text>12) Member of a labor union<jupyter_code>summarise_column(df, 'member of a labor union')<jupyter_output>Summary for column: member of a labor union
       member of a labor union
count                   199523
unique                       3
top            Not in universe
freq                    180459<jupyter_text>13) Reason for unemployment<jupyter_code>summarise_column(df, 'reason for unemployment')<jupyter_output>Summary for column: reason for unemployment
       reason for unemployment
count                   199523
unique                       6
top            Not in universe
freq                    193453<jupyter_text>14) Full or part time employment statThis column might need to be further split, since the Children element could be split. Armed forces get a salary, but children don't.<jupyter_code>summarise_column(df, 'full or part time employment stat')<jupyter_output>Summary for column: full or part time employment stat
       full or part time employment stat
count                             199523
unique                                 8
top             Children or Armed Forces
freq                              123769<jupyter_text>15) Capital gainsAs expected, most people don't invest in the stock market or have other investments generating gains or losses. Only a small proportion of the population does, so the histogram looks quite skewed.<jupyter_code>summarise_column(df, 'capital gains')<jupyter_output>Summary for column: capital gains
       capital gains
count   199523.00000
mean       434.71899
std       4697.53128
min          0.00000
25%          0.00000
50%          0.00000
75%          0.00000
99%       8614.00000
max      99999.00000<jupyter_text>16) Capital losses<jupyter_code>summarise_column(df, 'capital losses')<jupyter_output>Summary for column: capital losses
       capital losses
count   199523.000000
mean        37.313788
std        271.896428
min          0.000000
25%          0.000000
50%          0.000000
75%          0.000000
99%       1887.000000
max       4608.000000<jupyter_text>17) Dividends from stocksSame as with capital gains.<jupyter_code>summarise_column(df, 'dividends from stocks')<jupyter_output>Summary for column: dividends from stocks
       dividends from stocks
count          199523.000000
mean              197.529533
std              1984.163658
min                 0.000000
25%                 0.000000
50%                 0.000000
75%                 0.000000
99%              4550.000000
max             99999.000000<jupyter_text>18) Tax filer status<jupyter_code>summarise_column(df, 'tax filer stat')<jupyter_output>Summary for column: tax filer stat
       tax filer stat
count          199523
unique              6
top          Nonfiler
freq            75094<jupyter_text>19) region of previous residenceA lot of missing data here. Not sure if this indicates that people have not moved at all, or a mix between collection error and non-movers<jupyter_code>summarise_column(df, 'region of previous residence')<jupyter_output>Summary for column: region of previous residence
       region of previous residence
count                        199523
unique                            6
top                 Not in universe
freq                         183750<jupyter_text>20) state of previous residence<jupyter_code>summarise_column(df, 'state of previous residence')<jupyter_output>Summary for column: state of previous residence
       state of previous residence
count                       199523
unique                          51
top                Not in universe
freq                        183750<jupyter_text>21) detailed household and family stat<jupyter_code>summarise_column(df, 'detailed household and family stat')<jupyter_output>Summary for column: detailed household and family stat
       detailed household and family stat
count                              199523
unique                                 38
top                           Householder
freq                                53248<jupyter_text>22) detailed household summary in household<jupyter_code>summarise_column(df, 'detailed household summary in household')<jupyter_output>Summary for column: detailed household summary in household
       detailed household summary in household
count                                   199523
unique                                       8
top                                Householder
freq                                     75475<jupyter_text>23) migration code-change in msaSome missing values here as well, not sure MSA.<jupyter_code>summarise_column(df, 'migration code-change in msa')<jupyter_output>Summary for column: migration code-change in msa
       migration code-change in msa
count                        199523
unique                           10
top                               ?
freq                          99696<jupyter_text>24) migration code-change in reg<jupyter_code>summarise_column(df, 'migration code-change in reg')<jupyter_output>Summary for column: migration code-change in reg
       migration code-change in reg
count                        199523
unique                            9
top                               ?
freq                          99696<jupyter_text>25) migration code-move within reg<jupyter_code>summarise_column(df, 'migration code-move within reg')<jupyter_output>Summary for column: migration code-move within reg
       migration code-move within reg
count                          199523
unique                             10
top                                 ?
freq                            99696<jupyter_text>26) live in this house 1 year ago<jupyter_code>summarise_column(df, 'live in this house 1 year ago')<jupyter_output>Summary for column: live in this house 1 year ago
            live in this house 1 year ago
count                              199523
unique                                  3
top      Not in universe under 1 year old
freq                               101212<jupyter_text>27) migration prev res in sunbelt<jupyter_code>summarise_column(df, 'migration prev res in sunbelt')<jupyter_output>Summary for column: migration prev res in sunbelt
       migration prev res in sunbelt
count                         199523
unique                             4
top                                ?
freq                           99696<jupyter_text>28) num persons worked for employer<jupyter_code>summarise_column(df, 'num persons worked for employer')<jupyter_output>Summary for column: num persons worked for employer
       num persons worked for employer
count                    199523.000000
mean                          1.956180
std                           2.365126
min                           0.000000
25%                           0.000000
50%                           1.000000
75%                           4.000000
99%                           6.000000
max                           6.000000<jupyter_text>29) family members under 18<jupyter_code>summarise_column(df, 'family members under 18')<jupyter_output>Summary for column: family members under 18
       family members under 18
count                   199523
unique                       5
top            Not in universe
freq                    144232<jupyter_text>30) country of birth father<jupyter_code>summarise_column(df, 'country of birth father')<jupyter_output>Summary for column: country of birth father
       country of birth father
count                   199523
unique                      43
top              United-States
freq                    159163<jupyter_text>31) country of birth mother<jupyter_code>summarise_column(df, 'country of birth mother')<jupyter_output>Summary for column: country of birth mother
       country of birth mother
count                   199523
unique                      43
top              United-States
freq                    160479<jupyter_text>32) country of birth self<jupyter_code>summarise_column(df, 'country of birth self')<jupyter_output>Summary for column: country of birth self
       country of birth self
count                 199523
unique                    43
top            United-States
freq                  176989<jupyter_text>33) citizenship<jupyter_code>summarise_column(df, 'citizenship')<jupyter_output>Summary for column: citizenship
                               citizenship
count                               199523
unique                                   5
top      Native- Born in the United States
freq                                176992<jupyter_text>34) own business or self employed<jupyter_code>summarise_column(df, 'own business or self employed')<jupyter_output>Summary for column: own business or self employed
       own business or self employed
count                  199523.000000
mean                        0.175438
std                         0.553694
min                         0.000000
25%                         0.000000
50%                         0.000000
75%                         0.000000
99%                         2.000000
max                         2.000000<jupyter_text>35) fill inc questionnaire for veteran's admin<jupyter_code>summarise_column(df, "fill inc questionnaire for veteran's admin")<jupyter_output>Summary for column: fill inc questionnaire for veteran's admin
       fill inc questionnaire for veteran's admin
count                                      199523
unique                                          3
top                               Not in universe
freq                                       197539<jupyter_text>36) veterans benefits<jupyter_code>summarise_column(df, 'veterans benefits')<jupyter_output>Summary for column: veterans benefits
       veterans benefits
count      199523.000000
mean            1.514833
std             0.851473
min             0.000000
25%             2.000000
50%             2.000000
75%             2.000000
99%             2.000000
max             2.000000<jupyter_text>37) weeks worked in year<jupyter_code>summarise_column(df, 'weeks worked in year')<jupyter_output>Summary for column: weeks worked in year
       weeks worked in year
count         199523.000000
mean              23.174897
std               24.411488
min                0.000000
25%                0.000000
50%                8.000000
75%               52.000000
99%               52.000000
max               52.000000<jupyter_text>38) year<jupyter_code>summarise_column(df, 'year')<jupyter_output>Summary for column: year
                year
count  199523.000000
mean       94.499672
std         0.500001
min        94.000000
25%        94.000000
50%        94.000000
75%        95.000000
99%        95.000000
max        95.000000
{'fraction_non_alphanumeric': 0.04995766299745978, 'fraction_numerical': 0.0821337849280271, 'mean_word_length': 2.038963425840838, 'pattern_counts': {'":': 0, '<': 129, '<?xml version=': 0, '>': 129, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '41', 'dir': 'jupyter-structured-clean-dedup', 'id': '644304', 'license': 'MIT', 'n_tokens_mistral': 5246, 'n_tokens_neox': 4254, 'n_words': 1507, 'path': 'notebooks/1.0-lg-Initial exploration.ipynb', 'repo_name': 'lievcin/us_census', 'size': '16534'}