<jupyter_start><jupyter_text>Median Wagesresults and checks in notebook for internal review<jupyter_code>%load_ext autoreload
%autoreload 2

import sys
sys.path.append('../utils')
import wd_management
wd_management.set_wd_root()
from aggregate.PUMS.count_PUMS_economics import PUMSCountEconomics
from aggregate.PUMS.median_PUMS_economics import PUMSMedianEconomics
from aggregate.PUMS.economic_indicators import industry_mapper, occupation_mapper
import warnings
warnings.filterwarnings("ignore")
count_aggregator = PUMSCountEconomics(limited_PUMA=True)
counts = count_aggregator.aggregated
median_aggregator = PUMSMedianEconomics(limited_PUMA=True)
medians = median_aggregator.aggregated
medians<jupyter_output><empty_output><jupyter_text>Check occupationThis is first because fewer categories and thus easier Categories for reference:```pythonoccupation_mapper = {    "Management, Business, Science, and Arts Occupations": "mbsa",    "Service Occupations": "srvc",    "Sales and Office Occupations": "slsoff",    "Natural Resources, Construction, and Maintenance Occupations": "cstmnt",    "Production, Transportation, and Material Moving Occupations": "prdtrn",}``` We have 5 occupations, 5 races, and 3 measures per data point (median, CV, MOE).3\*5 non-crosstabbed occupatons + 5\*5\*3 crosstabbed occupations should come out to 90 occupation columns<jupyter_code>len([c for c in medians.columns if 'occupation' in c]) == 5*3 +  5*5*3<jupyter_output><empty_output><jupyter_text>First check that industries typically associated with higher wages do indeed have higher wages<jupyter_code>(medians['occupation-mbsa-wage-median'] - medians['occupation-srvc-wage-median'])<jupyter_output><empty_output><jupyter_text>Next look at an occupation-race crosstab with no workers. Does it have NaN median?<jupyter_code>display(counts['occupation-cstmnt-bnh-count'])
display(medians['occupation-cstmnt-wage-bnh-median'])<jupyter_output><empty_output><jupyter_text>Yes! Do PUMA-occupation-races with fewer people generally have higher variance on the median? First look at category with lots of people<jupyter_code>display(counts['occupation-slsoff-wnh-count'])
display(medians['occupation-slsoff-wage-wnh-median'])
display(medians['occupation-slsoff-wage-wnh-median-CV'])<jupyter_output><empty_output><jupyter_text>Then category with fewer<jupyter_code>display(counts['occupation-prdtrn-onh-count'])
display(medians['occupation-prdtrn-wage-onh-median'])
display(medians['occupation-prdtrn-wage-onh-median-CV'])<jupyter_output><empty_output><jupyter_text>Why do we get NaN for CV here?<jupyter_code>PUMS = median_aggregator.PUMS
PUMS[(PUMS['PUMA']=='4001') & (PUMS['race']=='onh')&(PUMS['occupation']=='occupation-prdtrn')& (PUMS['WAGP']>0)]<jupyter_output><empty_output><jupyter_text>I think our estimate of 13 workers comes from this one person with weight 13. I guess it can't get a measure of variance from one person. Is this worth asking population about?  Check industrySimilar checks<jupyter_code>industry_mapper = {
    "Agriculture, Forestry, Fishing and Hunting, and Mining": "AgFFHM",
    "Construction": "Cnstn",
    "Manufacturing": "MNfctr",
    "Wholesale Trade": "Whlsl",
    "Retail Trade": "Rtl",
    "Transportation and Warehousing, and Utilities": "TrWHUt",
    "Information": "Info",
    "Finance and Insurance,  and Real Estate and Rental and Leasing": "FIRE",
    "Professional, Scientific, and Management, and  Administrative and Waste Management Services": "PrfSMg",
    "Educational Services, and Health Care and Social Assistance": "EdHlth",
    "Arts, Entertainment, and Recreation, and  Accommodation and Food Services": "ArtEn",
    "Other Services (except Public Administration)": "Oth",
    "Public Administration": "PbAdm",
}
len(industry_mapper.keys())<jupyter_output><empty_output><jupyter_text>13 industries, same check for column count as above13\*3 columns for not crosstabbed by race13\*5\*3 for crosstabbed by race<jupyter_code>(13*3 + 13*5*3) - medians[[c for c in medians.columns if 'industry' in c]].shape[1]<jupyter_output><empty_output><jupyter_text>Off by eleven. Assume it's a race crosstab with zero people in it<jupyter_code>counts.loc[:,counts.sum() == 0].columns<jupyter_output><empty_output><jupyter_text>Check that these are indeed missing columns<jupyter_code>'industry-agffhm-wage-anh-median' in medians.columns
'industry-agffhm-wage-bnh-median' in medians.columns
'industry-agffhm-wage-onh-median' in medians.columns<jupyter_output><empty_output><jupyter_text>Just to be sure, hipanics and whites are present<jupyter_code>'industry-agffhm-wage-hsp-median' in medians.columns
'industry-agffhm-wage-wnh-median' in medians.columns<jupyter_output><empty_output><jupyter_text>Ok that accounts for nine of eleven missing columns. I assume we want these populated with NA?  Find missing industry columns the hard way Check for missing columns<jupyter_code>for indicator, mapper in [('occupation',occupation_mapper),('industry',industry_mapper)]:
    for k, i in mapper.items():
        for race in ['wnh','bnh','anh','onh','hsp']:
            for measure in ['median','median-CV','median-MOE']:
                col_name = f'{indicator}-{i.lower()}-wage-{race}-{measure}'
                if col_name not in medians.columns:
                    print(f'missing {col_name} ')<jupyter_output>missing industry-agffhm-wage-bnh-median 
missing industry-agffhm-wage-bnh-median-CV 
missing industry-agffhm-wage-bnh-median-MOE 
missing industry-agffhm-wage-anh-median 
missing industry-agffhm-wage-anh-median-CV 
missing industry-agffhm-wage-anh-median-MOE 
missing industry-agffhm-wage-onh-median 
missing industry-agffhm-wage-onh-median-CV 
missing industry-agffhm-wage-onh-median-MOE 
missing industry-agffhm-wage-hsp-median-CV 
missing industry-agffhm-wage-hsp-median-MOE<jupyter_text>Missing variance measure for hispanics?<jupyter_code>medians['industry-agffhm-wage-hsp-median']
counts['industry-agffhm-hsp-count']
PUMS[(PUMS['industry'] == 'industry-agffhm') & (PUMS['race']=='hsp')& (PUMS['WAGP']>0)]<jupyter_output><empty_output><jupyter_text>Ok that adds up. If it can't get variance measure it simply doesn't include them instead of adding columns of NaN like I think we want Check that if 0 workers in category then median wage is NA. We already saw this above but let's do fresh example to get away from that agriculture category that is giving us problems<jupyter_code>display(counts['industry-cnstn-anh-count'])
display(medians['industry-cnstn-wage-anh-median'])<jupyter_output><empty_output><jupyter_text>Ok but why is the 3701 median also NA?<jupyter_code>PUMS[(PUMS['industry']=='industry-cnstn')& (PUMS['race']=='anh')&(PUMS['PUMA']=='3701')]<jupyter_output><empty_output><jupyter_text>Ok great. Second person isn't in our universe at all for economics as they are not in labor force.First is in labor force and is in universe for counts but isn't in universe for wage medians as thier wage is 0. I read the "civilian employed population 16-64 **with earnings**; no percent calculated" in the denominator tab of the median wages in the data matrix as wages over 0. Maybe this was incorrect and this person should be counted and median wages for this data point will be zero. We should discuss This example is cleaner<jupyter_code>display(counts['industry-whlsl-bnh-count'])
display(medians['industry-whlsl-wage-bnh-median'])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.114397622906537, 'fraction_numerical': 0.007968665586169639, 'mean_word_length': 6.457200402819738, 'pattern_counts': {'":': 18, '<': 68, '<?xml version=': 0, '>': 70, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '418926', 'license': 'MIT', 'n_tokens_mistral': 2583, 'n_tokens_neox': 2432, 'n_words': 616, 'path': 'notebooks/Median_Wages.ipynb', 'repo_name': 'NYCPlanning/db-equitable-development-tool', 'size': '7404'}