<jupyter_start><jupyter_text>look at what was done in the Hillsboro dataset:<jupyter_code>pwd
main_dir='../..'
data_dir =main_dir + '/data'
hl_orig = pd.read_csv(data_dir + "/processed/hillsborough_fl_processed_2017_to_2019_20210225.csv")
hl_orig.shape

hl_expand = pd.read_csv(data_dir +"/processed/hillsborough_fl_processed_expanded_ACS_2017_to_2019_20210225.csv")

                        
hl_expand.shape
hl_expand.columns
hl_acs = pd.read_csv(data_dir + "/acs/hillsborough_acs5-2018_census.csv")
hl_acs.shape

hl_acs.columns

# which ACS columns were not added to the expanded dataset
set(hl_acs.columns.to_list()) - set(hl_expand.columns.to_list())  #looks like all but the index and GEOID<jupyter_output><empty_output><jupyter_text>tasks1. read docs to make sure GEOID in _processed_ and _ACS_ datasets are the same2. use data dict to make human readable column-names for ACS dataset3. understand what the processing step did so I don't undo it when adding extra (uprocesssed columns)4. merge the processed and human-readable ACS variables<jupyter_code># are the GEOID columns the same?
pd.DataFrame.equals(hl_acs.GEOID, hl_expand.census_tract_GEOID)
hl_acs.GEOID.shape

hl_expand.census_tract_GEOID.shape
hl_expand.census_tract_GEOID.head()
hl_acs.GEOID.head()
#ok, so maybe they are in different orders... lets see if they are the same sets
len(set(hl_acs.GEOID)) == len(hl_expand.census_tract_GEOID) # no dups
set(hl_acs.GEOID) == set(hl_expand.census_tract_GEOID) # ok so they are the same sets, differently ordered.<jupyter_output><empty_output><jupyter_text>seems safe to just merge acs and processed data on GEOID<jupyter_code>nyc_orig = pd.read_csv(data_dir + "/processed/nyc_processed_2017_to_2019_20210225.csv")
nyc_orig.shape
nyc_acs = pd.read_csv(data_dir + "/acs/nyc_acs5-2018_census.csv")
nyc_acs.shape
nyc_orig.head()
nyc_acs.head()
nyc_expand = pd.merge(nyc_orig, nyc_acs, left_on='census_tract_GEOID', right_on='GEOID', how='inner')
nyc_expand.shape
nyc_expand.to_csv(data_dir + "/processed/nyc_processed_expanded_ACS_2017_to_2019_20210225.csv")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09718913768461172, 'fraction_numerical': 0.037160552644116246, 'mean_word_length': 6.26643598615917, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '628084', 'license': 'MIT', 'n_tokens_mistral': 856, 'n_tokens_neox': 765, 'n_words': 166, 'path': 'scripts/_ahopejasen/add_ACS_columns_to_NYC_processed_data.ipynb', 'repo_name': 'ahopejasen/datakind-test', 'size': '2099'}