<jupyter_start><jupyter_text>To get city geodata:```state = cityList.loc[PLACENAME].statecounty = cityList.loc[PLACENAME].countytracts_YEAR[tracts_Year.NHGISST == state][tracts_Year.NHGISCTY == county]```remember to remove final 0 if pulling from 2010 tracts!<jupyter_code>reindex_1940 = tracts_1940.rename(columns={"TractID_1940": 'TractID'})
reindex_1940.set_index("TractID", inplace=True)
reindex_2010 = tracts_2010.rename(columns={"TractID_2010": 'TractID'})
reindex_2010.set_index("TractID", inplace=True)

print("1940 tracts before join: {}".format(reindex_1940.shape[0]))
print("2010 tracts before join: {}".format(reindex_2010.shape[0]))
joined = reindex_1940.join(reindex_2010, lsuffix="_1940", rsuffix="_2010")
print("tracts after join: {} ({}%)".format(joined.shape[0], np.round(joined.shape[0]/reindex_1940.shape[0] * 100, 1)))<jupyter_output>1940 tracts before join: 7563
2010 tracts before join: 73669
tracts after join: 7563 (100.0%)<jupyter_text>Although the join worked with 100% accuracy, we cannot conclude that the 2010 version of a tract occupies the same area as the 1940 version of that tract (see below)<jupyter_code>state = cityList.loc['NYC_Brooklyn'].state
county = cityList.loc['NYC_Brooklyn'].county
plot_friendly_1940 = gpd.read_file("nhgis0001_shape/1940_shapefiles/US_tract_1940.shp")
plot_friendly_2010 = gpd.read_file("nhgis0001_shape/2010_shapefiles/US_tract_2010.shp")

state, county
plot_friendly_1940[plot_friendly_1940.NHGISST == state][plot_friendly_1940.NHGISCTY == county].plot()
plot_friendly_2010[plot_friendly_2010.STATEFP10 == state[:-1]][plot_friendly_2010.COUNTYFP10 == county[:-1]].plot()<jupyter_output>/Users/arunavgupta/anaconda3/envs/tensorflow/lib/python3.6/site-packages/geopandas/geodataframe.py:576: UserWarning: Boolean Series key will be reindexed to match DataFrame index.
  result = super(GeoDataFrame, self).__getitem__(key)
/Users/arunavgupta/anaconda3/envs/tensorflow/lib/python3.6/site-packages/geopandas/geodataframe.py:576: UserWarning: Boolean Series key will be reindexed to match DataFrame index.
  result = super(GeoDataFrame, self).__getitem__(key)<jupyter_text>Strategy for building crosswalk (although each 1940 tract ID has a 2010 analogue, examination of the above map shows us that there are actually multiple 2010 tracts within a 1940 tract area. So, we must average the statistics for the subset of 2010 tracts in a given 1940 tract. Doing so requires building a centroid --> point in polygon algorithm):1. filter to just tracts from a given county2. get centroids of 2010 tracts, add those in column to 2010 tracts df3. for each 1940 tract, search through each 2010 tract (double for each)4. if centroid is in polygon for the tract, add 1940 tract ID | 2010 tract ID to crosswalk dataframe<jupyter_code># helper function to get centroid of polygon
centroid = lambda x: x.centroid
def get_county_crosswalk(placeName):
    # create place_specific crosswalk dataframe
    temp_crosswalk = pd.DataFrame({'Region':[], 'Tract_1940':[], 'Tract_2010':[]})
    
    # filter to just tracts from a given county
    state = cityList.loc[placeName].state
    county = cityList.loc[placeName].county
    filtered_1940 = tracts_1940[tracts_1940.State == state][tracts_1940.County == county]
    filtered_2010 = tracts_2010[tracts_2010.State == state[:-1]][tracts_2010.County == county[:-1]]
    # get centroids of 2010 tracts, add to df
    filtered_2010['Centroid'] = filtered_2010.Geometry_2010.apply(centroid)
    
    # for each 1940 tract
    print("Current region: {}...".format(placeName))
    for index, tract in tqdm(filtered_1940.iterrows()):
        # search through each 2010 tract
        for index10, tract10 in filtered_2010.iterrows():
            # if centroid of 2010 tract is in 1940 tract geometry, add to crosswalk df
            if tract10.Centroid.within(tract.Geometry_1940):
                temp_crosswalk = temp_crosswalk.append({'Region': placeName, 
                                       'Tract_1940': tract.TractID_1940,
                                      'Tract_2010': tract10.TractID_2010}, ignore_index=True)
    
    print("Generation for {} completed!".format(placeName))
    return temp_crosswalk

# get_county_crosswalk("San Francisco")
cityList
tract_crosswalk = pd.DataFrame({'Region':[], 'Tract_1940':[], 'Tract_2010':[]})

for index, region in cityList.iterrows():
    tract_crosswalk = tract_crosswalk.append(get_county_crosswalk(index))

tract_crosswalk<jupyter_output>/Users/arunavgupta/anaconda3/envs/tensorflow/lib/python3.6/site-packages/ipykernel_launcher.py:10: UserWarning: Boolean Series key will be reindexed to match DataFrame index.
  # Remove the CWD from sys.path while we load stuff.
/Users/arunavgupta/anaconda3/envs/tensorflow/lib/python3.6/site-packages/ipykernel_launcher.py:11: UserWarning: Boolean Series key will be reindexed to match DataFrame index.
  # This is added back by InteractiveShellApp.init_path()
4it [00:00, 31.37it/s]<jupyter_text>Note that oftentimes the 2010 tract did not fall within its 1940 region!<jupyter_code>tract_crosswalk.to_csv("1940-to-2010_tract_crosswalk.csv")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10692800310498739, 'fraction_numerical': 0.07704249951484572, 'mean_word_length': 5.247272727272727, 'pattern_counts': {'":': 2, '<': 14, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '563492', 'license': 'MIT', 'n_tokens_mistral': 1984, 'n_tokens_neox': 1637, 'n_words': 414, 'path': 'deprecated/generate-tract-crosswalk.ipynb', 'repo_name': 'agupta01/segregation-measure', 'size': '5153'}