<jupyter_start><jupyter_text>Match Cells with their Sub-RegionNeed the following files:* list_25_75.csv - contains all the cells rounded to .25 and .75 accuracy* tasday_thresh.csv - contains the subset of the cells we are working with.METHOD:Use the cells from `tasday_thresh.csv` to subset `list_25_75.csv` so that we just have data from the cells we want.<jupyter_code># Read in regions
regions = pd.read_csv("list_25_75.csv", index_col=0)
regions = regions.set_index(["lon_25_75", "lat_25_75"])
regions
len(np.unique(regions.index))
# Create a list of cells we want to work with
df = pd.read_csv("tasday_thresh.csv")
df = df.set_index(["lon", "lat"])
cells = np.unique(df.index)
len(cells)
# Subset regions using just the cells we want
regions_matched = regions.loc[cells]
regions_matched
# Create list of lon/lat pairs so that we can create new column 'urban_center_count'
# It's in ascending order and cells listed multiple times means there are multiple centers in that cell
cells_non_unique = list(regions_matched.index)
cells_non_unique
# The total number of urban centers in a cell is equivalent to the lenth - len() - of the dataframe we get
# if we just look at data for that cell - regions_matched.loc[cell]
regions_matched["urban_center_count"] = pd.Series(
    data=[len(regions_matched.loc[cell]) for cell in cells_non_unique],
    # Pass an index so that it matches up with regions_matched
    index=cells_non_unique,
)
# Check last 50 entries to make sure it worked
regions_matched.urban_center_count.tail(50)
# Check the unique regions listed
np.unique(regions_matched.GRGN_L2)
# Total number of urban centers in each region
grouped_by_region = regions_matched.groupby("GRGN_L2")
grouped_by_region.urban_center_count.count()
# Average number of urban centers per cell in each region
grouped_by_region.urban_center_count.mean()
regions_matched.to_csv("regions_matched.csv")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0828125, 'fraction_numerical': 0.016145833333333335, 'mean_word_length': 5.693379790940766, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '187761', 'license': 'MIT', 'n_tokens_mistral': 634, 'n_tokens_neox': 591, 'n_words': 208, 'path': '2_create_regions_matched_dataset/match_cells_to_subregions.ipynb', 'repo_name': 'codeananda/geographic_data_analysis', 'size': '1920'}