<jupyter_start><jupyter_text>Census Folium Tutorial <NAME>Released under the MIT license This tutorial demonstrates how to use the functions in census_folium_viewer.py to generate interactive zip-, county-, and state-level choropleth maps based on census data and shapefiles from the US Census Bureau. Within the tutorial, I will generate maps of two data types: (1) median household income and (2) the percentage of households that consist of a married couple with at least one child below the age of 18.The tutorial also demonstrates how to incorporate a custom vertical legend into Folium maps. These have advantages over the default horizontal legend in certain circumstances.**Note**: Some files in this project (such as the zip-code-level HTML maps) were too large to upload to GitHub. You can instead access those files via the following Google Drive folder: https://drive.google.com/drive/folders/11h1jnaVOA5A6ubbOJnC-kPEvdnJU00yv?usp=sharingCitation info for color_schemes_from_branca.json:Source: https://github.com/python-visualization/branca/blob/master/branca/_schemes.jsonI believe these schemes were originally created by <NAME>, and are licensed under the Apache License, Version 2.0. See http://www.personal.psu.edu/cab38/ColorBrewer/ColorBrewer_updates.html  Preliminary stepsFirst, you'll need to download zip code, county, and state shapefiles from the US Census bureau: https://www.census.gov/geographies/mapping-files/time-series/geo/tiger-line-file.htmlI used 2020 zip code shapefiles for this project, which are available here: https://www.census.gov/cgi-bin/geo/shapefiles/index.php?year=2020&layergroup=ZIP+Code+Tabulation+AreasSee this note regarding use of the shapefiles: https://www2.census.gov/geo/pdfs/maps-data/data/tiger/tgrshp2021/TGRSHP2021_TechDoc_Ch1.pdfOnce you download the shapefiles to your computer, extract them using an unzipping utility. The shapefile document within this unzipped folder ends in .shp; for the 2020 zip code data, the file name is tl_2020_us_zcta520.shp. It appears that the other files within this folder may also be used by Geopandas in the creation of GeoDataFrames, so I recommend accessing the file within the unzipped folder rather than copying it into your project folder.This project already contains zip-, county-, and state-level US Census data for various demographic variables. I created these data files via my Census query tutorial available at https://github.com/kburchfiel/census_query_tutorial . The zip- and county-level data comes from the 2019 American Community Survey (5-year estimates), and the state-level data comes from the 2019 American Community Survey (1-year estimates).<jupyter_code>import time
start_time = time.time()
import census_folium_viewer
import geopandas
import numpy as np
import pandas as pd
import folium<jupyter_output><empty_output><jupyter_text>Part 1: Zip Code-Level Maps The following code block uses census_folium_viewer.py's prepare_zip_table() function to create a GeoDataFrame storing both zip code and census data. It took 53.5 seconds for the function to create a table, whereas importing a saved version of this file took only 7.6 seconds. Therefore, to save time, I edited the code block so that it would only regenerate the GeoDataFrame if instructed to do so.I recommend reading the documentation in census_folium_viewer.py for prepare_zip_table so that you'll better understand the inputs used by this function.The zip code census data file only includes zip codes with at least 1,000 households, preventing outliers in the data related to low sample sizes.Note: on one day when using prepare_zip_table, I received the following error message:"ImportError: the 'read_file' function requires the 'fiona' package, but it is not installed or does not import correctly.Importing fiona resulted in: DLL load failed while importing ogrext: The specified module could not be found."I received this error while using version 3.4.0 of GDAL and version 1.8.20 of fiona.After trying a couple different uninstall/reinstall operations, I was able to resolve the error by force removing both GDAL and Fiona, then installing fiona 1.8.19 (as suggested by Abhiram at https://stackoverflow.com/a/69534619/13097194). Conda Forge installed GDAL 3.2.2 as part of this operation, and these two seem to work together well. (If you still run into issues with these package versions, try restarting your kernel and/or restarting your code editor.)<jupyter_code>create_new_zip_census_table = True

if create_new_zip_census_table == True:

    zip_and_census_table = census_folium_viewer.prepare_zip_table(
        shapefile_path = r'C:\Users\kburc\Downloads\tl_2020_us_zcta520\tl_2020_us_zcta520.shp',
        shape_feature_name = 'ZCTA5CE20', tolerance = 0.005, data_path =
        'census_query_results_1k_plus_households.csv',
        data_feature_name = 'Zip')
    print("Exporting data:")
    zip_and_census_table.to_file('zip_and_census_table.geojson',
    driver = 'GeoJSON') 
    # The above line exports the GeoDataFrame created by prepare_zip_table 
    # so that it can be imported back into the program, which takes less time
    # than does recreating the GeoDataFrame.<jupyter_output>Reading shape data:
Simplifying shape data:
Reading census data:
Merging shape and data tables:
Exporting data:<jupyter_text>After being stored within the project folder, this table gets re-read into Python. That way, the program will still function even if the above cell did not run.<jupyter_code>zip_and_census_table = geopandas.read_file('zip_and_census_table.geojson')<jupyter_output><empty_output><jupyter_text>First, I will create a map displaying median household income by zip code. (Note: I will exclude Puerto Rico from these maps so as to focus on the 50 US states and DC. Puerto Rico has the code 72 in the 'State' column of the merged zip code data table, hence the use of zip_and_census_table.query("State != 72") as the argument for the merged_data_table parameter.)If you haven't done so already, I highly recommend reading the documentation for generate_map so that you'll know what inputs are necessary for the function to run correctly.<jupyter_code>zip_median_hh_income_map = census_folium_viewer.generate_map(
    merged_data_table = zip_and_census_table.query("State != 72"),
    shape_feature_name = 'ZCTA5CE20',
    data_variable = "Median_household_income", feature_text = 'Zip Code',
    data_variable_text = 'Median Household Income',
    popup_variable_text = 'Income', 
    variable_decimals = None,
    map_name = 'zip_median_hh_income',
    fill_color = 'RdYlGn',
    html_save_path = r'C:\Users\kburc\D1V1\Documents\!Dell64docs\Programming\py\kjb3_programs\census_folium_tutorial',
    screenshot_save_path = 'census_folium_map_screenshots',
    bin_type = 'percentiles', rows_to_map = 0, multiply_data_by = 1, 
    vertical_legend = True)

# zip_median_hh_income_map<jupyter_output><empty_output><jupyter_text>Next, I'll create a table showing the proportion of households in each zip code in the dataset that consist of a married couple with at least one child.<jupyter_code>zip_married_couples_with_children_map = census_folium_viewer.generate_map(
    merged_data_table = zip_and_census_table.query("State != 72"),
    shape_feature_name = 'ZCTA5CE20',
    data_variable = "Married_couple_households_with_one_or_more_children_as_proportion_of_all_households",
    feature_text = 'Zip Code',
    data_variable_text= '% of households that consist of a married couple \
with at least one child',
    popup_variable_text = 'Percentage', 
    map_name = 'zip_married_couples_with_kids', fill_color = 'RdYlGn', 
    html_save_path = r'C:\Users\kburc\D1V1\Documents\!Dell64docs\Programming\py\kjb3_programs\census_folium_tutorial',
    screenshot_save_path = 'census_folium_map_screenshots',
    bin_type = 'percentiles', rows_to_map = 0, multiply_data_by = 100,
    variable_decimals = 2, vertical_legend = True)
# zip_married_couples_with_children_map<jupyter_output><empty_output><jupyter_text>Part 2: County-Level MapsThe steps for generating county-level maps are similar. Note that the county census data file being imported only contains counties with at least 1,000 households.<jupyter_code>county_and_census_table = census_folium_viewer.prepare_county_table(
    shapefile_path = r'C:\Users\kburc\Downloads\tl_2021_us_county\tl_2021_us_county.shp',
    shape_state_code_column = 'STATEFP', shape_county_code_column = 'COUNTYFP',
    tolerance = 0.005,
    data_path = 'census_county_query_results_1k_plus_households.csv',
    data_state_code_column = 'State', data_county_code_column = 'County_Code')
print("Exporting data:")
county_and_census_table.to_file('county_and_census_table.geojson',
    driver = 'GeoJSON') 
county_and_census_table
county_hh_income_map = census_folium_viewer.generate_map(
    merged_data_table = county_and_census_table.query("State != 72"), 
    shape_feature_name = 'County',
    data_variable = 'Median_household_income', feature_text = 'County',
    data_variable_text = 'Median Household Income',
    map_name = 'county_median_hh_income', 
    variable_decimals = None,
    html_save_path = r'C:\Users\kburc\D1V1\Documents\!Dell64docs\Programming\py\kjb3_programs\census_folium_tutorial',
    screenshot_save_path = 'census_folium_map_screenshots',
    popup_variable_text = 'Income (in 1000s)', fill_color = 'RdYlGn', 
    rows_to_map = 0, bin_type = 'percentiles', multiply_data_by = 1, 
    vertical_legend = True)

# county_hh_income_map
county_married_couples_with_children_map = census_folium_viewer.generate_map(
    merged_data_table = county_and_census_table.query("State != 72"),
    shape_feature_name = 'County',
    data_variable = 'Married_couple_households_with_one_or_more_children_as_proportion_of_all_households',
    feature_text = 'County', data_variable_text = '% of households \
that consist of a married couple with at least one child', 
    map_name = 'county_married_couples_with_kids', 
    html_save_path = r'C:\Users\kburc\D1V1\Documents\!Dell64docs\Programming\py\kjb3_programs\census_folium_tutorial', 
    screenshot_save_path = 'census_folium_map_screenshots', 
    popup_variable_text = 'Percentage', fill_color = 'RdYlGn', 
    rows_to_map = 0, bin_type = 'percentiles', multiply_data_by = 100,
    variable_decimals = 2, vertical_legend = True)

# county_married_couples_with_children_map<jupyter_output><empty_output><jupyter_text>Part 3: State-Level MapsFinally, I'll create state-level maps of median household income and married-couple family prevalence.<jupyter_code>state_and_census_table = census_folium_viewer.prepare_state_table(
    shapefile_path = r'C:\Users\kburc\Downloads\tl_2020_us_state\tl_2020_us_state.shp', 
    shape_feature_name = 'NAME', tolerance = 0.005, data_path = 
    'census_state_query_results.csv', data_feature_name = 'State')
print("Exporting data:")
state_and_census_table.to_file('state_and_census_table.geojson',
driver = 'GeoJSON') 
state_and_census_table
state_married_couples_with_children_map = census_folium_viewer.generate_map(
    merged_data_table = state_and_census_table.query("NAME != '<NAME>'"),
    shape_feature_name = 'NAME',
    data_variable = 'Married_couple_households_with_one_or_more_children_as_proportion_of_all_households', 
    feature_text = 'State', 
    data_variable_text = '% of households that consist of a married \
couple with at least one child', 
    map_name = 'state_married_couples_with_kids', 
    html_save_path = r'C:\Users\kburc\D1V1\Documents\!Dell64docs\Programming\py\kjb3_programs\census_folium_tutorial',
    screenshot_save_path = 'census_folium_map_screenshots', 
    popup_variable_text = 'Percentage', fill_color = 'RdYlGn', 
    rows_to_map = 0, bin_type = 'percentiles', bin_count = 8, 
    multiply_data_by = 100,
    variable_decimals = 2, vertical_legend = True)
state_median_hh_income_map = census_folium_viewer.generate_map(
    merged_data_table = state_and_census_table.query("NAME != '<NAME>'"), shape_feature_name = 'NAME',
    data_variable = 'Median_household_income', feature_text = 'State', 
    data_variable_text = 'Median Household Income (in 1000s)', 
    map_name = 'state_median_hh_income', 
    variable_decimals = None,
    html_save_path = r'C:\Users\kburc\D1V1\Documents\!Dell64docs\Programming\py\kjb3_programs\census_folium_tutorial', 
    screenshot_save_path = 'census_folium_map_screenshots', 
    popup_variable_text = 'Income (in 1000s)', fill_color = 'RdYlGn', 
    rows_to_map = 0, bin_count = 8,
    bin_type = 'percentiles', multiply_data_by = 1, vertical_legend = True, 
    generate_image = True)
end_time = time.time()
run_time = end_time - start_time
run_minutes = run_time // 60
run_seconds = run_time % 60
print("Completed run at",time.ctime(end_time),"(local time)")
print("Total run time:",'{:.2f}'.format(run_time),
"second(s) ("+str(run_minutes),"minute(s) and",'{:.2f}'.format(run_seconds),
"second(s))") 
# Only meaningful when the program is run nonstop from start to finish<jupyter_output>Completed run at Sat Jan 15 18:03:29 2022 (local time)
Total run time: 266.12 second(s) (4.0 minute(s) and 26.12 second(s))
{'fraction_non_alphanumeric': 0.07823310271420969, 'fraction_numerical': 0.019843381738006538, 'mean_word_length': 5.59679037111334, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 31, 'https://': 7, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '356016', 'license': 'MIT', 'n_tokens_mistral': 4474, 'n_tokens_neox': 4112, 'n_words': 1169, 'path': 'script_backups/backup_census_folium_tutorial_v5.ipynb', 'repo_name': 'kburchfiel/census_folium_tutorial', 'size': '13182'}