<jupyter_start><jupyter_text>[Get the data](get-the-data)<jupyter_code>%load_ext nb_black
%load_ext autoreload
%autoreload 2
from pathlib import Path

import pandas as pd
%aimport src.data_helpers
from src.data_helpers import get_data, get_shapefiles, get_geojson_files<jupyter_output><empty_output><jupyter_text>[Table of Contents](table-of-contents)0. [About](about)1. [User Inputs](user-inputs)2. [Get crime listings data](get-crime-listings-data)3. [Get weather data](get-weather-data)4. [Get Chicago boundary shapefiles](get-chicago-boundary-shapefiles)  0. [About](about) In this notebook, we will retrieve the following data- crime listing data from the Chicago Open data portal- weather data for the city of Chicago from the NOAA database for the [O'Hare Airport station in the city](https://www.ncdc.noaa.gov/cdo-web/datasets/GHCND/stations/GHCND:USW00094846/detail)- [boundary files](https://en.wikipedia.org/wiki/GIS_file_formatsVector) for the city of Chicago, to allow for producing [choropleth maps](https://en.wikipedia.org/wiki/Choropleth_map)  1. [User Inputs](user-inputs) We'll define below the variables and helper functions that are to be used throughout the code.<jupyter_code>data_dir = str(Path().cwd() / "data" / "raw")
crime_data_urls = {"2018": "3i3m-jwuy", "2019": "w98m-zvie"}
crime_data_prefix = (
    "https://data.cityofchicago.org/api/views/{}/rows.csv?accessType=DOWNLOAD"
)
shapefiles = {
    "Boundaries - Police Beats (current).zip": "https://data.cityofchicago.org/api/geospatial/aerh-rz74?method=export&format=Shapefile",
    "Boundaries - Community Areas (current).zip": "https://data.cityofchicago.org/api/geospatial/cauq-8yn6?method=export&format=Shapefile",
    "Boundaries - Neighborhoods.zip": "https://data.cityofchicago.org/api/geospatial/bbvz-uum9?method=export&format=Shapefile",
}

geojsonfiles = {
    "Boundaries - Community Areas (current).geojson": "https://data.cityofchicago.org/api/geospatial/bbvz-uum9?method=export&format=GeoJSON",
    "Boundaries - Neighborhoods.geojson": "https://data.cityofchicago.org/api/geospatial/cauq-8yn6?method=export&format=GeoJSON",
    "Boundaries - CPD districts.geojson": "https://data.cityofchicago.org/api/geospatial/7hhi-ktqw?method=export&format=GeoJSON",
}
force_download_crime_data = True
force_download_shape_files = True
force_download_geojson_files = True
data_dir = Path(data_dir)<jupyter_output><empty_output><jupyter_text>2. [Get crime listings data](get-crime-listings-data) We will begin by retrieving crime listings data from the [Chicago open data portal](https://data.cityofchicago.org/browse?limitTo=datasets) for the years [2018](https://data.cityofchicago.org/Public-Safety/Crimes-2018/3i3m-jwuy) and [2019](https://data.cityofchicago.org/Public-Safety/Crimes-2019/w98m-zvie).<jupyter_code>for year, id in crime_data_urls.items():
    file_url = crime_data_prefix.format(year, id)
    get_data(
        file_path=data_dir / f"Crimes_-_{year}.csv",
        url=crime_data_prefix.format(id),
        msg=f"Downloading crime data for {year} from {file_url}...",
        force_download=force_download_crime_data,
    )<jupyter_output><empty_output><jupyter_text>3. [Get weather data](get-weather-data) Next, we'll retrieve the weather data from the NOAA website for the city of Chicago as recorded at the Chicago O'Hare weather station. 1. First, we'll load the NOAA page for the [Chicago O'Hare weather station](https://www.ncdc.noaa.gov/cdo-web/datasets/GHCND/stations/GHCND:USW00094846/detail)2. Next, we will retrieve GHCN weather data, starting with the page for the weather station at Chicago O'Hare International Airport.   - As we are making a (free) purchase from the NOAA website, we first add the station record to our Shopping Cart by clicking the **ADD TO CART** button. Next, we load our shopping cart and, from the drop-down menus and text input box, we will make the following specifications and then click **Continue**     - Output format       - select "Custom GHCN-Daily CSV"     - Date Range (click on the dropdown menu)       - we want all dates from January 1, 2018 to the curent date of October 12, 20193. Next, from the Custom Options available, we'll make the following selections from the listed checkboxes and then click **Continue**   - Precipitation   - Air Temperature   - Wind   - Weather Type4. Finally, from the Review Order page, we'll simply enter our email address into the two required fields and click **Submit Order**  4. [Get Chicago boundary shapefiles and `geojson` files](get-chicago-boundary-shapefiles-and-geojson-files) Finally, we'll download various boundary files for the city of Chicago from the city's open data portal. To do this, we'll loop over all listed boundary files in the `shapefiles` and `geojsonfiles` dictionaries and1. download the zipped (for `shapefiles`) or `geojson` file2. (for `shapefiles`) create a dedicated folder for each type of boundary file3. (for `shapefiles`) [unzip with Python](https://stackoverflow.com/a/3451150/4057186) into a dedicated folder for each type of boundary file We'll start by retrieving the `shapefiles`<jupyter_code>get_shapefiles(
    data_dir=data_dir, shapefiles=shapefiles, force_download=force_download_shape_files
)<jupyter_output><empty_output><jupyter_text>Finally, we'll retrieve `geojson` files<jupyter_code>get_geojson_files(
    data_dir=data_dir,
    geojsonfiles=geojsonfiles,
    force_download=force_download_geojson_files,
)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10900127807193719, 'fraction_numerical': 0.018258170531312762, 'mean_word_length': 5.422039859320047, 'pattern_counts': {'":': 8, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 15, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '572701', 'license': 'MIT', 'n_tokens_mistral': 1871, 'n_tokens_neox': 1720, 'n_words': 482, 'path': '1_get_data.ipynb', 'repo_name': 'edesz/chicago-crime-predictor', 'size': '5477'}