<jupyter_start><jupyter_text>Load OSMNx graph for Kampala  Introduction  About this case study Caveats:1. The case study uses freely available data (e.g. Open Street Map building footprints). Open source data such as OSM is not necessarily high quality data (e.g., classification of building footprints in a parish may not be complete).2. The algorithms presented in this case study are not necessarily in their most efficient form.3. The case study will not provide answers to all users' questions, especially about OSMNx. Users must do a little bit more digging into the concepts presented to develop a better understanding of concepts being implemented and how these concepts are translated to Python code. This digging includes reading package manuals for GeoPandas, OSMNx, and Shapely and reading Stack Overflow and GIS Stack Exchange entries related to user questions (there's a chance someone asked these questions already).4. The case study provides insights into preparation of street network data and some basic descriptive analyses. More sophisticated analyses and visualizations could be done if data here were combined with other types.  About Uganda Uganda profile from the Uganda Bureau of Statistics (UBOS): https://www.ubos.org/uganda-profile/  Network graphs Review network graphs by going through 1854 Cholera Outbreak Advanced 1 notebooks: https://github.com/PHI-Case-Studies/1854-Cholera-Outbreak-London-Advanced-1The figure below is a rendering Kampala streets as a network graph. You can see the Python code to generate this graph in [Notebook Appendix 02](NBXX-Appendix02-Street-Network-Bearings.ipynb).  Resource stewardship  While at this workshop you will experience what it means to work with street networks. You will be downloading the street network graph for Kampala, which is about 4-6 GB in size, while working in a shared Jupyter environment with other users. Resources (CPU, RAM and disk space) that enable this Jupyter services are scarce and limited. We should strive to conserve as much of resources by doing the following:1. Keep one notebook open at a time. The notebooks are supposed to be run in sequence. Shut down earlier notebooks in the sequence once you are finished using them. You can close the tab then right-click / shutdown kernel on the filetree to your left. 2. This notebook - Notebook 1 - loads 2 graphs (projected and unprojected) but saves these immediately to disk upon opening. When you set `download_graph` to `True` it will use the Overpass API, a "free" service, to download the Kampala graphs to your notebook server. If you have the local copy of these graphs in your shared folder, use these copies - copy them to your data folder, and help keep the Overpass API open and readily available for new graph download requests.3. Watch out for memory consumption on the status bar below (where it says `Mem: xx GB`. If you are consuming > 5-6 GB shut down other notebooks so you only have 1 notebook running at a time (see 1).  Load OSMNx Graph  Check Python version<jupyter_code>!python --version<jupyter_output>Python 3.9.9<jupyter_text>Configure OSMNx To learn more about OSMNx settings see Python code here: https://github.com/gboeing/osmnx/blob/main/osmnx/settings.py<jupyter_code>import osmnx as ox

ox.settings.log_console=False 
ox.settings.use_cache=True
ox.settings.log_file=True
ox.settings.overpass_endpoint='https://overpass-api.de/api/interpreter'
ox.settings.overpass_rate_limit=True
ox.settings.timeout=240

ox.__version__<jupyter_output><empty_output><jupyter_text>Use a Bounding Box As a responsible netizen we will use only the resources we need. In this section, there are two things we will do to be a responsible netizen and:1. We will use a bounding box to limit the size of the graph, and hence the amount of data to request from the Overpass API.2. We will use the saved graph to avoid a trip to the Overpass API. If running the notebooks for the first time, set `download_graphml` to `True`. Loading the graph may take anywhere from 10-15 mins.**Optional:** If you want to track what how your notebook server is interacting with OSMNx and Overpass API, you can launch a terminal (from the JupyterLab Launcher), then do/type the following on the terminal command line:1. `cd notebooks/2021-HELINA-COVID-19-OSMNx-Workshop/logs`2. `ls -la`3. Look for the log file (`.log` extension) for today (log file will have the current date in the file name), copy the filename (highlight with your mouse and `Ctrl-C` to copy).4. Example: `tail -f osmnx_2021-10-17.log` (see sample log screenshot below)5. Note how log files are named by date, so be sure to `tail` the latest log file.<jupyter_code>%%time
import geopandas as gpd

# set download_graph to True if running notebook for the first time
download_graph = False

if download_graph:
    G = ox.graph_from_bbox(0.408513, 0.218915, 32.666921, 32.509538, network_type='all', simplify=False)
    G_proj = ox.project_graph(G)
    G_proj = ox.speed.add_edge_speeds(G_proj, precision=3)
    G_proj = ox.speed.add_edge_travel_times(G_proj, precision=3)
    G = ox.bearing.add_edge_bearings(G, precision=1)
    G = ox.distance.add_edge_lengths(G, precision=3)
    ox.save_graphml(G, 'data/g_unprojected.graphml', gephi=False)
    ox.save_graphml(G_proj, 'data/g_projected.graphml', gephi=False)
    # Create geodataframes for nodes and edges for projected and unprojected graph
    g_unprojected_nodes, g_unprojected_edges = ox.utils_graph.graph_to_gdfs(G, nodes=True, edges=True, node_geometry=True, fill_edge_geometry=True)
    g_projected_nodes, g_projected_edges = ox.utils_graph.graph_to_gdfs(G_proj, nodes=True, edges=True, node_geometry=True, fill_edge_geometry=True)
    # Save the nodes and edges to file using feather format
    g_unprojected_nodes.to_feather('data/g_unprojected_nodes.feather', index=True, compression='lz4')
    g_unprojected_edges.to_feather('data/g_unprojected_edges.feather', index=True, compression='lz4')
    g_projected_nodes.to_feather('data/g_projected_nodes.feather', index=True, compression='lz4')
    g_projected_edges.to_feather('data/g_projected_edges.feather', index=True, compression='lz4')
else:
    # Load unprojected graph, G, and undirected, unprojected graph, G_u
    g_unprojected_nodes = gpd.read_feather('data/g_unprojected_nodes.feather')
    g_unprojected_edges = gpd.read_feather('data/g_unprojected_edges.feather')
    G = ox.utils_graph.graph_from_gdfs(g_unprojected_nodes, g_unprojected_edges)
    G_u = ox.utils_graph.get_undirected(G)
    # Load projected graph, G_proj, and undirected, projected graph, G_proj_u
    g_projected_nodes = gpd.read_feather('data/g_projected_nodes.feather')
    g_projected_edges = gpd.read_feather('data/g_projected_edges.feather')
    G_proj = ox.utils_graph.graph_from_gdfs(g_projected_nodes, g_projected_edges)
    G_proj_u = ox.utils_graph.get_undirected(G_proj)<jupyter_output>CPU times: user 4min 48s, sys: 2.81 s, total: 4min 51s
Wall time: 4min 52s<jupyter_text>Upon running the notebook for the first time and with `download_graph` set to `True`, we did the following in the preceding code cell:1. We set the value of `download_graph` so we can either download a fresh graph from the Overpass API or reuse an existing saved graph.2. We downloaded the OSMNx graph for Kampala using a bounding box.3. We projected the graph so we can do distance measurements.4. We added some graph features so we can use them later for graph computatation steps (e.g., lengths, speeds, travel time).5. We saved the graph as GraphML so we can reuse the graph later.6. We timed the execution of code for the cell. OSMNx References:1. https://osmnx.readthedocs.io/en/stable/osmnx.htmlosmnx.graph.graph_from_bbox2. https://osmnx.readthedocs.io/en/stable/osmnx.htmlosmnx.projection.project_graph3. https://osmnx.readthedocs.io/en/stable/osmnx.htmlosmnx.distance.add_edge_lengths4. https://osmnx.readthedocs.io/en/stable/osmnx.htmlosmnx.speed.add_edge_speeds5. https://osmnx.readthedocs.io/en/stable/osmnx.htmlosmnx.speed.add_edge_travel_times6. https://osmnx.readthedocs.io/en/stable/osmnx.htmlosmnx.io.save_graphml7. https://osmnx.readthedocs.io/en/stable/osmnx.htmlosmnx.io.load_graphml  Graph Projections Why do we need to think about graph projections?<jupyter_code>g_crs = G.graph['crs']

g_crs
g_crs_prj = G_proj.graph['crs']

g_crs_prj<jupyter_output><empty_output><jupyter_text>In the code cells above we obtained the CRS values of the two graphs. What do you notice that's different about the two CRS values?  Obtain EPSG code for the projected graph What is EPSG?The projected graph and geodataframes we will create for residence centroids and testing facilities have to use the same planar projection. We obtain the projected graph's CRS below and obtain the corresponding EPSG authority code for use in Notebooks 2 and 3.<jupyter_code>from pyproj import CRS
import yaml, os

auth, crs = CRS.from_string(str(g_crs_prj)).to_authority()
crs_dict = {auth:crs}

print(crs_dict)

with open('proj_crs.yml', 'w') as outfile:
    yaml.dump(crs_dict, outfile, default_flow_style=False)
with open("proj_crs.yml", "r") as stream:
    try:
        epsg_dict = yaml.safe_load(stream)
    except yaml.YAMLError as exc:
        print(exc)
        
proj_epsg_str = str(epsg_dict).replace("{","").replace("}", "").replace("'","").replace(" ","")

proj_epsg_str<jupyter_output><empty_output><jupyter_text>Reference for 32636: https://epsg.io/32636  Graph Statistics<jupyter_code>%%time
G_stats_dict = ox.stats.basic_stats(G, area=None, clean_int_tol=None)

G_stats_dict
%%time
G_proj_stats_dict = ox.stats.basic_stats(G_proj, area=None, clean_int_tol=None)

G_proj_stats_dict<jupyter_output>CPU times: user 1min 54s, sys: 766 ms, total: 1min 54s
Wall time: 1min 55s<jupyter_text>Data Directory<jupyter_code>!ls -la data<jupyter_output>total 1044312
drwxr-xr-x  25 hermantolentino  staff        800 May 23 15:35 [34m.[m[m
drwxr-xr-x  28 hermantolentino  staff        896 Jun  3 20:09 [34m..[m[m
drwxr-xr-x   3 hermantolentino  staff         96 Jan 13 15:03 [34m.ipynb_checkpoints[m[m
-rw-r--r--   1 hermantolentino  staff       3049 Nov  3  2021 covid-testing-facilities.csv
-rw-r--r--   1 hermantolentino  staff      98304 Jan 31 23:52 filtered_testing_sites_4326_gdf.gpkg
-rw-r--r--   1 hermantolentino  staff   28372992 Nov 16  2021 footprints_gdf.gpkg
-rw-r--r--   1 hermantolentino  staff  163452187 Jan 29 12:28 g_projected.graphml
-rw-r--r--   1 hermantolentino  staff   18961666 Jan 29 12:30 g_projected_edges.feather
-rw-r--r--   1 hermantolentino  staff   12454914 Jan 29 12:30 g_projected_nodes.feather
-rw-r--r--   1 hermantolentino  staff  130597082 Jan 29 12:27 g_unprojected.graphml
-rw-r--r--   1 hermantolentino  staff   19749690 Jan 29 12:29 g_unprojected_edges.feather
-rw-r--r--   1 hermantolentino  staff  [...]
{'fraction_non_alphanumeric': 0.08011814657559535, 'fraction_numerical': 0.028521321764814474, 'mean_word_length': 4.819011815252416, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 26, 'https://': 12, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '277890', 'license': 'Apache-2.0', 'n_tokens_mistral': 3752, 'n_tokens_neox': 3466, 'n_words': 1199, 'path': 'NB01-Load-Graph.ipynb', 'repo_name': 'PHI-Case-Studies/2021-HELINA-COVID19-OSMNx-Workshop', 'size': '10834'}