<jupyter_start><jupyter_text>Step-by-step guide to generating an interactive climate map in Plotly (& Geopandas)- **With some specific boilerplate code already filled in.** ---  - **CREDITS**:        - This tutorial was adapted by [<NAME>](https://github.com/pawlodkowski) for the *Plotting on Maps* lesson @ Spiced Academy, based on original source code and lesson run by <NAME>, fellow Data Science Instructor @ Spiced Academy.        - **DATA**:     - The data for this particular lesson is **synthetic** / **fake!** It was originally scraped from [Berkeley Earth](http://berkeleyearth.lbl.gov/country-list/), but then regenerated based a simple polynomial trend and monthly seasonality modeled from the original data source & injected with random noise. Hence, the data in this repo is somewhat realistic but not authentic.- **INSTALLATION**:    - **Make sure you already have plotly and geopandas installed**! (e.g. `pip` or `conda`), e.g.:        - `pip install plotly`            - `pip install geopandas==0.8.0`     *Note*: this notebook was last run and tested on geopandas version 0.8.0 (released July 15, 2020). The latest version of plotly should work just fine. ---  STEP 1: Read in historical temperature data<jupyter_code>DATA = '../data/temp_data.csv'
#Read in the data with pandas
import pandas as pd
df = pd.read_csv(DATA)<jupyter_output><empty_output><jupyter_text>------  STEP 2: Read in the geographic data (geometric shapes of all countries in the world) - Hint: Use GeoPandas    - What is a **Shape file (.shp)?**        - https://en.wikipedia.org/wiki/ShapefileShapefile_shape_format_(.shp)- Publicly available GIS data downloaded from *Natural Earth*: https://www.naturalearthdata.com/downloads/110m-cultural-vectors/<jupyter_code>SHAPEFILE = '../data/ne_110m_admin_0_countries.shp'
#Read in the shapefile with geopandas
import geopandas as gpd
gdf = gpd.read_file(___)<jupyter_output><empty_output><jupyter_text>*What happens if I have trouble installing geopandas*?- Unfortunately, this happened a couple times in the past with students, as the conda / pip installation on some Windows machines has been somewhat unreliable.    - In this case, there have been some helpful posts from StackOverflow (for example, [this one](https://stackoverflow.com/questions/54734667/error-installing-geopandas-a-gdal-api-version-must-be-specified-in-anaconda)), which finally got the installation working.- However, if you still have trouble with the installation despite lots of trial and error, you may consider creating a virtual environment or building a custom Docker image that builds off another image with Jupyter (e.g. https://hub.docker.com/r/jupyter/datascience-notebook)    - If you choose to go this route, please reach out to a teacher for help. - If you would rather not try those options (for very understandable reasons), then don't worry; a teacher can provide you with a `.geojson` file you can use.    - After all, the only reason we use GeoPandas in the first place is to read in the data and generate these GeoJSON files, so you shouldn't let this hold you back -- it's not that important in the end!     - Also, in general, with enough time spent googling around, you can find GeoJSON files from pretty much everywhere! For example:        - https://github.com/johan/world.geo.json/tree/master/countries        - https://github.com/isellsoap/deutschlandGeoJSON/blob/master/2_bundeslaender/4_niedrig.geo.json        - https://github.com/funkeinteraktiv/Berlin-Geodaten/blob/master/berlin_bezirke.geojson        - _And many more! They're super easy to find online! The world of open source is truly wonderful..._ ------  STEP 3: Group / aggregate the temperature anomaly data by country, year- For simplicity, we're only interested in yearly averages<jupyter_code>df.groupby(...)<jupyter_output><empty_output><jupyter_text>STEP 4: Merge Data Sets.- We want to have our temperature data and geometric data in one place.- **Make sure you're still left with a GeoDataFrame at the end!**    - otherwise, if the resulting object is a regular Pandas dataframe, the ``.to_json()`` export will produce a normal ``.json`` file rather than a specialized ``.geojson`` file.    - A geojson file is very similar to a json file, with the exception that it's a bit more strict and specialized and works better with plotting libraries that usually expect the data to be in that format.<jupyter_code>gdf.merge(...)<jupyter_output><empty_output><jupyter_text>------ Time for Visualization with ------  STEP 5: Plot data on a map for a single year (we can make it interactive later)- Import the **"Plotly Express"** module (usually imported as ``px``), which contains high-level functions that can create entire figures at once. Plotly Express is a built-in part of the plotly library, and is the recommended starting point for creating most common figures.     - "Plotly Express is a terse, consistent, high-level API for creating figures." This means that it's a lot easier to use (less code) than the alternative option, which is to manually create, manipulate, configure, and render the underlying "Graph Objects" yourself.<jupyter_code>import plotly.express as px<jupyter_output><empty_output><jupyter_text>5a. Generate a GeoJSON string for a single year.- The Plotly library (as well as many other JavaScript-based mapping libraries) requires the data to be in GeoJSON format.    - **HINT**: How can you convert a GeoDataFrame into a GeoJSON? Think of how you might do this in regular pandas.- Let's use the year 2000 as an example.<jupyter_code>gdf_2000 = gdf[gdf['___'] == ___]
json_2000 = gdf_2000.to____()<jupyter_output><empty_output><jupyter_text>5b. Convert the GeoJSON string to an actual python dictionary- Strangely enough, the GeoJSON object that is generated from a GeoDataFrame is represented as a string.- Some plotting libraries can parse the string directly, while others (e.g. Plotly) require the GeoJSON data in python to be an actual dictionary.- We can use the built-in ``json`` library to convert this string into a dictionary, so that Plotly can understand it.<jupyter_code>import json
json_2000 = json.loads(json_2000)<jupyter_output><empty_output><jupyter_text>STEP 6: Generate an interactive choropleth map of the data for a single year- Use the convenient ``px.choropleth()`` method to generate the interactive figure, which you can even see directly in your Jupyter Notebook!<jupyter_code>fig = px.choropleth(
    data_frame = ____,               # use the year 2000 data
    geojson=____,    
    locations='____',                # name of the dataframe column that contains country names
    color='____',                    # name of the dataframe column that contains numerical data you want to display
    locationmode='country names',    # leave this as default
    scope='world',                   # change this to world, usa, ... 
    color_continuous_scale="thermal",
    range_color=(___, ___),
    color_continuous_midpoint=___
)

fig.update_layout(margin={"r":0,"t":0,"l":0,"b":0})

fig.show() #renders the figure directly in Jupyter Notebook!
#Hint: you may have to google around a bit if the figure doesn't render properly in Jupyter (e.g. you only see a blank screen)
#      otherwise, try exporting directly to HTML (next step)<jupyter_output><empty_output><jupyter_text>STEP 7: Export the figure to an HTML file, so you can open it in your web browser!- The reason the figure is interactive (e.g. you can zoom around, it has hover effects, etc.) is because there's a of front-end (i.e. client-side) JavaScript code that Plotly creates for you automatically.- Use the ``.write_html()`` method to export the file and open it up in a web browser.    - Bonus: Open the ``.html`` file in a text editor and see if you can understand any of it :)<jupyter_code>fig.write_html("2000_map.html", include_plotlyjs='cdn')
#bash command to open the file (should use your web browser by default)
!open 2000_map.html<jupyter_output><empty_output><jupyter_text>---  STEP 8: Add an interactive slider / animation to toggle between multiple years in a single file.- For this, we can use the convenient ``px.choropleth_mapbox()`` method that already has some built-in, front-end animation features right out of the box.- This time, pass the **entire DataFrame** (not just a single year slice) as an argument into the function.    - **WARNING**: While the HTML script seems to have fairly good performance right out-of-the-box, the resulting HTML file is pretty massive (~50 MB on hard disk), so you might want to delete the file afterwards!<jupyter_code>fig2 = px.choropleth_mapbox(
    mapbox_style='open-street-map',      # see other styles: https://plotly.com/python/mapbox-layers/
    data_frame = _____,                  # dataframe that contains all years
    geojson=json_2000,                   # we can still use the JSON data from 2000, assuming the countries are the same over time
    featureidkey='properties.____',      # name of JSON key within the "properties" value that contains country names
    locations='____',                    # name of the dataframe column that contains country names
    color='_____',                       # name of the dataframe column that contains numerical data you want to display
    center={"lat": 51.1657, "lon": 10.4515},
    zoom=1,
    animation_frame='_____',             # name of dataframe column that you want to make frames of
    animation_group='country',   
    color_continuous_scale="thermal",    # see other options: https://plotly.com/python/builtin-colorscales/
    range_color=(-50, 50),
    color_continuous_midpoint=0)
fig2.write_html("all_years_interactive.html", include_plotlyjs='cdn')
#this could take up to minute to generate -- file is very LARGE!
!open all_years_interactive.html<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0873588935218143, 'fraction_numerical': 0.011085121529543374, 'mean_word_length': 4.045664443304259, 'pattern_counts': {'":': 6, '<': 43, '<?xml version=': 0, '>': 43, 'https://': 10, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '388246', 'license': 'MIT', 'n_tokens_mistral': 2837, 'n_tokens_neox': 2663, 'n_words': 1185, 'path': 'plotly/plotly_tutorial.ipynb', 'repo_name': 'pawlodkowski/interactive_climate_map', 'size': '9848'}