<jupyter_start><jupyter_text>Plotting data as a heat mapHeatmaps are very useful ways of plotting data. Basically, each little color on the plot or the "map" is related to a calculated value for the area. In some cases this can be a two dimensional histogram, in other cases it can be a calculated statistic associated with a spatial value, or even something else.In this exercise we will plot the Global Earthquake Hazard Distribution - Peak Ground Acceleration 1976-2002. This data was collected by NASA and you can download it for free. We have downloaded it already for you and placed in the data folder as a zip file called `gdpga.zip`. **You will need to unzip the file yourself.** To download the file from NASA: [Download Here.](http://sedac.ciesin.columbia.edu/data/set/ndh-earthquake-distribution-peak-ground-acceleration/data-downloadclose)This data set has a few interesting characteristics so lets import it and check it out. In this exercise the data will be stored in a data folder but you can put it anywhere you want.*In this exercise, we went through the steps that we ACTUALLY went through to make the exercise. We did it this way instead of having every step working because we hope it shows that the scientific process is one of struggle.*<jupyter_code>%matplotlib inline
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
pwd
data_location = 'data/gdpga/gdpga.asc'

df = pd.read_csv(data_location)
df.head(6)<jupyter_output><empty_output><jupyter_text>Sometimes data can contain header rows that describe the data you have. In this case it tells us how many columns there are to be expected and how many rows to be expected and the lower left corner of the latitude and longitude. It also tells us how big the plotted value should be.<jupyter_code>df = pd.read_csv(data_location, skiprows=6, delim_whitespace=True,na_values=-9999)
df.columns.shape
df.index.shape
plt.pcolormesh(df.values)
df.values.max()<jupyter_output><empty_output><jupyter_text>What happened?!Data from NASA should be correct right? Maybe it downloaded wrong? The first thing I checked was the data I downloaded using a linux tool called `grep`:    cat gdpga.asc | grep '[0-9]'    This failed. Why? Basically what this does is search the file for values that contain numbers 0 to 9. But since I am clever, (not really see immediately previous mistake) I ammended this to only search for numbers 0 to 8.    cat gdpga.asc | grep '[0-8]'    This excludes any number that doesn't have 9 in it from being highlighted.![yay the data isn't wrong](http://i.imgur.com/DYBP5fE.png)Yay. Stuff is red. Red is good. The data isn't missing.So what was wrong? I'm not entirely sure. Let's try and load the data with numpy instead of pandas.<jupyter_code>data = np.loadtxt(data_location, skiprows=6)
data[0:5]
data.max()
data.min()<jupyter_output><empty_output><jupyter_text>Yay, there is data here.<jupyter_code>plt.pcolormesh(data)<jupyter_output><empty_output><jupyter_text>It annoys me to store data as -9999.0 instead of as NaN, lets get rid of that<jupyter_code>data[data==-9999.] = np.nan
plt.pcolormesh(data)<jupyter_output><empty_output><jupyter_text>Where did all our data go?<jupyter_code>data.max()<jupyter_output><empty_output><jupyter_text>Now we know why we originally had no data when importing via pandasSo we can look at what we used to import the data:    df = pd.read_csv(data_location, skiprows=6, delim_whitespace=True,na_values=-9999)    I just assumed that the header was correct that areas without data were -9999 and that pandas would *Do The Right Thing*. This obviously didn't happen. Note my error:    na_values=-9999If we get rid of this part pandas should work.<jupyter_code>df = pd.read_csv(data_location, skiprows=6, delim_whitespace=True)
df.values.max()
plt.pcolormesh(df.values)<jupyter_output><empty_output><jupyter_text>Yay it works!But this data is actually map data, and we shouldn't plot data when it doesn't exist.<jupyter_code>plt.pcolormesh(df.values, vmin=0, vmax=10)<jupyter_output><empty_output><jupyter_text>This looks better but it still is plotting data where there are zeros. So we should replace the -9999 with np.nan<jupyter_code>df.values[df.values==-9999.] = np.nan<jupyter_output><empty_output><jupyter_text>WHAT?!This is really strange. But it means we are probably storing everything as ints instead of floats. np.nans are floats so since we are aware of this perhaps we can fix it. But first lets check if that hypothesis is correct:<jupyter_code>masked_data = np.ma.masked_where(df.values<0,df.values)
plt.pcolormesh(masked_data, vmin=0, vmax=10)<jupyter_output><empty_output><jupyter_text>Now that is a good looking graph!But what is this masked business? Taken from the [scipy.org docs](https://docs.scipy.org/doc/numpy/reference/maskedarray.generic.htmlwhat-is-a-masked-array):    In many circumstances, datasets can be incomplete or tainted by the presence of invalid data. For example, a sensor may have failed to record a data, or recorded an invalid value. The numpy.ma module provides a convenient way to address this issue, by introducing masked arrays.    A masked array is the combination of a standard numpy.ndarray and a mask. A mask is either nomask, indicating that no value of the associated array is invalid, or an array of booleans that determines for each element of the associated array whether the value is valid or not.     What does this mean? Looking at a really simple example<jupyter_code>a = np.arange(4)
a
np.ma.masked_where(a<=2, a)<jupyter_output><empty_output><jupyter_text>Thus for the PGA data, we MASK all values below zero. Since we are using `pcolormesh`, we have a lot of behind the scenes magic that knows what do with masked data and we get:<jupyter_code>plt.pcolormesh(masked_data, vmin=0, vmax=10)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07348901098901099, 'fraction_numerical': 0.01201923076923077, 'mean_word_length': 5.263440860215054, 'pattern_counts': {'":': 0, '<': 51, '<?xml version=': 0, '>': 49, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '580820', 'license': 'MIT', 'n_tokens_mistral': 1825, 'n_tokens_neox': 1709, 'n_words': 745, 'path': 'example_notebooks/Introduction to plotting data as a heat map.ipynb', 'repo_name': 'mnky9800n/Python-course-materials-for-seismology-students', 'size': '5824'}