<jupyter_start><jupyter_text>Berlin Temperatures 2009-2019  Table of Contents1. Introduction2. Install & Import Packages3. Access Data from NOAA4. Plot Station Using mplleaflet5. Transform Data6. Plot Data7. Future Areas to Explore  1. IntroductionExtreme temperature is an increasingly important topic to understand. As Berlin is my home, it would be great to visualize how its temperatures fared in 2019. Let's create an engaging, multidimensional visual of 2019 record highs and lows compared to the maximum and minimum temperatures for dates the decade before (from 2009 through 2018). This could be useful for fellow Berliners, public policy planners, or specialists looking to communicate similar patterns and trends in other areas. We'll access data from the NOAA's NCDC website (https://www.ncdc.noaa.gov/cdo-web/) using the search tool for daily summaries from January 1, 2009 through December 31, 2019 for stations in Berlin, Germany. We'll select the Schonefeld Airport station (GME00127930) as it has 100% coverage for these dates. Let's select air tempature information and the output format as 'Custom GHCN-Daily CSV'. Voila, a few minutes later, it's in our email inbox. NOAA has data from well over 100,000 weather stations around the world if you'd like to understand how your hometown is faring. First, we'll create a quick mplleaflet geo plot of the station using its latitude and longitude. Next, in transforming the data, let's slice what we need, use datetime methods like to_datetime and strftime on the dates, remove all February 29s (leap year day), group by month_day using groupby, and determine the max and min highs and lows for each date using numpy functions .min() and .max(). We'll then perform a 'left' merge to get our final dataframe. Let's also create record high and low series for 2019 that include temperatures above and below the max and min, respectively, for each date the decade before (2009-2018). A quick check on the record high and low dataframes reveals that in 2019, Berlin had 45 days with record high temperatures and 21 days with record low temperatures. Lastly, for plotting, let's be sure to create a visual that's truthful (includes relevant data, labels, scaling), functional, beautiful (clean and engaging), and insightful for fellow Berliners. As the main purpose here is to create an engaging end-visual, there's still a ton more we could explore, including interactive visuals and web apps, distributions and spreads, additional groupings by dates, incorporating other Berlin weather station data and precipitation data, broadening the timeframe, and forecasting.   2. Install & Import Packages<jupyter_code>import pandas as pd
import numpy as np

import matplotlib.pyplot as plt
%matplotlib inline
!pip install mplleaflet # For mapping the station
import mplleaflet
import matplotlib.dates as dates
import matplotlib.ticker as ticker
from datetime import datetime<jupyter_output>Collecting mplleaflet
  Downloading https://files.pythonhosted.org/packages/05/b3/e68ff575ef0872e096878c75f602e76b06ca44d7038d69d867c7b800745b/mplleaflet-0.0.5.tar.gz
Requirement already satisfied: jinja2 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from mplleaflet) (2.10)
Requirement already satisfied: six in /opt/conda/envs/Python36/lib/python3.6/site-packages (from mplleaflet) (1.12.0)
Requirement already satisfied: MarkupSafe>=0.23 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from jinja2->mplleaflet) (1.1.0)
Building wheels for collected packages: mplleaflet
  Building wheel for mplleaflet (setup.py) ... [?25ldone
[?25h  Stored in directory: /home/dsxuser/.cache/pip/wheels/bf/fd/0b/0513de62c339c4a56e2234aa42a5d06e6660a4446fdae7b796
Successfully built mplleaflet
Installing collected packages: mplleaflet
Successfully installed mplleaflet-0.0.5<jupyter_text>3. Access Data from NOAA  We'll access data from the NOAA's NCDC website (https://www.ncdc.noaa.gov/cdo-web/) using the search tool for daily summaries from January 1, 2009 through December 31, 2019 for stations in Berlin, Germany. We'll select the Schonefeld Airport station (GME00127930) as it has 100% coverage for these dates. Let's select air tempature information and the output format as 'Custom GHCN-Daily CSV'. Voila, a few minutes later, it's in our email inbox. NOAA has data from well over 100,000 weather stations around the world if you'd like to understand how your hometown is faring.<jupyter_code># The code was removed by Watson Studio for sharing.
# Read in data. TMAX and TMIN refer to the high and low temperature, respectively, in degrees Celsius
df_raw = pd.read_csv(body)
df_raw.head()
# Check shape - we have 4017 observations and 8 features
df_raw.shape
# Check for null values - as expected, there are none as this station has 100% coverage
df_raw.isnull().sum()
# Check info and data types. Notice that DATE is the object data type - we'll need to convert this to datetime data type later
df_raw.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 4017 entries, 0 to 4016
Data columns (total 8 columns):
STATION      4017 non-null object
NAME         4017 non-null object
LATITUDE     4017 non-null float64
LONGITUDE    4017 non-null float64
ELEVATION    4017 non-null float64
DATE         4017 non-null object
TMAX         4017 non-null float64
TMIN         4017 non-null float64
dtypes: float64(5), object(3)
memory usage: 251.1+ KB<jupyter_text>4. Plot Station Using mplleaflet<jupyter_code># Latitude and longitude from above
lat = df_raw.loc[0, 'LATITUDE']
long = df_raw.loc[0, 'LONGITUDE']

# Plot
plt.figure(figsize=(10,10))
plt.scatter(long, lat, c='r', alpha=0.7, s=300)
mplleaflet.display()<jupyter_output>/opt/conda/envs/Python36/lib/python3.6/site-packages/IPython/core/display.py:689: UserWarning: Consider using IPython.display.IFrame instead
  warnings.warn("Consider using IPython.display.IFrame instead")<jupyter_text>5. Transform Data<jupyter_code># Keep only DATE, TMAX, and TMIN columns
data = df_raw.iloc[:,5:8]
data.head()
# Convert date data to datetime type
data['DATE'] = pd.to_datetime(data['DATE'])

# Create Year column
data['Year'] = data['DATE'].dt.year

# Create Month_Day column (strftime converts date time to string)
data['Month_Day'] = data['DATE'].dt.strftime('%m-%d')

# Remove February 29 (leap year day):
data = data[data['Month_Day']!='02-29']

# Check first few rows
data.head()
# Max and min data for TMAX and TMIN only for 2009 through 2018, grouping by each day
max_temp = data[(data.Year >= 2009) & (data.Year < 2019)].groupby(['Month_Day'])['TMAX'].max()
min_temp = data[(data.Year >= 2009) & (data.Year < 2019)].groupby(['Month_Day'])['TMIN'].min()
# Check max_temp series
max_temp.head()
# Check min_temp series
min_temp.head()
# Merge max_temp and min_temp  data with 'data' dataframe on 'Month_Day' column 
# Use left merge as we want to keep all rows on left ('data') (missing 'Month_Day' values in right are NaN)
data = data.merge(max_temp.reset_index(drop=False).rename(columns={'TMAX':'Max_temp'}), on='Month_Day', how='left')
data = data.merge(min_temp.reset_index(drop=False).rename(columns={'TMIN':'Min_temp'}), on='Month_Day', how='left')

# Record highs and lows for 2019 (2019 temps above and below the max and min for 2009 through 2018)
record_high = data[(data.Year==2019)&(data.TMAX > data.Max_temp)]
record_low = data[(data.Year==2019)&(data.TMIN < data.Min_temp)]
# Check first few rows of new merged data
data.head()
record_high_num = len(record_high)
print("In 2019, Berlin had {} days with record high temperatures relative to the previous decade".format(record_high_num))
record_low_num = len(record_low)
print("In 2019, Berlin had {} days with record low temperatures relative to the previous decade".format(record_low_num))<jupyter_output>In 2019, Berlin had 21 days with record low temperatures relative to the previous decade<jupyter_text>6. Plot Data<jupyter_code># x-axis spanning 2019 only
# Remember .arange is end exclusive, also specify datetime type otherwise .arange will read values as strings
date_index = np.arange('2019-01-01','2020-01-01', dtype='datetime64[D]')

# Figure size
plt.figure(figsize=(14,8))

# Line plot of max and min temps by date
plt.plot(date_index, max_temp, color='salmon', linewidth=1) 
plt.plot(date_index, min_temp,color='skyblue', linewidth=1) 

# Scatter plot of 2019 record highs and lows
plt.scatter(record_high.DATE.values, record_high.TMAX.values, color='red', s=15)
plt.scatter(record_low.DATE.values, record_low.TMIN.values, color='dodgerblue', s=15)

# Fill colour between highs and lows
ax = plt.gca() # gets current axes
ax.fill_between(date_index, max_temp, min_temp, facecolor='grey', alpha=0.20)

# Set axis names and title:
plt.xlabel('Date', fontsize=12)
plt.ylabel('° Celsius', fontsize=12)
plt.title('Berlin Temperatures (2009-2019)', fontsize=14)

# Create legend and title, loc=0 provides the best position for the legend
plt.legend(['Record high (2009-2018)','Record low (2009-2018)','Record-breaking high in 2019','Record-breaking low in 2019'], loc='lower center', fontsize=12, frameon=False)

# Where you locate the major and minor ticks
ax.xaxis.set_major_locator(dates.MonthLocator())
ax.xaxis.set_minor_locator(dates.MonthLocator(bymonthday=15)) # Put the label at the minor tick so it's in the center.

# What you put at the ticks
ax.xaxis.set_major_formatter(ticker.NullFormatter())
ax.xaxis.set_minor_formatter(dates.DateFormatter('%b')) #%b means abbreviated month name, e.g., Jan

# 1 refers to the bottom of the plot for xticks and the left for yticks
# 2 refers to the top of the plot for xticks and the right for yticks
for tick in ax.xaxis.get_minor_ticks():
    tick.tick1line.set_markersize(0) # Make small ticker disappear
    tick.label1.set_horizontalalignment('center')

# Remove top and right spines to clean it up
plt.gca().spines['top'].set_visible(False)
plt.gca().spines['right'].set_visible(False)<jupyter_output>/opt/conda/envs/Python36/lib/python3.6/site-packages/pandas/plotting/_converter.py:129: FutureWarning: Using an implicitly registered datetime converter for a matplotlib plotting method. The converter was registered by pandas on import. Future versions of pandas will require you to explicitly register matplotlib converters.

To register the converters:
	>>> from pandas.plotting import register_matplotlib_converters
	>>> register_matplotlib_converters()
  warnings.warn(msg, FutureWarning)
{'fraction_non_alphanumeric': 0.09217075038284839, 'fraction_numerical': 0.045941807044410414, 'mean_word_length': 5.340412621359223, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 28, 'https://': 3, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '407561', 'license': 'MIT', 'n_tokens_mistral': 3559, 'n_tokens_neox': 3138, 'n_words': 1200, 'path': 'Berlin Temperature 2009-2019.ipynb', 'repo_name': 'hrishipoola/Berlin_Temperature_Visualization', 'size': '10448'}