<jupyter_start><jupyter_text>About the DatasetThe dataset used in this project was obtained from the Environmental Protection Agency (EPA) and contains hourly measurements for air pollutant gases NO2, PM2.5 and PM10 for the period January 2019 to October 2020. The monitoring stations that collected the data are located in St. John's Road in Dublin City Centre.The air quality monitoring stations at St John's Road were selected, as they provided NO2, PM2.5 and PM10 data emissions over a continuous 22 month period. This data was not available for other monitoring stations across the country. Its location is opposite Heuston Station in Dublin City Centre, and as a monitoring location, it is highly affected by traffic emissions.The overall aim of this assignment is to determine if there was an effect on pollutant levels during the covid-19 restrictions compared to the same period in 2019. January to June 2020 pollutant levels will be compared to the same period in 2019.   ObjectivesThe goals of this assessment are as follows:- Preprocess the dataset so as to create a clean dataset for analysis- Look at descriptive statistics for the dataset- Visualise the dataset- Create a dashboard in Tableau - link belowhttps://dub01.online.tableau.com/t/airpollutantsjohnsroaddublin/views/StJohnsRoadPollutantLevels/Dashboard1?:showAppBanner=false&:display_count=n&:showVizHome=n&:origin=viz_share_link  Import Python LibrariesPython has a large number of libraries which makes it a powerful programming language for analysing data. The libraries that were imported and used in this project were NumPy, Pandas, Matplotlib and Seaborn..- NumPy - an array processing package.- Pandas - a data processing library which provides tools for shaping, merging, reshaping, and slicing data sets.- Matplotlib - a graphing library to create 2D plots and graphs on a variety of data formats.- Seaborn library - a plotting and styling library.<jupyter_code># for generating random variables
import numpy as np

# for plotting
import matplotlib.pyplot as plt 

# for managing data
import pandas as pd

# for plotting
import seaborn as sns 

# for converting date/time
from datetime import date, timedelta, datetime, time

%matplotlib inline

# to increase the size of the plots as plots are too small otherwise
plt.rcParams['figure.figsize'] = (20.0, 10.0)<jupyter_output><empty_output><jupyter_text>Data Preprocessing  Load the DatasetThe dataset was obtained from the Environmental Protection Agency (EPA). It is a csv file and contains 4 columns and 16081 rows. The csv file is imported into Python using Pandas.<jupyter_code># read in St Johns Road Hourly Data csv file using pandas
df = pd.read_csv("St Johns Road 2019 2020.csv", parse_dates=['DateTime'], encoding= 'unicode_escape')

# display the first 5 rows of the dataframe
df.head()
# check the datatypes
df.dtypes<jupyter_output><empty_output><jupyter_text>The types of data are datetime64 and float64. The datatypes do not need to be changed as they are correct. The three variables, PM10, PM2.5 and NO2 are all measured in µg/m³ and are type float64. The DateTime variable is of type datetime64 as required.<jupyter_code># print column names
df.columns
# check how many samples the dataset contains
df.shape
# check how many entries, columns and datatypes in the dataframe
df.info()
# check for any missing values in the dataset - a clean dataset is required for analysis
df.isnull().sum()<jupyter_output><empty_output><jupyter_text>There are a number of missing values in the dataset - all rows containing missing values will be dropped from the dataset.<jupyter_code># drop all rows with NaN values
df.dropna(axis=0,inplace=True)
# check to see if NaN values have been removed
df.isnull().sum()<jupyter_output><empty_output><jupyter_text>All missing values have been dropped from the dataset.<jupyter_code># change index of the dataframe to DateTime
df = df.set_index('DateTime')

# display first 3 rows of the dataframe
df.head(3)
# check the index column of the dataframe is now DateTime
df.info()
# check the index of the dataframe
df.index
# Add year, month, and weekday name columns to the dataframe
df['Year'] = df.index.year
df['Month'] = df.index.month
df['Weekday Name'] = df.index.weekday_name

# Display the first 5 rows of the dataframe
df.head()
# select a specific date using the loc command
df.loc['2019-01-04']
# display first 5 rows of the dataset
df.head()
# rename the column names
df.rename(columns = {'PM10 (µg/m³)':'PM10', 'PM2 5 (µg/m³)':'PM2 5', 'NO2 (µg/m³)':'NO2'}, inplace = True)

# check the column names have changed
df.head(3)
# rename the column name PM2.5 as not recognised when creating plots
df.rename(columns = {'PM2.5':'PM2 5'}, inplace = True)

# check the column name has changed
df.head(2)
# rename the column name PM2 5 as not recognised when creating plots
df.rename(columns = {'PM2 5':'PM25'}, inplace = True)

# check the column name has changed
df.head(2)<jupyter_output><empty_output><jupyter_text>Descriptive StatisticsNow that the dataset has been loaded into a Panda's dataframe and the data has been preprocessed, the dataset needs to be further analysed to learn more about what it contains.The Panda's describe function produces the descriptive statistics on the dataset. The mean, standard deviation, maximum number, minimum number, 25% quartile range (median), 50% quartile range and 75% quartile range are calculated for all variables.  NO2EU legislation inline with the World Health Organisation (WHO) guidelines, recommend limits of NO2 do not exceed 40 μg/m3 in any given hour.<jupyter_code># calculate descriptive statistics on the hourly time series data
df.describe()<jupyter_output><empty_output><jupyter_text>The descriptive statistics reveal that there are 15,821 samples for each variable in the hourly time series dataframe.EU legislation for NO2 limit values align with the World Health Organisation (WHO) guidelines, with 1 hour and 1 year limit values of 200 μg/m3 and 40 μg/m3 respectively. It can be seen from the table above that the hourly NO2 limit has not been exceeded between January 2019 and October 2020.  PM10World Health Organisation (WHO) guidelines provide for a daily average limit for PM10 of 50 µg/m3. The dataset is resampled to a daily frequency to determine whether this limit was exceeded.<jupyter_code># create a variable called dataColumns
dataColumns = ['PM10', 'PM25', 'NO2']

# resample to daily frequency, aggregating with mean
dfDaily = df[dataColumns].resample('D').mean()

# display first 5 rows
dfDaily.head()
# calculate descriptive statistics on the daily time series data
dfDaily.describe()<jupyter_output><empty_output><jupyter_text>The descriptive statistics reveal that by resampling the hourly dataset to a daily average the number of samples have been reduced to 667 for each variable.World Health Organisation (WHO) guidelines for PM10 is that levels should not exceed an average daily mean of 50 µg/m3. It can be seen from the table above that this value has been exceeded.  PM2.5World Health Organisation (WHO) guidelines provide for a yearly average limit for PM2.5 of 20 µg/m3. The dataset is resampled to a yearly averaged frequency to determine whether this limit was exceeded.<jupyter_code># create a variable called dataColumns1
dataColumns1 = ['PM10', 'PM25', 'NO2']

# resample to yearly frequency, aggregating with mean
dfYearly = df[dataColumns1].resample('Y').mean()

# display first 4 rows
dfYearly.head(2)
# calculate descriptive statistics on the yearly time series data
dfYearly.describe()<jupyter_output><empty_output><jupyter_text>The descriptive statistics reveal that by resampling the hourly dataset to a yearly mean the number of samples has been reduced to 2 for each variable.The World Health Organisation (WHO) guidelines for PM25 is that levels should not exceed an annual mean of 20 µg/m3. It can be seen from the above resampling dataframe that this limit has not been exceeded.  Data VisualisationIn this section the following plots will be created:  - Line plots for PM10, PM2.5 and NO2 from January 2019 to October 2020 for hourly time series data - Line plots for PM10, PM2.5 and NO2 from January 2019 to October 2020 for daily averaged time series data - Daily line plots for the variable NO2 selecting a working day during lockdown and the same working day from 2019<jupyter_code># experimenting with how data can accessed using the loc command
df.loc['2019-01']<jupyter_output><empty_output><jupyter_text>Line plots for PM10, PM2.5 and NO2 from January 2019 to October 2020First line plots are created on the hourly time series data for pollutants PM10, PM2.5 and NO2. There are 15,821 samples so these plots may not show any meaningful information.<jupyter_code># set the size of the plot
fig, ax = plt.subplots(figsize=(12,6))

# plot PM10 levels from January 2019 to October 2020
df.PM10.plot(lw=0.2, alpha=0.5)

# add title and axis labels to plot
ax.set(title='PM10 levels at St Johns Road in Dublin', xlabel='Date', ylabel='PM10')
sns.despine(offset=5, trim=True)       
# set the size of the plot
fig, ax = plt.subplots(figsize=(12,6))

# plot PM2.5 levels from January 2019 to October 2020
df.PM25.plot(lw=0.2, alpha=0.5, color='Green')

# add title and axis labels to plot
ax.set(title='PM2.5 levels at St Johns Road in Dublin', xlabel='Date', ylabel='PM2.5')
sns.despine(offset=5, trim=True)   
# set the size of the plot
fig, ax = plt.subplots(figsize=(12,6))

# plot NO2 levels from January 2019 to October 2020
df.NO2.plot(lw=0.2, alpha=0.5, color='red')

# add title and axis labels to plot
ax.set(title='NO2 levels at St Johns Road in Dublin', xlabel='Date', ylabel='NO2')
sns.despine(offset=5, trim=True)<jupyter_output><empty_output><jupyter_text>These line plots are not very intuitive as there are too many data points. The maximum and minimum values for the variables PM10, PM2.5 and NO2 can be seen but apart from that very little information is provided from these visualisations.   Resample Data to a Daily Mean Time SeriesAs the hourly dataset has a lot of values one solution is to resample the data from an hourly time series to a daily mean time series. This will reduce the number of data points considerably from 15,821 to 667.<jupyter_code># create a variable called dataColumns
dataColumns = ['PM10', 'PM25', 'NO2', 'Year', 'Month']

# resample to daily frequency, aggregating with mean
dfDaily = df[dataColumns].resample('D').mean()

# display first 4 rows
dfDaily.head(4)
# set the size of the plot
fig, ax = plt.subplots(figsize=(12,6))

# plot NO2 levels from January 2019 to October 2020
dfDaily.PM10.plot(lw=0.8, alpha=0.9)

# add title and axis labels to plot
ax.set(title='PM10 levels at St Johns Road in Dublin', xlabel='Date', ylabel='PM10')
sns.despine(offset=5, trim=True) 
# set the size of the plot
fig, ax = plt.subplots(figsize=(12,6))

# plot NO2 levels from January 2019 to October 2020
dfDaily.PM25.plot(lw=0.8, alpha=0.9, color='Green')

# add title and axis labels to plot
ax.set(title='PM25 levels at St Johns Road in Dublin', xlabel='Date', ylabel='PM25')
sns.despine(offset=5, trim=True)
# set the size of the plot
fig, ax = plt.subplots(figsize=(12,6))

# plot NO2 levels from January 2019 to October 2020
dfDaily.NO2.plot(lw=0.8, alpha=0.9, color='red')

# add title and axis labels to plot
ax.set(title='NO2 levels at St Johns Road in Dublin', xlabel='Date', ylabel='NO2')
sns.despine(offset=5, trim=True)<jupyter_output><empty_output><jupyter_text>The resampled plots are clearer due to the reduced number of data points. It can be seen that PM10, PM25 and NO2 levels are lower between January to June 2020 in comparison to the same period in 2019. PM10 and PM25 pollutants are primarily causing by burning coal, peat and wood. Levels did not decrease significantly for the period. This could be due to the fact that more people were working from home and lighting fires more frequently.The NO2 pollutant gas is primarily caused by diesel fuelled road traffic. The lockdown in Ireland commenced on the 12th March 2020 until 29th June 2020. The line graph shows NO2 levels reduced in this period in comparison to 2019. This was mainly due to the fact that there was less traffic on the roads due to the lockdown restrictions. Businesses and schools were closed and people were advised to work from home. The NO2 levels continue to be lower from June to October 2020 in comparison to the same period in 2019.  Daily line plots for NO2A line plot for the pollutant gas NO2 for 30th April 2020, a day in full lockdown, will be compared to the same day in 2019 to determine if there is a significant difference in the pollutant levels. PM10 and PM25 will not be analysed this way as there was not a significant difference in 2019 and 2020 levels as shown in plots previously.<jupyter_code># display values for working day Thursday 30th April 2020 to check that have 24 values
df.loc['2020-04-30']
# display values for working day Tuesday 30th April 2019 to check that have 24 values
df.loc['2019-04-30']
# set the size of the plot
fig, ax = plt.subplots(figsize=(12,6))

# plot NO2 levels for Thursday 30th April 2019
ax = df.loc['2019-04-30', 'NO2'].plot()

# add title and axis labels to plot
ax.set(title='NO2 levels at St Johns Road in Dublin on the 30th April 2019', xlabel='Time of Day', ylabel='NO2')
sns.despine(offset=5, trim=True)
# set the size of the plot
fig, ax = plt.subplots(figsize=(12,6))

# plot NO2 levels for Thursday 30th April 2020
ax = df.loc['2020-04-30', 'NO2'].plot(color='red')

# add title and axis labels to plot
ax.set(title='NO2 levels at St Johns Road in Dublin on the 30th April 2020', xlabel='Time of Day', ylabel='NO2')
sns.despine(offset=5, trim=True)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07095134350036311, 'fraction_numerical': 0.040595497458242556, 'mean_word_length': 5.005669428696032, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 49, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '169300', 'license': 'Apache-2.0', 'n_tokens_mistral': 4346, 'n_tokens_neox': 3824, 'n_words': 1910, 'path': "St John's Road Monitoring Station.ipynb", 'repo_name': 'beccycole/Data-Visualisation-Project', 'size': '13770'}