<jupyter_start><jupyter_text>Programming for Data Analysis Project ** Introduction** The weather, the obsession of all Irish adults. From a conversation starter to the dreaded threat of weddings and summer bar-be-cues.Meteireann has the dubious task of trying to give accurate and timely forecasts for the nation, while more often than not being the butt of the joke due to each individual taking a wholistic forecast as a personal gaurantee of weather certainty.The following project is going to analyse the percipitation (Rain Fall) from the Cork Airport weather station in County Cork under the following headings;- Date- Percipitation Amount (mm)- Maximum Air Temperature (C)- Minimum Air Temperature (C)- Mean Wind Speed (Knot)- Wind Direction at Max 10 min Mean (deg)- Mean CBL Pressure (hpa) ** Aim**The aim of this project is to explore the creation of an accurate localised weather predictor by investigating if known historical weather characterisitics have a dependency and relationship with each other. ** Method **By analysing historical weather data and its associated characteristics in relation to percipitation amount it is proposed that if the correct weather characteristics and data for the local area are entered into the model i.e. Ballincollig, County Cork, an accurate determination if percipition (rain) is forecast for your locality will be determined.As the Cork Airport weather station is the closest weather station with enough variable data to create a model it was used to gather historical data points.For the purpose of this project random data will be generated using numpy.random to generate a data sets to mimick the characteristics of the variables taken from the Cork Airport weather station. i.e.- Date- Percipitation Amount (mm)- Maximum Air Temperature (C)- Minimum Air Temperature (C)- Mean Wind Speed (Knot)- Wind Direction at Max 10 min Mean (deg)- Mean CBL Pressure (hpa)<jupyter_code>##Importing pandas library
import pandas as pd
##Importing numpy library
import numpy as np
##Importing matplotlib library
import matplotlib.pyplot as plt
##Importing scipy library
from scipy import stats
##Importing scipy stats library
from scipy.stats import kurtosis, skew
## Required to prevent warning in relation to repeat label use in Matplotlib
import warnings
import matplotlib.cbook
warnings.filterwarnings("ignore",category=matplotlib.cbook.mplDeprecation)<jupyter_output><empty_output><jupyter_text>**Definition of the Data Frame**To provide clarity when further analysis is conducted and to be able to differentiate between the original and random data sets we will define the original Cork airport data frame as "CorkAirportData"<jupyter_code>## Assigning the WeatherData Frame for Cork Airort Data as "CorkAirportData"
CorkAirportData = pd.read_csv('C://Users//Shane//Desktop//GMIT//meteireann//CorkAirportDataone.txt')
CorkAirportData<jupyter_output><empty_output><jupyter_text>** Summary Descriptive Statistics for the Data Frame**To generate random data to match the characteristics of the cork airport weather data I first need to determine the descriptive statistics for the data set. To generate a randm data set that closely aligns to the cork airport data set I am going to calculate the following statistical parameters.- Mean- Standard Deviation (std)- Skewness- Kurtosis<jupyter_code>##Calculating the Mean Values from the CorkAirportData Frame
"Mean Data"
CorkAirportData.mean()
##Calculating the Standard Deviation Values from the CorkAirportData Frame
CorkAirportData.std()
##Calculating the Skewness Values from the CorkAirportData Frame
CorkAirportData.skew()
##Calculating the Kurtosis Values from the CorkAirportData Frame
CorkAirportData.kurtosis()
##Calculating the Min Values from the CorkAirportData Frame
CorkAirportData.min()
##Calculating the Max Values from the CorkAirportData Frame
CorkAirportData.max()
##Calculating the Median Values from the CorkAirportData Frame
CorkAirportData.median()<jupyter_output><empty_output><jupyter_text>** Defining Each Column Within the Cork Airport Data Frame**To compare the data and graphs of the Cork airport data and the randomly generated data I will define each column within the data frame so it can be graphed. I will use the iloc function to define each column of data so they can be plotted on as subplots on bar chart graphs.<jupyter_code>## Creating a Max Temperature Histogram of the CorkAirportData but as a Subplot
plt.subplot(2,3,1)
MaxTemp=CorkAirportData.iloc[0:365,1]
plt.hist(MaxTemp)
plt.title("MaxTemp")
plt.xlabel("Qty")
plt.ylabel("Temperature")
## Creating a Min Temperature Histogram of the CorkAirportData but as a Subplot
plt.subplot(2,3,2)
MinTemp=CorkAirportData.iloc[0:365,2]
plt.hist(MinTemp)
plt.title("MinTemp")
plt.xlabel("Qty")
plt.ylabel("Temperature")
## Creating a Rain Histogram of the CorkAirportData but as a Subplot
plt.subplot(2,3,3)
Rain=CorkAirportData.iloc[0:365,3]
plt.hist(Rain)
plt.title("Percipitation (mm)")
plt.xlabel("Qty")
plt.ylabel("Volume")
## Creating a Pressure Temperature Histogram of the CorkAirportData but as a Subplot
plt.subplot(2,3,4)
Pressure=CorkAirportData.iloc[0:365,4]
plt.hist(Pressure)
plt.title("Pressure (Hpa)")
plt.xlabel("Qty")
plt.ylabel("Pressure")
## Creating a Wind Speed Histogram of the CorkAirportData but as a Subplot
plt.subplot(2,3,5)
WindSpeed=CorkAirportData.iloc[0:365,5]
plt.hist(WindSpeed)
plt.title("Wind Speed (knots)")
plt.xlabel("Qty")
plt.ylabel("Speed Knots")
## Creating a Wind Direction Histogram of the CorkAirportData but as a Subplot
plt.subplot(2,3,6)
WindDirection=CorkAirportData.iloc[0:365,6]
plt.hist(WindDirection)
plt.title("Wind Direction (deg)")
plt.xlabel("Qty")
plt.ylabel("Wind Direction (deg)")
## Displaying the Subplots
plt.tight_layout() ## Fix for Over Lapping Axis and Labels
plt.show()<jupyter_output><empty_output><jupyter_text>** Generatation of Random Data to Simulation the Cork Airport Weather Data**To analyse the data and to determine if there is any dependancies between the variables and rainfall, random data will be generated who's distributions approximate the real world weather data from Cork Airport. Below we detail the distributions assigned to each of the weather variables selected and a brief justification for selecting the specific distribution for each variable.           Weather Variable         Designation        Selected Distribution     - Max Temperature          RandMaxTemp        Normal Distribution     - Min Temperature          RandMinTemp        Normal Distribution     - Percipitation            RandRain           Gamma Distribution     - Pressure                 RandPressure       Gumbel Distribution     - Wind Speed               RandWindSpeed      Gumbel Distribution     - Wind Direction           RandWindDir        Gumbel Distribtion     As the original data visually illustrated the Max and Min temperatures displayed an approximation to the normal distribution the random data for Max and Min temperatures will be generated following the normal distribution.On review of literature it was discovered a number of distributions can be used to model percipitation amounts. As the original data set was right skewed the Gamma distribution was selected to model the random data for percipitation (Rain Fall)All other variables display extreme values and for this reason the Gumbel distributions were selected for them.      **RandMaxTemp**<jupyter_code>## Creating a Random Data Set for the Max Temp following the Normal Distribution
RandMaxTemp=np.random.normal(13.301366,5.517347,365)
plt.subplot(2,3,1)
plt.hist(RandMaxTemp)
plt.title("Daily Max Temperature")
plt.xlabel("Qty")
plt.ylabel("Temperature")<jupyter_output><empty_output><jupyter_text>**RandMinTemp**<jupyter_code>## Creating a Random Data Set for the Min Temp following the Normal Distribution
RandMinTemp=np.random.normal(6.671858,4.669434,365)
plt.subplot(2,3,2)
plt.hist(RandMinTemp)
plt.title("Daily Min Temperature")
plt.xlabel("Qty")
plt.ylabel("Temperature")<jupyter_output><empty_output><jupyter_text>**RandRain**<jupyter_code>## Creating a Random Data Set for Rain following the Gamma Distribution
RandRain = np.random.gamma(0.4, 5.256508, 365)
plt.subplot(2,3,3)
plt.hist(RandRain)
plt.title("Daily Rain Fall")
plt.xlabel("Amount")
plt.ylabel("Rain mm")<jupyter_output><empty_output><jupyter_text>**RandPressure**<jupyter_code>## Creating a Random Data Set for Pressure following the Gumbel Distribution
RandPressure = np.random.gumbel(996.85, 11.382705, 365)
plt.subplot(2,3,4)
plt.hist(RandPressure)
plt.title("Daily Pressure")
plt.xlabel("Pressure")
plt.ylabel("Pressure Frequency Hpa")<jupyter_output><empty_output><jupyter_text>**RandWindSpeed**<jupyter_code>## Creating a Random Data Set for Wind Speed following the Gumbel Distribution
RandWindSpeed = np.random.gumbel(8.9, 3.611673, 365)
plt.subplot(2,3,5)
plt.hist(RandWindSpeed)
plt.title("Daily Wind Speed")
plt.xlabel("Wind Speed")
plt.ylabel("Wind Speed Frequency Knots")<jupyter_output><empty_output><jupyter_text>**RandWindDir**<jupyter_code>## Creating a Random Data Set for Wind Direction following the Gumbel Distribution
RandWindDir = np.random.gumbel(230, 89.321351, 365)
plt.subplot(2,3,6)
plt.hist(RandWindDir)
plt.title("Daily Wind Direction")
plt.xlabel("Wind Direction")
plt.ylabel("Wind Direction Degs")
## Displaying SubPlots
plt.tight_layout() ## Fix for Over Lapping Axis and Labels
plt.show()
## Including a range of integers for use in the scatter plots for an "X" value
DayNumber = range(1,366)<jupyter_output><empty_output><jupyter_text>**Random Data, Data Frame Creation**Now that we have created random data to simulate the real world weather data from Cork airport, we will create a data from to store our results.I will define and name the data frame RanWeatherData to differenciate from the original data set from Cork airport.<jupyter_code>## Assigning the name "RanWeatherData" to the random weather data frame
RanWeatherData = pd.DataFrame({"Day Number": DayNumber,"Random Max Temp":RandMaxTemp, "Random Min Temp": RandMinTemp, "Random Rain": RandRain, "Random Pressure": RandPressure, "Random Wind Speed": RandWindSpeed, "Random Wind Direction": RandWindDir})
## Displaying the Random Data, Data Frame
RanWeatherData<jupyter_output><empty_output><jupyter_text>** Comparison of Original Data and Random Generated Data** To compare and contrast the original Cork Airport weather data and the random generated data, I will generate box plots to compare and contrast the data sets. The box plot will allow a standardized way of displaying the distribution of the data based on five descriptive statistics, namely minimum value, first quartile, median value, third quartile, and maximum value.<jupyter_code>## Creating a Boxplot for the Max Temp Data Sets as a Subplot
plt.subplot(1,2,1)
MaxTempBoxPlot=[RandMaxTemp, MaxTemp]
plt.title("Max Temperature Comparison")
plt.boxplot(MaxTempBoxPlot)
plt.xlabel('Data Points')
plt.ylabel('Temperature')
plt.xticks([1, 2], ['RandMaxTemp', 'MaxTemp'])
## Creating a Boxplot for the Min Temp Data Sets as a Subplot
plt.subplot(1,2,2)
MinTempBoxPlot=[RandMinTemp, MinTemp]
plt.title("Min Temperature Comparison")
plt.boxplot(MinTempBoxPlot)
plt.xlabel('Data Points')
plt.ylabel('Temperature')
plt.xticks([1, 2], ['RandMinTemp', 'MinTemp'])
plt.show()
## Creating a Boxplot for the Pressure Data Sets as a Subplot
PressureBoxPlot=[RandPressure, Pressure]
plt.title("Pressure Comparison")
plt.boxplot(PressureBoxPlot)
plt.xlabel('Data Points')
plt.ylabel('Pressure')
plt.xticks([1, 2], ['Rand Pressure','Pressure cbl Hpa'])
plt.show()
## Creating a Boxplot for the Rain Data Sets as a Subplot
RainBoxPlot=[RandRain, Rain]
plt.title("Rain Comparison")
plt.boxplot(RainBoxPlot)
plt.xlabel('Data Points')
plt.ylabel('Rain')
plt.xticks([1, 2], ['Random Rain','Rain'])
plt.show()
## Creating a Boxplot for the Wind Speed Data Sets as a Subplot
WindSpeedBoxPlot=[RandWindSpeed, WindSpeed]
plt.title("Wind Speed Comparison")
plt.boxplot(WindSpeedBoxPlot)
plt.xlabel('Data Points')
plt.ylabel('Wind Speed')
plt.xticks([1, 2], ['Random Wind Speed','Wind Speed'])
plt.show()
## Creating a Boxplot for the Wind Direction Data Sets as a Subplot
WindDirectionBoxPlot=[RandWindDir, WindDirection]
plt.title("Wind Direction Comparison")
plt.boxplot(WindDirectionBoxPlot)
plt.xlabel('Data Points')
plt.ylabel('Wind Direction')
plt.xticks([1, 2], ['Random Wind Direction','Wind Direction'])
plt.show()<jupyter_output><empty_output><jupyter_text>**Box Plot Comparison Results**As can be seen from the box plots the random data generated for the majority of the weather variables mimick the distribution of the Cork Airport weather data, except for Wind Direction. Possibly the incorrect distribution for generating the Wind Direction random data was selected and this would explain the differences in the random data generated. As can be seen from the box plot the random data is right skewed while the airport data is left skewed, which indicates opposing distributions. **Generation of Scatter Plots**To use PearsonR correlation analysis tool a fundamental requirement is that the data is linear. I will now generate scatter plots for each of the random data weather variables to graphically display the data sets.<jupyter_code>## Creating a scatter plot of the Max Temp Random Data Set
plt.scatter(DayNumber,RandMaxTemp, s=100, marker=(5, 2))
plt.title('Scatter Plot of Random Data Max Temperature')
plt.xlabel('Day Number')
plt.ylabel('Temperature')
plt.show()
## Creating a scatter plot of the Min Temp Random Data Set
plt.scatter(DayNumber,RandMinTemp, s=100, marker=(5, 2))
plt.title('Scatter Plot of Random Data Min Temperature')
plt.xlabel('Day Number')
plt.ylabel('Temperature')
plt.show()
## Creating a scatter plot of the Pressure Random Data Set
plt.scatter(DayNumber,RandPressure, s=100, marker=(5, 2))
plt.title('Scatter Plot of Random Data Atmospheric Pressure')
plt.xlabel('Day Number')
plt.ylabel('Pressure cbl Hpa')
plt.show()
## Creating a scatter plot of the Rain Random Data Set
plt.scatter(DayNumber,RandRain, s=100, marker=(5, 2))
plt.title('Scatter Plot of Random Data Percipitation')
plt.xlabel('Day Number')
plt.ylabel('Rain Fall mm')
plt.show()
## Creating a scatter plot of the Wind Speed Random Data Set
plt.scatter(DayNumber,RandWindSpeed, s=100, marker=(5, 2))
plt.title('Scatter Plot of Random Data Wind Speed')
plt.xlabel('Day Number')
plt.ylabel('Wind Speed knots')
plt.show()
## Creating a scatter plot of the Wind Direction Random Data Set
plt.scatter(DayNumber,RandWindDir, s=100, marker=(5, 2))
plt.title('Scatter Plot of Random Data Wind Direction')
plt.xlabel('Day Number')
plt.ylabel('Wind Direction degrees')
plt.show()<jupyter_output><empty_output><jupyter_text>** Correlation Analysis of Weather Variables**The statistical evaluation of the relationship between two numerical continuous variables is called Correlation Analysis. Every day examples would be a persons height to weight ratio and a persons exam results in relation to study time.For the purpose of this project, below, we are going to perform correlation analysis on the following variables against rain fall to see if there is a relationship between the amount of rain fallen and any of the other weather variables. - Maximum Air Temperature (C)- Minimum Air Temperature (C)- Mean Wind Speed (Knot)- Mean CBL Pressure (hpa) There is a number of different commands that could be used to determine if a correlation exists between rain fall and any of the other variables. These include;- np.corrcoef- stats.pearsonrThe correlation coefficient can range in value from -1 to 1. The larger the coefficient value the stronger the relationship between the variables. A perfect linear correlation is indicated by an absolute value of 1. The sign of the coefficient correlation value indicates the direction of the relationship. A positive value indicates both variables tend to increase or decrease together. If one variable increases and the other decreases a negative correlation coefficient is produced.For the purpose of this project the stats.pearsonr command will be utilised. There are a number of assumptions that need to be met before the Pearsons correlation method can be used but for the purpose of the project each varialbe will be analysed using the Pearson method.The correlation coefficient result is outputted in the following for;(x,y)where x if the correlation coefficient and y is the p-value.The significance of the p-value is important. The p-value indicates if the correlation coefficient is significantly different from 0. A coefficient of 0 indicates no linear relationship.If the p-value is less than or equal to the specified alpha level (0.05) it can be concluded that the correlation is significantly different from 0. If the p-value is greater than the alpha level it cannot be concluded that the correlation is significant from 0. ** Rain vs Pressure **<jupyter_code>## Correlation analysis of Rain vs Pressure using the Pearsonr Correlation Method
stats.pearsonr(RandRain, RandPressure)
## Scatter plot of Rain vs Pressure
plt.scatter(RandRain, RandPressure)
plt.show()<jupyter_output><empty_output><jupyter_text>**Analysis Interpretation**The first value listed 0.067132682332380747 is the Pearson Coefficient number and the second value is the p-value. As the pearson number is close to zero indicates there is no correlation between rain fall and pressure. As the p-value is greater than 0.05 it cannot be concluded that the correlation is different from 0. **Rain vs Maximum Temperature**<jupyter_code>## Correlation analysis of Rain vs Max Temp using the Pearsonr Correlation Method
stats.pearsonr(RandRain, RandMaxTemp)
## Scatter Plot of Rain vs Max Temp 
plt.scatter(RandRain, RandMaxTemp)
plt.show()<jupyter_output><empty_output><jupyter_text>**Analysis Interpretation**The first value listed 0.14261847330715821 is the Pearson Coefficient number and the second value is the p-value. As the pearson number is close to zero indicates there is no correlation between rain fall and maximum temperature. As the p-value is less than 0.05 it can be concluded that the correlation is different from 0. ** Rain vs Minimum Temperature**<jupyter_code>## Correlation analysis of Rain vs Min Temp using the Pearsonr Correlation Method
stats.pearsonr(RandRain, RandMinTemp)
plt.scatter(RandRain, RandMinTemp)
plt.show()<jupyter_output><empty_output><jupyter_text>**Analysis Interpretation**The first value listed -0.032223282535898991 is the Pearson Coefficient number and the second value is the p-value. As the pearson number is close to zero indicates there is no correlation between rain fall and minimum temperature. As the p-value is greater than 0.05 it cannot be concluded that the correlation is different from 0. ** Rain vs Wind Speed **<jupyter_code>## Correlation analysis of Rain vs Wind Speed using the Pearsonr Correlation Method
stats.pearsonr(RandRain, RandWindSpeed)
plt.scatter(RandRain, RandWindSpeed)
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07696346073078539, 'fraction_numerical': 0.015172196556068879, 'mean_word_length': 5.842313218390805, 'pattern_counts': {'":': 7, '<': 69, '<?xml version=': 0, '>': 69, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '604483', 'license': 'Apache-2.0', 'n_tokens_mistral': 5773, 'n_tokens_neox': 5126, 'n_words': 2261, 'path': 'Programming for Data Analysis Project.ipynb', 'repo_name': 'shanecanny1/Program-for-Data-Analysis-Project', 'size': '19048'}