<jupyter_start><jupyter_text>Fundamentals of Data Analysis Project 2020Student : <NAME> Task details : Perform and explain simple linear regression using Python on the powerproduction datasetGoal : Accurately predict wind turbine output from wind speed values using the data set as a basis.Using GIT repository  for the submission including the following items : 1. Jupyter notebook that performs simple linear regression on the data set2. An explanation of your regression and an analysis of the accuracy3. Standard items in a git repository such as a README4. Compare a simple linear regression to other types of regression on this data set The Powerproduction Dataset .........  1.Introduction  $^{1}$For this project , We will start/ use a simple linear regression to determine whether it can fit the particular type of curve in the dataset. Simple linear regression is a statistical model that allows us to summarize and study relationships between 2 continuous variables. In our dataset case the powerproduction of wine turbines,  the variables are wind speed and the turbine power output. Variables defined :X variable is wind speed and this can be regarded as the predictor or independent variableY variable is the response or dependent variable and here it represents the power output of the turbine.The image below depicts simple linear regression and shows the equation of the line ![Linear%20regression.png](attachment:Linear%20regression.png)   4 Assumptions of linear regression $^{2}$: 1. Linear Relationship : Is there a linear relationship between the independent variable x, and the dependent variable , y ? In order to determine this we create a scatter plot of x vs y. So for the dataset powerproduction , we created a scatter plot between the wind speed and power. We see that the points on the plot do not fall on a straight line which indicates that it is not a linear relationship but that there is a relationship between the two variables. There are two options here  :    -  To apply a nonlinear transformation to the independent and/or dependent variables. examples include taking the log or squareroot of either variable     - Add another independent variable to the model  2. Independence$^{13}$Are the residuals independent? When a simple linear regression is done, you get a line of best fit. The points dont fall exactly on the regression line , they are scatter about. A residual is the difference between the observed value and the mean value that the model predicts for that observation , negative if its below , positive if above and it zero residual if its on the line itself. Residuals are sometimes referred to as errors , which is an unexplained difference between the datapoint and the line.  ![residuals.webp](attachment:residuals.webp)3. HomoscedasticityDo the residuals have a constant variance at every level of x ? The opposite case to this is called heteroscedastivity and the results of the analysis become hard to trust, as it increases the variance of the regression coefficient estimates.To detect this , a fitted value vs, residual plot is create. If you see a cone shape in the results were the residuals spread out as the fitted value increases . To overcome this :  - Transform the dependent variable : taking the log of the dependent variable for example - Redefine the dependent variable - use a weighted regression  4. Normality  Are the residuals normally distributed ?  This can be checked using Q-Q plots or using formal statiscal tests like Shapiro-Wilk ( sensitive to larger sample sizes).  4 steps to build a Simple Linear Regression Model $^{3}$: Step 1. Importing the datasetStep 2. Data Pre-processingStep 3. Visualize the dataStep 4. Fitting a best fit line to the dataset dfpower  Step 1 Importing the PowerProduction Dataset Import the pandas library - this provides a easy to use data structures and data analysis tools<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

dfpower =pd.read_csv("https://raw.githubusercontent.com/ianmcloughlin/2020A-machstat-project/master/dataset/powerproduction.csv")
dfpower[0:6]

#dfpower.shape
dfpower.describe<jupyter_output><empty_output><jupyter_text>Step 2  Data Checking and  pre-processing**Data Checking Check that the dataset imported correctly by looking at the Head and Tail values. Also using the describe() to have an overview of the statistics of the dataset.<jupyter_code>#First 6 entries of the dataset powerproduction
dfpower.head()
# Last 6 entries of the dataset powerproduction
dfpower.tail()
# Statistic review of the dataset powerproduction
dfpower.describe()<jupyter_output><empty_output><jupyter_text>**Data Processing $^{4}$This is not necessary for this dataset - However we did examine the data to see if there was any outliers using scipy-stats .Using Z- scores and interquartile range methods to check if outliers are present.Using both methods resulted in the same conclusion , there was no outliers present to remove from the data as the shape returned ( 500,2) for all.<jupyter_code>import scipy.stats as stats

#find absolute value of z-score for each observation
z = np.abs(stats.zscore(dfpower))


#only keep rows in dataframe with all z-scores less than absolute value of 3 
data_clean = dfpower[(z<3).all(axis=1)]

#find how many rows are left in the dataframe 
data_clean.shape
#find Q1, Q3, and interquartile range for each column
Q1 = dfpower.quantile(q=.25)
Q3 = dfpower.quantile(q=.75)
IQR = dfpower.apply(stats.iqr)

#only keep rows in dataframe that have values within 1.5*IQR of Q1 and Q3
data_clean = dfpower[~((dfpower < (Q1-1.5*IQR)) | (dfpower > (Q3+1.5*IQR))).any(axis=1)]

#find how many rows are left in the dataframe 
data_clean.shape<jupyter_output><empty_output><jupyter_text>Step 3  Visualising the dataset $^{5}$<jupyter_code>import matplotlib.pyplot as plt

plt.scatter(dfpower.speed, dfpower.power)
plt.title('Power vs. Wind speed')
plt.xlabel('Speed (m/s)')
plt.ylabel('Power (kW)')
plt.show()

plt.style.use("ggplot")

# Bigger plots.
plt.rcParams["figure.figsize"] = (18,10)<jupyter_output><empty_output><jupyter_text>From the scatter plot above , there does not seem to be a classic linear graph but it does demonstrate that there is a relationship between wind speed and power , where the power increases as the windspeed increase, however it is not obviously linear. The first observation would be that the curve is  a sigmoid curve, shaped. Next is to try and best fit a line to the data points.   Numpy.polyfit  - Its importance $^{6}$In calculating the best fit line in this simple linear regression we use numpy.polyfit. It uses ordinary least squares method. This method does the following :  - It calculates the errors between all the data points and the regression model, where error is the difference between the y value that is estimated and the y value in the dataset   - It squares each of the error values  - Takes the sum of these values   - Find the line where this sum of the squared errors is the smallest possible value  Calculate the Coefficients of the line (m and c)$^{7}$The equation of a line for linear regression is  y = mx + c, where x is the WindSpeed ( independent Variable), and y = power ( dependent variable). m is the slope and c is the intercept.<jupyter_code>import numpy as np

#Converting df columns into np arrays for x and y
x=np.array(dfpower.speed)
y=np.array(dfpower.power)

print(type(x))
print(type(y))

dfpower =pd.read_csv("https://raw.githubusercontent.com/ianmcloughlin/2020A-machstat-project/master/dataset/powerproduction.csv")
x= dfpower.speed
y=dfpower.power

m = np.polyfit(x, y, 1)[0]
c = np.polyfit(x,y, 1)[1]
print(f'm(slope): {m:.3f}    c(intercept): {c:.3f}')#round the m and c values to 3 decimal places<jupyter_output><class 'numpy.ndarray'>
<class 'numpy.ndarray'>
m(slope): 4.918    c(intercept): -13.900<jupyter_text>Step 4 Plotting the line : best fit $^{8}$,$^{9}$Using the values of m and c<jupyter_code># Plot the best fit line.
plt.figure(figsize=(12,8))

x= dfpower.speed
y=dfpower.power

plt.plot(x, y, '.b', label='Original')
plt.plot(x, m * x + c, 'r-', label='Fitted line')

# Add axis labels and a legend.
plt.xlabel('WindSpeed (m/s)')
plt.ylabel('Power (kW)')
plt.legend()

# Show the plot.
plt.show();<jupyter_output><empty_output><jupyter_text>**Correlation coefficients : $^{9}$R-squared calculates how similar a regression line is to the data it’s fitted to.These quantify the relationships between the variables of the dataset. Results below are positive and close to 1. This indicates a positive relationship between the variables, that is when one variable increases so will the other.There are three examples of correlation coefficients :    - Pearson’s r  - measures linear correlation between 2 variables, value of +1 ( total positive linear )and -1 ( total   negative linear) and 0- no correlation. - Spearman’s rho - measures monotonic relationships ( whether linear or not). Compares the ranks of data  - Kendall’s tau - measures the ordinal association between two measured quantities.Compares the ranks of dataUsing SciPys statistics routine to calculate these three coeefficient versions :<jupyter_code>import scipy.stats
scipy.stats.pearsonr(x, y) # Pearson's r
scipy.stats.spearmanr(x, y)   # Spearman's rho
scipy.stats.kendalltau(x, y)  # Kendall's tau
The R value results for all three examples are postive correlation indicating that as x windspeed increases so does y power.<jupyter_output><empty_output><jupyter_text>**Simple Linear Regression conclusions :$^{15}$ The graph above shows us that there is very little power output when the speed is less than 10 meters per second and after 17 meters per second you are not generating any extra power. One would think the faster the wind turbine the more power you would generate but this is not the case. The most power generated is between the speeds of 10 and less than 20 meters per second. The curve looks more like a sigmoid curve ( s-shaped curve) rather than an obvious straight line. However the R2 values are all close to one.   **Advanced statistical parameters and residual plots for a Linear regression model:$^{9}$$^{14}$This can be achieved using the statsmodel package , it gives you a summary of results for the linear regression using the OlS() function ( ordinary least squares ) and creates residual plotsThis is method to check if the linear regression model is accurate and that the best possible estimates can be achieved. Residual plots show if the residuals of the linear regression model are normally distributed and are homoscedastic.Import statsmodels.api as smfrom statsmodels.formula.api import ols<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import statsmodels.api as sm
from statsmodels.formula.api import ols
import scipy.stats

dfpower =pd.read_csv("https://raw.githubusercontent.com/ianmcloughlin/2020A-machstat-project/master/dataset/powerproduction.csv")

x = dfpower.speed
y = dfpower.power


#fit simple linear regression model
model = ols('speed ~ power', data=dfpower).fit()

#view model summary
print(model.summary())

#define figure size
fig = plt.figure(figsize=(12,8))

#produce regression plots
fig = sm.graphics.plot_regress_exog(model, 'power', fig=fig)

Conclusion to OLS summary and plots : 
   
From the summary above , we can determine that : 
    
power (y) = 5.473 +0.1482*(speed) which can be used in prediction.
r2 value is 0.729  which is in line with the value achieved in above linear regression coefficent calculations. 

Four residual plots are produced. 
The plot of residual vs power shows that there is pattern , that it is not Homoscedasticity but heteroscedastivity. This is 
one of the assumptions of linear regression. Inorder to correct this  we could add in another independent variable or look at
polynomial regression or an interaction term.<jupyter_output><empty_output><jupyter_text>Other Regressions and methods : $^{10}$,$^{11}$.$^{12}$The above scatter plot shows that the relationship between the variables are not a simple linear one . Other types of regressions are polynomial and logisical .NB: We look at the method of R-squared value to calculate the accuracy of the model used. This value is a number between 0 and 1. , and the closer you are to 1 , the more accurate the linear regression model is. Below we examine another example of regression : Polynomial. There is various degrees of polynomial , starting at 1 which is your simple linear regression. Its important to note that as the degrees increases , the model gets increasing more complicated. This could lead to overfitting of the data or underfitting too. 1. Polynomial Regression Using NumPy Polyfit() Method to fit our data inside a polynomial function. It least squares the polynomial fit. It fits a polynomial p(x) of degree deg to points(x,y). This method returns a vector of coefficients p that minimises the squared error in the order of deg, deg-1, ... 0.The equation for Polynomial is : ![-2n](attachment:-2n)The syntax:numpy.polyfit (X, Y, deg, rcond=None, full=False, w=None, cov=False)  Calculating value of coefficients in case of linear polynomial - 2 constants1. z = np.polyfit(x, y, 1)  the last argument is the degree of polynomial<jupyter_code>import pandas as pd

import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score, mean_squared_error 
import sklearn.model_selection as mod
import numpy as np

dfpower =pd.read_csv("https://raw.githubusercontent.com/ianmcloughlin/2020A-machstat-project/master/dataset/powerproduction.csv")
dfpower[0:6]
x = dfpower.speed
y = dfpower.power
# calculating value of coefficients in case of linear polynomial - 2 constants
z = np.polyfit(x, y, 1) # the last argument is the degree of polynomial
#A one-dimensional polynomial class function created
poly1d_fn = np.poly1d(z)
print("\ncoefficient value in case of linear polynomial:\n", z)

plt.plot(x,y, 'yo', x, poly1d_fn(x), '--k')

#calculating the accuracy of the linear polynomial model using r2 values

predict=np.poly1d(z)
from sklearn.metrics import r2_score
r2_score(y, predict(x))<jupyter_output><empty_output><jupyter_text>Calculating value of coefficients in case of Cubic polynomial - 3 constantsz = np.polyfit(x,y,2)<jupyter_code>x = dfpower.speed
y = dfpower.power

z = np.polyfit(x,y,2)
poly1d_fn = np.poly1d(z) 
print(poly1d_fn)
# poly1d_fn is now a function which takes in x and returns an estimate for y

plt.plot(x,y, 'yo', x, poly1d_fn(x), '--k')
print("\ncoefficient value in case of linear polynomial:\n", z)

#calculating the accuracy of the cubic polynomial model using r2 values
predict=np.poly1d(z)
from sklearn.metrics import  r2_score
r2_score(y, predict(x))<jupyter_output><empty_output><jupyter_text>Calculating value of coefficient in case of quadratic polynomial - 4 constantsz = np.polyfit(x,y,3)<jupyter_code>import numpy as np
z = np.polyfit(x,y,4)
poly1d_fn = np.poly1d(z) 
print(poly1d_fn)
# poly1d_fn is now a function which takes in x and returns an estimate for y

plt.plot(x,y, 'yo', x, poly1d_fn(x), '--k')
print("\ncoefficient value in case of quadratic polynomial:\n", z)    
#calculating the accuracy of the quadratic polynomial model using r2 values
predict=np.poly1d(z)
from sklearn.metrics import r2_score
r2_score(y, predict(x))<jupyter_output><empty_output><jupyter_text>Comparison of the different degrees of polynomial regression : **linear polynomial - 2 constants Coefficient value in case of linear polynomial:[  4.91759567 -13.89990263]    r2 :0.7289360258564073 **Cubic polynomial - 3 constants          coefficient value in case of linear polynomial:[ -0.07021949   6.67418543 -21.22684391]    r2: 0.7351940425885428    **Quadratic polynomial - 4 constants        coefficient value in case of quadratic polynomial: [ -0.05241927   1.89485751 -12.94285866  19.20130246]    r2:0.8796883953739737From the r values above , the best fit line looks like the Quadratic polynomial - Degree of 4. Most of the data points are on the line. As defined above R-squared calculates how similar a regression line is to the data it’s fitted to.   Comparison of Linear Regression and Polynomial Regression  $^{11}$,$^{15}$$^{16}$To compare power output predictions using both Linear regression and Polynomial Regressions using SKlearn<jupyter_code># Import necessary libraries

import matplotlib.pyplot as plt
import numpy as np
from sklearn import datasets, linear_model
from sklearn.metrics import mean_squared_error, r2_score
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import PolynomialFeatures


import matplotlib.pyplot as plt


#import dataset
dfpower =pd.read_csv("https://raw.githubusercontent.com/ianmcloughlin/2020A-machstat-project/master/dataset/powerproduction.csv")

x = dfpower.speed
y = dfpower.power

#reshape() is used to convert from 1-D to 2-D
#x = x.values.reshape(-1, 1)
#y = y.values.reshape(-1, 1)

#Converting df columns into np arrays for x and y
x=np.array(dfpower.speed).reshape(-1, 1)
y=np.array(dfpower.power).reshape(-1, 1)

#fitting the linear regression model to the dfpower through polynomial features in sklearn

poly_reg = PolynomialFeatures(degree=1) # Degree 1 is a simple linear regression
x_poly = poly_reg.fit_transform(x) # transforming the x variables data to polynomial features
pol_reg = linear_model.LinearRegression(fit_intercept=False) # choosing the polynomial regression model
pol_reg.fit(x_poly, y)
print('Coefficients = ', lin_reg.coef_)
y_pred = pol_reg.predict(poly_reg.fit_transform(x))
r2 = r2_score(y,y_pred)
print('Coefficient of determination: %.2f'% r2)
print('Mean squared error: %.2f'
      % mean_squared_error(y,y_pred))


# visualising the linear regression model
plt.scatter(x,y, color='red')
plt.plot(x, pol_reg.predict(poly_reg.fit_transform(x)),color='blue')
plt.title("Windspeed vs Power (Linear)")
plt.xlabel('speed')
plt.ylabel('power')
plt.show()

import matplotlib.pyplot as plt
import numpy as np
from sklearn import datasets, linear_model
from sklearn.metrics import mean_squared_error, r2_score
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import PolynomialFeatures

#Converting df columns into np arrays for x and y
x=np.array(dfpower.speed).reshape(-1, 1)
y=np.array(dfpower.power).reshape(-1, 1)

poly_reg = PolynomialFeatures(degree=4)
x_poly = poly_reg.fit_transform(x) 

pol_reg = linear_model.LinearRegression(fit_intercept=False)
pol_reg.fit(x_poly, y)
print('Coefficients = ', pol_reg.coef_)
y_pred = pol_reg.predict(poly_reg.fit_transform(x))
r2 = r2_score(y,y_pred)
print('Coefficient of determination: %.2f'% r2)
print('Mean squared error: %.2f'
      % mean_squared_error(y,y_pred)) 
    
# Visualising the regression results( for higher resolution and smoother curve)
x_grid = np.arange(min(x),max(x),0.1)
x_grid = x_grid.reshape(len(x_grid),1) 
plt.scatter(x,y, color='red') # original values
 
plt.plot(x_grid, pol_reg.predict(poly_reg.fit_transform(x_grid)),color='blue') # using the original x values and the predicted y values ( from the transformed x values)
 
plt.title("Windspeed vs Power (Polynomial)")
plt.xlabel('speed')
plt.ylabel('power')
plt.show()
#predictions : 
lin_reg.predict([[15]])
lin_reg2.predict( poly_reg.fit_transform([[15]]))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07691910854312835, 'fraction_numerical': 0.020068097399917457, 'mean_word_length': 5.063497028464186, 'pattern_counts': {'":': 0, '<': 53, '<?xml version=': 0, '>': 52, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '392714', 'license': 'MIT', 'n_tokens_mistral': 5913, 'n_tokens_neox': 5458, 'n_words': 2403, 'path': 'Project 2020.ipynb', 'repo_name': 'g00387856/Fundamentals-of-Data-Analysis-Project-2020', 'size': '19392'}