<jupyter_start><jupyter_text>MACHINE LEARNING AND STATISTICS PROJECT 2020In this notebook I am am creating my project for the Machine Learning and Statistics 2020 module.Find in markdown cells below a description of the project( instructions for the project to be deleted before last commit), references and the code explained with markdown cells, plus commented code cells explaining what is being done. ***References- [Fundamentals DA project's repository ](https://github.com/Ainara12/Fundamentals-Project/blob/master/Fundamentals%20DA-Project.ipynb)- [Introduction to Linear and Polynomial Regression](https://towardsdatascience.com/introduction-to-linear-regression-and-polynomial-regression-f8adc96f31cb)- [Numpy polyfit function documentation](https://numpy.org/doc/stable/reference/generated/numpy.polyfit.html)- [Numpy poly1d function documentation](https://numpy.org/doc/stable/reference/generated/numpy.poly1d.html)- [Lecturer's simple linear regression Jupyter notebook](https://github.com/ianmcloughlin/jupyter-teaching-notebooks/blob/master/simple-linear-regression.ipynb)- [Machine Learning Polynomial Regression](https://www.w3schools.com/python/python_ml_polynomial_regression.asp)- [Introduction to Keras](https://keras.io/getting_started/intro_to_keras_for_engineers/)- [Lecturer's linear regression in Keras Jupyter notebook](https://github.com/ianmcloughlin/jupyter-teaching-notebooks/blob/master/keras-linear.ipynb)- [Neural network with Keras tutorial](https://machinelearningmastery.com/tutorial-first-neural-network-python-keras/)- [Classification and regression using Keras tutorial](https://stackabuse.com/tensorflow-2-0-solving-classification-and-regression-problems/) - [Linear regression using Python Sklearn video tutorial](https://www.youtube.com/watch?v=b0L47BeklTE&ab_channel=RylanFowers)    ***  Project instructions![Project%20Instructions.jpg](attachment:Project%20Instructions.jpg)  Overall purpose of this projectMy goal with this project is , following instructions above, use machine learning to make predictions using the 'powerproduction' dataset. I have some works done for this in my project for the module Fundamentals of Data Analysis  that I will use as reference to create a model that predicts output wind turbine power from wind speed values.Once this model is complete I will create a script that runs a web service based on this model and a Dockerfile to build and run the web service in a container. To achieve this goal I am using the information and knowledge gathered during the course of this module along with the references consulted and detailed in this document.   Dataset analysis Loading and observing raw datasetBefore we get into create a prediction model, I am looking into the raw dataset to get a feel into how it looks and see some descriptive statistics. See works in code cells below:<jupyter_code>#Importing the modules I am going to use

import pandas as pd #to load and organize dataset
import matplotlib.pyplot as plt #for visualisation
import numpy as np #to work with arrays and apply regression functions

#loading dataset

data=pd.read_csv(r'C:\Users\fainl\OneDrive\Documents\GMITSEP2020\Machine Learning -Project\Powerproduction dataset.csv', header=0) 

#converting this dataset into dataframe with pandas 

powerdata=pd.DataFrame (data, columns = ['speed','power'])

powerdata=powerdata.iloc[1:]

print(powerdata)

#initially describing dataset using pandas functionalities

powerdata.describe()<jupyter_output><empty_output><jupyter_text>The code cells above show us the structure of this dataset. This dataset is based on a real life scenario in which we have 499 rows ( initially 500 but I dropped first row as had no information on it) and two columns for wind speed and power generated. The dataset shows the relation between the wind speed and the power output derived from this speed.Using command describe we get some general information such as the mean or average for 'speed' being 12.62 while power has 48.11, the std and minimum and max values. Now on the cells belows I am plotting this data so we can see a clearer picture of the correlation of these 2 variables.<jupyter_code>#Plotting dataset initially in its 'raw' form: 

# Matplotlib interactive plots

%matplotlib inline
#resizing
plt.rcParams['figure.figsize']=([15, 10])



#plotting, speed is shown blue dots while power is the green section
plt.plot( 'speed', 'y1', data=powerdata, marker='o', markerfacecolor='blue', markersize=3, color='skyblue', linewidth=4)
plt.plot( 'power', 'y2', data=powerdata, marker='2', color='olive', linewidth=4)


plt.legend()

plt.show()<jupyter_output><empty_output><jupyter_text>As seen above and considering my analysis in [Fundamentals-DA project's repository](https://github.com/Ainara12/Fundamentals-Project/blob/master/Fundamentals%20DA-Project.ipynb), I have concluded that this dataset seems to fit in a Polynomial Linear Regression. A Polynomial Linear Regression, See below an example on how I created the curve that fits this data and found the **R-Squared value** to be very high ( closer to 1). In order to apply the polynomial regression to our plot I  am creating a variable that will use **numpy poly1d** and **numpy polyfit** functions to generate this curve to fit the data.<jupyter_code>#Applying Polynomial regression to our plot:

#Creating variable with numpy functions

#first I separate the 2 variables to represent them 

power=powerdata['speed']
speed=powerdata['power']


Polynom=np.poly1d(np.polyfit(speed, power,deg=3))

xline=np.linspace(0.0, 120, 200) # creatingand even spaced axis

#plotting these elements with a scatter plot
#resizing
plt.rcParams['figure.figsize']=([15, 10])

plt.scatter( speed, power)# Using scatterplot to represent the two variables 
plt.plot(xline, Polynom(xline), 'r*')#using function to add our xline along wit the result of the poly1d and polyfit functions 

plt.show()<jupyter_output><empty_output><jupyter_text>Now we calculate **R-squared value** also called **Coefficient of determination** , this value measures how well a regression model fits to the data. We can calculate using the **numpy polyfit function** and then proceed to square this value. This coefficient can be positive or negative. **Numpy polyfit function** uses Pearson's correlation coefficient , which differs from the **coefficient of determination** above described as it measures the strength of the linear relationship between 2 sets of observations, in this case how much the output power depends on the wind speed;power output is the dependent variable (*y* value) . It also tells whether the relationship is positive or negative. In this case since we have a Polynomial linear regression , I am using **Sklearn** to obtain the **coefficient of determination** .I followed steps found on this [guide](https://www.w3schools.com/python/python_ml_polynomial_regression.asp)I am describing the process on cell below.<jupyter_code>#calculating R-squared value with sklearn

#first, I import sklearn and specific module

from sklearn.metrics import r2_score

#Using r2_score functionality with my previous created variable 'Polynom' 
r2=r2_score(power,Polynom(speed) )

print('The R-squared value is: ',r2)<jupyter_output>The R-squared value is:  0.7321827537382541<jupyter_text>How we interpret the R-squared value? The closest a value is to 1 the better fit, in this case with 0.7321827537382541 we have a moderate to very good fit. This implies that most of the changes in the dependent variable or *y* (power) are explained by the corresponding changes in *x* or independent variable (speed).  Presenting and training *Model*In this section I am going to create a model to predict values, based on what we have learn in previous sections about this dataset. Once I created the model I will analyse its accuracy.   Presenting modelMy model is based on this [guide/tutorial](https://www.w3schools.com/python/python_ml_polynomial_regression.asp) in which after confirming that Polynomial Regression fits the data very well, I use module **r2_score** from **sklearn** in order to predict how much power will be generated from a specific wind speed value. In this example we are going to test the model considering that wind speed is :2.179.Let's see the steps I have taken below and what output power result we obtain:<jupyter_code>#Using my variable Polynom created in previous steps

Polynom=np.poly1d(np.polyfit(speed, power,deg=3))

#I enter the value that I want to find out with this model

Power_generated = Polynom(2.179)

print('This is the amount of power generated considering win speed is 2.179 mph:')
print(Power_generated)<jupyter_output>This is the amount of power generated considering win speed is 2.179 mph:
5.912036065122072<jupyter_text>Experimenting with Keras and Neural networks model In this section of the project I am experimenting with sklearn neural networks, following [lecture's tutorials](https://web.microsoftstream.com/video/b3c0a6ba-86b6-4f4a-bc1d-48d26c868bea). My intention is again to predict what power output we obtain from an specific wind speed as we tried above using **sklearn r2_score**. My main foucs here is to have a portion of the data that will be part of the train model and reach then more accurate results. See my attemp below:   What is Keras for Machine learning?Keras for machine learning is source Also using lecturer's videoand Jupyter notebook.<jupyter_code>#Now using dataset I am creating a new neural network with two neurons ( one for wind
#speed the other for Power generated )

# using this tutorial: https://machinelearningmastery.com/tutorial-first-neural-network-python-keras/


# Neural networks.
import tensorflow.keras as kr

# Plot style.
plt.style.use("ggplot")

# Plot size.
plt.rcParams['figure.figsize'] = [14, 18]

#loading dataset
Powerdataset=pd.read_csv('Powerproduction dataset.csv', delimiter=',')


x=Powerdataset['speed']
y=Powerdataset['power']

#As this dataset fits better a polynomial regression I am trying this approach

f = lambda x: 2.0 * x**2 + 3.0 * x + 4.0

poly = pd.DataFrame()
poly['x'] = Powerdataset['speed']
poly['y'] = poly['x'].apply(f)

# Creating model using sequential
model = kr.models.Sequential()
model.add(kr.layers.Dense(50, input_shape=(1,), activation='sigmoid', kernel_initializer="glorot_uniform", bias_initializer="glorot_uniform"))
model.add(kr.layers.Dense(1, activation='linear', kernel_initializer="glorot_uniform", bias_initializer="glorot_uniform"))
model.compile(kr.optimizers.Adam(lr=0.001), loss='mean_squared_error')#Adjusting learn rate  to 0.001 for more accuracy

# Fit the data.
model.fit(poly['x'], poly['y'], epochs=1000, batch_size=10) # batch_size chooses how many values are going to be send to add the weights


#Now we plot the result showing in red for actual and blue for the prediction obtained

#resizing
plt.rcParams['figure.figsize']=([15, 10])

xline=np.linspace(0.00,1,100)

plt.plot(xline)
plt.plot(poly['x'], poly['y'], label='actual')
plt.plot(poly['x'], model.predict(poly['x']), label='prediction')
plt.legend();

#making predictions using this model:

# Let's prepare the model to receive new instances ( new values for wind speed)which are unknown for the model:
Xnew, a = make_regression(n_samples=3, n_features=2, noise=0.1, random_state=1)
Xnew = scalarX.transform(Xnew)

# make a prediction
ynew = model.predict(Xnew)
# show the inputs and predicted outputs

for i in range(len(Xnew)):
    print("X=%s, Predicted=%s" % (Xnew[i], ynew[i]))<jupyter_output><empty_output><jupyter_text>Now I am using the same method than lecturer to evaluate if this model is working as intended to predict the 'y' values.<jupyter_code>#we can take a sample of values and see the predictions

model.predict([1.0,2.0,3.0,4.0])

#now I use this piece of code to see what 'f' says the values are

np.array([[f(i)] for i in [1.0, 2.0, 3.0, 4.0]])<jupyter_output><empty_output><jupyter_text>Conclusion about this ModelAfter plotting the results  and evaluate the model as in the cells direcly above this one, we can see that the model is not working 100% although based on the (talk about laminar flow and turbulent when find good references) and the prediction does not seem to be accurate to what we could expect to happen with the power output. In this case we haven't trained the data.   Using a different approach with KerasLet's try a different approach next , training our data. I am using this[tutorial](https://stackabuse.com/tensorflow-2-0-solving-classification-and-regression-problems/) as reference.See steps below.- First I import the necessary 'Sklearn' modules to split the data into training and test sets.- I import 'Tensorflow keras layers' and 'Tensorflow keras models' to create and train the model.<jupyter_code>#First I import the needed modules to divide dataset into training and 
#test sets

from sklearn.model_selection import train_test_split
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.20, random_state=0)

from tensorflow.keras.layers import Input, Dense, Activation,Dropout
from tensorflow.keras.models import Model<jupyter_output><empty_output><jupyter_text>- Next step is to create the model using 'Sequential'. There are two ways to build Keras models ( Functional and Sequential) and in this case I am using same way as the tutorial as this is a simplest type of model, with a linear stock of layers. For this dataset I think is enough to use Sequential.<jupyter_code>#Creating model

model = kr.models.Sequential()
model.add(kr.layers.Dense(2, input_shape=(1,), activation='sigmoid', kernel_initializer="glorot_uniform", bias_initializer="glorot_uniform"))
model.add(kr.layers.Dense(1, activation='linear', kernel_initializer="glorot_uniform", bias_initializer="glorot_uniform"))
model.compile(kr.optimizers.Adam(lr=0.001), loss='mean_squared_error')#Adjusting learn rate  to 0.001 for more accuracy<jupyter_output><empty_output><jupyter_text>In the next step we are training the model, I selected batch size 10 which means the model will pass 10 values everytime and an epoch of 500.<jupyter_code>#training model

history= power_model = model.fit(x_train, y_train, batch_size=10, epochs=1000, verbose=1, validation_split=0.2)<jupyter_output>Epoch 1/1000
32/32 [==============================] - 0s 4ms/step - loss: 242.0819 - val_loss: 243.7709
Epoch 2/1000
32/32 [==============================] - 0s 2ms/step - loss: 242.0652 - val_loss: 243.7523
Epoch 3/1000
32/32 [==============================] - 0s 2ms/step - loss: 242.0866 - val_loss: 243.6749
Epoch 4/1000
32/32 [==============================] - 0s 2ms/step - loss: 242.0461 - val_loss: 243.7570
Epoch 5/1000
32/32 [==============================] - 0s 3ms/step - loss: 242.1178 - val_loss: 243.7638
Epoch 6/1000
32/32 [==============================] - 0s 2ms/step - loss: 242.0587 - val_loss: 243.7550
Epoch 7/1000
32/32 [==============================] - 0s 2ms/step - loss: 242.0300 - val_loss: 243.6990
Epoch 8/1000
32/32 [==============================] - 0s 3ms/step - loss: 242.0920 - val_loss: 243.7663
Epoch 9/1000
32/32 [==============================] - 0s 3ms/step - loss: 242.0689 - val_loss: 243.6947
Epoch 10/1000
32/32 [==============================] - 0s 3ms/st[...]<jupyter_text>Now we plot the result. And can see that the result changes as the model learn everytime.<jupyter_code>#plotting results

#resizing
plt.rcParams['figure.figsize']=([15, 10])

plt.plot(poly['x'], poly['y'], label='actual')
plt.plot(poly['x'], model.predict(poly['x']), label='prediction')

plt.legend()
#checking History object to see record of the loss values 
#of the loss values and metric values during training 

'''Source:https://www.tensorflow.org/guide/keras/train_and_evaluate '''

history.history
# Making predictions with this model:


prediction=model.predict([4.0,0.234])

print('This is the amount of power generated for the wind speed you have provided:')
print(prediction)<jupyter_output>This is the amount of power generated for the wind speed(s) you have provided:
[[4.3476424]
 [4.3300843]]<jupyter_text>Again, to evaluate this model. I use the method shown in this tutorial, which is the root mean squared error method. This method consist of  he square root of the mean of the square of all of the error.<jupyter_code>#Evaluating the performance of a regression model on test set using 
#root mean squared error method

from sklearn.metrics import mean_squared_error
from math import sqrt

pred_train = model.predict(x_train)
print(np.sqrt(mean_squared_error(y_train,pred_train)))

pred = model.predict(x_test)
print(np.sqrt(mean_squared_error(y_test,pred)))<jupyter_output>15.996012304201834
17.039363794649415<jupyter_text>Conclusion about this modelLooking at the shape that the prediction is creating it seems that this prediction model is more accurate.Cosidering the evaluation method we have performed in the markdown cell above using root mean squared error, we can see that both the train and the set model are performing good as they both give similar values. In cases where for example, the train model is performing better than the test model we talk about 'overfitting'. 'Overfitting' means that the model has an excessively complex structure and learns both the existing relations among data and noise. On the other hand 'Underfitting'is often the consequence of model being unable to encapsulate the relations among data.[source](https://realpython.com/train-test-split-python-data/underfitting-and-overfitting).   Creating *Model* using SklearnAfter my approaches made in previous sections, I have decided to use one last approach with module 'Sklearn'. Using Keras and Tensorflow to cerate ad trai models using this dataset have been good practice since I have never used this before and for sure I will keep learning about it. I would like now to use this model for linear regression using [this tutorial]() which I think it might be easier and more accurate to make predictions as my model to be included on the second  part of this project which is the web service which will be encapsulated into my Dockerfile ( review this text after completion). I will provide details of the process in markdown and code cells below:<jupyter_code>#First we import the modules we are going to use
from sklearn.linear_model import LinearRegression #import linear regression model
from sklearn.model_selection import train_test_split #to divide data
#loading data in this section for better understanding

Powerdataset=pd.read_csv('Powerproduction dataset.csv', delimiter=',')


x=Powerdataset['speed']
y=Powerdataset['power']

#plotting dataset to have this included on this section: 

Powerdataset.plot(kind='scatter', x= 'speed', y='power')

plt.show()


#Let's create our linear regression model
#doing test train split

X_train, X_test, y_train, y_test= train_test_split(Powerdataset.speed, Powerdataset.power)

#Let's see how this split looks

plt.scatter(X_train, y_train, label='Training Data', color='g', alpha=.7)
plt.scatter(X_test, y_test, label='Test Data', color= 'r', alpha=.7)
plt.legend()

plt.title('Test train split')

plt.show()<jupyter_output><empty_output><jupyter_text>Once we have our data split so one part is ised for training and the other part is used for testing, as we can see in the visualization above this cell, we move onto the Model creation section.<jupyter_code>#Model creation
#naming model LR as Linear Regression as in the tutorial for 
#easier understanding

LR=LinearRegression()
LR.fit(X_train.values.reshape(-1,1), y_train.values)#Adding x_train and y_train values and reshaping X_train values as they need to be in
#a 1d shape for this to work<jupyter_output><empty_output><jupyter_text>The next step is to use the model to predict in out test data. See below:<jupyter_code># Predicting

prediction=LR.predict(X_test.values.reshape(-1,1))

#Plotting X_test against prediction results in same plot
plt.plot(X_test, prediction, label='Linear Regression', color='r')
plt.scatter(X_test, y_test, label='Actual Test Data', color='b', alpha=.7)
plt.legend()

plt.show()<jupyter_output><empty_output><jupyter_text>As seen in previous sections where we found the **R squared value** for Simple Linear regression model used on this dataset is  0.531347729791333, which is a low to moderate fit. Let's try this model to predict the power output based on the wind speed values we enter as input on our model.<jupyter_code>#Making predictions for specific values
#using command predict we enter a sample wind speed: 

print('This is the power generated considering your input: ')
LR.predict(np.array([[25.00]]))[0]<jupyter_output>This is the power generated considering your input:<jupyter_text>We can see that the power generated result we received is the same as the power result we can see in our plot above. The model seems to be accurate enough.Finally we will 'Sklearn' score function to evaluate this model's accuracy.<jupyter_code>#score function

LR.score(X_test.values.reshape(-1,1),y_test.values)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09809225512528474, 'fraction_numerical': 0.028568716780561883, 'mean_word_length': 5.581199250468457, 'pattern_counts': {'":': 0, '<': 63, '<?xml version=': 0, '>': 63, 'https://': 19, 'lorem ipsum': 0, 'www.': 5, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '316116', 'license': 'MIT', 'n_tokens_mistral': 6554, 'n_tokens_neox': 5949, 'n_words': 2454, 'path': '.ipynb_checkpoints/Machine Learning & Statistics-Project-checkpoint.ipynb', 'repo_name': 'Ainara12/Machine-Learning-Statistics-Project2020', 'size': '21072'}