<jupyter_start><jupyter_text>1. What is Simple Linear Regression*** Linear Regression Linear Regression is a statistical technique which is used to find the linear relationship between dependent and one or more independent variables. Simple Linear Regression model employs one independent and one dependent variable. It is the simplest form of Linear Regression where we fit a straight line to the data. Independent variable is the variable whose value does not change by the effect of other variables. It is denoted as X.Dependent variable is the one whose value changes when there is a change in the value of the independent variable. It is denoted by Y.Linear regression can be used in various fields, for example, in Machine Learning Algorithms, like, predicting a country's economic growth or GDP, prediction of a price change in future on a specific product, estimates used in sports, for example, a certain team or person's performance, score, etc.Regression equationIn its simplest form, the formula for the Simple Linear Regression is:y = m*x + c, wherey - dependent variablex - independent variablem - slope of the linec - intercept of the line.In simple terms, intercept  c is the point where the regression line crosses the y axis and the slope m means that for every 1 change of x there is times change of y (for example, if m = 3 that means that for every 1 change of x there is 3 times as much change of y).See drawing below.![regr.PNG](attachment:regr.PNG) In the drawing above, the the intercept is positive, but it can be negative as well, crossing the y line below zero.The bigger the slope value is, the steeper the graph line is directed towards the top. On the other hand, the smaller the value, the more it is going to be directed towards the ground. It can be zero as well, then the regression line will be parallel to the x axis, or even negative. Then the regression line will point downwards and will cross the x axis at some point.A slope of zero implies there is NO LINEAR RELATIONSHIP between x and y and that x in its linear form is of no use in explaining y.  Reference - Simple Linear Regression Testing for Significance   - https://www.youtube.com/watch?v=GOkM7gBqrGA Values m and c can be readily computed using particular formulae, that require to have values of x and y (and to compute their square roots subsequently) and the total number n of valuables. I believe and hope that showing these formulae is outside of scope of our project.As the line will not go through all the data points on the chart, Simple Linear Regression equation should also take into account the residual errors, that are calculated by comparing and analysing the distance between predicted y values (brown dots on the chart below) and the actual ones (blue dots). "The best fit line" should then be adjusted to have the least sum of squares of these errors (ordinary least squares method).![error.PNG](attachment:error.PNG)***  2. Simple Linear Regression plotting in Python  2.1 DataFrame scatter plot*** Now we will produce a scatter plot for the powerproduction dataframe, we will use the file that we have downloaded previously from the Moodle page and saved as a csv.file in the same folder where our jupyter Notebook file is kept.Reference: https://www.techwalla.com/articles/how-to-save-a-text-file-as-csvWe need to import necessary libraries first of all<jupyter_code>import numpy as np # provides operating functions and fast numerical array structure
import pandas as pd # provides tools for data storage, manipulation and analysis tasks
import matplotlib.pyplot as plt # provides tools for making plots
# create a DataFrame by importing our csv file 
df = pd.read_csv("powerproduction.csv")
df<jupyter_output><empty_output><jupyter_text>We can see that the DataFrame has 500 rows and 2 columns, so purely technically it is suitable for Simple Linear Regression, where we can use first column (speed) as the independent variable x and then use the 2nd column (power) as the dependent variable y and then analysing the character of correlation between the two.The way the dataframe is shown in Jupyter notebook by Pandas now is that we can only see its first 5 rows and last 5 rows.In these rows all the POWER values equal to zero, irrespective of SPEED values. Looking at and analysing these rows alone, there, obviously, does not seem to be any correlation between the two values, that is POWER values do not seem to DEPEND on SPEED values.Let us call out some other rows to have a look at their values.<jupyter_code># calling rows 200 to 250 (iloc function - calling by indices)  
# (Reference - Pandas plotting tutorial w/ Tatplotlib and Pandas   https://www.youtube.com/watch?v=0P7QnIQDBJY)
df.iloc[200:251]<jupyter_output><empty_output><jupyter_text>Now we can see some kind of correlation - the POWER values generally seem to increase when SPEED value increases. General common sense from our life experience tells us that that would be somehow expected.That kind of "increase - increase" correlation would give us positive slope, that is, when Simple Linear Regression is plotted, the graph line should go up from left to right. Now we will try and plot all the DataFrame.<jupyter_code># We have already imported all the necessary libraries.
# Let us only add matplotlib "magic function" be able to call out the plot without plt.show() function
%matplotlib inline
plt.title("Correlation of wind speed to power output",fontsize=18)   # Title of the plot
plt.xlabel("Wind speed",fontsize=17)   # independent variable - Wind speed
plt.ylabel("Power output",fontsize=17)  # dependent variable - Power output
plt.scatter(df.speed,df.power, color="green");   # scatter plotting.   ";" in the end to cancel PathCollection line<jupyter_output><empty_output><jupyter_text>Now we can have a good look at the plot above. We can see that except for few stray values it has a very well defined pattern, we can definitely see a positive linear relationship between x and y (particularly, in the middle section of the plot), that is, it looks like with the increase of wind speed there can be predicted certain amount of increase of power output. It does not seem to apply to the first part (looking at the dataset - first 5 values) of the plot and the last part of the plot (last 10 values). In these parts there is no linear relationship between x and y. It looks like if the wind speed is less than about 0.276 or more than 2.4, it is unable to generate any power output.Overall, still, as mentioned, there is quite a solid pattern, so we  can apply Simple Linear Regression model to it and then analyse it. Reference: Regression output explained  - https://www.youtube.com/watch?v=VvlqA-iO2HA  2.2 Simple Linear Regression Modelling***<jupyter_code># Importing linear_model from scikit-learn library
from sklearn import linear_model<jupyter_output><empty_output><jupyter_text>First we create Linear Regression object.We will draw sklearn.linear_model.Linear_Regression(). It applies Ordinary Least Squares algorithm in its model, (calculates minimum sum of squares between the actual targets and the predicted by the model ones).Reference: https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html?highlight=linear%20regressionsklearn.linear_model.LinearRegression<jupyter_code>reg = linear_model.LinearRegression()<jupyter_output><empty_output><jupyter_text>Now we fit out data, that is we train the Regression model using available datapoints Reference: https://scikit-learn.org/stable/auto_examples/linear_model/plot_ols.htmlsphx-glr-auto-examples-linear-model-plot-ols-py<jupyter_code>reg.fit(df[["speed"]],df.power)<jupyter_output><empty_output><jupyter_text>Now we are going to predict our Simple Linear Regresion line<jupyter_code># We draw our scatter plot again
plt.title("Correlation of wind speed to power output",fontsize=18)   # Title of the plot
plt.xlabel("Wind speed",fontsize=17)   # independent variable - Wind speed
plt.ylabel("Power output",fontsize=17)  # dependent variable - Power output
plt.scatter(df.speed,df.power, color="green");   # scatter plotting.   ";" in the end to cancel PathCollection line

# Model has processed the data and trained itself, now we use predict() function to draw a Simple Linear Regression Line.
plt.plot(df.speed,reg.predict(df[["speed"]]),color="red",linewidth=4);

# Now let us check the accuracy of prediction
reg.score(df[["speed"]],df.power)
# It is around 72.9%<jupyter_output><empty_output><jupyter_text>3. Dataset values validity evaluation  When we look at the full POWERPRODUCTION dataset, there is one thing that really stands out. It is the fact that some POWER values have zero value even with decent wind speed. I am talking about the ones from the middle of the dataset, not the end ones, where the wind is too strong for power output.So, to me, these middle zero values look  like some kind of fault and definitely deserve considering to be excluded from the dataset altogether. Let us try that scenario and create a new dataset without these values in it.<jupyter_code># import libraries again
import numpy as np 
import pandas as pd 
import matplotlib.pyplot as plt 
%matplotlib inline
# create a DataFrame by importing our csv file 
df = pd.read_csv("powerproduction.csv")
df<jupyter_output><empty_output><jupyter_text>It can be done in steps. Step 1: create part of the dataset that will have POWER values of more than zero and SPEED value more than 0.275<jupyter_code>new_df_part2 = df.loc[(df["power"] > 0) & (df["speed"] > 0.275)]

# Reference: Complete Python Pandas Data Science Tutorial https://www.youtube.com/watch?v=vmEHCJofslg
new_df_part2<jupyter_output><empty_output><jupyter_text>Step 2: draw a dataset made of the first 5 rows from our original dataset - the ones that have zero POWER value<jupyter_code>new_df_part1 = df[0:5]
new_df_part1<jupyter_output><empty_output><jupyter_text>Step 3: draw a dataset made of the last 10 rows of the original dataset (zero values)<jupyter_code>new_df_part3 = df.tail(10)
new_df_part3<jupyter_output><empty_output><jupyter_text>Step 4: concatenate 3 new datasets into a new one.<jupyter_code>new_df = pd.concat([new_df_part1,new_df_part2,new_df_part3])
new_df<jupyter_output><empty_output><jupyter_text>Let us also save that dataset as separate csv file<jupyter_code># save as csv file
new_df.to_csv("powerproduction2.csv")<jupyter_output><empty_output><jupyter_text>Now we have a dataset that does not include somewhat erroneous zero values.We can plot it the same way we did it with full df dataset.<jupyter_code>plt.title("Correlation of wind speed to power output, zero values in the middle not incl.",fontsize=15)   # Title of the plot
plt.xlabel("Wind speed",fontsize=14)   # independent variable - Wind speed
plt.ylabel("Power output",fontsize=14)  # dependent variable - Power output
plt.scatter(new_df.speed,new_df.power, color="grey");   # scatter plotting<jupyter_output><empty_output><jupyter_text>Now we can create a Simple Lenear Regression prediction as before.<jupyter_code># Importing linear_model from scikit-learn library
from sklearn import linear_model
reg = linear_model.LinearRegression()
reg.fit(new_df[["speed"]],new_df.power)
plt.title("Correlation of wind speed to power output, zero values in the middle not incl.",fontsize=15)   # Title of the plot
plt.xlabel("Wind speed",fontsize=14)   # independent variable - Wind speed
plt.ylabel("Power output",fontsize=14)  # dependent variable - Power output
plt.scatter(new_df.speed,new_df.power, color="grey"); 
plt.plot(new_df.speed,reg.predict(new_df[["speed"]]),color="red",linewidth=4);<jupyter_output><empty_output><jupyter_text>4. Analyzing the prediction  4.1 Coefficient of the line (slope). One of the parameters that will give us certain idea of correlation between our x and y values is the coefficent of the line (or slope). it is letter m in our formula y = m*x + c, mentioned earlier.<jupyter_code># to draw coefficient of the line
reg.coef_<jupyter_output><empty_output><jupyter_text>As we can see, our slope is 4.96915322. That means that for every 1 unit of change on axis x there is about 5.56 change on unit y.  4.2 Intercept Letter c in the formula y = m*x + c<jupyter_code># to draw intercept
reg.intercept_<jupyter_output><empty_output><jupyter_text>Intercept has negative value, that is the prediction line crosses y axis below zero point. Now we can predict the power output at certain wind turbine speed, using our formula. For example, we have a weather forecast and we know that the wind speed at a particular day is going to be of 10.0 value. We can predict the power output.<jupyter_code># y = m*x + c
y = 4.96915322*10.0 - 13.66943465384908
y<jupyter_output><empty_output><jupyter_text>The model gives us predicted output of around 36.0221 units.  4.3 Coefficient of determination (r squared). R squared or coefficient of determination is one of the most important parameters that allows us to analyze how precise our regression model is.It can be measured using Sum of Squares formula.SS = SUM(y - y(mean))squaredThe formula measures the sum of squares of the difference between each y variable of the dataset and the mean of variable y. Then we can see percentage variation in y which can be explained by all the x variables together. Higher the better. It is always between 0 and 1. It can never be negative – since it is a squared value.In simple terms, R square value is  the number of correct predictions divided by the total number of predictions of our model.References - Regression output explained  - https://www.youtube.com/watch?v=VvlqA-iO2HAhttps://stackoverflow.com/questions/38325068/regr-score-and-r2-score-give-different-values<jupyter_code># There is a function .score() in sklearn package that draws R squared value. 
# Let us see how high or low it is for our Regression model.
reg.score(new_df[["speed"]],new_df.power)<jupyter_output><empty_output><jupyter_text>So, the R squared value is about 0.735248. It means that 73.5% of variation of y variables (sum of squares value) is explained by our x variable. It is actually quite a high number, so we can say to a certain degree of confidence that our regression model is quite a reliable one. The dataset we analyzed is new_df - the one that does not include middle-set zero power output values. Let us compare that figure to the original dataset df, that has 500 values.<jupyter_code>reg.score(df[["speed"]],df.power)<jupyter_output><empty_output><jupyter_text>The figure is slghtly lower, around 72.8%. we can see that our prediction model will be around  0.7% more precise if we eliminate middle-set zero values.  4.4 Pearson's correlation coefficient r. Another parameter that will allow us to estimate our prediction model is Pearson's correlation coefficient r.Like coefficient of determination, it shows the degree of relationship between variables x and y. It can go between -1 and 1.  1 indicates that the two variables are moving perfectly in unison. They rise and fall together and have perfect correlation. The relationship will exhibit positive linear relationship. -1 means that the two variables are in perfect opposites. One goes up and other goes down, in perfect negative way. Any two variables can be argued to have a correlation value. If they are not correlated then the correlation value can still be measured and would be 0. r is simply a square root of R square.So, once we know R squared now, r can be easily computed.Reference: Difference between correlation coefficient and coefficient of determination   https://blog.uwgb.edu/bansalg/statistics-data-analytics/linear-regression/what-is-the-difference-between-coefficient-of-determination-and-coefficient-of-correlation/:~:text=Coefficient%20of%20correlation%20is%20%E2%80%9CR,table%20in%20the%20Regression%20output.&text=In%20other%20words%20Coefficient%20of,all%20the%20x%20variables%20together. Original 500 values dataset dfr = √R = √ 0.7284082420220264 = 0.85346836 Again, we see that the value (around 0.85) is quite high, it is very good, very close to 1. That means that the regression model shows very solid POSITIVE (as the value is between 0 and 1) linear relationship between variables x and y. Dataset new_df that does not include middle-set zero valuesr = √R = √ 0.7352481534105535 = 0.857466 Dataset new_df has even higher r value, it is even closer to 1 (1 is indication of perfect positive linear relationship).  5. Eliminating zero y values in the beginning and at the end of the dataset. We have already mentioned that looking at the scatter plot we can see that it does seem to be representing solid positive linear relationship in the middle part of it, but there is no any relationship whatsoever in the beginning (first 5 data points) and at the end (last 10 datapoints).  In our opinion, it is worth considering to eliminate these zero values from the dataset and to create and analyze a new scatter plot without these values. There is obviously no power generated if the wind speed is below at least 0.276 units and above 24.498 units so no any prediction model needs to be applied to these values.I think that by eliminating these "unnecessary" datapoints we can create even more reliable prediction model. Let us do that and then analyze and compare it with the previous one.  We load up new data set - eliminating all the "power" == 0 values (the ones at the beginning and at the end of the dataset as well as the ones in the middle which we considered "erroneous", see. section 3).Reference: Deleting DataFrame row in Pandas based on column value -   https://stackoverflow.com/questions/18172851/deleting-dataframe-row-in-pandas-based-on-column-value<jupyter_code>df_no_any_zeros = df[df.power != 0]
df_no_any_zeros
# Let us also save that dataset as separate csv file
df_no_any_zeros.to_csv("powerproduction3.csv")<jupyter_output><empty_output><jupyter_text>Now we can repeat all the steps again to do the scatter plot<jupyter_code>plt.title("Correlation of wind speed to power output, no any zero power values incl.",fontsize=16)   # Title of the plot
plt.xlabel("Wind speed",fontsize=15)   # independent variable - Wind speed
plt.ylabel("Power output",fontsize=15)  # dependent variable - Power output
plt.scatter(df_no_any_zeros.speed,df_no_any_zeros.power, color="blue");   # scatter plotting<jupyter_output><empty_output><jupyter_text>Analysis: The scatter plot looks the same as the previous one, eliminating the middle-set zero power values did not seem to have changed its pattern.  6. Analyzing separate segments of the dataset Let us break the dataset into separate segments and analyze them separately.<jupyter_code># First 100 datapoints
df100 = df.head(100)
df100
plt.title("Correlation of wind speed to power output, first 100 data points",fontsize=16)   # Title of the plot
plt.xlabel("Wind speed 0 to 5.280",fontsize=15)   # independent variable - Wind speed
plt.ylabel("Power output",fontsize=15)  # dependent variable - Power output
plt.scatter(df100.speed,df100.power, color="black");   # scatter plotting<jupyter_output><empty_output><jupyter_text>First 100 data points analysis - the scatter plot does not seem to have any linear relationship between x and y values, it is not suitable for Simple Linear regression prediction model, its correlation coefficient r will be close to zero.<jupyter_code># Data points 101 to 200
df101to200 = df.iloc[100:200]
df101to200
plt.title("Correlation of wind speed to power output, data points 101 to 200",fontsize=16)   # Title of the plot
plt.xlabel("Wind speed 5.305 to 10.185",fontsize=15)   # independent variable - Wind speed
plt.ylabel("Power output",fontsize=15)  # dependent variable - Power output
plt.scatter(df101to200.speed,df101to200.power, color="black");   # scatter plotting<jupyter_output><empty_output><jupyter_text>We can see that this segment shows much stronger positive linear relationship! Let us plot the Simple Linear Regression line and analyze it!<jupyter_code># Importing linear_model from scikit-learn library
from sklearn import linear_model
reg = linear_model.LinearRegression()
reg.fit(df101to200[["speed"]],df101to200.power)
plt.title("Correlation of wind speed to power output, data points 101 to 200",fontsize=16)   # Title of the plot
plt.xlabel("Wind speed 5.305 10.185",fontsize=15)   # independent variable - Wind speed
plt.ylabel("Power output",fontsize=15)  # dependent variable - Power output
plt.scatter(df101to200.speed,df101to200.power, color="black"); 
plt.plot(df101to200.speed,reg.predict(df101to200[["speed"]]),color="red",linewidth=4);
# Coefficient of determination R square
reg.score(df101to200[["speed"]],df101to200.power)<jupyter_output><empty_output><jupyter_text>Analysis: the segment has the coefficient of determination 0.5930 (59.8%), which is quite a good figure.<jupyter_code># Data points 201 to 300
df201to300 = df.iloc[200:300]
df201to300
plt.title("Correlation of wind speed to power output, data points 201 to 300",fontsize=16)   # Title of the plot
plt.xlabel("Wind speed 10.210 to 15.265",fontsize=15)   # independent variable - Wind speed
plt.ylabel("Power output",fontsize=15)  # dependent variable - Power output
plt.scatter(df201to300.speed,df201to300.power, color="black");   # scatter plotting<jupyter_output><empty_output><jupyter_text>Again, we can see very strong positive linear relationship.Let us see the Simple Linear Regression line.<jupyter_code># Importing linear_model from scikit-learn library
from sklearn import linear_model
reg = linear_model.LinearRegression()
reg.fit(df201to300[["speed"]],df201to300.power)
plt.title("Correlation of wind speed to power output, data points 201 to 300",fontsize=16)   # Title of the plot
plt.xlabel("Wind speed 10.210 to 15.265",fontsize=15)   # independent variable - Wind speed
plt.ylabel("Power output",fontsize=15)  # dependent variable - Power output
plt.scatter(df201to300.speed,df201to300.power, color="black");   # scatter plotting
plt.plot(df201to300.speed,reg.predict(df201to300[["speed"]]),color="red",linewidth=4);
# Coefficient of determination R square
reg.score(df201to300[["speed"]],df201to300.power)<jupyter_output><empty_output><jupyter_text>We can see very very strong figure of R square = 0.930495!<jupyter_code># Data points 301 to 400
df301to400 = df.iloc[300:400]
df301to400 
plt.title("Correlation of wind speed to power output, data points 301 to 400",fontsize=16)   # Title of the plot
plt.xlabel("Wind speed 15.315 to 19.895",fontsize=15)   # independent variable - Wind speed
plt.ylabel("Power output",fontsize=15)  # dependent variable - Power output
plt.scatter(df301to400.speed,df301to400.power, color="black");   # scatter plotting
# Importing linear_model from scikit-learn library
from sklearn import linear_model
reg = linear_model.LinearRegression()
reg.fit(df301to400[["speed"]],df301to400.power)
plt.title("Correlation of wind speed to power output, data points 301 to 400",fontsize=16)   # Title of the plot
plt.xlabel("Wind speed 15.315 to 19.895",fontsize=15)   # independent variable - Wind speed
plt.ylabel("Power output",fontsize=15)  # dependent variable - Power output
plt.scatter(df301to400.speed,df301to400.power, color="black");   # scatter plotting
plt.plot(df301to400.speed,reg.predict(df301to400[["speed"]]),color="red",linewidth=4);
# Coefficient of determination R square
reg.score(df301to400[["speed"]],df301to400.power)<jupyter_output><empty_output><jupyter_text>R square is 0.18536 The R square value is very low for that type of nicely patterned scatter plot. Let us explore further and see if it is due to the 1 stray zero value. Let us find it.<jupyter_code># Draw first 50 rows and we can see the zero value at index 340.
df301to400.head(50)
# draw new dataset without zero value.
df301to400_updated = df301to400[df301to400.power != 0]
# Draw first 50 rows again to double-check that zero value gone.
# It is  - index 340 gone.
df301to400_updated.head(50)<jupyter_output><empty_output><jupyter_text>Create Simple Linear Regression line again.<jupyter_code># Importing linear_model from scikit-learn library
from sklearn import linear_model
reg = linear_model.LinearRegression()
reg.fit(df301to400_updated[["speed"]],df301to400_updated.power)
plt.title("Correlation of wind speed to power output, data points 301 to 400",fontsize=16)   # Title of the plot
plt.xlabel("Wind speed 15.315 to 19.895",fontsize=15)   # independent variable - Wind speed
plt.ylabel("Power output",fontsize=15)  # dependent variable - Power output
plt.scatter(df301to400_updated.speed,df301to400_updated.power, color="black");   # scatter plotting
plt.plot(df301to400_updated.speed,reg.predict(df301to400_updated[["speed"]]),color="red",linewidth=4);<jupyter_output><empty_output><jupyter_text>Scatter plot looks different, but it is only because of the difference in scale (we have eliminated the distant value so the scale is larger now, appears to be more scattering).<jupyter_code># Coefficient of determination R square
reg.score(df301to400_updated[["speed"]],df301to400_updated.power)<jupyter_output><empty_output><jupyter_text>Analysis: now the R squared value jumped from 0.1761 to 0.49669! Let us have a look at the last segment, we will exclude zero power output values from it<jupyter_code># Draw all data points 401 to 500 first
df401to500 = df.iloc[400:]
# Then draw new dataset without zero values.
df401to500_updated = df401to500[df401to500.power != 0]
df401to500_updated
plt.title("Correlation of wind speed to power output, data points 401 to 500",fontsize=16)   # Title of the plot
plt.xlabel("Wind speed 19.895 to 25",fontsize=15)   # independent variable - Wind speed
plt.ylabel("Power output",fontsize=15)  # dependent variable - Power output
plt.scatter(df401to500_updated.speed,df401to500_updated.power, color="black");   # scatter plotting
plt.title("Correlation of wind speed to power output, data points 401 to 500",fontsize=16)   # Title of the plot
plt.xlabel("Wind speed 19.895 to 25",fontsize=15)   # independent variable - Wind speed
plt.ylabel("Power output",fontsize=15)  # dependent variable - Power output
plt.scatter(df401to500_updated.speed,df401to500_updated.power, color="black");   # scatter plotting
plt.plot(df401to500_updated.speed,reg.predict(df401to500_updated[["speed"]]),color="red",linewidth=4);<jupyter_output><empty_output><jupyter_text>We can see that this part of the segment does not seem to exhibit any linear relationship between x and y variables and is not suitable for creating a reliable simple linear regression prediction model.  Overall analysis of the separate segments of the simple regression line. Breaking the regression line into separate segments showed that there is quite a difference in strength of linear relationship in different parts of the plot  - from R square value as high as 0.93 at 201 to 300 data points segment to the one that does not exhibit any linear relationship at all in 401 to 500 data points segment.   7. Train and test split method The train-test split procedure is used to estimate the performance of machine learning algorithms when they are used to make predictions on data not used to train the model. The model is split in 2 subsets and being trained on one subset only, the other one tests the model for prediction efficacy then. The whole idea of it is that the model is to be tested on the data that it did not see before, but not the data that it has already used for creating its prediction model. The results of this procedure allow to compare the performance of machine learning algorithms for our predictive modeling problem. The procedure is simple enough to use and interpret through sklearn, however there are times when the procedure should not be used, such as when you have a small dataset and situations where additional configuration is required, such as when it is used for classification and the dataset is not balanced.In my opinion our dataset will be suitable for the procedure. Let us perform it now.Reference:  https://medium.com/@contactsunny/how-to-split-your-dataset-to-train-and-test-datasets-using-scikit-learn-e7cf6eb5e0dTrain/ test split in sklearn  -  https://www.youtube.com/watch?v=lSwvUmZCvco We import all ther libraries as before<jupyter_code>import numpy as np # provides operating functions and fast numerical array structure
import pandas as pd # provides tools for data storage, manipulation and analysis tasks
import matplotlib.pyplot as plt # provides tools for making plots
import seaborn as sns
%matplotlib inline
dftest = pd.read_csv("powerproduction.csv")<jupyter_output><empty_output><jupyter_text>Draw the dataset as before<jupyter_code>dftest<jupyter_output><empty_output><jupyter_text>Draw the scatterplot<jupyter_code>plt.title("Correlation of wind speed to power output",fontsize=18)   # Title of the plot
plt.xlabel("Wind speed",fontsize=17)   # independent variable - Wind speed
plt.ylabel("Power output",fontsize=17)  # dependent variable - Power output
plt.scatter(dftest.speed,dftest.power, color="green");
# use reshape function to create two separate arrays - X and y for independent and dependent variables
X = dftest["speed"]
X = np.array(X).reshape(-1,1)
y = dftest["power"]
y = np.array(y).reshape(-1,1)
# we are going to use train_test_split method
from sklearn.model_selection import train_test_split<jupyter_output><empty_output><jupyter_text>The method will split both X and y arrays into train and test sections.It takes few parameters:    - test_size will specify the ratio of train and test arrays. If none specified, default is 25% test and 75% train. We shall specify test as 20%.    Random state=10 parameter would fix the random data. We shall leave it out for our example.<jupyter_code># we call the train_test_split method to call out 4 parameters, train to test ration is 80/20
X_train, X_test, y_train, y_test = train_test_split(X,y, test_size=0.2) # if add random state=10 - random data fixed
#checking length of train subset now - and it is 80% (400 out of total 500 values)
len(X_train)
# checking length of test subset- and it is 20% (100 out of 500 values)
len(X_test)
# checking content - and it is randomly chosen data points. We did not choose random data=10 parameter, so every time we hit RUN button, new set of data is randomly generated.
X_train
# We use our Linear regression model - importing Linear regression class
from sklearn.linear_model import LinearRegression

# and create classifier
clf = LinearRegression()
# we use fit method to train our model
clf.fit(X_train, y_train)
# Model is trained now. We can call predict method on our actual test dataset
clf.predict(X_test)
# we can have a look at our y test
y_test
# checking accuracy of the model by calling score method now. It will predict X_test values, 
# then compare them against y_test values and calculate the accuracy.
clf.score(X_test,y_test)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07126682194616978, 'fraction_numerical': 0.034873188405797104, 'mean_word_length': 5.3528565556925605, 'pattern_counts': {'":': 0, '<': 145, '<?xml version=': 0, '>': 147, 'https://': 13, 'lorem ipsum': 0, 'www.': 7, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '36', 'dir': 'jupyter-structured-clean-dedup', 'id': '601491', 'license': 'MIT', 'n_tokens_mistral': 9497, 'n_tokens_neox': 8507, 'n_words': 3854, 'path': 'Powerproduction.ipynb', 'repo_name': 'dmitrijsg123/Fundamentals_of_Data_Analysis_Project', 'size': '30912'}