<jupyter_start><jupyter_text>Supervised Learning - Linear Regression Example Project In this section we will be going over linear regression. We'll be going over how to use the scikit-learn regression model, as well as how to train the regressor using the fit() method, and how to predict new labels using the predict() method. We'll be analysing a data set consisting of house prices in Boston. We'll start off with a single variable linear regression using numpy and then move on to using scikit learn. We'll do an overview of the mathematics behind the method we're using, but mostly we'll dive deeper into pratical "hands-on" coding lessons.If you're interested in the deeper mathematics of linear regession methods, check out the wikipedia page and also check out <NAME>'s wonderful lectures for free on youtube.In this section we will be working through linear regression with the following steps:Step 1: Getting and setting up the data.Step 2: Visualizing current data.Step 3: The mathematics behind the Least Squares Method.Step 4: Using Numpy for a Univariate Linear Regression.Step 5: Getting the error.Step 6: Using scikit learn to implement a multivariate regression.Step 7: Using Training and Validation.Step 8: Predicting PricesStep 9 : Residual Plots<jupyter_code>import numpy as np
import pandas as pd
from pandas import Series,DataFrame
import matplotlib.pyplot as plt
import seaborn as sns

sns.set_style('whitegrid')
%matplotlib inline
from sklearn.datasets import load_boston
boston = load_boston()
print(boston.DESCR)
plt.hist(boston.target,bins=50)

plt.xlabel('Prices in $1000s')
plt.ylabel('Number of houses')
plt.scatter(boston.data[:,5],boston.target)

plt.ylabel('Price in $1000s')
plt.xlabel('Number of rooms')
boston_df = DataFrame(boston.data)

boston_df.columns = boston.feature_names

boston_df.head()
boston_df['Price'] = boston.target
boston_df.head()
sns.lmplot('RM','Price',data=boston_df)<jupyter_output>C:\Users\mrloc\anaconda3\lib\site-packages\seaborn\_decorators.py:43: FutureWarning: Pass the following variables as keyword args: x, y. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  FutureWarning<jupyter_text>The mathematics behind the Least Squares Method In this particular lecture we'll use the least squares method as the way to estimate the coefficients. Here's a quick breakdown of how this method works mathematically:Take a quick look at the plot we created above using seaborn. Now consider each point, and know that they each have a coordinate in the form (X,Y). Now draw an imaginary line between each point and our current "best-fit" line. We'll call the distanace between each point and our current best-fit line, D. To get a quick image of what we're currently trying to visualize, take a look at the picture below:<jupyter_code># Quick display of image form wikipedia
from IPython.display import Image
url = 'https://upload.wikimedia.org/wikipedia/commons/thumb/b/b0/Linear_least_squares_example2.svg/220px-Linear_least_squares_example2.svg.png'
Image(url)
X = boston_df.RM
X.shape
X
X = np.vstack([boston_df.RM,np.ones(len(boston_df.RM))]).T
X.shape
Y = boston_df.Price
X
# Create the X array in the form [X 1]
X = np.array( [ [value,1] for value in X ], dtype=object )
m,b = np.linalg.lstsq(X,Y,rcond=None)[0]
print(m,b)
plt.plot(boston_df.RM,boston_df.Price,'o')
x = boston_df.RM
plt.plot(x, m*x + b,'r',label="Best Fit Line")<jupyter_output><empty_output><jupyter_text>Getting The Error Great! We've just completed a single variable regression using the least squares method with Python! Let's see if we can find the error in our fitted line. Checking out the documentation here, we see that the resulting array has the total squared error. For each element, it checks the the difference between the line and the true value (our original D value), squares it, and returns the sum of all these. This was the summed D^2 value we discussed earlier.It's probably easier to understand the root mean squared error, which is similar to the standard deviation. In this case, to find the root mean square error we divide by the number of elements and then take the square root. There is also an issue of bias and an unbiased regression, but we'll delve into those topics later.For now let's see how we can get the root mean squared error of the line we just fitted.<jupyter_code>result = np.linalg.lstsq(X,Y,rcond=None)

error_total = result[1]

rmse = np.sqrt(error_total/len(X))

print('The root mean square error was %.2f' %rmse)<jupyter_output>The root mean square error was 6.60<jupyter_text>Using scikit learn to implement a multivariate regression Now, we'll keep moving along with using scikit learn to do a multi variable regression. This will be a similar apporach to the above example, but sci kit learn will be able to take into account more than just a single data variable effecting the target!We'll start by importing the linear regression library from the sklearn module.The sklearn.linear_model.LinearRegression class is an estimator. Estimators predict a value based on the observed data. In scikit-learn, all estimators implement the fit() and predict() methods. The former method is used to learn the parameters of a model, and the latter method is used to predict the value of a response variable for an explanatory variable using the learned parameters. It is easy to experiment with different models using scikit-learn because all estimators implement the fit and predict methods.<jupyter_code>import sklearn
from sklearn.linear_model import LinearRegression
lreg = LinearRegression()
X_multi = boston_df.drop('Price',1)

Y_target = boston_df.Price
lreg.fit(X_multi,Y_target)
print('The estimated intercept coefficient is %.2f ' % lreg.intercept_)

print('The number of coefficients used was %d ' %len(lreg.coef_))
coeff_df = DataFrame(boston_df.columns)
coeff_df.columns = ['Features']

coeff_df['Coefficient Estimate'] = Series(lreg.coef_)
coeff_df<jupyter_output><empty_output><jupyter_text>Using Training and Validation In a dataset a training set is implemented to build up a model, while a validation set is used to validate the model built. Data points in the training set are excluded from the validation set. The correct way to pick out samples from your dataset to be part either the training or validation (also called test) set is randomly.Fortunately, scikit learn has a built in function specifically for this called train_test_split.The parameters passed are your X and Y, then optionally test_size parameter, representing the proportion of the dataset to include in the test split. As well a train_size parameter. ou can learn more about these parameters here<jupyter_code>from sklearn.model_selection import train_test_split

X_train,X_test,Y_train,Y_test = train_test_split(X,boston_df.Price)
print(X_train.shape, X_test.shape, Y_train.shape, Y_test.shape)
lreg = LinearRegression()

lreg.fit(X_train,Y_train)
pred_train = lreg.predict(X_train)
pred_test = lreg.predict(X_test)
print("Fit a model X_train and calculate the MSE with Y_train: %.2f " % np.mean(Y_train-pred_train)**2)
print("Fit a model X_train and calculate the MSE with X_test and Y_test: %.2f " % np.mean(Y_test - pred_test)**2)
train = plt.scatter(pred_train,(pred_train - Y_train),c='b',alpha=0.5)

test = plt.scatter(pred_test,(pred_test-Y_test),c='r',alpha=0.5)

plt.hlines(y=0,xmin=-10,xmax=10)

plt.legend((train,test),('Training','Test'),loc='lower left')

plt.title('Residual Plots')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07425417269023525, 'fraction_numerical': 0.007359705611775529, 'mean_word_length': 5.515410958904109, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 20, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '272259', 'license': 'MIT', 'n_tokens_mistral': 2315, 'n_tokens_neox': 2188, 'n_words': 953, 'path': 'linear-regression/Supervised Learning - Linear Regression Example Project.ipynb', 'repo_name': 'locspoc/data_analysis_and_visualisation', 'size': '7612'}