<jupyter_start><jupyter_text><jupyter_code>import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

from sklearn import preprocessing
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures
from sklearn.preprocessing import FunctionTransformer
from sklearn.metrics import r2_score


from sklearn.pipeline import make_pipeline,Pipeline
from sklearn.model_selection import cross_val_score

import sys
import os
import seaborn as sns<jupyter_output><empty_output><jupyter_text>Linear regression 1. Load the provided .npy files. You can load it with numpy.* each file contains one vector, X and y* visualize X vs y on a scatter plot* fit an  y=w_0 + w_1⋅X + w_2⋅X^2  linear regression using `sklearn`<jupyter_code>x = np.load("X.npy")
y = np.load("y.npy")

plt.figure(figsize=(8,8))
plt.title("Not So Random Points", fontsize=22)
plt.scatter(x,y,color="blue", marker="o")
plt.xlabel("X axis", fontsize=20)
plt.ylabel("Y axis", fontsize=20)
plt.grid()
plt.show()
#i had to look around and then puff
#https://scikit-learn.org/0.23/auto_examples/model_selection/plot_underfitting_overfitting.html

deg = 2
polyreg = Pipeline([("polynomial_features", PolynomialFeatures(degree=2, include_bias=False)),
                         ("linear_regression", LinearRegression())])

polyreg.fit(x[:, np.newaxis],y)
w1,w2 = polyreg[1].coef_
w0 = polyreg[1].intercept_

scores = cross_val_score(polyreg, x[:, np.newaxis], y,
                             scoring="neg_mean_squared_error", cv=10)
print(scores)

x_test = np.linspace(-100,100,1000)

#now let's utilize these for a plot

fig,ax = plt.subplots(1,1,figsize=(14,10))

ax.set_title("The fitted params are: {0:.3f} + {1:.3f} $\cdot$ x + {2:.3f} $\cdot$ x$^2$".format(w0,w1,w2),fontsize=23)
ax.scatter(x,y,color="blue", marker="o",alpha=0.3, label="Raw")
ax.plot(x_test, polyreg.predict(x_test[:,np.newaxis]),color="red",linestyle="--",linewidth=5, label="Prediction")
ax.set_xlabel("X axis", fontsize=20)
ax.set_ylabel("Y axis", fontsize=20)
ax.legend(loc="best",fontsize=22)
ax.tick_params(axis="both",labelsize=16)
ax.grid()

plt.show()<jupyter_output>[-5590.64124118 -4125.62531932 -4553.07748319 -5242.30507949
 -4689.51458145 -5083.32366064 -4425.73546224 -5252.05763162
 -4630.2140137  -4793.10525627]<jupyter_text>`NOTE`: After looking at the data, it was very obvious thata high degree polynom could fit all points, but that would make any sense, as it will be overfitting. A second degree polynom is the right guess here!  2. Using different features* plot the residuals (the difference between the prediction and the actual  y ) vs the original  y * a non-random-noise like pattern suggests non-linear connection between the features and the predictions* someone told us that the connection between X and y is  y=A⋅X+B⋅cos^3(X)+C⋅X^2+D   * using sklearn's linear regression estimate  A,B,C,D !* plot the residuals again! is it better now?<jupyter_code>differences = y - polyreg.predict(x[:, np.newaxis])  #get differences
remainer = np.sqrt(np.mean(differences**2))          #some good measurement for catching the better fit

fig,ax = plt.subplots(2,1,figsize=(14,14))
ax[0].set_title("RMS: {0:.3f}".format(remainer),fontsize=22)
ax[0].plot(x,differences," ro",markersize=2)
ax[0].set_xlabel("X axis",fontsize=20)
ax[0].set_ylabel("Differences",fontsize=20)
ax[0].tick_params(axis="both",labelsize=14)
ax[0].grid()

ax[1].set_title("RMS: {0:.3f}".format(remainer),fontsize=22)
ax[1].plot(x,differences," ro",markersize=2)
ax[1].set_xlabel("X axis",fontsize=20)
ax[1].set_ylabel("Differences",fontsize=20)
ax[1].tick_params(axis="both",labelsize=14)
ax[1].set_xlim(-100,-75)
ax[1].grid()

plt.show()<jupyter_output><empty_output><jupyter_text>`TRICK`: see a trend globally and locally too. IF something is visibly noticable, focus on it!It is very visible that something is going on... something systematic... Wait a second!<jupyter_code>def poly2(X):
    """
    Returns a transformed array
    A * X**2 + B * COS(X)**3 + C*X + D 
    """
    return np.hstack((np.cos(X)**3,X,X**2))
#https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.FunctionTransformer.html
polyreg2 = Pipeline([("custom polynom", FunctionTransformer(poly2)),
                         ("linear_regression", LinearRegression())])

polyreg2.fit(x[:,np.newaxis],y)
c,b,a = polyreg2[1].coef_
d = polyreg2[1].intercept_

x_test = np.linspace(-100,100,2000)

#now let's utilize these for a plot

fig,ax = plt.subplots(1,1,figsize=(14,10))

ax.set_title("The fitted params are: $({0:.3f}\cdot x^2) + ({1:.3f}\cdot x) + ({2:.3f}\cdot \cos^3(x)) + ({3:.3f})$".format(a,b,c,d),fontsize=23)
ax.scatter(x,y,color="blue", marker="o",alpha=0.3, label="Raw")
ax.plot(x_test, polyreg2.predict(x_test[:,np.newaxis]),color="red",linestyle="--",linewidth=5, label="Prediction")
ax.set_xlabel("X axis", fontsize=20)
ax.set_ylabel("Y axis", fontsize=20)
ax.legend(loc="best",fontsize=22)
ax.tick_params(axis="both",labelsize=16)
ax.grid()

fig.tight_layout()

plt.show()
differences = y - polyreg2.predict(x[:, np.newaxis])  #get differences
remainer = np.sqrt(np.mean(differences**2))          #some good measurement for catching the better fit

fig,ax = plt.subplots(2,1,figsize=(14,14))
ax[0].set_title("RMS: {0:.3f}".format(remainer),fontsize=22)
ax[0].plot(x,differences," ro",markersize=2)
ax[0].set_xlabel("X axis",fontsize=20)
ax[0].set_ylabel("Differences",fontsize=20)
ax[0].tick_params(axis="both",labelsize=14)
ax[0].grid()

ax[1].set_title("RMS: {0:.3f}".format(remainer),fontsize=22)
ax[1].plot(x,differences," ro",markersize=2)
ax[1].set_xlabel("X axis",fontsize=20)
ax[1].set_ylabel("Differences",fontsize=20)
ax[1].tick_params(axis="both",labelsize=14)
ax[1].set_xlim(-100,-75)
ax[1].grid()

fig.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>Now its much closer to a noise like thing, rather then a systematic error...:o  3. Other methdods than sklearn for linear regression* using the statsmodels package perform the same linear regression as in 2.) (hint: use statsmodels.api.OLS)* is the result the same? if not guess, why? (did you not forget to add the constant term?)* try to get the same results with statsmodels as with sklearn!* using the analytic solution formula shown during the lecture, calculate the coefficients (A, B, C, D). are they the same compared to the two previous methods?<jupyter_code>#https://www.statsmodels.org/stable/index.html

import statsmodels.api as sm
import statsmodels.formula.api as smf
def poly1(x,a,b,c):
    return a*x**2 + b*x + c
def poly2v2(x,a,b,c,d):
    return a*x**2 + b*x + c*np.cos(x)**3 + d
#this is some interesting stuff... seems easier then sklearn

X = PolynomialFeatures(degree=deg, include_bias=False).fit_transform(x[:,np.newaxis])
X = sm.add_constant(X)
model = sm.OLS(y,X)
results = model.fit()
display(results.summary())
c,b,a = results.params

#plot the results
fig,ax = plt.subplots(1,1,figsize=(14,10))

ax.set_title("The fitted params are: {0:.3f} + {1:.3f} $\cdot$ x + {2:.3f} $\cdot$ x$^2$".format(c,b,a),fontsize=23)
ax.scatter(x,y,color="blue", marker="o",alpha=0.3, label="Raw")
ax.plot(x_test, poly1(x_test,a,b,c),color="red",linestyle="--",linewidth=5, label="Prediction")
ax.set_xlabel("X axis", fontsize=20)
ax.set_ylabel("Y axis", fontsize=20)
ax.legend(loc="best",fontsize=22)
ax.tick_params(axis="both",labelsize=16)
ax.grid()

fig.tight_layout()
plt.show()

differences = y - poly1(x,a,b,c)  #get differences
remainer = np.sqrt(np.mean(differences**2))          #some good measurement for catching the better fit

fig,ax = plt.subplots(2,1,figsize=(14,14))
ax[0].set_title("RMS: {0:.3f}".format(remainer),fontsize=22)
ax[0].plot(x,differences," ro",markersize=2)
ax[0].set_xlabel("X axis",fontsize=20)
ax[0].set_ylabel("Differences",fontsize=20)
ax[0].tick_params(axis="both",labelsize=14)
ax[0].grid()

ax[1].set_title("RMS: {0:.3f}".format(remainer),fontsize=22)
ax[1].plot(x,differences," ro",markersize=2)
ax[1].set_xlabel("X axis",fontsize=20)
ax[1].set_ylabel("Differences",fontsize=20)
ax[1].tick_params(axis="both",labelsize=14)
ax[1].set_xlim(-100,-75)
ax[1].grid()

fig.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>So it seems it performs on the same level. Onto the next polynom!<jupyter_code>X = FunctionTransformer(poly2).fit_transform(x[:,np.newaxis])
X = sm.add_constant(X)
model = sm.OLS(y,X)
results = model.fit()
display(results.summary())
d,c,b,a = results.params

fig,ax = plt.subplots(1,1,figsize=(14,10))

ax.set_title("The fitted params are: $({0:.3f}\cdot x^2) + ({1:.3f}\cdot x) + ({2:.3f}\cdot \cos^3(x)) + ({3:.3f})$".format(a,b,c,d),
             fontsize=23)
ax.scatter(x,y,color="blue", marker="o",alpha=0.3, label="Raw")
ax.plot(x_test, poly2v2(x_test,a,b,c,d),color="red",linestyle="--",linewidth=5, label="Prediction")
ax.set_xlabel("X axis", fontsize=20)
ax.set_ylabel("Y axis", fontsize=20)
ax.legend(loc="best",fontsize=22)
ax.tick_params(axis="both",labelsize=16)
ax.grid()

fig.tight_layout()
plt.show()

differences = y - poly2v2(x,a,b,c,d)  #get differences
remainer = np.sqrt(np.mean(differences**2))          #some good measurement for catching the better fit

fig,ax = plt.subplots(2,1,figsize=(14,14))
ax[0].set_title("RMS: {0:.3f}".format(remainer),fontsize=22)
ax[0].plot(x,differences," ro",markersize=2)
ax[0].set_xlabel("X axis",fontsize=20)
ax[0].set_ylabel("Differences",fontsize=20)
ax[0].tick_params(axis="both",labelsize=14)
ax[0].grid()

ax[1].set_title("RMS: {0:.3f}".format(remainer),fontsize=22)
ax[1].plot(x,differences," ro",markersize=2)
ax[1].set_xlabel("X axis",fontsize=20)
ax[1].set_ylabel("Differences",fontsize=20)
ax[1].tick_params(axis="both",labelsize=14)
ax[1].set_xlim(-100,-75)
ax[1].grid()

fig.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>Performs really well, almost the same!<jupyter_code>X_a = np.array((x**0,x,np.cos(x)**3,x**2)).T                                   #prepare for analytic solution
analytic_coefs = np.dot(np.linalg.inv(np.dot(X_a.T,X_a)),np.dot(X_a.T,y))    #get the coefs by multiplying
display("{0:.3f},{1:.3f},{2:.3f},{3:.3f}".format(*analytic_coefs))           #see them!<jupyter_output><empty_output><jupyter_text>4.* load the [real_estate](https://gist.github.com/qbeer/f356d7144543cbb09c9792c34b8ad722) data to a pandas dataframedrop the ID column and the geographic location columnsfit a linear regression model to predict the unit price using sklearn At this point, the stastmodel api is very good, very easy and very tempting to be utilized... But anyway, the exercise asks for sklearn, going to do sklearn. We have to do a prediction. We did regression before, which meant fitting on data to get coefficients. Now my guess is that we could do the same, but I feel like, there has to be a splitting involved, so that's what I am gonna do:1. Do a splitting2. Train data used for coeffs3. Compare prediction with originals<jupyter_code>from sklearn.model_selection import train_test_split
def line_reg(x,a,b):
    return a*x + b
#load it in

real_estate_data = pd.read_csv("real_estate.csv",delimiter=",")  #no NaN values, weird change
real_estate_data = real_estate_data.drop(columns=["No","X5 latitude", "X6 longitude"])
display(real_estate_data)

#split
X_train, X_test, Y_train, Y_test = train_test_split(real_estate_data[real_estate_data.columns[:-1]],
                                                    real_estate_data[real_estate_data.columns[-1]],
                                                    train_size=0.8)

#see it
#display(X_train.head)
#display(X_test.head)
#display(Y_train.head)
#display(Y_test.head)

X_train = X_train.values
X_test = X_test.values
Y_train = Y_train.values
Y_test = Y_test.values

#get coeffs
lin_reg = LinearRegression().fit(X_train,Y_train) 
Y_pred = lin_reg.predict(X_test)

display("{0:.3f},{1:.3f},{2:.3f},{3:.3f}".format(*lin_reg.coef_))

x_lin = np.linspace(0,80,100)

fig,ax = plt.subplots(1,1,figsize=(8,8))

ax.set_title("Prediction with spliting \n Accuracy: {0:.3f}%".format(r2_score(Y_test,Y_pred)*100),fontsize=24)
ax.scatter(Y_test,Y_pred,color="red",alpha=0.7)
ax.plot(x_lin, line_reg(x_lin,1,0), "--")
ax.set_xlabel("Y_test",fontsize=22)
ax.set_ylabel("Y_pred",fontsize=22)
ax.grid()

fig.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>5.* interpret the coefficients and their meaning shortly with your own words* plot the residuals for the predictions. if you had to decide only on this information, which house would you buy?--- My interpretation of the coefficients (with the help of he graphs): 1. The transation age does not impact thw price of the house, but a newly transacted house tends to go for a higher price (val:5.794)2. The age of the house is inversely impacts the value of the house, while the graph hows some parabola like behavior, this more info should be gathered regarding this parameter (val:-0.257)3. Distance from the closes MRT station greatly impacts the value and it is inversely proportional to its price. While it has the an exponantial look, a mid ranged house should be choosen (val:-0.006)4. The amount of nearby convinience stores greatly increases the value.This analysis means that 3. and 4. parameters should be the main parameters to choose a house. Others are less important. This way of choosing a house still not a guarantee to buy a good one, but the indicators could be used for a good pricing and great value (thus cheapest).<jupyter_code>fig,axi = plt.subplots(2,2,figsize=(12,12))

for i in range(2):
    for j in range(2):
        axi[i,j].scatter(X_train[:,i*2+j],Y_train)
        """
        DO NOT EVEN!
        x_line = np.linspace(min(X_train[:,i*2+j]),max(X_train[:,i*2+j]),100)
        axi[i,j].plot(x_line, line_reg(x_line,lin_reg.coef_[i*2+j],0))
        """
        axi[i,j].grid()
axi[0,0].set_title("X1 transaction date",fontsize=22)
axi[0,1].set_title("X2 house age",fontsize=22)
axi[1,0].set_title("X3 distance to the nearest MRT station",fontsize=22)
axi[1,1].set_title("X4 number of convenience stores",fontsize=22)

fig.tight_layout()
plt.show()

X_train[i,:].shape<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15999436143219622, 'fraction_numerical': 0.044403721454750494, 'mean_word_length': 5.6427902621722845, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 36, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '310586', 'license': 'MIT', 'n_tokens_mistral': 5606, 'n_tokens_neox': 5261, 'n_words': 1177, 'path': 'hw4/HW_04.ipynb', 'repo_name': 'AdamGTaylor/DataMining_2021', 'size': '14188'}