<jupyter_start><jupyter_text>Polynomial RegressionWhat if your data is actually more complex than a simple straight line? Surprisingly,you can actually use a linear model to fit nonlinear data. A simple way to do this is toadd powers of each feature as new features, then train a linear model on this extendedset of features. This technique is called Polynomial Regression.<jupyter_code>import numpy as np
import pandas as pd 
import seaborn as sns
import matplotlib.pyplot as plt

m = 100
X = 6 * np.random.rand(m, 1) - 3
Y = 0.5 * X**2 + X + 2 + np.random.randn(m, 1)

X=X.reshape(X.shape[0], )
Y=Y.reshape(X.shape[0], )

X=pd.Series(X)
Y=pd.Series(Y)
sns.scatterplot(x=X,y=Y)<jupyter_output><empty_output><jupyter_text>Clearly, a straight line will never fit this data properly. So let’s use Scikit-Learn’s PolynomialFeatures class to transform our training data, adding the square (2nd-degreepolynomial) of each feature in the training set as new features (in this case there isjust one feature):<jupyter_code>from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
LinearRegression
poly_features = PolynomialFeatures(degree=2, include_bias=False)

X=pd.DataFrame(X)

X_poly = poly_features.fit_transform(X)

X_new=np.linspace(-3, 3, 100).reshape(100, 1)
lin_reg = LinearRegression()
lin_reg.fit(X_poly, Y)
lin_reg.intercept_, lin_reg.coef_
X_new_poly = poly_features.transform(X_new)
y_new = lin_reg.predict(X_new_poly)
plt.plot(X.to_numpy(), Y, "b.")
plt.plot(X_new, y_new, "r-", linewidth=2, label="Predictions")
plt.xlabel("$x_1$", fontsize=18)
plt.ylabel("$y$", rotation=0, fontsize=18)
plt.legend(loc="upper left", fontsize=14)
plt.axis([-3, 3, 0, 10])
plt.show()<jupyter_output><empty_output><jupyter_text>Not bad: the model estimates y = 0 . 56x1 2 + 0 . 93x1 + 1 . 78 when in fact the originalfunction was y = 0 . 5x1 2 + 1 . 0x1 + 2 . 0 + Gaussian noise.Note that when there are multiple features, Polynomial Regression is capable of findingrelationships between features (which is something a plain Linear Regressionmodel cannot do). This is made possible by the fact that PolynomialFeatures alsoadds all combinations of features up to the given degree.  Learning Curves If a model performs well on the training data but generalizes poorlyaccording to the cross-validation metrics, then your model is overfitting. If it performspoorly on both, then it is underfitting. This is one way to tell when a model istoo simple or too complex.Another way is to look at the learning curves: these are plots of the model’s performanceon the training set and the validation set as a function of the training set size.To generate the plots, simply train the model several times on different sized subsetsof the training set.LC for linear regression model is as follows: ![image.png](attachment:image.png) First, let’s look at the performance on the trainingdata: when there are just one or two instances in the training set, the model can fitthem perfectly, which is why the curve starts at zero. But as new instances are addedto the training set, it becomes impossible for the model to fit the training data perfectly,both because the data is noisy and because it is not linear at all. So the error onthe training data goes up until it reaches a plateau, at which point adding new instancesto the training set doesn’t make the average error much better or worse. Now let’slook at the performance of the model on the validation data. When the model istrained on very few training instances, it is incapable of generalizing properly, whichis why the validation error is initially quite big. Then as the model is shown moretraining examples, it learns and thus the validation error slowly goes down. However,once again a straight line cannot do a good job modeling the data, so the error endsup at a plateau, very close to the other curve.These learning curves are typical of an underfitting model. Both curves have reacheda plateau; they are close and fairly high. Polynomial 10 degree curve The error on the training data is much lower than with the Linear Regressionmodel.There is a gap between the curves. This means that the model performs significantlybetter on the training data than on the validation data, which is the hallmarkof an overfitting model. However, if you used a much larger training set,the two curves would continue to get closer.![image.png](attachment:image.png) ![image.png](attachment:image.png) ![image-2.png](attachment:image-2.png) Regularized Linear Models Ridge RegressionRidge Regression (also called Tikhonov regularization) is a regularized version of LinearRegression: a regularization term is added to the cost function.This forces the learning algorithm to not only fit the data but also keep the modelweights as small as possible. ![image.png](attachment:image.png) Note:It is quite common for the cost function used during training to bedifferent from the performance measure used for testing. Apartfrom regularization, another reason why they might be different isthat a good training cost function should have optimizationfriendlyderivatives, while the performance measure used for testingshould be as close as possible to the final objective. A goodexample of this is a classifier trained using a cost function such asthe log loss (discussed in a moment) but evaluated using precision/recall.The hyperparameter α controls how much you want to regularize the model. If α = 0then Ridge Regression is just Linear Regression. If α is very large, then all weights endup very close to zero and the result is a flat line going through the data’s mean.It is important to scale the data (e.g., using a StandardScaler)before performing Ridge Regression, as it is sensitive to the sca Lasso RegressionLeast Absolute Shrinkage and Selection Operator Regression (simply called LassoRegression) is another regularized version of Linear Regression: just like RidgeRegression, it adds a regularization term to the cost function, but it uses the ℓ1 normof the weight vector instead of half the square of the ℓ2 normAn important characteristic of Lasso Regression is that it tends to completely eliminatethe weights of the least important features (i.e., set them to zero). For example,the dashed line in the right plot on Figure 4-18 (with α = 10-7) looks quadratic, almostlinear: all the weights for the high-degree polynomial features are equal to zero. Inother words, Lasso Regression automatically performs feature selection and outputs asparse model Elastic NetElastic Net is a middle ground between Ridge Regression and Lasso Regression. Theregularization term is a simple mix of both Ridge and Lasso’s regularization terms,and you can control the mix ratio r. When r = 0, Elastic Net is equivalent to RidgeRegression, and when r = 1, it is equivalent to Lasso RegressionSo when should you use Linear Regression, Ridge, Lasso, or Elastic Net?It is almost always preferable to have at least a little bit of regularization, so generally you shouldavoid plain Linear Regression. Ridge is a good default, but if you suspect that only afew features are actually useful, you should prefer Lasso or Elastic Net since they tendto reduce the useless features’ weights down to zero as we have discussed. In general,Elastic Net is preferred over Lasso since Lasso may behave erratically when the numberof features is greater than the number of training instances or when several featuresare strongly correlated Early StoppingA very different way to regularize iterative learning algorithms such as GradientDescent is to stop training as soon as the validation error reaches a minimum. This iscalled early stopping.With Stochastic and Mini-batch Gradient Descent, the curves arenot so smooth, and it may be hard to know whether you havereached the minimum or not. One solution is to stop only after thevalidation error has been above the minimum for some time (whenyou are confident that the model will not do any better), then rollback the model parameters to the point where the validation errorwas at a minimum. Logistic RegressionLogistic Regression is commonly used to estimate the probability that an instance belongs to a particular class Estimating ProbabilitiesSo how does it work? Just like a Linear Regression model, a Logistic Regressionmodel computes a weighted sum of the input features (plus a bias term), but insteadof outputting the result directly like the Linear Regression model does, it outputs thelogistic of this resultThe logistic—also called the logit, noted σ(·)—is a sigmoid function (i.e., S-shaped)that outputs a number between 0 and 1![image.png](attachment:image.png) Training and Cost FunctionGood, now you know how a Logistic Regression model estimates probabilities andmakes predictions. But how is it trained? The objective of training is to set the parametervector θ so that the model estimates high probabilities for positive instances (y =1) and low probabilities for negative instances (y = 0). This idea is captured by thecost function shown in Equation 4-16 for a single training instance x.![image.png](attachment:image.png) This cost function makes sense because – log(t) grows very large when t approaches0, so the cost will be large if the model estimates a probability close to 0 for a positive instance, and it will also be very large if the model estimates a probability close to 1for a negative instance. On the other hand, – log(t) is close to 0 when t is close to 1, sothe cost will be close to 0 if the estimated probability is close to 0 for a negativeinstance or close to 1 for a positive instance, which is precisely what we want.The cost function over the whole training set is simply the average cost over all traininginstances. It can be written in a single expression (as you can verify easily), calledthe log loss.![image.png](attachment:image.png) Once you have the gradient vector containing allthe partial derivatives you can use it in the Batch Gradient Descent algorithm. That’sit: you now know how to train a Logistic Regression model. For Stochastic GD youwould of course just take one instance at a time, and for Mini-batch GD you woulduse a mini-batch at a time. Decision BoundariesLet’s use the iris dataset to illustrate Logistic Regression. This is a famous dataset thatcontains the sepal and petal length and width of 150 iris flowers of three differentspecies: Iris-Setosa, Iris-Versicolor, and Iris-Virginica<jupyter_code>from sklearn import datasets
iris = datasets.load_iris()
df=pd.DataFrame(iris['data'],columns=iris['feature_names'])
df.describe()
y =(iris["target"] == 2).astype(np.int)
y=y.reshape(y.shape[0],)
from sklearn.linear_model import LogisticRegression

X=df.copy()
log_reg = LogisticRegression()
log_reg.fit(X, y)
X_new = [[np.linspace(5, 10, 1000)],[np.linspace(2, 5, 1000)],
         [np.linspace(1, 8, 1000)],[np.linspace(0.1, 3, 1000)]]

X_df=[]
for i in range(1000):
    X_df.append([X_new[0][0][i],X_new[1][0][i],X_new[2][0][i],X_new[3][0][i]])
X_df=pd.DataFrame(X_df)
y_proba = log_reg.predict_proba(X_df)
plt.plot(X_df[3], y_proba[:, 1], "g-", label="Iris-Virginica")
plt.plot(X_df[3], y_proba[:, 0], "b--", label="Not Iris-Virginica")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.05822131220510995, 'fraction_numerical': 0.012107184189441823, 'mean_word_length': 5.2272727272727275, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '530533', 'license': 'MIT', 'n_tokens_mistral': 3104, 'n_tokens_neox': 2914, 'n_words': 1583, 'path': 'Hands on ML with Scikit Learn and Tenserflow/Chapter 4 Training Models/4.2 Polynomial, Regularized and Logistic Regression.ipynb', 'repo_name': 'insomniac-klutz/Data-Science-Referentica', 'size': '11233'}