<jupyter_start><jupyter_text>Import Packages<jupyter_code>import os
import sys
import numpy as np
import pandas as pd

%matplotlib inline
import matplotlib as mpl
import matplotlib.pyplot as plt
import seaborn as sns

np.random.seed(42)<jupyter_output><empty_output><jupyter_text>Linear Regression using Normal EquationTo find the value of ${\theta}$ that minimizes the cost function, there is a closed-form solution—in other words, a mathematical equation that gives the result directly. This is calledthe Normal Equation$$\hat{\theta} = (X^{T}X)^{-1}X^{T}Y$$*  $\hat{\theta}$ is the value of ${\theta}$ that minimizes the cost function.*  $Y$ is the vector of target values containing $y^{(1)}$ to $y^{(m)}$.Note that,$$Y = X\theta + \epsilon,\quad N(0, \sigma^2)$$$$\hat{Y} = h_\theta(X) = X\hat{\theta}$$$$J(\theta) = {1\over m} \sum_{}^m (h_\theta(X)-Y)^2$$$$\hat{\theta} = argmax_\theta J(\theta)$$<jupyter_code>X = 2 * np.random.rand(100,1)
y = 4 + 3*X + np.random.randn(100, 1)
plt.plot(X, y, "b.")
plt.xlabel("$x_1$", fontsize=15)
plt.ylabel("$y$", fontsize=15)
plt.axis([0,2,0,15])
plt.show()
X_b = np.c_[np.ones((100,1)), X]
theta_hat = np.linalg.inv(X_b.T.dot(X_b)).dot(X_b.T).dot(y)
theta_hat
X_new = np.array([[0],[2]])
X_new_b = np.c_[np.ones((2,1)), X_new]
y_pred = X_new_b.dot(theta_hat)
plt.plot(X, y, "b.")
plt.plot(X_new, y_pred, "r-")
plt.xlabel("$x_1$", fontsize=15)
plt.ylabel("$y$", fontsize=15)
plt.axis([0,2,0,15])
plt.show()<jupyter_output><empty_output><jupyter_text>using Scikit-learn<jupyter_code>from sklearn.linear_model import LinearRegression

lin_reg = LinearRegression()
lin_reg.fit(X, y)
lin_reg.intercept_, lin_reg.coef_
theta_hat<jupyter_output><empty_output><jupyter_text>Computational ComplexityThe Normal Equation **computes the inverse of $X^{T}X$, which is an (n + 1) × (n + 1)matrix** (where n is the number of features). The computational complexity of invertingsuch a matrix is typically about $O(n^{2.4})$ to $O(n^{3})$ (depending on the implementation).In other words, if you double the number of features, you multiply the computationtime by roughly $2^{2.4} = 5.3$ to $2^{3} = 8$.  Batch Gradient DescentTo implement Gradient Descent, you need to compute the gradient of the cost functionwith regards to each model parameter θj. In other words, you need to calculatehow much the cost function will change if you change θj just a little bit. This is calleda partial derivative. It is like asking “what is the slope of the mountain under my feetif I face east?” and then asking the same question facing north (and so on for all otherdimensions, if you can imagine a universe with more than three dimensions). Equationbelow computes the partial derivative of the cost function with regards to parameterθj, noted ${\partial \over \partial \theta} J(\theta)$Notice that this formula involves calculations over the full trainingset X, at each Gradient Descent step! This is why the algorithm iscalled Batch Gradient Descent: it uses the whole batch of trainingdata at every step (actually, Full Gradient Descent would probablybe a better name). As a result it is terribly slow on very large trainingsets (but we will see much faster Gradient Descent algorithmsshortly). However, Gradient Descent scales well with the number offeatures; training a Linear Regression model when there are hundredsof thousands of features is much faster using GradientDescent than using the Normal Equation or SVD decomposition. $$J(\theta) = {1\over m} \sum_{}^m (h_\theta(X)-Y)^2$$$${\partial \over \partial \theta} J(\theta) = {2\over m} X^{T}(X\theta - Y)$$<jupyter_code>eta = 0.1 # learning rate
n_iterations = 1000
m = 100 # number of instances

theta = np.random.randn(2,1) # random initialization

for iteration in range(n_iterations):
    grad = 2/m * X_b.T.dot(X_b.dot(theta) - y)
    theta = theta -eta * grad
theta
theta_hat
theta_path_bgd = []

def plot_gradient_descent(theta, eta, theta_path=None):
    m = len(X_b)
    plt.plot(X, y, "b.")
    n_iterations = 1000
    for iteration in range(n_iterations):
        if iteration < 10:
            y_predict = X_new_b.dot(theta)
            style = "b-" if iteration > 0 else "r--"
            plt.plot(X_new, y_predict, style)
        gradients = 2/m * X_b.T.dot(X_b.dot(theta) - y)
        theta = theta - eta * gradients
        if theta_path is not None:
            theta_path.append(theta)
    plt.xlabel("$x_1$", fontsize=15)
    plt.axis([0, 2, 0, 15])
    plt.title(r"$\eta = {}$".format(eta), fontsize=16)
np.random.seed(42)
theta = np.random.randn(2,1)  # random initialization

plt.figure(figsize=(10,4))
plt.subplot(131); plot_gradient_descent(theta, eta=0.02)
plt.ylabel("$y$", rotation=0, fontsize=15)
plt.subplot(132); plot_gradient_descent(theta, eta=0.1, theta_path=theta_path_bgd)
plt.subplot(133); plot_gradient_descent(theta, eta=0.5)

plt.show()<jupyter_output><empty_output><jupyter_text>Scaling for Gradient Descent In fact, the cost function has the shape of a bowl, but it can be an elongated bowl ifthe features have very different scales. Figure 4-7 shows Gradient Descent on a trainingset where features 1 and 2 have the same scale (on the left), and on a training setwhere feature 1 has much smaller values than feature 2 (on the right).As you can see, on the left the Gradient Descent algorithm goes straight toward theminimum, thereby reaching it quickly, whereas on the right it first goes in a directionalmost orthogonal to the direction of the global minimum, and it ends with a longmarch down an almost flat valley. It will eventually reach the minimum, but it willtake a long time.![image.png](attachment:image.png)When using Gradient Descent, you should ensure that **all featureshave a similar scale** (e.g., using Scikit-Learn’s StandardScalerclass), or else it will take much longer to converge.  Stochastic Gradient Descent The main problem with Batch Gradient Descent is the fact that it uses the wholetraining set to compute the gradients at every step, which makes it very slow whenthe training set is large. At the opposite extreme, Stochastic Gradient Descent justpicks a random instance in the training set at every step and computes the gradientsbased only on that single instance. Obviously this makes the algorithm much fastersince it has very little data to manipulate at every iteration. **It also makes it possible totrain on huge training sets, since only one instance needs to be in memory at eachiteration (SGD can be implemented as an out-of-core algorithm)**On the other hand, due to its stochastic (i.e., random) nature, this algorithm is muchless regular than Batch Gradient Descent: instead of gently decreasing until it reachesthe minimum, **the cost function will bounce up and down, decreasing only on average.**Over time it will end up very close to the minimum, but once it gets there it willcontinue to bounce around, never settling down (see Figure 4-9). So once the algorithmstops, the final parameter values are good, but not optimal.![image.png](attachment:image.png)When the cost function is very irregular, this can actually help thealgorithm jump out of local minima, so Stochastic Gradient Descent has a betterchance of finding the global minimum than Batch Gradient Descent does.<jupyter_code>theta_path_sgd = []
m = len(X_b)
np.random.seed(42)
n_epochs = 50
t0, t1 = 5, 50  # learning schedule hyperparameters

def learning_schedule(t):
    return t0 / (t + t1)

theta = np.random.randn(2,1)  # random initialization

for epoch in range(n_epochs):
    for i in range(m):
        if epoch == 0 and i < 20:                    # not shown in the book
            y_predict = X_new_b.dot(theta)           # not shown
            style = "b-" if i > 0 else "r--"         # not shown
            plt.plot(X_new, y_predict, style)        # not shown
        random_index = np.random.randint(m)
        xi = X_b[random_index:random_index+1]
        yi = y[random_index:random_index+1]
        gradients = 2 * xi.T.dot(xi.dot(theta) - yi)
        eta = learning_schedule(epoch * m + i)
        theta = theta - eta * gradients
        theta_path_sgd.append(theta)                 # not shown

plt.plot(X, y, "b.")
plt.xlabel("$x_1$", fontsize=18)                     # not shown
plt.ylabel("$y$", rotation=0, fontsize=18)           # not shown
plt.axis([0, 2, 0, 15])                              # not shown           
plt.show()
theta
theta_hat
y.shape
y.ravel().shape
from sklearn.linear_model import SGDRegressor

sgd_reg = SGDRegressor(max_iter=100, tol=1e-3, penalty=None, eta0=0.1, random_state=42)
sgd_reg.fit(X, y.ravel())
sgd_reg.intercept_, sgd_reg.coef_
theta_hat<jupyter_output><empty_output><jupyter_text>When using Stochastic Gradient Descent, the training instancesmust be independent and identically distributed (IID), to ensurethat the parameters get pulled towards the global optimum, onaverage. A simple way to ensure this is **to shuffle the instances duringtraining (e.g., pick each instance randomly, or shuffle the trainingset at the beginning of each epoch).** If you do not do this, forexample if the instances are sorted by label, then SGD will start byoptimizing for one label, then the next, and so on, and it will notsettle close to the global minimum.  Mini-batch Gradient DescentThe last Gradient Descent algorithm we will look at is called Mini-batch GradientDescent. It is quite simple to understand once you know Batch and Stochastic GradientDescent: at each step, instead of computing the gradients based on the full trainingset (as in Batch GD) or based on just one instance (as in Stochastic GD), MinibatchGD computes the gradients on small random sets of instances called minibatches.**The main advantage of Mini-batch GD over Stochastic GD is that you canget a performance boost from hardware optimization of matrix operations, especiallywhen using GPUs.**The algorithm’s progress in parameter space is less erratic than with SGD, especiallywith fairly large mini-batches. As a result, Mini-batch GD will end up walkingaround a bit closer to the minimum than SGD. But, on the other hand, it may beharder for it to escape from local minima (in the case of problems that suffer fromlocal minima, unlike Linear Regression as we saw earlier). Figure below shows thepaths taken by the three Gradient Descent algorithms in parameter space duringtraining. They all end up near the minimum, but Batch GD’s path actually stops at theminimum, while both Stochastic GD and Mini-batch GD continue to walk around.However, don’t forget that Batch GD takes a lot of time to take each step, and StochasticGD and Mini-batch GD would also reach the minimum if you used a good learningschedule.<jupyter_code>theta_path_mgd = []
m = len(X_b)

n_iterations = 100
batch_size = 20

theta = np.random.randn(2,1)

t0, t1 = 200, 1000
def learning_schedule(t): return t0 / (t+t1)

t = 0
for epoch in range(n_iterations):
    shuffled_indices = np.random.permutation(m)
    X_b_shuffled= X_b[shuffled_indices]
    y_shuffled = y[shuffled_indices]
    
    for i in range(0, m, batch_size):
        t += 1
        xi = X_b_shuffled[i:i+batch_size]
        yi = y_shuffled[i:i+batch_size]
        
        grad = 2/batch_size * xi.T.dot(xi.dot(theta) - yi)
        eta = learning_schedule(t)
        theta = theta - eta * grad
        theta_path_mgd.append(theta)
theta
theta_hat<jupyter_output><empty_output><jupyter_text>Full GD, Stochastic GD, Mini-batch GD<jupyter_code>theta_path_bgd = np.array(theta_path_bgd)
theta_path_sgd = np.array(theta_path_sgd)
theta_path_mgd = np.array(theta_path_mgd)
plt.figure(figsize=(10,8))
plt.plot(theta_path_sgd[:, 0], theta_path_sgd[:, 1], "r-s", linewidth=1, label="Stochastic")
plt.plot(theta_path_mgd[:, 0], theta_path_mgd[:, 1], "g-+", linewidth=1, label="Mini-batch")
plt.plot(theta_path_bgd[:, 0], theta_path_bgd[:, 1], "b-o", linewidth=1, label="Batch")
plt.legend(loc='best', fontsize=12)
plt.xlabel(r"$\theta_0$", fontsize=15)
plt.ylabel(r"$\theta_1$   ", fontsize=15)
plt.axis([2.5, 4.5, 2.3, 3.9])
plt.show()<jupyter_output><empty_output><jupyter_text>Polynomial Regression<jupyter_code>np.random.seed(42)

m = 100
X = 6 * np.random.rand(m, 1) - 3
y = 0.5 * X**2 + X + 2 + np.random.randn(m, 1)
plt.plot(X, y, "b.")
plt.xlabel("$x_1$", fontsize=15)
plt.ylabel("$y$", rotation=0, fontsize=15)
plt.axis([-3, 3, 0, 10])
plt.show()
from sklearn.preprocessing import PolynomialFeatures

poly_features = PolynomialFeatures(degree=2, include_bias=False)
X_poly = poly_features.fit_transform(X)
X[0]
X_poly[0]
lin_reg = LinearRegression()
lin_reg.fit(X_poly, y)
lin_reg.intercept_, lin_reg.coef_
X_new=np.linspace(-3, 3, 100).reshape(100, 1)
X_new_poly = poly_features.transform(X_new)
y_new = lin_reg.predict(X_new_poly)
plt.plot(X, y, "b.")
plt.plot(X_new, y_new, "r-", linewidth=2, label="Predictions")
plt.xlabel("$x_1$", fontsize=15)
plt.ylabel("$y$", rotation=0, fontsize=15)
plt.legend(loc="upper left", fontsize=12)
plt.axis([-3, 3, 0, 10])
plt.show()
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

for style, width, degree in (("g-", 1, 300), ("b--", 2, 2), ("r-+", 2, 1)):
    polybig_features = PolynomialFeatures(degree=degree, include_bias=False)
    std_scaler = StandardScaler()
    lin_reg = LinearRegression()
    polynomial_regression = Pipeline([
            ("poly_features", polybig_features),
            ("std_scaler", std_scaler),
            ("lin_reg", lin_reg),
        ])
    polynomial_regression.fit(X, y)
    y_newbig = polynomial_regression.predict(X_new)
    plt.plot(X_new, y_newbig, style, label=str(degree), linewidth=width)

plt.plot(X, y, "b.", linewidth=3)
plt.legend(loc="upper left")
plt.xlabel("$x_1$", fontsize=15)
plt.ylabel("$y$", rotation=0, fontsize=15)
plt.axis([-3, 3, 0, 10])

plt.show()
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split

def plot_learning_curves(model, X, y):
    X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=10)
    train_errors, val_errors = [], []
    for m in range(1, len(X_train)):
        model.fit(X_train[:m], y_train[:m])
        y_train_predict = model.predict(X_train[:m])
        y_val_predict = model.predict(X_val)
        train_errors.append(mean_squared_error(y_train[:m], y_train_predict))
        val_errors.append(mean_squared_error(y_val, y_val_predict))

    plt.plot(np.sqrt(train_errors), "r-+", linewidth=2, label="train")
    plt.plot(np.sqrt(val_errors), "b-", linewidth=3, label="val")
    plt.legend(loc="upper right", fontsize=14)   # not shown in the book
    plt.xlabel("Training set size", fontsize=14) # not shown
    plt.ylabel("RMSE", fontsize=14) 
lin_reg = LinearRegression()
plot_learning_curves(lin_reg, X, y)
plt.axis([0, 80, 0, 3])                         # not shown in the book

plt.show()                                      # not shown
from sklearn.pipeline import Pipeline

polynomial_regression = Pipeline([
        ("poly_features", PolynomialFeatures(degree=10, include_bias=False)),
        ("lin_reg", LinearRegression()),
    ])

plot_learning_curves(polynomial_regression, X, y)
plt.axis([0, 80, 0, 3])           # not shown

plt.show()                        # not shown<jupyter_output><empty_output><jupyter_text>These learning curves look a bit like the previous ones, but there are two very importantdifferences:*  The error on the training data is much lower than with the Linear Regressionmodel.*  There is a gap between the curves. This means that the model performs significantlybetter on the training data than on the validation data, which is the hallmarkof an **overfitting** model. However, if you used a much larger training set,the two curves would continue to get closer.  Regularized Regression<jupyter_code>np.random.seed(42)
m = 20
X = 3 * np.random.rand(m, 1)
y = 1 + 0.5 * X + np.random.randn(m, 1) / 1.5
X_new = np.linspace(0, 3, 100).reshape(100, 1)
from sklearn.linear_model import Ridge
ridge_reg = Ridge(alpha=1, solver="cholesky", random_state=42)
ridge_reg.fit(X, y)
from sklearn.linear_model import Ridge

def plot_model(model_class, polynomial, alphas, **model_kargs):
    for alpha, style in zip(alphas, ("b-", "g--", "r:")):
        model = model_class(alpha, **model_kargs) if alpha > 0 else LinearRegression()
        if polynomial:
            model = Pipeline([
                    ("poly_features", PolynomialFeatures(degree=10, include_bias=False)),
                    ("std_scaler", StandardScaler()),
                    ("regul_reg", model),
                ])
        model.fit(X, y)
        y_new_regul = model.predict(X_new)
        lw = 2 if alpha > 0 else 1
        plt.plot(X_new, y_new_regul, style, linewidth=lw, label=r"$\alpha = {}$".format(alpha))
    plt.plot(X, y, "b.", linewidth=3)
    plt.legend(loc="upper left", fontsize=10)
    plt.xlabel("$x_1$", fontsize=15)
    plt.axis([0, 3, 0, 4])

plt.figure(figsize=(8,4))
plt.subplot(121)
plot_model(Ridge, polynomial=False, alphas=(0, 10, 100), random_state=42)
plt.ylabel("$y$", rotation=0, fontsize=15)
plt.subplot(122)
plot_model(Ridge, polynomial=True, alphas=(0, 10**-5, 1), random_state=42)

plt.show()
sgd_reg = SGDRegressor(penalty="l2", max_iter=1000, tol=1e-3, random_state=42)
sgd_reg.fit(X, y.ravel())
sgd_reg.intercept_, sgd_reg.coef_
from sklearn.linear_model import Lasso

plt.figure(figsize=(8,4))
plt.subplot(121)
plot_model(Lasso, polynomial=False, alphas=(0, 0.1, 1), random_state=42)
plt.ylabel("$y$", rotation=0, fontsize=15)
plt.subplot(122)
plot_model(Lasso, polynomial=True, alphas=(0, 10**-7, 1), random_state=42)

plt.show()<jupyter_output>C:\Users\Choi\Anaconda3\lib\site-packages\sklearn\linear_model\_coordinate_descent.py:476: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations. Duality gap: 2.802867703827423, tolerance: 0.0009294783355207351
  positive)<jupyter_text>Ridge vs Lasso vs Elastic NetSo when should you use plain Linear Regression (i.e., without any regularization),Ridge, Lasso, or Elastic Net? It is almost always preferable to have at least a little bit ofregularization, so generally you should avoid plain Linear Regression. Ridge is a gooddefault, but if you suspect that only a few features are actually useful, you should preferLasso or Elastic Net since they tend to reduce the useless features’ weights down tozero as we have discussed. **In general, Elastic Net is preferred over Lasso since Lassomay behave erratically when the number of features is greater than the number oftraining instances or when several features are strongly correlated.**  Early Stopping<jupyter_code>np.random.seed(42)
m = 100
X = 6 * np.random.rand(m, 1) - 3
y = 2 + X + 0.5 * X**2 + np.random.randn(m, 1)

X_train, X_val, y_train, y_val = train_test_split(X[:50], y[:50].ravel(), test_size=0.5, random_state=10)
from sklearn.base import clone

poly_scaler = Pipeline([
        ("poly_features", PolynomialFeatures(degree=90, include_bias=False)),
        ("std_scaler", StandardScaler())
    ])

X_train_poly_scaled = poly_scaler.fit_transform(X_train)
X_val_poly_scaled = poly_scaler.transform(X_val)

sgd_reg = SGDRegressor(max_iter=1, tol=-np.infty, warm_start=True,
                       penalty=None, learning_rate="constant", eta0=0.0005, random_state=42)

minimum_val_error = float("inf")
best_epoch = None
best_model = None
for epoch in range(1000):
    sgd_reg.fit(X_train_poly_scaled, y_train)  # continues where it left off
    y_val_predict = sgd_reg.predict(X_val_poly_scaled)
    val_error = mean_squared_error(y_val, y_val_predict)
    if val_error < minimum_val_error:
        minimum_val_error = val_error
        best_epoch = epoch
        best_model = clone(sgd_reg)
sgd_reg = SGDRegressor(max_iter=1, tol=-np.infty, warm_start=True,
                       penalty=None, learning_rate="constant", eta0=0.0005, random_state=42)

n_epochs = 500
train_errors, val_errors = [], []
for epoch in range(n_epochs):
    sgd_reg.fit(X_train_poly_scaled, y_train)
    y_train_predict = sgd_reg.predict(X_train_poly_scaled)
    y_val_predict = sgd_reg.predict(X_val_poly_scaled)
    train_errors.append(mean_squared_error(y_train, y_train_predict))
    val_errors.append(mean_squared_error(y_val, y_val_predict))

best_epoch = np.argmin(val_errors)
best_val_rmse = np.sqrt(val_errors[best_epoch])

plt.annotate('Best model',
             xy=(best_epoch, best_val_rmse),
             xytext=(best_epoch, best_val_rmse + 1),
             ha="center",
             arrowprops=dict(facecolor='black', shrink=0.05),
             fontsize=16,
            )

best_val_rmse -= 0.03  # just to make the graph look better
plt.plot([0, n_epochs], [best_val_rmse, best_val_rmse], "k:", linewidth=2)
plt.plot(np.sqrt(val_errors), "b-", linewidth=3, label="Validation set")
plt.plot(np.sqrt(train_errors), "r--", linewidth=2, label="Training set")
plt.legend(loc="upper right", fontsize=14)
plt.xlabel("Epoch", fontsize=14)
plt.ylabel("RMSE", fontsize=14)

plt.show()
best_epoch, best_model<jupyter_output><empty_output><jupyter_text>Logistic Regression<jupyter_code>t = np.linspace(-10, 10, 100)
sig = 1 / (1 + np.exp(-t))
plt.figure(figsize=(10, 5))
plt.plot(t, sig, "b-", linewidth=2, label=r"$\sigma(t) = \frac{1}{1 + e^{-t}}$")
plt.xlabel("t")
plt.legend(loc="upper left", fontsize=20)
plt.axis([-10, 10, -0.1, 1.1])
plt.grid(True)
plt.show()
from sklearn import datasets
iris = datasets.load_iris()
list(iris.keys())
X = iris["data"][:, 3:]  # petal width
y = (iris["target"] == 2).astype(np.int)  # 1 if Iris virginica, else 0
from sklearn.linear_model import LogisticRegression
log_reg = LogisticRegression(random_state=42)
log_reg.fit(X, y)
X_new = np.linspace(0, 3, 1000).reshape(-1, 1)
y_proba = log_reg.predict_proba(X_new)

plt.plot(X_new, y_proba[:, 1], "g-", linewidth=2, label="Iris virginica")
plt.plot(X_new, y_proba[:, 0], "b--", linewidth=2, label="Not Iris virginica")
X_new = np.linspace(0, 3, 1000).reshape(-1, 1)
y_proba = log_reg.predict_proba(X_new)
decision_boundary = X_new[y_proba[:, 1] >= 0.5][0]

plt.figure(figsize=(8, 3))
plt.plot(X[y==0], y[y==0], "bs")
plt.plot(X[y==1], y[y==1], "g^")
plt.plot([decision_boundary, decision_boundary], [-1, 2], "k:", linewidth=2)
plt.plot(X_new, y_proba[:, 1], "g-", linewidth=2, label="Iris virginica")
plt.plot(X_new, y_proba[:, 0], "b--", linewidth=2, label="Not Iris virginica")
plt.text(decision_boundary+0.02, 0.15, "Decision  boundary", fontsize=14, color="k", ha="center")

plt.xlabel("Petal width (cm)", fontsize=14)
plt.ylabel("Probability", fontsize=14)
plt.legend(loc="center left", fontsize=14)
plt.axis([0, 3, -0.02, 1.02])

plt.show()<jupyter_output><empty_output><jupyter_text>Softmax Regression<jupyter_code>X = iris["data"][:, (2, 3)]  # petal length, petal width
y = iris["target"]

softmax_reg = LogisticRegression(multi_class="multinomial",solver="lbfgs", C=10, random_state=42)
softmax_reg.fit(X, y)
x0, x1 = np.meshgrid(
        np.linspace(0, 8, 500).reshape(-1, 1),
        np.linspace(0, 3.5, 200).reshape(-1, 1),
    )
X_new = np.c_[x0.ravel(), x1.ravel()]


y_proba = softmax_reg.predict_proba(X_new)
y_predict = softmax_reg.predict(X_new)

zz1 = y_proba[:, 1].reshape(x0.shape)
zz = y_predict.reshape(x0.shape)

plt.figure(figsize=(10, 4))
plt.plot(X[y==2, 0], X[y==2, 1], "g^", label="Iris virginica")
plt.plot(X[y==1, 0], X[y==1, 1], "bs", label="Iris versicolor")
plt.plot(X[y==0, 0], X[y==0, 1], "yo", label="Iris setosa")

from matplotlib.colors import ListedColormap
custom_cmap = ListedColormap(['#fafab0','#9898ff','#a0faa0'])

plt.contourf(x0, x1, zz, cmap=custom_cmap)
contour = plt.contour(x0, x1, zz1, cmap=plt.cm.brg)
plt.clabel(contour, inline=1, fontsize=12)
plt.xlabel("Petal length", fontsize=14)
plt.ylabel("Petal width", fontsize=14)
plt.legend(loc="center left", fontsize=14)
plt.axis([0, 7, 0, 3.5])

plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1265584469918835, 'fraction_numerical': 0.02840766463057485, 'mean_word_length': 4.5152284263959395, 'pattern_counts': {'":': 0, '<': 51, '<?xml version=': 0, '>': 53, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '236545', 'license': 'MIT', 'n_tokens_mistral': 8843, 'n_tokens_neox': 8236, 'n_words': 2402, 'path': '00_04.ipynb', 'repo_name': 'laplaisanterie/Hands-on-ML', 'size': '23902'}