<jupyter_start><jupyter_text>Beyond linear separation in classificationAs we saw in the regression section, the linear classification modelexpects the data to be linearly separable. When this assumption does nothold, the model is not expressive enough to properly fit the data.Therefore, we need to apply the same tricks as in regression: featureaugmentation (potentially using expert-knowledge) or using akernel-based method.We will provide examples where we will use a kernel support vector machineto perform classification on some toy-datasets where it is impossible tofind a perfect linear separation.We will generate a first dataset where the data are represented as twointerlaced half circle. This dataset is generated using the function[`sklearn.datasets.make_moons`](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_moons.html).<jupyter_code>import numpy as np
import pandas as pd
from sklearn.datasets import make_moons

feature_names = ["Feature #0", "Features #1"]
target_name = "class"

X, y = make_moons(n_samples=100, noise=0.13, random_state=42)

# We store both the data and target in a dataframe to ease plotting
moons = pd.DataFrame(np.concatenate([X, y[:, np.newaxis]], axis=1),
                     columns=feature_names + [target_name])
data_moons, target_moons = moons[feature_names], moons[target_name]<jupyter_output><empty_output><jupyter_text>Since the dataset contains only two features, we can make a scatter plot tohave a look at it.<jupyter_code>import matplotlib.pyplot as plt
import seaborn as sns

sns.scatterplot(data=moons, x=feature_names[0], y=feature_names[1],
                hue=target_moons, palette=["tab:red", "tab:blue"])
_ = plt.title("Illustration of the moons dataset")<jupyter_output><empty_output><jupyter_text>From the intuitions that we got by studying linear model, it should beobvious that a linear classifier will not be able to find a perfect decisionfunction to separate the two classes.Let's try to see what is the decision boundary of such a linear classifier.We will create a predictive model by standardizing the dataset followed bya linear support vector machine classifier.<jupyter_code>import sklearn
sklearn.set_config(display="diagram")
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

linear_model = make_pipeline(StandardScaler(), SVC(kernel="linear"))
linear_model.fit(data_moons, target_moons)<jupyter_output><empty_output><jupyter_text>WarningBe aware that we fit and will check the boundary decision of the classifieron the same dataset without splitting the dataset into a training set and atesting set. While this is a bad practice, we use it for the sake ofsimplicity to depict the model behavior. Always use cross-validation whenyou want to assess the generalization performance of a machine-learning model. Let's check the decision boundary of such a linear model on this dataset.<jupyter_code>from helpers.plotting import DecisionBoundaryDisplay

DecisionBoundaryDisplay.from_estimator(
    linear_model, data_moons, response_method="predict", cmap="RdBu", alpha=0.5
)
sns.scatterplot(data=moons, x=feature_names[0], y=feature_names[1],
                hue=target_moons, palette=["tab:red", "tab:blue"])
_ = plt.title("Decision boundary of a linear model")<jupyter_output><empty_output><jupyter_text>As expected, a linear decision boundary is not enough flexible to split thetwo classes.To push this example to the limit, we will create another dataset wheresamples of a class will be surrounded by samples from the other class.<jupyter_code>from sklearn.datasets import make_gaussian_quantiles

feature_names = ["Feature #0", "Features #1"]
target_name = "class"

X, y = make_gaussian_quantiles(
    n_samples=100, n_features=2, n_classes=2, random_state=42)
gauss = pd.DataFrame(np.concatenate([X, y[:, np.newaxis]], axis=1),
                     columns=feature_names + [target_name])
data_gauss, target_gauss = gauss[feature_names], gauss[target_name]
ax = sns.scatterplot(data=gauss, x=feature_names[0], y=feature_names[1],
                     hue=target_gauss, palette=["tab:red", "tab:blue"])
_ = plt.title("Illustration of the Gaussian quantiles dataset")<jupyter_output><empty_output><jupyter_text>Here, this is even more obvious that a linear decision function is notadapted. We can check what decision function, a linear support vector machinewill find.<jupyter_code>linear_model.fit(data_gauss, target_gauss)
DecisionBoundaryDisplay.from_estimator(
    linear_model, data_gauss, response_method="predict", cmap="RdBu", alpha=0.5
)
sns.scatterplot(data=gauss, x=feature_names[0], y=feature_names[1],
                hue=target_gauss, palette=["tab:red", "tab:blue"])
_ = plt.title("Decision boundary of a linear model")<jupyter_output><empty_output><jupyter_text>As expected, a linear separation cannot be used to separate the classesproperly: the model will under-fit as it will make errors even onthe training set.In the section about linear regression, we saw that we could use severaltricks to make a linear model more flexible by augmenting features orusing a kernel. Here, we will use the later solution by using a radial basisfunction (RBF) kernel together with a support vector machine classifier.We will repeat the two previous experiments and check the obtained decisionfunction.<jupyter_code>kernel_model = make_pipeline(StandardScaler(), SVC(kernel="rbf", gamma=5))
kernel_model.fit(data_moons, target_moons)
DecisionBoundaryDisplay.from_estimator(
    kernel_model, data_moons, response_method="predict", cmap="RdBu", alpha=0.5
)
sns.scatterplot(data=moons, x=feature_names[0], y=feature_names[1],
                hue=target_moons, palette=["tab:red", "tab:blue"])
_ = plt.title("Decision boundary with a model using an RBF kernel")<jupyter_output><empty_output><jupyter_text>We see that the decision boundary is not anymore a straight line. Indeed,an area is defined around the red samples and we could imagine that thisclassifier should be able to generalize on unseen data.Let's check the decision function on the second dataset.<jupyter_code>kernel_model.fit(data_gauss, target_gauss)
DecisionBoundaryDisplay.from_estimator(
    kernel_model, data_gauss, response_method="predict", cmap="RdBu", alpha=0.5
)
ax = sns.scatterplot(data=gauss, x=feature_names[0], y=feature_names[1],
                     hue=target_gauss, palette=["tab:red", "tab:blue"])
_ = plt.title("Decision boundary with a model using an RBF kernel")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08351145038167938, 'fraction_numerical': 0.006412213740458015, 'mean_word_length': 5.9765708200212995, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '159718', 'license': 'MIT', 'n_tokens_mistral': 1944, 'n_tokens_neox': 1899, 'n_words': 648, 'path': 'sklearn/notes/logistic_regression_non_linear.ipynb', 'repo_name': 'shamik-biswas-rft/CodeSnippets', 'size': '6550'}