<jupyter_start><jupyter_text>Perceptron algorithm in plain PythonThe perceptron is a simple supervised machine learning algorithm and one of the earliest **neural network** architectures. It was introduced by Rosenblatt in the late 1950s. A perceptron represents a **binary linear classifier** that maps a set of training examples (of $d$ dimensional input vectors) onto binary output values using a $d-1$ dimensional hyperplane.The perceptron as follows.**Given:** - dataset $\{(\boldsymbol{x}^{(1)}, y^{(1)}), ..., (\boldsymbol{x}^{(m)}, y^{(m)})\}$- with $\boldsymbol{x}^{(i)}$ being a $d-$dimensional vector $\boldsymbol{x}^i = (x^{(i)}_1, ..., x^{(i)}_d)$- $y^{(i)}$ being a binary target variable, $y^{(i)} \in \{0,1\}$The perceptron is a very simple neural network:- it has a real-valued weight vector $\boldsymbol{w}= (w^{(1)}, ..., w^{(d)})$- it has a real-valued bias $b$- it uses the Heaviside step function (unit step function) as its activation function* * *A perceptron is trained using **gradient descent**. The training algorithm has different steps. In the beginning (step 0) the model parameters are initialized. The other steps (see below) are repeated for a specified number of training iterations or until the parameters have converged.**Step 0: ** Initialize the weight vector and bias with zeros (or small random values).* * ***Step 1: ** Compute a linear combination of the input features and weights. This can be done in one step for all training examples, using vectorization and broadcasting:$\boldsymbol{a} = \boldsymbol{X} \cdot \boldsymbol{w} + b$where $\boldsymbol{X}$ is a matrix of shape $(n_{samples}, n_{features})$ that holds all training examples, and $\cdot$ denotes the dot product.* * ***Step 2: ** Apply the Heaviside step function, which returns binary values:$\hat{y}^{(i)} = 1 \, if \, a^{(i)} \geq 0, \, else \, 0$* * *** Step 3: ** Compute the weight updates using the perceptron learning rule\begin{equation}\Delta \boldsymbol{w} = \eta \, \boldsymbol{X}^T \cdot \big(\boldsymbol{\hat{y}} - \boldsymbol{y} \big)\end{equation}$$ \Delta b = \eta \, \big(\boldsymbol{\hat{y}} - \boldsymbol{y} \big) $$where $\eta$ is the learning rate.* * *** Step 4: ** Update the weights and bias\begin{equation}\boldsymbol{w} = \boldsymbol{w} + \Delta \boldsymbol{w}\end{equation}$$b = b  + \Delta b$$<jupyter_code>import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_blobs
from sklearn.model_selection import train_test_split
np.random.seed(123)

# % matplotlib inline<jupyter_output>(3, 1)<jupyter_text>Perceptron model class<jupyter_code>class Perceptron():

    def __init__(self):
        pass

    def train(self, X, y, learning_rate=0.05, n_iters=100):
        n_samples, n_features = X.shape

        # Step 0: Initialize the parameters
        self.weights = np.zeros((n_features,1))
        self.bias = 0

        for i in range(n_iters):
            # Step 1: Compute the activation
            a = np.dot(X, self.weights) + self.bias

            # Step 2: Compute the output
            y_predict = self.step_function(a)

            # Step 3: Compute weight updates
            delta_w = learning_rate * np.dot(X.T, (y - y_predict))
            delta_b = learning_rate * np.sum(y - y_predict)

            # Step 4: Update the parameters
            self.weights += delta_w
            self.bias += delta_b

        return self.weights, self.bias

    def step_function(self, x):
        return np.array([1 if elem >= 0 else 0 for elem in x])[:, np.newaxis]

    def predict(self, X):
        a = np.dot(X, self.weights) + self.bias
        return self.step_function(a)<jupyter_output><empty_output><jupyter_text>Dataset<jupyter_code>X, y = make_blobs(n_samples=1000, centers=2)
fig = plt.figure(figsize=(8,6))
plt.scatter(X[:,0], X[:,1], c=y)
plt.title("Dataset")
plt.xlabel("First feature")
plt.ylabel("Second feature")
plt.show()
y_true = y[:, np.newaxis]

X_train, X_test, y_train, y_test = train_test_split(X, y_true)
print(f'Shape X_train: {X_train.shape}')
print(f'Shape y_train: {y_train.shape})')
print(f'Shape X_test: {X_test.shape}')
print(f'Shape y_test: {y_test.shape}')<jupyter_output>Shape X_train: (750, 2)
Shape y_train: (750, 1))
Shape X_test: (250, 2)
Shape y_test: (250, 1)<jupyter_text>Initialization and training the model<jupyter_code>p = Perceptron()
w_trained, b_trained = p.train(X_train, y_train,learning_rate=0.05, n_iters=500)<jupyter_output><empty_output><jupyter_text>Testing<jupyter_code>y_p_train = p.predict(X_train)
y_p_test = p.predict(X_test)

print(f"training accuracy: {100 - np.mean(np.abs(y_p_train - y_train)) * 100}%")
print(f"test accuracy: {100 - np.mean(np.abs(y_p_test - y_test)) * 100}%")<jupyter_output>training accuracy: 100.0%
test accuracy: 100.0%<jupyter_text>Visualize decision boundary<jupyter_code>def plot_hyperplane(X, y, weights, bias):
    """
    Plots the dataset and the estimated decision hyperplane
    """
    slope = - weights[0]/weights[1]
    intercept = - bias/weights[1]
    x_hyperplane = np.linspace(-10,10,10)
    y_hyperplane = slope * x_hyperplane + intercept
    fig = plt.figure(figsize=(8,6))
    plt.scatter(X[:,0], X[:,1], c=y)
    plt.plot(x_hyperplane, y_hyperplane, '-')
    plt.title("Dataset and fitted decision hyperplane")
    plt.xlabel("First feature")
    plt.ylabel("Second feature")
    plt.show()
plot_hyperplane(X, y, w_trained, b_trained)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14522058823529413, 'fraction_numerical': 0.019301470588235295, 'mean_word_length': 4.201720841300191, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 23, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '50125', 'license': 'MIT', 'n_tokens_mistral': 1861, 'n_tokens_neox': 1754, 'n_words': 524, 'path': 'single_layer_perceptron_1.ipynb', 'repo_name': 'louis-xu-ustc/machine_learning_basics', 'size': '5440'}