<jupyter_start><jupyter_text>Introduction to neural nets The input is a set of feature vectors and the known answers for each vector - targets. Neural nets take input data that is observable, recordable, and by extension knowable and transform it into a logical model that can predict any future data.The neural network is simply a one or more weights which can be multiplied by the input vector to output a prediction or a class label. Thus, neural nets are good both for the prediction and for the classification.  Is the prediction always correct? No, the network can make mistakes. But it can learn from its mistakes.How does the network learn?Trial and error! First, it tries to make a prediction. Then, it sees whether it was too high or too low. Finally, it changes the weight (up or down) to predict moreaccurately the next time it sees the same input. The algorithm consists of several iterations (*epochs) of **predict-compare-learn**.   1. Perceptron We will demonstrate this idea with the simplest possible neural network which contains a single processing neuron. This self-learning network is called a *perceptron*. A feature vector is presented to this neuron. For each feature there is a weight, and the network predicts a target variable by multiplying each feature value by its weight and producing a weighted sum. For a simple case of binary classification, we will use an activation function *sign*: if the weighted sum is positive the neuron predicts class yes(1), and if the sum is negative it predicts no(0).        Fig 1. Simple perceptron for classifying a vector of 2 features. Tranforms a weighted sum of features into a class using the sign function.  1.1. Implementation of a simple Perceptron<jupyter_code>import numpy as np


class Perceptron:
    """ A basic Perceptron"""

    def __init__(self, inputs, targets):
        """ Constructor - setups dimensions and initializes weights"""
        # Set up network size
        if np.ndim(inputs) > 1:
            self.nIn = np.shape(inputs)[1]
        else:
            self.nIn = 1

        if np.ndim(targets) > 1:
            self.nOut = np.shape(targets)[1]
        else:
            self.nOut = 1

        self.nData = np.shape(inputs)[0]

        # Initialise network weights - random guess
        self.weights = np.random.rand(self.nIn + 1, self.nOut) * 0.1 - 0.05

        
    # Use this to predict target for a given feature vector 'point'
    def predict(self, point):
        input_with_bias = np.concatenate((point, -np.ones((1, 1))), axis=1)
        activations = self.forward(input_with_bias)
        print("Prediction for input {} is {}:".format(point, activations))

        
    # Train network
    def train(self, inputs, targets, eta, nIterations):
        """ Train the thing """
        # Add one bias node
        inputs = np.concatenate((inputs, -np.ones((self.nData, 1))), axis=1)

        # Training: activation function - sign       
        activations = self.forward(inputs)
        for n in range(nIterations):
            print("Iteration: ", n+1)
            print("Current weights:", self.weights.tolist())

            print("Current activations:",  activations.tolist())
            print("Targets:", targets.tolist())
            print()

            total_error = np.sum(activations - targets)
            if total_error == 0:
                break

            # update weights
            self.weights -= eta * np.dot(np.transpose(inputs), activations - targets)
            activations = self.forward(inputs)

        print("Targets:", targets.tolist())
        print("Final predictions:", activations.tolist())

    
    def forward(self, inputs):
        """ Run the network forward """
        # Compute activations
        activations = np.dot(inputs, self.weights)

        # Threshold the activations
        return np.where(activations > 0, 1, 0)

    def confusion_matrix(self, inputs, targets):
        """Confusion matrix"""

        # Add the bias node
        inputs = np.concatenate((inputs, -np.ones((self.nData, 1))), axis=1)

        outputs = np.dot(inputs, self.weights)

        nClasses = np.shape(targets)[1]

        if nClasses == 1:
            nClasses = 2
            outputs = np.where(outputs > 0, 1, 0)
        else:
            # 1-of-N encoding
            outputs = np.argmax(outputs, 1)
            targets = np.argmax(targets, 1)

        cm = np.zeros((nClasses, nClasses))
        for i in range(nClasses):
            for j in range(nClasses):
                cm[i, j] = np.sum(np.where(outputs == i, 1, 0) * np.where(targets == j, 1, 0))

        print("Confusion matrix:")
        print(cm)
        print("Accuracy:", np.trace(cm) / np.sum(cm))<jupyter_output><empty_output><jupyter_text>1.2. Learning concept of OR        Fig 2. Perceptron can learn a decision boundary for OR data.<jupyter_code>a = np.array([[0, 0, 0], [0, 1, 1], [1, 0, 1], [1, 1, 1]])

# set up the size of the input vector and initial weights
p = Perceptron(a[:, 0:2], a[:, 2:])<jupyter_output><empty_output><jupyter_text>Prediction before learning:<jupyter_code>print("Before training:")
test = np.array([[0, 0]])
p.predict(test)
print()
test = np.array([[5, 0]])
p.predict(test)
print()
test = np.array([[0, 3]])
p.predict(test)<jupyter_output>Before training:
Prediction for input [[0 0]] is [[0]]:

Prediction for input [[5 0]] is [[0]]:

Prediction for input [[0 3]] is [[1]]:<jupyter_text>Now learns from data:<jupyter_code>p.train(a[:, 0:2], a[:, 2:], 0.25, 10)
p.confusion_matrix(a[:, 0:2], a[:, 2:])<jupyter_output>Confusion matrix:
[[1. 0.]
 [0. 3.]]
Accuracy: 1.0<jupyter_text>Now can classify any input of OR:<jupyter_code>print("After training:")
test = np.array([[0, 0]])
p.predict(test)
print()
test = np.array([[5, 0]])
p.predict(test)
print()
test = np.array([[0, 3]])
p.predict(test)<jupyter_output>After training:
Prediction for input [[0 0]] is [[0]]:

Prediction for input [[5 0]] is [[1]]:

Prediction for input [[0 3]] is [[1]]:<jupyter_text>1.3. Learning concept of AND        Fig 3. Perceptron can learn a decision boundary for AND data.<jupyter_code>a = np.array([[0, 0, 0], [0, 1, 0], [1, 0, 0], [1, 1, 1]])

test = np.array([[0, 1]]) 
print("Before training:")
p.predict(test)
p.train(a[:, 0:2], a[:, 2:], 0.25, 10)
p.confusion_matrix(a[:, 0:2], a[:, 2:])
print("After training:")
p.predict(test)<jupyter_output>After training:
Prediction for input [[0 1]] is [[0]]:<jupyter_text>1.4. Concept of $x_1$ AND NOT $x_2$        Fig 4. Perceptron should be able to learn a decision boundary for $x_1$ AND NOT $x_2$.  1.5. Concept of exclusive OR (XOR)        Fig 5. Decision boundary for $x_1$ XOR $x_2$.<jupyter_code>a = np.array([[0, 0, 1], [0, 1, 0], [1, 0, 0], [1, 1, 1]])

test = np.array([[0, 0]]) 
print("Before training:")
p.predict(test)
p.train(a[:, 0:2], a[:, 2:], 0.25, 10)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13089691322073385, 'fraction_numerical': 0.025480489225393126, 'mean_word_length': 3.0838287752675386, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '471144', 'license': 'MIT', 'n_tokens_mistral': 2285, 'n_tokens_neox': 2132, 'n_words': 763, 'path': 'basic_perceptron.ipynb', 'repo_name': 'mgbarsky/ml_neural_nets_demo', 'size': '6868'}