<jupyter_start><jupyter_text>Neural Networks 101: Through the Plane and Back!In the last class we covered a lot of ground so that we could understand the algebra and geometry underlying neural networks. We also left a little challenge, with which we will make most of our progress in this class. In this class, we are going to gradually build the basic ideas of neural networks so that we can get hacking!Let's import our libraries and get plotting:<jupyter_code>%matplotlib inline
import math 
import numpy as np 
from ipywidgets import *
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>The challenge we left in the last class was to find a way to separate the following plot using just a single straight line. Obviously, there is no way to perfectly do it because the red area comes from a curve that is non linear.![Challenge for linear separation](images/TargetSeparate.png)In the usual Machine Learning vocabulary, we say this kind of set is not linearly separable. It just means that the red points can't be separated from the blue ones by using planes alone. You probably considered just trying to get as much of one of the two colors right. This might give you a good enough approximation, but we are looking for something more general here. Let's look at our plot, place a plane and see how much we can get right:<jupyter_code>X = np.random.random((10000, 2)) * 7 - (3.5, 0)
y = np.asarray([0 if x[1] < x[0]**2 else 1 for x in X]) # our binary labels -- 1 is red, 0 otherwise
xx = np.linspace(-3.5, 3.5)

def color_fn(a, b):
    '''Color to be assigned for a point given the predicted class (a) and the real class (b):
    
    green (g) if the class is unknown (-1).
    yellow (y) if the predicted class is different from the real one.
    red (r) if the predicted and the real class match and the class is not 0, blue (b) otherwise
    '''
    if a == -1:
        return 'g'
    if a != b:
        return 'y'
    else:
        return 'r' if a else 'b'

def plot_hyperplane(x, y, c):
    '''Plot a plane with weights x and y and bias c.'''
    if y:
        a = -x / y
        yy = a * xx - c / y
        plt.plot(xx, yy, 'k-')
    else:
        s = np.linalg.norm([x, y])
        plt.vlines([-c / s], ymin=-7, ymax=7, colors='k')
    plt.show()
    
    
def update_parabola(u, w, b):
    '''Plot the parabola dataset and the separating plane, showing classification accuracy statistics.'''
    v = [u, w]
    
    # compute prediction
    y_pred = np.dot(X, v) + b > 0
    c = [color_fn(l, r) for l, r in zip(y_pred, y)]
    
    # plot it out
    plt.gca().set_xlim(-3.5, 3.5)
    plt.gca().set_ylim( 0.0, 7.0)
    plt.scatter(X[:, 0], X[:, 1], marker='.', c=c)
    plot_hyperplane(v[0], v[1], b)
    
    # count percentages of instances
    wrong = 100.0 * sum(l == 'y' for l in c) / len(c)
    reds  = 100.0 * sum(l == 'r' for l in c) / sum(1 for l in y if l)
    blues = 100.0 * sum(l == 'b' for l in c) / sum(1 for l in y if not l)
    print('We got {:.2f}% of the points wrong, correctly matching {:.2f}% of the red points and {:.2f}% of the blue ones.'.format(wrong, reds, blues))

    
interact(update_parabola, 
         u=widgets.FloatText(value=1.0, step=0.01, description='X comp: '),
         w=widgets.FloatText(value=0.0, step=0.01, description='Y comp: '),
         b=widgets.FloatText(value=0.0, step=0.01, description='Bias: '));<jupyter_output><empty_output><jupyter_text>If you spend some time playing around with the components, you can get a result that separates the red points well enough, but you are going to have some difficulty with the blue ones. Since there are roughly the same amount of reds and blues, you're at most going to get $\frac{3}{4}$ths of the space filled with the correct color. For instance, with $\langle 2, 0.58 \rangle$ as our weight vector and a bias of $1.5$ and we will get something that looks like this: ![Separation example with fixed parameters](images/ExampleSeparate.png)Don't get scared by the weird brackets: it just means our $x$ component is $2$ and our $y$ component is $0.58$. With this setup, only 26.24% of the points are misclassified, and we correctly classify 98.86% of the red points and 48.70% of the blue ones. Aside from not being totally accurate, it doesn't tell us how far off we are! A first way of measuring this is by adding some uncertainty boundaries, where we will classify points as unknown. Admitting that we don't know whether a point should be red or blue is not a big deal for this problem. However, in  contexts like medicine or finance our models can hurt people or lose money. **Measuring uncertainty lets us handle these situations**, perhaps by making so that the system asks for expert supervision. Let's add uncertainty to our model:<jupyter_code>def update_uncertain_parabola(u, w, b, d):
    '''Plot the parabola dataset and the separating plane (with uncertainty ranges),
    showing classification accuracy statistics.'''
    v = [u, w]
    
    # compute prediction
    y_pred = [-1 if np.abs(x) < d else int(x > 0) for x in np.dot(X, v) + b]
    c = [color_fn(l, r) for l, r in zip(y_pred, y)]
    
    # plot it out
    plt.gca().set_xlim(-3.5, 3.5)
    plt.gca().set_ylim( 0.0, 7.0)
    plt.scatter(X[:, 0], X[:, 1], marker='.', c=c)
    plot_hyperplane(v[0], v[1], b)
    
    # count percentages of instances
    wrong = 100.0 * sum(l == 'y' for l in c) / len(c)
    unknn = 100.0 * sum(l == 'g' for l in c) / len(c)
    reds  = 100.0 * sum(l == 'r' for l in c) / sum(1 for l in y if l)
    blues = 100.0 * sum(l == 'b' for l in c) / sum(1 for l in y if not l)
    print('We got {:.2f}% of the points wrong and {:.2f}% unknown, correctly matching {:.2f}% of the red points and {:.2f}% of the blue ones.'.format(wrong, unknn, reds, blues))

    
interact(update_uncertain_parabola, 
         u=widgets.FloatText(value=1.0, step=0.01, description='X comp: '),
         w=widgets.FloatText(value=0.0, step=0.01, description='Y comp: '),
         b=widgets.FloatText(value=0.0, step=0.01, description='Bias: '),
         d=widgets.FloatText(value=0.1, step=0.01, description='Uncertainty: '));<jupyter_output><empty_output><jupyter_text>If you play around with the weights and bias, you will notice that this method of representing uncertainty around the decision boundary is not very understandable. In particular, the uncertainty scale depends on the norm of the weight vector. Furthermore, the way we say points are unknown has no clear-cut interpretation. Instead, we would want to know that the blue patch that we misclassify as 'very red' is just very wrong. A way to work around this is to represent uncertainty in some other way. Our plane, in the end, is giving us a number from which we are taking the sign. Let's word our task more carefully: we want to separate blue and red points, be able to distinguish them. In short, we want to classify the points that are red and those that are blue. It would be nice to find a way that represents **how red and not blue a point is** as a probability. Then, the notion of uncertainty would be baked into it.Hold that thought, let's go off-track for a second. Our plane gives us a boundary depending on the sign of the output. Is there a way to take this output and represent **how certain by how large the results are**? Let's plot two functions and try to get a quick intuition:<jupyter_code>space = np.linspace(-10, 10)
f, axarr = plt.subplots(2)
axarr[0].grid(True)
axarr[1].grid(True)
axarr[0].plot(space, np.tanh(space))
axarr[0].set_title('Two special functions: tanh (top) and sigmoid (bottom)')
axarr[1].plot(space, 1 / (1 + np.exp(-space)))
plt.show()<jupyter_output><empty_output><jupyter_text>What we have introduced is two nonlinearities: the hyperbolic tangent and the sigmoid function. The first one will take our output and map it into the interval from -1 to 1. The second one, instead, will wrap it into the interval from 0 to 1. Let's see what we get if we apply one of them to our plane. In particular, since we want to use a probability, let's go with the sigmoid. We can see that values rapidly become either 0 or 1, and that when the value of $x$ is 0, the value of the sigmoid is 0.5. We will plot all the points, colored according to the sigmoid of the output of our plane:<jupyter_code>def sigmoid(x):
    '''Implementation of the sigmoid function using NumPy.'''
    return 1 / (1 + np.exp(-x))

def update_sigmoid(u, w, b):
    '''Plot the parabola dataset and the separating plane (smoothing outputs with sigmoid),
    showing classification accuracy statistics.'''
    v = [u, w]
    
    # compute prediction
    y_pred = sigmoid(np.dot(X, v) + b)
    c = 1 - y_pred # the color map we are using is from red to blue, so we flip the prediction
    c_overlay = [color_fn(i, i) for i in y]
    
    # plot it out on 3 layers to represent boundary & figure
    plt.gca().set_xlim(-3.5, 3.5)
    plt.gca().set_ylim( 0.0, 7.0)
    plt.scatter(X[:, 0], X[:, 1], marker='.', c=c_overlay)
    plt.scatter(X[:, 0], X[:, 1], marker='.', c=c, cmap='RdBu', alpha=0.8)
    plt.scatter(X[:, 0], X[:, 1], marker='.', c=c_overlay, alpha=0.08)
    plot_hyperplane(v[0], v[1], b)

    
interact(update_sigmoid, 
         u=widgets.FloatText(value=1.0, step=0.01, description='X comp: '),
         w=widgets.FloatText(value=0.0, step=0.01, description='Y comp: '),
         b=widgets.FloatText(value=0.0, step=0.01, description='Bias: '));<jupyter_output><empty_output><jupyter_text>The plot we have just seen is actually encoding something more: we have a smooth transition from red to blue! This gives us a measure of how uncertain our decision is. Clearer tones in our picture represent more uncertainty when a point is close to our plane.Our function is still the same plane equation: $f(x) = dot(w, x) + b$ with our weights $w$ and our bias $b$. However, we are wrapping it up with $sigmoid(x)$ — often written $\sigma(x)$, since $\sigma$ is the greek letter 'sigma'. That is, we now use $g(x) = \sigma(dot(w, x) + b)$.If you know some statistics or have done some machine learning before, you might recognize that we are now computing a **logistic regression**! Logistic regressions simply take an input vector and produce a probability between 0 and 1. As you can see, they are a simple way of classifying two sets of points, and giving a probability in the process. In our case, this represents the probability of belonging to the group of red points, since they are the class we are putting on the positive side.What a ride! We have gotten very far with our little example: we now can encode how certain we are of our predictions with our plane! Furthermore, we have inadvertedly introduced the last concept we need to define a neural network. Let's welcome the simplest neural net, a single perceptron:![Single Perceptron](images/SinglePerceptron.png)which in terms of code, it is simply:<jupyter_code>def perceptron(x, weight, bias, activation):
    '''Feed-forward single perceptron.'''
    dot = np.dot(weight, x)
    with_bias = dot + bias
    return activation(with_bias)<jupyter_output><empty_output><jupyter_text>That looks familiar: we have $n$ different inputs, each of which are weighted and added together with a bias. Finally we apply the activation function to the result of the sum! We can see that a logistic regression is just a single perceptron with a sigmoid activation! Of course, there can be many different activation functions, some beyond the two that we have seen. If the activation is a differentiable function, we can also compute the error between what we desired and our prediction — and then send it back to adjust the values of the weight vector and bias! This is how neural networks learn: by adjusting the weights to reduce the error. Of course, we have to define a way of measuring the error, which is called the **loss function**, and then select an algorithm to minimize it over our network, called the **optimizer**. The whole process of sending error backwards is aptly called backpropagation and, in modern frameworks, it comes for free. We will not worry about the specifics of backpropagation in this course, so **you just need to grasp this general intuition about automatically minimizing some sort of error by adjusting the parameters of our model**.Let's train our first neural network using Keras, specifying our structure, our loss function, our optimizer and setting it up to get going:<jupyter_code>from keras.models import Model
from keras.layers import Input, Dense

# define the input -- a two component vector
inp = Input((2,), name='input_pair')

# put the layer -- a dense layer with just 1 unit
out = Dense(1, name='output_f', activation='sigmoid')(inp)

# define the model
model = Model(inputs=inp, outputs=out)
model.compile(loss='binary_crossentropy', optimizer='sgd', metrics=['accuracy'])
model.summary()
model.fit(X, y, epochs=20, batch_size=500)<jupyter_output><empty_output><jupyter_text>There are a few details we glossed through: this is where a hacker like you plays around with the code to really get what is going on. First of all, our model has an `Input`, which defines the shape of the input vector, and then a `Dense`, which really doesn't sound like anything we have seen. Our loss function has a fancy name, `binary_crossentropy`. Our optimizer sounds like a secret police force, `sgd`. We are defining a number of `epochs` and a `batch_size`. What's the deal with all that? Let me try to answer those questions one by one:1. We define our model by stacking layers. The first layer, as you've seen, is an input with 2 components. The second and last layer, which is the output of the model, is formed by a **Dense** layer with a single component. A dense layer is simply a layer that has weights for all the components of its input. This is why it is called dense! Since a single perceptron weights every input, it is exactly what we need! In general, weights are first set to the values specified by using some **initializer** function. In the case of Dense layers in Keras, weights are initialized using **random uniform** numbers!2. On the output, we want to classify whether or not a point is red. This is a **binary** classification problem: our points are labelled with a $1$ if they are red and with a $0$ if they are blue. We are using a sigmoid output, which can be understood as a probability. To decrease our error, we try to reduce the difference between the probabilities of what we get and what we expect. This is done by reducing the **binary cross-entropy**, a function that compares how far a probability between 0 and 1 is from a given binary label. In general, functions to measure the error between what we want and our predictions are called **loss functions**.3. **SGD** simply stands for **Stochastic Gradient Descent**. It as the algorithm that tries to find better values for our model. Algorithms like SGD are called optimizers, and they work by using the output of our loss function on batches of data to estimate the amount of error. With the computed error, they change the parameters of the model slightly. The scale of this change is controlled by the **learning rate**. In our case, we have not specified the learning rate, leaving it to its default value.4. The **epochs** are the number of times we will go through the dataset. It will be processed in **batches**, the samples of data that we mentioned in the previous bulletpoint, each of a fixed, given size. If we don't fix the random seed, the batches that we will get every time we train will be different. This means that different executions are likely to producWith that out of the way, let's look at the plane resulting from our hard work and reap the fruits of our labor:<jupyter_code># get the vector and bias
(v, b) = model.get_layer('output_f').get_weights()

# plot the prediction and the separating hyperplane
y_pred = model.predict(X).round()
c = [color_fn(l, r) for l, r in zip(y_pred, y)]
plt.scatter(X[:, 0], X[:, 1], marker='.', c=c)
plot_hyperplane(v[0], v[1], b)

# report some stats
wrong = 100.0 * sum(l == 'y' for l in c) / len(c)
reds  = 100.0 * sum(l == 'r' for l in c) / sum(1 for l in y if l)
blues = 100.0 * sum(l == 'b' for l in c) / sum(1 for l in y if not l)
print('We got {:.2f}% of the points wrong, correctly matching {:.2f}% of the red points and {:.2f}% of the blue ones.'.format(wrong, reds, blues))<jupyter_output><empty_output><jupyter_text>Well, that was a lot of work for nothing — we did much better by hand! However, we have to keep in mind that we still only have one single plane and now we have many parameters to play with. Before proceeding into the next section, change up the parameters of the network: perhaps try more epochs or smaller batches. Play around with Keras, and perhaps check out the [fantastic documentation](https://keras.io). Then leap forward.Our initial question is still unanswered: how do we effectively separate the blue from the red?![Challenge for linear separation](images/TargetSeparate.png)  Putting the Network in Neural Networks!Our single perceptron, the simplest possible neural network, is really not much of a network. Indeed, we can see it as just a single node, receiving a vector and giving us an output. So far, however, we have seen little network action in our classes! Indeed, if we look at the picture of our single perceptron, it is just a single node taking a few inputs and producing some output.If you tend to think out of the box, you might have thought of separating the red and the blue in the challenge plot by just adding more planes. If you did, you were absolutely right! We can try to approximate the overall space with linear approximations. However, allow me to spark your imagination with a new friend: the multilayer perceptron!![Multilayer Perceptron](images/MultiLayerPerceptron.png)To avoid clutter, we are not drawing the bias components here. Aside from that, it is a simple construct. We have 3 input components, from which we are trying to find a single output label. That is what we see at the left and right of the diagram, respectively. However, we have something in the middle: those are called hidden layers. In essence, we want our system to learn some relations from the 3 input values, encode them as 4 intermediate numbers and then correctly predict the label just using those!In our problem, this translates to adding more planes to help classify the shape we have! Before we can deep dive into model parameters, let's introduce two more activation functions to compliment $tanh$ and $sigmoid$:<jupyter_code>space = np.linspace(-10, 10)
f, axarr = plt.subplots(2)
axarr[0].grid(True)
axarr[1].grid(True)
axarr[0].plot(space, np.maximum(0, space))
axarr[0].set_title('Two new activations: ReLU (top) and Linear (bottom)')
axarr[1].plot(space, space)
plt.show()<jupyter_output><empty_output><jupyter_text>These two functions are called ReLU and Linear, respectively. ReLU stands for Rectified Linear Unit, and simply turns any negative number into 0. Linear is the simplest possible activation: it does absolutely nothing! Why introduce those dead simple activations now? They will be part of our interactive activation zoo, in which we will try to tune our deep red-and-blue model!Let's start by playing around with the number of units in the hidden layer and the activation function that applies to it:<jupyter_code>def make_deep_model(num_epochs, hidden_activation, hidden_size, hidden_number):
    '''Create a deep neural model with:
    
        - an activation function for its units,
        - the size of all hiden layers and
        - a number of hidden layers (all of the same size)
        
    and train it for a number of epochs.'''
    # define the input -- two component vector
    inp = Input((2,), name='input_pair')
    
    # clamp to minimum one layer, one unit/layer
    hidden_size = max(1, hidden_size)
    hidden_number = max(1, hidden_number)
    
    # put the layers
    mod = inp
    for i in range(1, hidden_number + 1):
        mod = Dense(hidden_size, name='hidden_{}'.format(i), activation=hidden_activation)(mod)
    out = Dense(1, name='output_f', activation='sigmoid')(mod)

    # define the model
    model = Model(inputs=inp, outputs=out)
    model.compile(loss='binary_crossentropy', optimizer='sgd', metrics=['accuracy'])
    model.summary()
    model.fit(X, y, epochs=num_epochs, batch_size=2000, verbose=0)
    
    # plot the predictions
    y_pred = model.predict(X).round()
    c = [color_fn(l, r) for l, r in zip(y_pred, y)]
    plt.scatter(X[:, 0], X[:, 1], marker='.', c=c)
    
    # report some stats
    wrong = 100.0 * sum(l == 'y' for l in c) / len(c)
    reds  = 100.0 * sum(l == 'r' for l in c) / sum(1 for l in y if l)
    blues = 100.0 * sum(l == 'b' for l in c) / sum(1 for l in y if not l)
    print('We got {:.2f}% of the points wrong, correctly matching {:.2f}% of the red points and {:.2f}% of the blue ones.'.format(wrong, reds, blues))

activations = ['linear', 'relu', 'sigmoid', 'tanh']
interact(make_deep_model, 
         num_epochs=widgets.BoundedIntText(value=25, step=1, min=1, max=10000, description='Epochs: '),
         hidden_activation=widgets.Select(value='linear', options=activations, description='Activation: '),
         hidden_size=widgets.BoundedIntText(value=4, step=1, min=1, max=1000, description='Hidden size: '),
         hidden_number=fixed(1));<jupyter_output><empty_output><jupyter_text>After tinkering for a while, you should see that too many hidden units is not helpful for this problem. We don't have so much data after all! Similarly, too few of them and we can't represent a lick of what we want to capture. Furthermore, the choice of activation really matters a lot. Each training run is different, so the same parameters also give us all kinds of results. If we run few epochs, training takes little time but the network doesn't learn much. If instead we set a high number of epochs, it takes noticeably long.In general, we are not getting impressive results. Can we generalize the idea of this **hidden layer**? Could we put several of them? Let's give that a shot, selecting the number of hidden layers, their activation and their number of units:<jupyter_code>activations = ['linear', 'relu', 'sigmoid', 'tanh']
interact(make_deep_model, 
         num_epochs=widgets.BoundedIntText(value=25, step=1, min=1, max=10000, description='Epochs: '),
         hidden_activation=widgets.Select(value='linear', options=activations, description='Activation: '),
         hidden_size=widgets.BoundedIntText(value=4, step=1, min=1, max=1000, description='Hidden size: '),
         hidden_number=widgets.BoundedIntText(value=1, step=1, min=1, description='Nr. of layers: '),);<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09154342388203801, 'fraction_numerical': 0.014745249484355115, 'mean_word_length': 4.016068677085626, 'pattern_counts': {'":': 0, '<': 47, '<?xml version=': 0, '>': 47, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '617531', 'license': 'MIT', 'n_tokens_mistral': 6862, 'n_tokens_neox': 6530, 'n_words': 3281, 'path': 'Lesson 1 - Neural Networks 101/1 - Neural Networks 101.ipynb', 'repo_name': 'nur-ag/neural-networks-for-nlp-course', 'size': '22787'}