<jupyter_start><jupyter_text>Deep Learning From ScratchPerceptron        <NAME>        www.data4sci.com            @bgoncalves, @data4sci<jupyter_code>import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D

import watermark

%load_ext watermark
%matplotlib inline
%watermark -i -n -v -m -g -iv<jupyter_output>numpy      1.16.2
matplotlib 3.1.0
watermark  1.8.1
Tue Sep 24 2019 2019-09-24T11:07:59-04:00

CPython 3.7.3
IPython 6.2.1

compiler   : Clang 4.0.1 (tags/RELEASE_401/final)
system     : Darwin
release    : 18.7.0
machine    : x86_64
processor  : i386
CPU cores  : 8
interpreter: 64bit
Git hash   : 46e8ada9ae78bc5ab280631b85d64d9ea1891da7<jupyter_text>Logic functions Let's start by setting up our training examples. We'll consider the simple 4 point binary example for binary logic<jupyter_code>X = np.ones((4, 3), dtype='float')
X[1, 2] = 0
X[2, 1] = 0
X[3, 1] = 0
X[3, 2] = 0<jupyter_output><empty_output><jupyter_text>The first column is just the bias, always set to 1.<jupyter_code>X<jupyter_output><empty_output><jupyter_text>We'll take a look at two examples, *AND* and *OR*<jupyter_code>y_AND = [1, 0, 0, 0]
y_OR = [1, 1, 1, 0]<jupyter_output><empty_output><jupyter_text>Prediction and Training The prediction function is simple. Just predict 1 if the activation value is larger than 0 and 0 otherwise<jupyter_code>def predict(weights, inputs):
    return int(np.dot(weights, inputs) > 0)<jupyter_output><empty_output><jupyter_text>The training algorithm is also simple:- If the prediction is correct, do nothing- If the prediction is wrong, add/subtract the input vector<jupyter_code>def train(weights, X, y, epochs = 100):
    for _ in range(epochs):
        for i in range(len(y)):
            inputs = X[i, :]
            label = y[i]
            
            prediction = predict(weights, inputs)
            weights += (label - prediction) * inputs<jupyter_output><empty_output><jupyter_text>Perceptron In this simple case, our _perceptron_ is just a vector of 3 weights that we initialize to zero<jupyter_code>weights_AND = np.zeros(3)
weights_OR = np.zeros(3)<jupyter_output><empty_output><jupyter_text>We can train them both quickly<jupyter_code>train(weights_AND, X, y_AND)
train(weights_OR, X, y_OR)<jupyter_output><empty_output><jupyter_text>And take a look at the resulting weights<jupyter_code>weights_AND
weights_OR<jupyter_output><empty_output><jupyter_text>Visualization Let's define some helper functions. First, one to draw the decision surface<jupyter_code>def surface(weights, n=20):
    points = np.linspace(0, 1, n)
    xs = []
    ys = []
    zs = []
    
    for i in range(n):
        x = points[i]
        for j in range(n):
            y = points[j]
            
            point = [1, x, y]
            
            xs.append(x)
            ys.append(y)
            zs.append(np.dot(weights, point))
            
    return np.array(xs), np.array(ys), np.array(zs)<jupyter_output><empty_output><jupyter_text>And a function to plot the perceptron output<jupyter_code>def plot_output(weights, X, y, level=0, label='AND function'):
    fig = plt.figure()
    ax = fig.add_subplot(111, projection='3d')

    xs, ys, zs = surface(weights)

    colors = np.array(['blue']*xs.shape[0])
    selector = (zs>=-1) & (zs<=1)
    colors[zs>0] = 'red'
    
    ax.scatter(X[:, 1], X[:, 2], y, c='gold', marker='*', s=1000, depthshade=False)
    ax.scatter(xs[selector], ys[selector], zs[selector], c=colors[selector], marker='.')
    grids = np.linspace(0, 1, 6)
    
    for i in range(6):
        ax.plot([0, 1], [grids[i], grids[i]], [level, level], 'darkgray')
        ax.plot([grids[i], grids[i]], [0, 1], [level, level], 'darkgray')

    ax.set_xlabel('X')
    ax.set_ylabel('Y')
    ax.set_zlabel('output')
    ax.set_title(label)

    fig.set_size_inches(11, 11)<jupyter_output><empty_output><jupyter_text>And Function<jupyter_code>plot_output(weights_AND, X, y_AND, 0, 'AND function')<jupyter_output><empty_output><jupyter_text>OR function<jupyter_code>plot_output(weights_OR, X, y_OR, 1, 'OR function')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12840466926070038, 'fraction_numerical': 0.03769455252918288, 'mean_word_length': 3.908114558472554, 'pattern_counts': {'":': 0, '<': 54, '<?xml version=': 0, '>': 56, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '187902', 'license': 'MIT', 'n_tokens_mistral': 1631, 'n_tokens_neox': 1528, 'n_words': 367, 'path': 'Perceptron.ipynb', 'repo_name': 'annalizaantonio/DeepLearning', 'size': '4121'}