<jupyter_start><jupyter_text>Neural Boolean Connectives 1** November 2017 **** <NAME> @ [AndrewRib.com](http://www.andrewrib.com) **A notebook on understanding how a single layer neural network can represent the operation of common logical operators. ** Note: ** This notebook contains interactive elements and certain latex snippets that will not render in github markdown. You must run this notebook on your local Jupyter notebook environment for interactive elements or render or if you wish to render just the latex by using the url of this repo with the [online NBViewer](https://nbviewer.jupyter.org/). Libraries<jupyter_code>import numpy as np

#Interactive Components
from ipywidgets import interact<jupyter_output><empty_output><jupyter_text>Boolean ConnectivesBoolean connectives are logical operators that connect two boolean statements in a particular way. Amoung the fundamental few are and, or, implication, and xor. We can define the operations by truth tables as seen here:A | B | A ∧ B--- | --- | ---0 | 0 | 00 | 1 | 01 | 0 | 0 1 | 1 | 1A | B | A ∨ B--- | --- | ---0 | 0 | 00 | 1 | 11 | 0 | 11 | 1 | 1A | B | A → B--- | --- | ---0 | 0 | 10 | 1 | 11 | 0 | 01 | 1 | 1A | B | A ⊻ B--- | --- | ---0 | 0 | 00 | 1 | 11 | 0 | 11 | 1 | 0  Learning XORHere we simply implement the results in section 6.1 of the book. We are not really learning XOR here. We are using the results of some learning process to show that given some matrix X, we can transform it into a vector which represents the XOR operation on each row of X.<jupyter_code>def relu(z):
    return np.maximum(0,z)

def linLayer(x,w,b):
    return x.T*w + b

# x: 2 by 4
# W: 2 by 2
# c: 2 by 1
# w: 2 by 1
# b: scalar

def fullNetwork(x,W,c,w,b):
    h = relu( linLayer(W,x,c) )
    return linLayer(w, h, b)
W = np.matrix([[1,1],[1,1]])
x = np.matrix([[0,0,1,1],[0,1,0,1]])
c = np.matrix([0,-1]).T

print("x.T\n")
print(x.T)
print("\nx.T*W\n")
print(x.T*W)
linLayH = linLayer(W,x,c)
print("x.T*W + c\n")
print(linLayH.T)
h = relu( linLayH )
print("relu( x.T*W + c )\n")
print(h.T)
w = np.matrix([1,-2]).T
b = 0

f = linLayer(w, h, b)

print("w.T*relu( x.T*W + c )+b\n")
print(f.T)<jupyter_output>w.T*relu( x.T*W + c )+b

[[0]
 [1]
 [1]
 [0]]<jupyter_text>If learning XOR was a linear transformation there would be an M such that x.T * M = [ 0,1,1,0]. There is no such M, however. Below you will find an interactive section for exploring M space ( 1 by 2 matrix ).<jupyter_code>def exploreM(m1=1,m2=-2):
    M = np.matrix([m1,m2]).T
    print("M:")
    print(M)
    print("\nx:")
    print(x.T)
    print("\nDesired x.T * M:")
    print(np.matrix([0,1,1,0]).T)
    print("\nx.T * M:")
    print(x.T*M)

interact(exploreM,m1=[-10,10],m2=[-10,10])<jupyter_output>M:
[[ 1]
 [-2]]

x:
[[0 0]
 [0 1]
 [1 0]
 [1 1]]

Desired x.T * M:
[[0]
 [1]
 [1]
 [0]]

x.T * M:
[[ 0]
 [-2]
 [ 1]
 [-1]]<jupyter_text>Representing ANDWe'd like to transform matrix X representing all possible values for the opperands of AND( ∧ ) into a vector representing the value of applying AND to those opperands:$$\begin{bmatrix} 0 & 0 \\0 & 1 \\1 & 0 \\1 & 1 \\\end{bmatrix}\overset{∧}{\rightarrow}\begin{bmatrix} 0  \\0 \\0  \\1  \\\end{bmatrix}$$$$w^\intercal relu(\begin{bmatrix} 0 & 0 \\0 & 1 \\1 & 0 \\1 & 1 \\\end{bmatrix}^\intercal W + c)+b = \begin{bmatrix} 0 & 0 & 0 & 1 \end{bmatrix}^\intercal$$As you can see, the task is to find matricies W,w,c, and scalar b which transforms our x matrix into the solution desired. Let's try to do this by hand to explore how the various variables behave in respect to transforming X.We first look at how a matrix of ones is transformed by different values of W.<jupyter_code>x = np.matrix([[1,1,1,1],[1,1,1,1]])<jupyter_output><empty_output><jupyter_text>W<jupyter_code>def wInt(w1,w2,w3,w4):
    W = np.matrix([[w1,w2],
                   [w3, w4]])
    print(x.T*W)
    
interact(wInt,w1=[-5,5],w2=[-5,5],w3=[-5,5],w4=[-5,5])<jupyter_output>[[0 0]
 [0 0]
 [0 0]
 [0 0]]<jupyter_text>From experimenting above we can see that: * w1: Modifies the first column values. * w3: Modifies the first column values. * w2: Modifies the second column values* w4: Modifies the second column values Now let's do the same experiment with the x matrix set to the desired x.<jupyter_code>x = np.matrix([[0,0,1,1],[0,1,0,1]])
interact(wInt,w1=[-5,5],w2=[-5,5],w3=[-5,5],w4=[-5,5])<jupyter_output>[[0 0]
 [0 0]
 [0 0]
 [0 0]]<jupyter_text>From experimenting above we can see that: * w1: Modifies the first column values in **the last two rows**. * w2: Modifies the second column values in **the last two rows**.* w3: Modifies the first column values in **the second and forth rows**.* w4: Modifies the second column values in **the second and forth rows**.By experimenting with the applet above, we can find a matrix that makes the last column distinctly larger than all others. Since we'd like to transform this matrix to [0,0,0,1], this is a property we need. I found a W matrix of the form [0,1],[0,1] that seems to do the job.<jupyter_code>W = np.matrix([[0,1],[0,1]])
print(x.T*W)<jupyter_output>[[0 0]
 [0 1]
 [0 1]
 [0 2]]<jupyter_text>cC acts as a sort of filter on W. We can use this to get rid of the 1's above and keep the last row a 1.<jupyter_code>c = np.matrix([0,-1]).T
linLayH = linLayer(W,x,c)
print("x.T*W + c\n")
print(linLayH.T)<jupyter_output>x.T*W + c

[[ 0 -1]
 [ 0  0]
 [ 0  0]
 [ 0  1]]<jupyter_text>Nice! We can see that when applying an element wise relu will get us what we'd like.<jupyter_code>h = relu(linLayH)
print(h.T)<jupyter_output>[[0 0]
 [0 0]
 [0 0]
 [0 1]]<jupyter_text>Now we must figure out the other variable values to flatten this matrix to a vector.  wGiven the results from above, we know we can ignore b and just find a w that turns the matrix above to a vector. We do that with a column vector [0,1].T<jupyter_code>w = np.matrix([0,1]).T
fn = linLayer(w, h, 0)
print(fn.T)<jupyter_output>[[0]
 [0]
 [0]
 [1]]
{'fraction_non_alphanumeric': 0.1576637407157326, 'fraction_numerical': 0.0462525320729237, 'mean_word_length': 3.73621103117506, 'pattern_counts': {'":': 0, '<': 34, '<?xml version=': 0, '>': 34, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '464418', 'license': 'MIT', 'n_tokens_mistral': 2415, 'n_tokens_neox': 2246, 'n_words': 803, 'path': 'Notebooks/Neural Boolean Connectives 1.ipynb', 'repo_name': 'Andrewnetwork/WorkshopScipy', 'size': '5932'}