<jupyter_start><jupyter_text>- title: Individual Neurons- summary: Simple math- date: 2019-03-03- image: /images/singleneuron1.png- status: draft Artificial Neural Networks apply simple math, over and over, to achieve their fantastic results. Their individual pieces are simple, and they combine in a simple way. Let's look at one of those pieces. A single neuronHere's a schematic picture of a single neuron:![Single Neuron IO]({static}/images/singleneuron1.png)Its mechanics are inspired by biological neurons such as those in your brain, each of which aggregates input from some other neurons and then decides whether or not to "fire" its output.An artificial neuron's inputs are numbers (here $x_1$, $x_2$, $x_3$...), and the output is a single number (here $y$). It does very simple math to decide if the incoming inputs are enough for it to activate and fire its own output. What each neuron does1. Each input has its own "weight", which is a number that determines how important that input is to this neuron. Weighing an input is as simple as multiplying the input by its associated weight. In this example, $x_1\times w_1$, $x_2 \times w_2$, and $x_3 \times w_3$ give us the weighted versions of each input:![Single Neuron Weights]({static}/images/singleneuron2.png)2. Then the neuron adds all of the weighted inputs together. $x_1w_1+x_2w_2+x_3w_3$:![Single Neuron Summation]({static}/images/singleneuron3.png)3. The last thing it does is wrap the whole sum in some non-linear function, e.g. $f(x_1w_1+x_2w_2+x_3w_3)$. This function can be anything non-linear (that is, the function _isn't_ a line), but the most popular one is also super simple: $f(z)=max(0,z)$. That is, it replaces all negative numbers with zero so the output of the neuron can never be negative. The function $f(z)=max(0,z)$ is called a Rectified Linear Unit (ReLU). You'll see this a lot.![Single Neuron Nonlinearity]({static}/images/singleneuron4.png)4. And that's it. The output of a single neuron is ultimately $y = f(\sum\limits_{i} w_ix_i)$![Single Neuron Math]({static}/images/singleneuron5.png)An artificial neuron _is_ this equation $y = f(\sum\limits_{i} w_ix_i)$. It is not more complicated than that:<jupyter_code># We'll do examples with these soon

def relu(z):
    return max(z,0)

def neuron (w1,w2,w3, x1,x2,x3):
    f = relu
    return f(x1*w1 + x2*w2 + x3*w3)<jupyter_output><empty_output><jupyter_text>How could that _possibly_ be enough?This may seem quite anticlimactic, and like nothing much has been explained. After all, neural networks can drive cars, play video games, recognize (and generate) human faces, and understand human speech, etc. If each neuron is so simple, and a neural network (as we will see) is just a bunch of neurons arranged in layers, then where does the magic happen?Since this is the post on individual neurons, I'll give the single-neuron answer to that now, and the many-neurons answer to that later. In short, you can shove a lot of computational power into that simple equation, including logic gates and the weighing of evidence. Neurons can implement universal logicIt's just a matter of wiring things up and picking the right weights.You can implement AND and NOT gates with a single neuron apiece. This is significant, because the combination of these two gates is universal for computation. That is, you can build a whole computer if you just have these two gates, and we do.![AND Gate]({static}/images/AND_gate.png)<jupyter_code>AND = lambda IN1,IN2: neuron(w1=-1,w2=1,w3=1, x1=1,x2=IN1,x3=IN2)

print("AND(0,0) =", AND(0,0))
print("AND(0,1) =", AND(0,1))
print("AND(1,0) =", AND(1,0))
print("AND(1,1) =", AND(1,1))<jupyter_output>AND(0,0) = 0
AND(0,1) = 0
AND(1,0) = 0
AND(1,1) = 1<jupyter_text>![NOT Gate]({static}/images/NOT_gate.png)<jupyter_code>NOT = lambda IN1: neuron(w1=1,w2=0,w3=-1, x1=1,x2=0,x3=IN1)

print("NOT(0) =", NOT(0))
print("NOT(1) =", NOT(1))<jupyter_output>NOT(0) = 1
NOT(1) = 0<jupyter_text>Neurons can represent the weighing of evidenceNeurons are also capable of expressing _more_ than binary logic. They can represent gradations of certainty and real values. Here a neuron weighs the evidence that the incoming numbers represent a cat, and outputs not just a binary determination, but a degree of "catness".Again, it's just a matter of wiring things up and picking the right weights.![Catness Components]({static}/images/catness_components.png)<jupyter_code>catness = lambda pt, fm, bdn: neuron(w1=0.5,w2=0.2,w3=-0.5, x1=pt,x2=fm,x3=bdn)

# Say some image has these values...
pointy_topness = 70
fuzzy_middleness = 80
big_dark_noseness = 2

result = catness(pointy_topness, fuzzy_middleness, big_dark_noseness)

print("catness =", result)<jupyter_output>catness = 50.0
{'fraction_non_alphanumeric': 0.10141559264736953, 'fraction_numerical': 0.028945700401436722, 'mean_word_length': 5.2701986754966885, 'pattern_counts': {'":': 0, '<': 14, '<?xml version=': 0, '>': 14, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '432491', 'license': 'MIT', 'n_tokens_mistral': 1591, 'n_tokens_neox': 1500, 'n_words': 617, 'path': 'content/icebox/Individual Neurons.ipynb', 'repo_name': 'computableai/computableai.github.io', 'size': '4733'}