<jupyter_start><jupyter_text>Neural Networks: IntroductionThis is the start of a guide I wrote to help people get into neural networks. In this part we'll start by having a look at what it means to "learn from data", and we'll also look at the basic feed forward network's structure. Further reading* An introduction to modeling data, from [Introduction to Statistical Learning with R](https://www.youtube.com/playlist?list=PL5-da3qGB5ICcUhueCyu25slvsGp8IDTa)* Andrew Ng Machine Learning Coursera, [weeks 4 and 5](https://www.coursera.org/learn/machine-learningsyllabus)* For a good foundation in machine learning, watch all of Andrew Ng's course on [Coursera](https://www.coursera.org/learn/machine-learning) or [Youtube](https://www.youtube.com/playlist?list=PLZ9qNFMHZ-A4rycgrgOYma6zxF4BZGGPW)* Elements of Statistical Learning: Chapter 11, Neural Networks ([get a copy](https://web.stanford.edu/~hastie/ElemStatLearn/))* Computer Age Statistical Inference: Chapter 18, Neural Networks and Deep Learning (p. 351) ([get a copy](https://web.stanford.edu/~hastie/CASI/))* [3Blue2Brown's neural network videos.](https://www.youtube.com/playlist?list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi) Introduction What this coversThis covers the classic neural network, called the mutli-layer perceptron model, the feed-forward model, or the densely-connected neural network. The feed forward model is a network .What's the point of the feed forward model? It acts like an overcharged logistic regression, powerful enough to do some basic machine learning tasks. It's a favourite project for beginners because of its basic construction.The feed forward is related to the logistic regression, a model that learns to return a true/false depending on the data it's seeing. When many of these logistic regressions are combined together, they form a feed forward network with extra power. This extra power lets the neural network reach more complicated true/false decisions.Training multiple logistic regressions simultaneously is challenging, which is why the gradient descent algorithm was invented to train these models little-by-little.![Regression Relation](Images/intro_logit_and_mlp.png "Regression Relation")The feed forward model's power is also a curse. Because of its added complexity, it's difficult to know exactly what the feed forward model is up to. In comparison, it's easy to see what a single logistic regression is doing.The feed forward network's power is sometimes not enough. For example, in image recognition tasks it is overtaken by convolutional neural networks, which can process visual data a lot better.Here is a short glossary you can refer to if you run into unknown terms here or elsewhere:* Model: a reproduction of a real-life phenomenom. A painter can paint a landscape, and a writer can write a novel about it. The raw material for the statistical model is data. Using modelling techniques, it is possible to find the patterns in data and create a simulation of it. The feed forward network is a model that learns to simulate data.* Loss, cost, objective function: how to measure error. For example, the mean-squared error treats error as the squared ($x^2$) difference between the modeled data and real data. With squared error, the cost increases faster and faster as the difference between prediction and truth grows.* Gradient descent: the mathematical technology used to train neural networks. Performing gradient descent once will reduce model error by a *very* small amount. Performing gradient descent thousands of times *usually* gives a well-trained neural network.* Neuron, perceptron: neurons/perceptrons are the building blocks of the neural network. **Caution**: a perceptron looks identical to the logistic regression, but the two have different philosophies: perceptrons are used to train machines, logistic regressions are used for statistics. Importing librariesTo get started, we'll need to import `numpy` to deal with all the matrices involved.Each NN library you use will have a way of handling matrices. They tend to be similar and might even just work with `numpy` matrices seemlessly.<jupyter_code>import numpy as np
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Neural networks simulate realityWhile the goal of machine learning is to teach machines, machines so far mostly just learn to simulate data. In this way, they're related to other statistical modelling techniques, and it helps comprehension to think of them this way.The first part of statistical modeling is in the data itself: its underlying patterns and relationships. Statistical modelling algorithms are designed to detect these patterns and recreate them as mathematical formulas. These formulas, just like neural networks, can then be used to do useful things.What we'll be talking about here is supervised learning. With supervised learning, you generally have $X$ features/inputs and target/output $Y$ variable(s).When training the model, you have both $x$ and $y$. After training is done, you will only have $x$, and you will want to predict $y$ using only $x$. The $x$ are usually things that are easy to get, like pictures, while $y$ is something more difficult to find, like whether these pictures have cats or not.I will give you an example of an exploitable pattern between $x$ and $y$. I will take $x$ to mean **age of random people** and $y$ to represent **height of these random people**.<jupyter_code># create random Xs
X = np.random.uniform(2, 22, size=(100, 1))

# create y as a combination of X, a minimum height, and some random noise
y = 50 + 0.05*X + 0.2*(X**2) + np.random.normal(loc=0.0, scale=10.0, size=(100, 1))

plt.scatter(X, y)
plt.xlabel("age (years)")
plt.ylabel("height (cm)")
plt.show()<jupyter_output><empty_output><jupyter_text>This data is a little weight since there are some very tall toddlers, but there you go: some data.Here is a glossary of some statistical concepts at play here.* Population: the small amount of people we're looking at comes from a larger population. If we wanted to look at another 100 people, we'd find another 100 in the population of humans on Earth. In machine learning, you will probably find your populations through Google searches: either there is a website/API you can scrape or a respository of datasets you can download from. In either case, there is probably always going to be more data *out there*. That's the concept of population.* Data generation process: we generated fake data with the formula $50 + 0.05 \cdot X + 0.2 \cdot (X^2) + noise$. **In reality this formula is unknown.** Here the relationship between age and height we could write down with math; in fact, even the "unexplainable" we approximated with random noise. The data generation process can be very complicated, such as with photos, which is the reason people use neural networks. Statistical modeling and machine learning are able to predict new data because they learn to simulate these data generation processes.* The model: when you look at the graph above, you see a shape. When a modeling technique is applied to the data, it also sees a pattern. When you have access to the data but *not the data generation formula*, you can still say something meaningful about what you see in the data in front of you. This is statistical modeling and machine learning.The data generation process *is* reality; the statistical model *approximates* reality.When you use the power of the neural networks, you are using them to approximate a very complicated reality.To close off this section, I'll model the data above with a statistical modeling technique called the linear regression. By looking at the graph, I can already guess that the model will have the shape $\hat{y} = \alpha + \beta_1 \cdot X + \beta_2 \cdot X^2$.I hoping to get $\alpha = 50$, $\beta_1 = 0.05$, and $\beta_2 = 0.2$.<jupyter_code># Set up my guesstimated formula
X_train = np.concatenate((np.ones((100, 1)), X, X**2), axis=1)

# I will use something called the normal equation to estimate the linear regression model
coefficients = np.linalg.inv(X_train.transpose() @ X_train) @ X_train.transpose() @ y

# As you can see, the linear regression will probably get some coefficients right
print("alpha = {0}, beta_1 = {1}, beta_2 = {2}".format(coefficients[0,0], 
                                                       coefficients[1,0], 
                                                       coefficients[2,0]))<jupyter_output>alpha = 61.50628830146805, beta_1 = -2.0252335788412568, beta_2 = 0.2771090271209202<jupyter_text>How the feed forward is constructed The basic unit: the perceptronThe following diagram shows a perceptron.![Just a lonely neuron](Images/intro_perceptron.png "Just a lonely neuron")It's a building block that "learns" to return a true/false signal. The perceptron is a collections of parameters (synapses) and an activation function (sigmoid here). Data comes in and data comes out.The perceptron is like a logistic regression, except that the perceptron was designed to be modular and stack on top of other perceptrons. Feed forward basicsA neural network is a stack of layers of perceptrons working together, and the term feed forward refers to "feeding" data through this network from start to finish. As you can see with the arbitrary example below, that can be a lot of feeding.![A lot](Images/intro_ManyNeurons.png "A lot")The diagram above is just to scare you. Neural networks typically have *a lot* more connections![*Drawn with this python script*](https://gist.github.com/craffel/2d727968c3aaebd10359file-draw_neural_net-py)We'll try to avoid these diagrams when we can. They can get very busy.  Values flowingIn a neural network, values flow through layers of synapses and neurons. This is called feed-forward.Everything is passed forward to whatever is down the line. A neuron like the one below feeds information to many other neurons in the network.![Everything flows](Images/intro_perceptron_feed.png "Everything flows")To feed-forward data through a neural network is to pass data through the network's weights and activation functions to create an output. The feed-forward receives its data at the input layer, a copy of the input. The activity begins at the first hidden layer, when the input signal is passed through synapses (weights), and adjusted (bias) and transformed (activation function) by the neuron. Here is what it does: First hidden layer1. Signal $x$ passing though weights $w_1$: $x \times w_1$2. Signal adjusted by neuron bias $b_1$: $x \times w_1 + b_1 = z_1$3. Signal shaped by activation function $\sigma$: $a_1 = \sigma(z_1)$4. Signal ready for rest of networkLet's look at that more closely.The $w_1$ matrix (weights) is the collection of "synapses" of the "neurons": they are the connections the neurons use to pull in data. These synapses can be increased to amplify an incoming variable, set to zero to ignore one, or made negative to invert the inbound signal. During training they are tuned by the neurons to help the NN minimize prediction error. Most of the work is done by these weights.The biases $b_1$ are unique to each neuron. They're used by the neuron to adjust what they receive.The activation function $\sigma$, the sigmoid function, causes the neuron to output a binary signal. Unlike a digital computer though, the signal can range *between* 0 and 1 if the neuron is unsure.When you look at the graph below, the result of the weights and bias applied to the data is the x-axis, and the resulting sigmoid signal is seen along the y-axis. The higher $x$ is, the closer $y$ gets to 1.<jupyter_code>x=np.linspace(-10,10,100)
plt.plot(x,1/(1 + np.exp(-x)))
plt.title('Sigmoid activation function')
plt.show()<jupyter_output><empty_output><jupyter_text>* Synapses strengthen or weaken incoming variables with their weights* Biases adjust the sum of these weighted signals* Neuron does a weighted sum of everything and transforms it with its activation function Final point: the neural network as a functionA function is a mathematical invention that maps one space into another. The neural network is a function because it takes data from its inputs and transforms it into output data.$$NN(x) = \hat{y}$$People call neural networks "universal function approximators", meaning that if you made a neural network big enough it'd be able to recreate anything. You can see this below.Let's set up a simple neural network:$$NN(x) = \sigma( \sigma(x \cdot W_1 + b_1) \cdot W_{out} + b_{out} ) = \hat{y}$$If you keep re-running the cell below, it'll keep coming up with weird functions.<jupyter_code>def sigmoid(x): return 1/(1 + np.exp(-x))

b1 = np.random.uniform(low=-5, high=5, size=(1,1)) # Bias: 1 x layer_1_size
w1 = np.random.uniform(low=-5, high=5, size=(1,3)) # Weight: input_vars x layer_1_size
b_out = np.random.uniform(low=-1, high=1, size=(1,1)) # Bias: 1 x output_size
w_out = np.random.uniform(low=-2, high=2, size=(3,1)) # Weight: layer_1_size x output_size

x = np.linspace(-10,10,100)
x_data = np.reshape(x, (100,1))
plt.plot(x, sigmoid(sigmoid(x_data.dot(w1) + b1).dot(w_out) + b_out) )
plt.title('Sigmoid activation function')
plt.show()<jupyter_output><empty_output><jupyter_text>Actually, let me drive the point home by adding another hidden layer to the above and increasing the neuron count to 200+.<jupyter_code>def sigmoid(x): return 1/(1 + np.exp(-x))

b1 = np.random.uniform(low=-10, high=10, size=(1,100)) # Bias: 1 x layer_1_size
w1 = np.random.uniform(low=-10, high=10, size=(1,100)) # Weight: input_vars x layer_1_size
b_2 = np.random.uniform(low=-10, high=10, size=(1,100)) # Bias: 1 x layer_2_size
w_2 = np.random.uniform(low=-10, high=10, size=(100,100)) # Weight: layer_1_size x layer_2_size
b_out = np.random.uniform(low=-0.1, high=0.1, size=(1,1)) # Bias: 1 x output_size
w_out = np.random.uniform(low=-0.1, high=0.1, size=(100,1)) # Weight: layer_2_size x output_size

x = np.linspace(-10,10,100)
x_data = np.reshape(x, (100,1))
plt.plot(x, sigmoid(sigmoid(sigmoid(x_data.dot(w1) + b1).dot(w_2) + b_2).dot(w_out) + b_out) )
plt.title('Sigmoid activation function')
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07773776742215632, 'fraction_numerical': 0.021817411565346324, 'mean_word_length': 4.845645893520429, 'pattern_counts': {'":': 1, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 8, 'lorem ipsum': 0, 'www.': 5, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '583539', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 4109, 'n_tokens_neox': 3764, 'n_words': 1929, 'path': 'Neural Networks/1_Introduction.ipynb', 'repo_name': 'sjchiass/ml_guides', 'size': '14163'}