<jupyter_start><jupyter_text>Grokking Deep Learning> <NAME> by <NAME> ------ Chapter 2: Fundamental Concepts* What are deep learning, machine learning, and artificial intelligence?* What are parametric models and nonparametric models?* What are supervised learning and unsupervised learning?* How can machines learn? --- What is deep learning?* Deep learning is a subset of machine learning  * Subset of methods in the ML toolbox primarily using ANNs What is machine learning?* Subfield of computer science* Deals with teaching machines to perform tasks for which they were not explicitly programmed* Machines observe a pattern and attempt to imitate it, either directly or indirectly  * i.e. supervised or unsupervised --- Supervised vs unsupervised Supervised* Transform one dataset into another  * Input is what is known  * Output is what one wants to know   Unsupervised* Also transforms one dataset into another* However, the output dataset is not previously known* No "right answer"  * Finding patterns in the data* Ex: clustering datapoints into cluster labels> In general, all forms of unsupervised learning are ~forms of clustering --- Parametric vs nonparametric learning* Supervision ~ the type of pattern being learned* Parametricism ~ the way the learning is stored and the method for learning* Characterization by number of parameters  * Parametric model has a fixed number of parameters    * Tend to use trial and error  * Nonparametric model has potentially infinite number of parameters    * Tend to use counting     Supervised parametric learning* "Trial and error learning using knobs"  * Trial-and-error is a common property of parametric models (with exceptions)* Machines with a fixed number of knobs (parameters)  * Learning means turning the knobs* Input data is processed based on the settings of the knobs, transformed into a prediction* Steps  * 1. Predict  * 2. Compare to truth pattern  * 3. Learn the pattern   Unsupervised parametric learning* Uses knobs to group data  * Knobs in each group map the input's "affinity" to a particular group* Each group's machine transforms the input data to number 0-1  * Probability that the input is a member of that group Nonparametric learning* "Counting-based methods"* Number of parameters/knobs is based on the data, rather than predefined* There is some overlap / a blurry boundary between (non)parametric  * Because the number of parameters in parametric models will be influenced by the number of classes in the data* "Parameters" is a generic term  * Set of numbers used to model a pattern --- Review and solifidy> Record myself explaining (ELI5) the following topics* [ ] Machine learning and deep learning* [ ] Supervised vs unsupervised learning* [ ] Parametric vs nonparametric learning ------ Chapter 3: Forward Propogation* A simple network making a prediction* What is a neural network and what does it do?* Making a prediction with...  * Multiple inputs  * Multiple outputs  * Multiple inputs and outputs* Predicting on predictions --- Step 1: Predict* Number of datapoints processed at a time does a lot to determine the structure of the NN* How many to process at a time?  * Can the neural network be accurate with the (batch of) data it's given?  * E.g. predicting on an image requires the entire image data, not just a single pixel  * Rule of thumb: always present "enough" (how much a human might need to make the same prediction)* Network defined by the shape of the input and output  * Start with a single datapoint making a single prediction  Making a prediction with a single input<jupyter_code># === First neural network === #
weight = 0.1

def neural_network(input, weight):
    pred = input * weight
    return pred
number_of_fingers = [9.5, 10, 15.5, 11]
input = number_of_fingers[0]

pred = neural_network(input, weight)
pred<jupyter_output><empty_output><jupyter_text>What does this NN do?* Multiplies the input by a weight  * Scales the input by a certain amount* Input variable = information* Weight variable = knowledge* Prediction = output> Though they might get more complex, this same underlying concept always applies.* Another way to look at it:  * Weight value is a measure of sensitivity b/w input and the net's prediction    * Large weights amplify the range of predictions    * Small weights "understate" or reduce range of preds  * "Volume knob"* Inputs, predictions can be negative  Making a prediction with multiple inputs* Combining intelligence from multiple datapoints  * Multiply each input by its own weight  * Sum the predictions (weighted sum)    * Dot product<jupyter_code>def w_sum(a, b):
    assert(len(a) == len(b))
    output = 0
    for i in range(len(a)):
        output += (a[i] * b[i])
    return output

def neural_network(input, weights):
    pred = w_sum(input, weights)
    return pred
weights = [0.1, 0.2, 0]

toes = [8.5, 9.5, 9.9, 9.0]
wlrec = [0.65, 0.8, 0.8, 0.9]
nfans = [1.2, 1.3, 0.5, 1.0]

input = [toes[0], wlrec[0], nfans[0]]

pred = neural_network(input, weights)
pred<jupyter_output><empty_output><jupyter_text>Vector math* Elementwise operation  * Mathematical operation between two vectors of equal length  * Pair up values according to their positions in their vectors<jupyter_code>def elementwise_multiplication(vec_a, vec_b):
    assert len(vec_a) == len(vec_b)
    vec_out = [0 for i in range(len(vec_a))]
    for i in range(len(vec_a)):
        vec_out[i] = vec_a[i] * vec_b[i]
    return vec_out
def elementwise_addition(vec_a, vec_b):
    assert len(vec_a) == len(vec_b)
    vec_out = [0 for i in range(len(vec_a))]
    for i in range(len(vec_a)):
        vec_out[i] = vec_a[i] + vec_b[i]
    return vec_out
def vector_sum(vec_a):
    output = 0
    for v in vec_a:
        output += v
    return output
def vector_average(vec_a):
    output = 0
    for v in vec_a:
        output += v
    return output / len(vec_a)
def dot_product(vec_a, vec_b):
    vec_c = elementwise_multiplication(vec_a, vec_b)
    return vector_sum(vec_c)<jupyter_output><empty_output><jupyter_text>Dot product intuitionThe intuition behind how and why a dot product (weighted sum) works is one of the most important parts of understanding how NNs make predictions.> A dot product gives you a notion of similarity between two vectorsThe highest dot product is between two identical vectors. What does it mean when a neural network makes a prediction?> The network gives a high score of the inputs based on how similar they are to the weights.<jupyter_code># === Multiple inputs using numpy === #
import numpy as np

weights = np.array([0.1, 0.2, 0])
def neural_network(input, weights):
    pred = input.dot(weights)
    return pred

toes = np.array([8.5, 9.5, 9.9, 9.0])
wlrec = np.array([0.65, 0.8, 0.8, 0.9])
nfans = np.array([1.2, 1.3, 0.5, 1.0])

input = np.array([toes[0], wlrec[0], nfans[0]])
pred = neural_network(input, weights)
print(pred)<jupyter_output>0.9800000000000001<jupyter_text>Making a prediction with multiple outputsI.e. 3 disconnected single-weight networks.<jupyter_code>def ele_mul(number,vector):
    output = [0] * len(vector)
    assert(len(output) == len(vector))
    for i in range(len(vector)):
        output[i] = number * vector[i]
    return output

def neural_network(input, weights):
    pred = ele_mul(input, weights)
    return pred

weights = [0.3, 0.2, 0.9]

wlrec = [0.65, 0.8, 0.8, 0.9]
input = wlrec[0]
pred = neural_network(input, weights)
print(pred)<jupyter_output>[0.195, 0.13, 0.5850000000000001]<jupyter_text>Multiple inputs and multiple outputsThree independent weighted sums of the input -> three predictions.<jupyter_code>def w_sum(a, b):
    assert len(a) == len(b)
    output = 0
    for i in range(len(a)):
        output += (a[i] * b[i])
    return output

def vect_mat_mul(vect, matrix):
    assert len(vect) == len(matrix)
    output = [0] * len(vect)
    for i in range(len(vect)):
        output[i] = w_sum(vect, matrix[i])
    return output

def neural_network(input, weights):
    pred = vect_mat_mul(input, weights)
    return pred

weights = [
    [0.1, 0.1, -0.3], # hurt?
    [0.1, 0.2, 0.0],  # win?
    [0.0, 1.3, 0.1],  # sad?
]

toes = [8.5, 9.5, 9.9, 9.0]
wlrec = [0.65, 0.8, 0.8, 0.9]
nfans = [1.2, 1.3, 0.5, 1.0]

input = [toes[0], wlrec[0], nfans[0]]
pred = neural_network(input, weights)
pred<jupyter_output><empty_output><jupyter_text>Predictions on predictionsTake the output of one network and use it as input to another.<jupyter_code>ih_wgt = [
    [0.1, 0.2, -0.1], # hid[0]
    [-0.1,0.1, 0.9],  # hid[1]
    [0.1, 0.4, 0.1],  # hid[2]
]

hp_wgt = [
    [0.3, 1.1, -0.3], # hurt?
    [0.1, 0.2, 0.0],  # win?
    [0.0, 1.3, 0.1],  # sad?
]

weights = [ih_wgt, hp_wgt]

def neural_network(input, weights):
    hid = vect_mat_mul(input, weights[0])
    pred = vect_mat_mul(hid, weights[1])
    return pred

neural_network(input, weights)
# === numpy version === #
import numpy as np

ih_wgt = np.array([
    [0.1, 0.2, -0.1], # hid[0]
    [-0.1,0.1, 0.9],  # hid[1]
    [0.1, 0.4, 0.1],  # hid[2]
])

hp_wgt = np.array([
    [0.3, 1.1, -0.3], # hurt?
    [0.1, 0.2, 0.0],  # win?
    [0.0, 1.3, 0.1],  # sad?
])

weights = [ih_wgt, hp_wgt]

input = np.array([toes[0], wlrec[0], nfans[0]])

def neural_network(input, weights):
    hid = input.dot(weights[0])
    pred = hid.dot(weights[1])
    return pred

neural_network(input, weights)<jupyter_output><empty_output><jupyter_text>For dot product: put the (rows, columns) shapes next to each other - the inner ones should match. --- Chapter 3: Review and solifidy> Record myself explaining (ELI5) the following topics* [ ] What is a neural network and what does it do?* [ ] What is the intuition behind the dot product of two vectors?* [ ] What dimensions have to match when taking the dot product? ------ Chapter 4: Gradient Descent* Do neural networks make accurate predictions?* Why measure error?* Hot and cold learning* Calculating both direction and amount from error* Gradient descent* Learning is just reducing error* Derivatives and how to use them to learn* Divergence and alpha> Or, the "compare" and "learn" part of the "predict, compare, and learn" process.Compare provides a measurement of how far off a prediction was, while learning tells each weight how it can change to reduce the error.  CompareDoes your network make good predictions?* Measuring error simplifies the problem* Different ways of measuring error prioritize error differently  * Mean squared error amplifies large error and minimizes small error* Also ensures that the error is always positive (there can't be negative error!)<jupyter_code>knob_weight = 0.5
input = 0.5
goal_pred = 0.8

pred = input * knob_weight
error = (pred - goal_pred) ** 2

print(error)<jupyter_output>0.30250000000000005<jupyter_text>Calculating both direction and amount from errorAKA gradient descent!* Stopping: if `input` is 0 then `direction_and_amount` is also 0 - there's nothing to learn* Negative reversal: multiplying pure error by `input` will reverse the sign of `direction_and_amount` if `input` is negative  * Ensures `weight` moves in the right direction even if `input` is negative* Scaling: if `input` is large, so should be `weight`<jupyter_code>weight = 0.5
goal_pred = 0.8
input = 0.5

for iteration in range(20):
    pred = input * weight
    # `(pred - goal_pred)` is the pure error
    error = (pred - goal_pred) ** 2
    # input: scaling, negative reversal, and stopping
    direction_and_amount = (pred - goal_pred) * input
    # Recalculate weight
    weight = weight - direction_and_amount
    
    print(f"Error: {error}; Prediction: {pred}")
# Perform weight update / gradient descent on single training example

weight = 0.1
alpha = 0.01

def neural_network(input, weight):
    prediction = input * weight
    return prediction

weight, goal_pred, input = (0.0, 0.8, 0.5)

for iteration in range(4):
    # Make prediction, calculate error and delta
    pred = neural_network(input, weight)
    error = (pred - goal_pred) ** 2
    delta = pred - goal_pred
    weight_delta = input * delta
    weight = weight - weight_delta
    print(f"Error: {error}; Prediction: {pred}")<jupyter_output>Error: 0.6400000000000001; Prediction: 0.0
Error: 0.3600000000000001; Prediction: 0.2
Error: 0.2025; Prediction: 0.35000000000000003
Error: 0.11390625000000001; Prediction: 0.4625<jupyter_text>What is `weight_delta`, really?What is a function, and how can one be understood?* A function defines some sort of relationship between the input and output* Every function has moving parts that can be tweaked to change the output    error = ((input * weight) - goal_pred) ** 2`weight` is the only part of the function above that can be changed while still conforming to the patterns in the data> Key point: learning is adjusting `weight` to reduce `error` to 0.  DerivativesThe derivative describes how one variable moves when another is changed. Or, it's the slope describing the relationship.This slope can be used to figure out how to force one to change in a specific direction by modifying the other.The important thing is to understand what the derivative represents.Again again again! — the relationship between two variables in a function, describing how much one changes in response to the other changing. It's the sensitivity between two variables. Using a derivative to learn> `weight_delta` is the derivative!To use the derivative in finding the minimum `error`, move (change `weight`) in the opposite direction of the slope (derivative). Divergence and alphaWith big `input`, the prediction is very sensitive to changes in `weight`. Small changes in `weight` will lead to massive swings in the prediction.This can be combatted using alpha, or the learning rate. I.e. multiply the `weight_delta` by a fraction.<jupyter_code>weight = 0.5
goal_pred = 0.8
# Earlier implementation would break with this input
# but this one doesn't, thanks to our friend alpha!
input = 2
alpha = 0.1

for iteration in range(20):
    pred = input * weight
    error = (pred - goal_pred) ** 2
    derivative = input * (pred - goal_pred)
    weight = weight - (alpha * derivative)
    print("Error:" + str(error) + " Prediction:" + str(pred))<jupyter_output>Error:0.03999999999999998 Prediction:1.0
Error:0.0144 Prediction:0.92
Error:0.005183999999999993 Prediction:0.872
Error:0.0018662400000000014 Prediction:0.8432000000000001
Error:0.0006718464000000028 Prediction:0.8259200000000001
Error:0.00024186470400000033 Prediction:0.815552
Error:8.70712934399997e-05 Prediction:0.8093312
Error:3.134566563839939e-05 Prediction:0.80559872
Error:1.1284439629823931e-05 Prediction:0.803359232
Error:4.062398266736526e-06 Prediction:0.8020155392
Error:1.4624633760252567e-06 Prediction:0.8012093235200001
Error:5.264868153690924e-07 Prediction:0.8007255941120001
Error:1.8953525353291194e-07 Prediction:0.8004353564672001
Error:6.82326912718715e-08 Prediction:0.8002612138803201
Error:2.456376885786678e-08 Prediction:0.8001567283281921
Error:8.842956788836216e-09 Prediction:0.8000940369969153
Error:3.1834644439835434e-09 Prediction:0.8000564221981492
Error:1.1460471998340758e-09 Prediction:0.8000338533188895
Error:4.125769919393652e-10 Prediction:0.80002031199[...]<jupyter_text>--- Chapter 4: Review and solifidy- [x] Build the code for the above neural network implementation from scratch, without looking at notes  - Leave comments ELI5 every single line<jupyter_code># Basic single-neuron network, from memory

def network(input, weight):
    pred = input * weight
    return pred

input, weight, output, alpha = 4, 0.5, 2.3, 0.1

for iteration in range(16):
    # Get the prediction from the network
    pred = network(input, weight)
    # Calculate error - this uses mean squared error
    error = (pred - output) ** 2
    # Calculate delta (pure error)
    delta = pred - output
    # Calculate the weight delta / derivative
    # how much the weight should be updated and in what direction
    derivative = input * delta
    # Calculate new weight
    weight = weight - (alpha * derivative)
    print(f"Prediction: {pred}; error: {error}")
input, weight, truth, alpha = 3, 0.2, 42.1, 0.1

for iteration in range(16):
    # Make prediction by multiplying the input by the neuron's weight
    pred = input * weight
    # Calculate error (using mean squared error)
    error = (pred - truth) ** 2
    # Calcualte derivative: direction and amount weight should change to reduce error
    derivative = input * (pred - truth)
    # Use derivative and alpha to update the neuron's weight
    weight = weight - (alpha * derivative)
    print(f"Truth: {truth}; Prediction: {pred}; error: {error}")
# Attempt #3 - got it first try!
input, weight, output, alpha = 8, 0.8, 16.61, 0.01

for i in range(8):
    pred = input * weight  # Make prediction
    error = (pred - output) ** 2  # Calculate error
    derivative = input * (pred - output)  # Calculate derivative
    weight = weight - (derivative * alpha)  # Calculate new weight
    print(f"Pred: {pred:,f}; error: {error:,f}")<jupyter_output>Pred: 6.400000; error: 104.244100
Pred: 12.934400; error: 13.510035
Pred: 15.286784; error: 1.750901
Pred: 16.133642; error: 0.226917
Pred: 16.438511; error: 0.029408
Pred: 16.548264; error: 0.003811
Pred: 16.587775; error: 0.000494
Pred: 16.601999; error: 0.000064
{'fraction_non_alphanumeric': 0.09721498465636037, 'fraction_numerical': 0.06751201435933067, 'mean_word_length': 4.1542823037899135, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 52, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 13, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '236798', 'license': 'MIT', 'n_tokens_mistral': 6088, 'n_tokens_neox': 5335, 'n_words': 2080, 'path': 'ds/kb/books/grokking_deep_learning.ipynb', 'repo_name': 'tobias-fyi/vela', 'size': '17289'}