<jupyter_start><jupyter_text>Training Neural Networks, How do neural networks learn from input data We often hear deep learning is used to create many great tools in modern technology. .  But what makes it so special is it's ability to create function from data, you just need to tell what your input is in a language your neural network can understand and perform some math on it and return the output in a way you wanted it to present. So how does it actually create the function from the input we provide? Let us try to understand the building blocks of neural networks by designing one to create a small function.<jupyter_code>def f(x):
    return (3*x)**2 + (2*x) + 1<jupyter_output><empty_output><jupyter_text>Let us assume we want our neural network to create the above function. For that we need bunch of input data so that on performing some simple math on the input few time it should be able to create a function that will be as close to the above function as possible So let us create some random input data for our neural network<jupyter_code>input = torch.randn(20,1)
input<jupyter_output><empty_output><jupyter_text>There are few ways the neural networks try to create a function called supervise learning, unsupervised learning etc. We will try to create our function with supervised learning techinque. Incase of supervised learning technique we should also have the corresponding output data for your inputs so that we can tell if your neural network is going in right direction for creating your function. We can get the outputs to our input data by passing our inputs through our function.<jupyter_code>output = f(input)
output<jupyter_output><empty_output><jupyter_text>Let us plot our function on a graph for vizualization<jupyter_code>plt.scatter(input,output,color='blue')<jupyter_output><empty_output><jupyter_text>Now we need to create a neural network that can create a function that looks similar to the above graph. This is can be done by repeating 3 steps for few times.1. Make a prediction by performing multiplication and addition.2. Check how bad the prediction is from the output.3. Change some values accordingly so that your prediction gets better. Let us go through these steps one by one.  Forward Propagation We start with create a random function by multiplying your input by a parameter called weights and adding it to a parameter called bias. This process of predicting a function is called forward propagation. We start with random values for these weights and bias and our goal is to adjust these parameters so that we can create a function that is similar to the one we wanted to create.<jupyter_code>def param_init(size):
    return torch.randn(size)
weights = param_init(1)
bias = param_init(1)
weights.shape, bias.shape<jupyter_output><empty_output><jupyter_text>The number of weights and bias to use depends on your input and output dimensions and your network architecture We make a prediction by multiplying our input with the weight and adding it to bias<jupyter_code>pred = input@weights + bias
pred<jupyter_output><empty_output><jupyter_text>"@" symbol represents matrix multiplication between our input and weightsThis is referred as single layer neural network<jupyter_code>pred,output<jupyter_output><empty_output><jupyter_text>These are our prediction and actual output for our input. Now we need a method that tells us how bad our prediction is from our actual output and helps the neural network to make changes so that our predictions become much better. This method is called loss function and this varies depending on what function you want your neural network to create. For our function we need a loss function that tells us how far our predicted output is from the real out. There is a function that helps us to achieve this called mean squared error.<jupyter_code>def mse(pred,target):
    return ((pred-target)**2).mean().sqrt()
mse(pred,output)<jupyter_output><empty_output><jupyter_text>Now we need to reduce this loss as much as possible so that our predictions will be as close to our real outputs, there by resulting in a function close to the one we wanted to create.  Back propagation We can reduce our loss with the help of calculus. Since we got the loss from the prediction by multiplying our input to weights and adding them with bias, by calculating gradient of the loss with respect to particular parameter we get how much change we need to make to the parameter so that the loss will be reduced.  This process of calculating the gradients and updating the parameters is called back propagation and the process of reducing the loss by subtracting the gradients from their parameters is called gradient descent. Using these new adjusted parameters we do a forward propagation and calculate the loss and do a back propagation. Pytorch is a machine learning library will help us to calculate these gradients easily. For that we need to modify our init_params function.<jupyter_code>def param_init(size):
    return torch.randn(size).requires_grad_()
weights = param_init(1)
bias = param_init(1)
pred = input@weights + bias
pred
loss = mse(pred,output)
loss<jupyter_output><empty_output><jupyter_text>We can caluculate the gradient of each parameter by calling backward on the loss<jupyter_code>loss.backward()
print(weights.data)
weights.data -= weights.grad.data * 0.01
weights.data
bias.data -= bias.grad.data * 0.01
bias<jupyter_output><empty_output><jupyter_text>We subtract the gradient from its parameter so that it results in reducing loss and 0.01 we multipled is referred as learning rate that indicates how fast we want to reduce it<jupyter_code>pred = input@weights + bias
pred
loss = mse(pred,output)
loss<jupyter_output><empty_output><jupyter_text>By subtracting the gradient from our parameters and predicting again using new parameters we have seen that the loss is bit reduced. We need to perform this operation few times so that we can obtain a function that is close to one we wanted to create.  Putting it all together<jupyter_code>weights = param_init(1)
bias = param_init(1)
def model(input):
    return input@weights + bias
for i in range(100):
    preds = model(input)
    loss = mse(preds,output)
    loss.backward()
    weights.data = (weights.data - weights.grad.data)*0.1
    bias.data = (bias.data - bias.grad.data)*0.1
    weights.grad = None
    bias.grad = None
    
    if i % 10 == 0:
        print(f'loss at {i} epoch is {loss.item()}')
preds = model(input)
plt.scatter(input,output,color='blue')
plt.scatter(input,to_np(preds),color='red')<jupyter_output><empty_output><jupyter_text>The function created by our neural network is not close to the one we wanted it to create. We can add more layers to our neural network so that it can create more complex functions<jupyter_code>weights1 = param_init((1,3))
bias1 = param_init(1)
weights2 = param_init((3,1))
bias2 = param_init(1)
def model(x):
    res = x@weights1 + bias1
    res = res.max(tensor(0.0))
    res = res@weights2 + bias2
    return res<jupyter_output><empty_output><jupyter_text>In order to add more layers in our neural networks we need to add non linearity between each layer because without non linearity adding any number of layers result in a single layer with different values res.max(tensor(0.0)) This is called as relu activation (rectified linear unit). This return the value of the input if it is great than 0 else it return 0 There are also other activation function. These help us to add any number of layers to our neural network We will try to create a function using new neural network<jupyter_code>for i in range(100):
    preds = model(input)
    loss = mse(preds,output)
    loss.backward()
    weights1.data -= weights1.grad.data*0.1
    bias1.data -= bias1.grad.data*0.1
    weights2.data -= weights2.grad.data*0.1
    bias2.data -= bias2.grad.data*0.1
    weights1.grad = None
    bias1.grad = None
    weights2.grad = None
    bias2.grad = None
    
    if i % 10 == 0:
        print(f'loss at {i} epoch is {loss.item()}')
preds = model(input)
plt.scatter(input,output,color='blue')
plt.scatter(input,to_np(preds),color='red')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06033848417954378, 'fraction_numerical': 0.010179053225410841, 'mean_word_length': 4.829163688348821, 'pattern_counts': {'":': 0, '<': 57, '<?xml version=': 0, '>': 57, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '404765', 'license': 'Apache-2.0', 'n_tokens_mistral': 2233, 'n_tokens_neox': 2169, 'n_words': 1083, 'path': '_notebooks/2021-06-07-blog-1.ipynb', 'repo_name': 'AdityaVarmaUddaraju/ai_blogs', 'size': '8154'}