<jupyter_start><jupyter_text>Backpropagation ExerciseIn this exercise we will use backpropagation to train a multi-layer perceptron (with a single hidden layer).  We will experiment with different patterns and see how quickly or slowly the weights converge.  We will see the impact and interplay of different parameters such as learning rate, number of iterations, and number of data points.<jupyter_code>#Preliminaries
from sklearn.metrics import accuracy_score
from __future__ import division, print_function
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline<jupyter_output><empty_output><jupyter_text>ALGORITHM STEPS Fill out the code below so that it creates a multi-layer perceptron with a single hidden layer (with 4 nodes) and trains it via back-propagation.  Specifically your code should:1. Initialize the weights to random values between -1 and 11. Perform the feed-forward computation1. Compute the loss function1. Calculate the gradients for all the weights via back-propagation1. Update the weight matrices (using a learning_rate parameter)1. Execute steps 2-5 for a fixed number of iterations1. Plot the accuracies and log loss and observe how they change over time  QUESTIONSOnce your code is running, try it for the different patterns below.- Which patterns was the neural network able to learn quickly and which took longer?- What learning rates and numbers of iterations worked well?- If you have time, try varying the size of the hidden layer and experiment with different activation functions (e.g. ReLu) 1 - Which patterns was the neural network able to learn quickly and which took longer?- Diamond Pattern- Circle pattern - Centered square - Thick Right Angle pattern - Thin right angle pattern Observando os gráficos gerados com cada pattern, tendo 0.03 como coeficiente de aprendizado e 250 como número de iterações, pudemos observar que a aprendizagem dos modelos se comportam de modo bem parecido, logo é difícil classificar o mais rápido e o mais lento, mas podemos afirmar que o pattern com acurácia e log loss mais regulares é o Thick Right Angle pattern, suas curvas foram suaves e com poucas oscilações. 2 - What learning rates and numbers of iterations worked well?Utilizamos em nossos aprendizados de rede o pattern Diamond Pattern. *   NUMBER_INTERACTIONS = 250 LEARNING_RATE = 0.9*   Não convergiu---*   NUMBER_INTERACTIONS = 250 LEARNING_RATE = 0.5*   Não convergiu---*   NUMBER_INTERACTIONS = 250 LEARNING_RATE = 0.3*   Não convergiu---*   NUMBER_INTERACTIONS = 250 LEARNING_RATE = 0.1*   Covergiu mas teve muitas variações no gráfico*   A acurácia teve muita variação---*   NUMBER_INTERACTIONS = 250 LEARNING_RATE = 0.01*   Demorou a Covergir e teve muitas variações no gráfico *   A acurácia teve muita variação---*   NUMBER_INTERACTIONS = 250 LEARNING_RATE = 0.03*   Convergiu rapidamente*   A acurácia variou muito ---*   NUMBER_INTERACTIONS = 1000 LEARNING_RATE = 0.03*   Convergiu rapidamente*   Boa acurácia, o gráfico da acurácia teve algumas variações---*   NUMBER_INTERACTIONS = 5000 LEARNING_RATE = 0.03*   Convergiu rapidamente*   Boa acurácia, o gráfico da acurácia ficou bem regulado---*   NUMBER_INTERACTIONS = 10000 LEARNING_RATE = 0.03*   Convergiu rapidamente, bem semelhante aos resultados anteriores*   Boa acurácia, o gráfico da acurácia ficou mais ajustado em alguns pontos**Melhor learning rates and numbers of iterations: 10000 e 0.03**<jupyter_code>## This code below generates two x values and a y value according to different patterns
## It also creates a "bias" term (a vector of 1s)
## The goal is then to learn the mapping from x to y using a neural network via back-propagation

NUM_OBS = 500
x_mat_1 = np.random.uniform(-1,1,size = (NUM_OBS,2))
x_mat_bias = np.ones((NUM_OBS,1))
x_mat_full = np.concatenate( (x_mat_1,x_mat_bias), axis=1)

# PICK ONE PATTERN BELOW and comment out the rest.

# # Circle pattern
#y = (np.sqrt(x_mat_full[:,0]**2 + x_mat_full[:,1]**2)<.75).astype(int)

# # oficial pattern
# # Diamond Pattern
y = ((np.abs(x_mat_full[:,0]) + np.abs(x_mat_full[:,1]))<1).astype(int)

# # Centered square
#y = ((np.maximum(np.abs(x_mat_full[:,0]), np.abs(x_mat_full[:,1])))<.5).astype(int)

# # Thick Right Angle pattern
#y = (((np.maximum((x_mat_full[:,0]), (x_mat_full[:,1])))<.5) & ((np.maximum((x_mat_full[:,0]), (x_mat_full[:,1])))>-.5)).astype(int)

# # Thin right angle pattern
#y = (((np.maximum((x_mat_full[:,0]), (x_mat_full[:,1])))<.5) & ((np.maximum((x_mat_full[:,0]), (x_mat_full[:,1])))>0)).astype(int)


print('shape of x_mat_full is {}'.format(x_mat_full.shape))
print('shape of y is {}'.format(y.shape))

fig, ax = plt.subplots(figsize=(5, 5))
ax.plot(x_mat_full[y==1, 0],x_mat_full[y==1, 1], 'ro', label='class 1', color='darkslateblue')
ax.plot(x_mat_full[y==0, 0],x_mat_full[y==0, 1], 'bx', label='class 0', color='chocolate')
# ax.grid(True)
ax.legend(loc='best')
ax.axis('equal');<jupyter_output>shape of x_mat_full is (500, 3)
shape of y is (500,)<jupyter_text>Here are some helper functions<jupyter_code>def sigmoid(x):
    """
    Sigmoid function
    """
    return 1.0 / (1.0 + np.exp(-x))
def loss_fn(y_true, y_pred, eps=1e-16):
    """
    Loss function we would like to optimize (minimize)
    We are using Logarithmic Loss
    http://scikit-learn.org/stable/modules/model_evaluation.html#log-loss
    """
    y_pred = np.maximum(y_pred,eps)
    y_pred = np.minimum(y_pred,(1-eps))
    return -(np.sum(y_true * np.log(y_pred)) + np.sum((1-y_true)*np.log(1-y_pred)))/len(y_true)
def forward_pass(W1, W2):
    """
    Does a forward computation of the neural network
    Takes the input `x_mat` (global variable) and produces the output `y_pred`
    Also produces the gradient of the log loss function
    """
    global x_mat
    global y
    global num_
    # First, compute the new predictions `y_pred`
    z_2 = np.dot(x_mat, W_1)
    a_2 = sigmoid(z_2)
    z_3 = np.dot(a_2, W_2)
    y_pred = sigmoid(z_3).reshape((len(x_mat),))
    # Now compute the gradient
    J_z_3_grad = -y + y_pred
    J_W_2_grad = np.dot(J_z_3_grad, a_2)
    a_2_z_2_grad = sigmoid(z_2)*(1-sigmoid(z_2))
    J_W_1_grad = (np.dot((J_z_3_grad).reshape(-1,1), W_2.reshape(-1,1).T)*a_2_z_2_grad).T.dot(x_mat).T
    gradient = (J_W_1_grad, J_W_2_grad)
    
    # return
    return y_pred, gradient
def plot_loss_accuracy(loss_vals, accuracies):
    fig = plt.figure(figsize=(16, 8))
    fig.suptitle('Log Loss and Accuracy over iterations')
    
    ax = fig.add_subplot(1, 2, 1)
    ax.plot(loss_vals)
    ax.grid(True)
    ax.set(xlabel='iterations', title='Log Loss')
    
    ax = fig.add_subplot(1, 2, 2)
    ax.plot(accuracies)
    ax.grid(True)
    ax.set(xlabel='iterations', title='Accuracy');
def sum_mean_square_error(previsto, y):
    return np.array([(y_i - y_previsto) ** 2 for y_previsto, y_i in zip(previsto,y)]).mean()
def classification_metric_values(y_pred):
  return np.where(y_pred > 0.5, 1, 0)<jupyter_output><empty_output><jupyter_text>Complete the pseudocode below<jupyter_code>#### Initialize the network parameters

np.random.seed(1241)

W_1 = np.random.uniform(-1,1, size = (3,NUM_OBS))
W_2 = np.random.uniform(-1,1, size = (NUM_OBS,1))
NUMBER_INTERACTIONS = 250
LEARNIG_RATE = 0.03
PRINT_STEP = 1000
x_mat = x_mat_full


loss_vals, accuracies = [], []
for i in range(NUMBER_INTERACTIONS):

    ### Do a forward computation, and get the gradient
  
    y_pred, gradient = forward_pass(W_1, W_2)
  
    ## Update the weight matrices

    W_1 -= LEARNIG_RATE * gradient[0]
    W_2 -= LEARNIG_RATE * gradient[1].reshape((500,1))

    ### Compute the loss

    loss_vals.append(loss_fn(y, y_pred))

    ### Compute the loss and accuracy

    accuracies.append(accuracy_score(y,classification_metric_values(y_pred)))
    # accuracies.append(sum_mean_square_error(classification_metric_values(y_pred), y))

    ## Print the loss and accuracy for every 200th iteration

    #if (i+1) % PRINT_STEP == 0:
      # plot_loss_accuracy(loss_vals, accuracies)

plot_loss_accuracy(loss_vals, accuracies)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11243632749409864, 'fraction_numerical': 0.028202261150453472, 'mean_word_length': 4.517477724468814, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 19, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '166402', 'license': 'MIT', 'n_tokens_mistral': 2997, 'n_tokens_neox': 2835, 'n_words': 839, 'path': 'Backprop_Exercise_HW.ipynb', 'repo_name': 'fsluizvictor/backpropagation', 'size': '8049'}