<jupyter_start><jupyter_text>INFO This is my solution for the fourth homework problem.  **SOLUTION**  Description I will use network with:- input layer with **2 neurons** (two input variables)- **one** hidden layer with **2 neurons** (I need to split the plane in a nonlinear way, creating a U-shaped plane containing the diagonal points)- output layer with 1 neuron (result - active or inactive) Also, as an activation function I will use a sigmoid function - simple, values between (0, 1) and with simple derivative  CODE<jupyter_code>import numpy as np<jupyter_output><empty_output><jupyter_text>Let's define our sigmoid function and its derivative<jupyter_code>def sigmoid(x, derivative=False):
  if derivative:
    return x * (1 - x)
  else:
    return 1 / (1 + np.exp(-x))<jupyter_output><empty_output><jupyter_text>Now, number of neurons per layer<jupyter_code>layers_sizes = np.array([2, 2, 1])<jupyter_output><empty_output><jupyter_text>And layers initialization function<jupyter_code>def init_layers(sizes):
  weights = [np.random.uniform(size=size) for size in zip(sizes[0:-1], sizes[1:])]
  biases = [np.random.uniform(size=(size, 1)) for size in sizes[1:]]

  return weights, biases<jupyter_output><empty_output><jupyter_text>Function which execute network (forward propagation).Takes input layer, following layers weights and biases and activation function. Returns layers outputs.<jupyter_code>def execute(input, weights, biases, activation_f):
  result = [input]
  previous_layer = input

  for weight, bias in zip(weights, biases):
    executed_layer = execute_layer(previous_layer, weight, bias, activation_f)

    previous_layer = executed_layer
    result.append(executed_layer)

  return result

 
def execute_layer(input_layer, weight, bias, activation_f):
  layer_activation = np.dot(input_layer.T, weight).T + bias

  return activation_f(layer_activation)<jupyter_output><empty_output><jupyter_text>And time for the backpropagation function.Function takes layers outputs, weights, biases and activation function, expected output and learning rate.<jupyter_code>def backpropagation(layers_outputs,
                    weights,
                    biases,
                    activation_f,
                    expected_output,
                    learning_rate):

  updated_weights = weights.copy()
  updated_biases = biases.copy()

  predicted_output = layers_outputs[-1]

  output_error = 2 * (expected_output - predicted_output)
  output_delta = output_error * activation_f(predicted_output, True)

  updated_weights[-1] += layers_outputs[-2].dot(output_delta.T) * learning_rate
  updated_biases[-1] += output_delta * learning_rate

  next_layer_delta = output_delta

  for layer_id in reversed(range(1, len(layers_outputs)-1)):
    weight_id = layer_id - 1

    error = np.dot(weights[weight_id+1], next_layer_delta)
    delta = error * activation_f(layers_outputs[layer_id], True)
    
    updated_weights[weight_id] += layers_outputs[layer_id-1].dot(delta.T) * learning_rate
    updated_biases[weight_id] += delta * learning_rate

    next_layer_delta = delta

  return updated_weights, updated_biases<jupyter_output><empty_output><jupyter_text>---Create test set:<jupyter_code>test_set_X = [np.array([[0], [0]]), np.array([[1], [0]]), np.array([[0], [1]]), np.array([[1], [1]])]
test_set_Y = [np.array([[0]]), np.array([[1]]), np.array([[1]]), np.array([[0]])]<jupyter_output><empty_output><jupyter_text>And training parameters:<jupyter_code>learning_rate = 0.07
number_of_iterations = 30000<jupyter_output><empty_output><jupyter_text>And train out model:<jupyter_code>weights, biases = init_layers(layers_sizes)

errors = []

for iteration in range(number_of_iterations):
  error = 0
  for test_x, test_y in zip(test_set_X, test_set_Y):
    values = execute(test_x, weights, biases, sigmoid)
    predicted_y = values[-1]

    error += np.sum((predicted_y - test_y) ** 2) / len(test_y)

    new_weights, new_biases = backpropagation(values, weights, biases, sigmoid, test_y, learning_rate)
    weights = new_weights
    biases = new_biases

  print("iteration number {} done! Error: {}".format(iteration, error / len(test_set_X)))
  errors.append(error / len(test_set_X))<jupyter_output>[1;30;43mStreaming output truncated to the last 5000 lines.[0m
iteration number 25000 done! Error: 0.00045371713494963964
iteration number 25001 done! Error: 0.00045369499548500995
iteration number 25002 done! Error: 0.0004536728581278429
iteration number 25003 done! Error: 0.00045365072287784287
iteration number 25004 done! Error: 0.00045362858973471134
iteration number 25005 done! Error: 0.0004536064586981529
iteration number 25006 done! Error: 0.0004535843297678677
iteration number 25007 done! Error: 0.00045356220294356253
iteration number 25008 done! Error: 0.00045354007822493683
iteration number 25009 done! Error: 0.0004535179556116912
iteration number 25010 done! Error: 0.00045349583510353704
iteration number 25011 done! Error: 0.00045347371670017195
iteration number 25012 done! Error: 0.0004534516004012991
iteration number 25013 done! Error: 0.00045342948620661966
iteration number 25014 done! Error: 0.00045340737411584303
iteration number 25015 done! Error: 0.00045338526412866[...]<jupyter_text>And plot the error over iterations<jupyter_code>import matplotlib.pyplot as plt

plt.plot(errors)
plt.ylabel('error vs iteration')
plt.show()<jupyter_output><empty_output><jupyter_text>And print results<jupyter_code>print("iterations: {}, learning rate: {}".format(number_of_iterations, learning_rate))
for test_x, test_y in zip(test_set_X, test_set_Y):
  values = execute(test_x, weights, biases, sigmoid)
  predicted_y = values[-1]
  print("{} xor {} = {} ({} confidence)".format(test_x[0][0], test_x[1][0], round(predicted_y[0][0]), predicted_y))<jupyter_output>iterations: 30000, learning rate: 0.07
0 xor 0 = 0 ([[0.02099635]] confidence)
1 xor 0 = 1 ([[0.98180765]] confidence)
0 xor 1 = 1 ([[0.98183589]] confidence)
1 xor 1 = 0 ([[0.01885256]] confidence)
{'fraction_non_alphanumeric': 0.11153717905968656, 'fraction_numerical': 0.08836278759586529, 'mean_word_length': 5.570646221248631, 'pattern_counts': {'":': 0, '<': 43, '<?xml version=': 0, '>': 43, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '15052', 'license': 'MIT', 'n_tokens_mistral': 2356, 'n_tokens_neox': 1991, 'n_words': 496, 'path': 'sem6/sus/homework4/src/homework4.ipynb', 'repo_name': 'abrams27/mimuw', 'size': '5998'}