<jupyter_start><jupyter_text>Element-wise operations `relu` operationThe `relu` operation is an _element-wise_ operation, i.e. applied independently to each entry in the tensor. It is therefore amenable to massively parallel (_vectorized_) implementations Naive implementation of an element-wise relu operation:<jupyter_code>import numpy as np
def naive_relu(x):
  #First check that it is a 2-D tensor
  assert len(x.shape)==2

  x = x.copy()#Avoid overwriting the inut 
  for i in range(x.shape[0]):
    for j in range(x.shape[1]):
      x[i,j] = max(x[i,j], 0)
  return x<jupyter_output><empty_output><jupyter_text>Let's check the `naive_relu` function on x.<jupyter_code>x = np.array([[5, -78, 2, 34, 0],
              [6, 79, 3, -35, 1],
              [7, 80, 4, 36, 2]])
x.ndim
naive_relu(x)<jupyter_output><empty_output><jupyter_text>We see that the negative elements have been replaced by 0  Naive implementation of addition operation<jupyter_code>def naive_add(x, y):
  assert len(x.shape)==2
  assert x.shape==y.shape

  x = x.copy()
  for i in range(x.shape[0]):
    for j in range(x.shape[1]):
      x[i, j] += y[i, j]
  return x
x = np.array([[5, -78, 2, 34, 0],
              [6, 79, 3, -35, 1],
              [7, 80, 4, 36, 2]])
y = np.ones(shape=(3,5))
print(x, "\n", y, "\n", "x+y: \n", naive_add(x,y))<jupyter_output>[[  5 -78   2  34   0]
 [  6  79   3 -35   1]
 [  7  80   4  36   2]] 
 [[1. 1. 1. 1. 1.]
 [1. 1. 1. 1. 1.]
 [1. 1. 1. 1. 1.]] 
 x+y: 
 [[  6 -77   3  35   1]
 [  7  80   4 -34   2]
 [  8  81   5  37   3]]<jupyter_text>`relu`in `numpy`<jupyter_code>x = np.array([[5, -78, 2, 34, 0],
              [6, 79, 3, -35, 1],
              [7, 80, 4, 36, 2]])

np.maximum(x, np.zeros(shape = x.shape))<jupyter_output><empty_output><jupyter_text>Tensor broadcastingWhat happens with addition when the shapes of the two tensorsbeing added differ?When possible, and if there’s no ambiguity, the smaller tensor will be broadcasted to match the shape of the larger tensor. Broadcasting consists of two steps:1. Axes (called broadcast axes) are added to the smaller tensor to match the ndim of the larger tensor.2. The smaller tensor is repeated alongside these new axes to match the full shape of the larger tensor.<jupyter_code>def naive_add_matrix_and_vector(x, y):
  assert len(x.shape)==2
  assert len(y.shape)==1
  assert x.shape[1]==y.shape[0]
  
  x = x.copy()
  for i in range(x.shape[0]):
    for j in range(x.shape[1]):
      x[i, j] += y[j]
  return x
x = np.array([[5, -78, 2, 34, 0],
              [6, 79, 3, -35, 1],
              [7, 80, 4, 36, 2]])
y = np.ones(shape=(x.shape[1],))
naive_add_matrix_and_vector(x, y)<jupyter_output><empty_output><jupyter_text>Tensor dot product<jupyter_code>import numpy as np
np.dot(x, y)<jupyter_output><empty_output><jupyter_text>Let's break down the dot product for vectors<jupyter_code>def naive_vector_dot(x,y):
  assert len(x.shape)==1
  assert len(y.shape)==1
  assert x.shape[0]==y.shape[0]
  z=0
  for i in range(x.shape[0]):
    z += x[i]*y[i]
  return z<jupyter_output><empty_output><jupyter_text>Dot product for a matrix and a vector<jupyter_code>def naive_matrix_vector_dot(x,y):
  assert len(x.shape)==2
  assert len(y.shape)==1
  assert x.shape[1]==y.shape[0]
  z = np.zeros(shape=(x.shape[0]))
  print(z)
  for i in range(x.shape[0]):
    print(f"i={i}")
    for j in range(x.shape[1]):
      print(f"j={j}")
      z[i] += x[i,j]*y[j]
  return z
x = np.array([[5, -78, 2, 34, 0],
              [6, 79, 3, -35, 1],
              [7, 80, 4, 36, 2]])
print(f"Shape of x: {x.shape} \n shape of y: {y.shape}")
y = np.ones(shape=x.shape[1])
naive_matrix_vector_dot(x,y)
range(x.shape[0])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.16148228293210712, 'fraction_numerical': 0.05328644847173384, 'mean_word_length': 3.34037558685446, 'pattern_counts': {'":': 0, '<': 32, '<?xml version=': 0, '>': 32, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '452642', 'license': 'MIT', 'n_tokens_mistral': 1625, 'n_tokens_neox': 1487, 'n_words': 425, 'path': '2.3.Tensor_Operations.ipynb', 'repo_name': 'sri-spirited/fchollet-book-deep-learning-with-python-notebooks', 'size': '3697'}