<jupyter_start><jupyter_text>* The above code shows us how ReLU function works. Of cource we can write this code more simply:<jupyter_code>inputs = [ 0, 2, -1, 3.3, -2.7, 1.1, 2.2, -100]

output = []
for i in inputs:
    output.append(max(0, i))

print(output)<jupyter_output>[0, 2, 0, 3.3, 0, 1.1, 2.2, 0]<jupyter_text>* But this is code also not sortest code. We can use Numpy as well.<jupyter_code>import numpy as np

inputs = [ 0, 2, -1, 3.3, -2.7, 1.1, 2.2, -100]
output = np.maximum(0, inputs)

print(output)<jupyter_output>[0.  2.  0.  3.3 0.  1.1 2.2 0. ]<jupyter_text>* This method compares each element of the input list (or an array) and returns an object of the same shape filled with new values. ---  Define ReLU Class<jupyter_code>class Activation_ReLU:
    
    # Forward pass
    def forward(self, inputs):
        #calculate output values from input
        self.output = np.maximum(0, inputs)<jupyter_output><empty_output><jupyter_text>Define Dense Layer<jupyter_code>class Layer_Dense:
    
    def __init__(self, n_inputs, n_neurons):
        self.weights = 0.01 * np.random.randn(n_inputs, n_neurons)
        self.biases = np.zeros((1, n_neurons))
        
    def forward(self, inputs):
        self.output = np.dot(inputs, self.weights) + self.biases<jupyter_output><empty_output><jupyter_text>Let's apply this activation function to the dense layer's outputs in our code:<jupyter_code># Create dataset
import numpy as np
import matplotlib.pyplot as plt
import nnfs

from nnfs.datasets import spiral_data

##########################################
# YOU CAN FIND MORE INFORMATION ABOUT THIS DATA FROM
# --> Training Data for Dense Layer Notebook.
##########################################


nnfs.init()
# The nnfs.init() does three things: 
# it sets the random seed to 0 (by the default),
# creates afloat32 dtype default, and overrides 
# the original dot product from NumPy. 
# All of these are meant to ensure repeatable results 
# for following along.


X, y = spiral_data(samples=100, classes=3)
# The spiral_data function allows us to create a dataset
# with as many classes as we want. The function has parameters
# to choose the number of classes and the number of 
# points/observationsper class in the resulting non-linear dataset.


# Create Dense layer with 2 input features and 3 output values
dense1 = Layer_Dense(2, 3)

# Make a forward pass of our training data through this layer
dense1.forward(X)

# Create ReLU activation (to be used with Dense Layer)
activation1 = Activation_ReLU()

# Forward pass through activation function
# Takes in output from previous layer
activation1.forward(dense1.output)

# Let's see output of the first few samples:
print(activation1.output[:5])<jupyter_output>[[0.         0.         0.        ]
 [0.         0.00011395 0.        ]
 [0.         0.00031729 0.        ]
 [0.         0.00052666 0.        ]
 [0.         0.00071401 0.        ]]
{'fraction_non_alphanumeric': 0.12341446691806651, 'fraction_numerical': 0.04182379156667809, 'mean_word_length': 3.4754601226993866, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 19, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '333859', 'license': 'MIT', 'n_tokens_mistral': 1044, 'n_tokens_neox': 942, 'n_words': 338, 'path': 'ReLU Activation Function Code.ipynb', 'repo_name': 'gryhkn/neural_networks_from_scratch', 'size': '2917'}