<jupyter_start><jupyter_text>Task 1Implement basic layer which will consist of simple logical neurons. Allow your solution to stack multiple layers to form MLP network. Perform forward propagation through your network.<jupyter_code># Import
import numpy as np
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Main Processing CellPerform following operations:__A__: Declare a simple model consisting of Input Layer, followed by 1 Dense Layer with single neuron. Perform forward pass for the data example __xIn__ of batch size = 1. Plot the results. Repeat the process using all 3 activation functions.__B__: Declare a simple model consisting of Input Layer, followed by 3 Dense Layers of arbitrary size and 1 Dense Layer with 2 neurons. Declare your own input data with batch size 16 and vector size of 10. Perform forward pass through the network and visualize activations of each resulting neuron.<jupyter_code>#----------
# TODO: PRE KAZDY LAYER NECH SA VIE NASTAVIT AKTIVACNA F
# TODO: ma zbehnut Task1 aj Task2 zaroven
# 
#----------

# Main processing
# Run this cel as last

# Task A:
#xIn = np.array([[-4, -3, -2, -1, 0, 1, 2, 3, 4, 5]])
###>>> start of solution

#model = Model("relu")
#model.initialize()
#y = model.predict(xIn)

###<<< end of solution
#plt.plot(xIn[0], y[0])
#plt.show()



#xIn = np.array([[-4, -3, -2, -1, 0, 1, 2, 3, 4, 5]])
###>>> start of solution

###<<< end of solution
#plt.plot(xIn[0], y[0])
#plt.show()

#xIn = np.array([[-4, -3, -2, -1, 0, 1, 2, 3, 4, 5]])
###>>> start of solution

###<<< end of solution
#plt.plot(xIn[0], y[0])
#plt.show()



# Task B:
xIn = np.random.randn(16, 10)
###>>> start of solution
print(xIn)

model = Model("relu")
model.initialize()

y = model.predict(xIn)



###<<< end of solution
plt.plot(y[0])
plt.plot(y[1])
plt.show()


#16 polii 10 cisel
# dolezite su atributy, poc. zaznamov nas netrapi. Atributy urcuju velkost matice

# shape neuronu (matica) musi byt POCET ATRIBUTOV x POCET NEURONOV V PREDCHADZAJUCEJ
# -> Aby sa dali nasobit matice (musia mat tvar 16x10 10xYY)<jupyter_output>[[-0.130933   -1.31686596 -0.53403496  0.73260734 -0.48151154 -1.24260825
   1.49567064 -0.9789524   1.00828316 -0.1106754 ]
 [ 0.4481412  -1.69389871 -0.26772823 -0.39793885  0.73360778 -2.32565301
  -0.02053764 -0.63580525 -0.88261725  0.19073068]
 [-0.30704137 -1.41657648  0.75468355 -1.41325298 -0.41893641  1.30869713
   0.13283716  0.35857967  0.851453    0.42454691]
 [ 0.68303888  0.4634035  -1.65877033  1.80208614 -0.44214272 -0.35051264
   1.37489827 -0.72704239  0.52648123  0.5734369 ]
 [-0.47431344 -0.38378286 -0.44707081 -1.17901097  0.04911351 -0.20335789
  -0.06104448 -0.8969035   0.08992018  0.2222575 ]
 [-0.07684607  1.08385781 -0.30525926 -0.07082582  0.5516573  -1.71589682
  -0.97429762  1.04931132 -0.69005613  0.96179667]
 [ 0.45961051  0.94339125 -1.81959974  1.17503453  0.34877943 -1.30130326
   0.74003366 -1.16295234 -1.64150407  0.24012839]
 [ 0.69514806 -0.74288906 -0.45082876  1.26100819  1.69122255 -0.93347541
  -1.59859375 -2.64965898 -0.25407856  0.52546257]
[...]<jupyter_text>LayerThis is the main class which can hold different types of layers and provide us with standard tasks like forward propagation. Perform the variable declaration, provide your solution with variable initialization when later called by your model and also functionality to perform forward pass. We added another class of Neuron to the Dense Layer for simplicity. You can perform the logic of a layer per single neuron using lists and for cycles. You can declare activation function in your __init__ function of the parent class (Layer) and call it later when you perform forward pass, or perform it in one of the child classes.nUnits - number of neuron units in your layerprevLayer - previous layer (need it to know the shape of it to create appropriate number of weights for you to use in current layer)nX - number of units (in case of input layer its batch size or size of input data)<jupyter_code>#------------------------------------------------------------------------------
#   Layer class
#------------------------------------------------------------------------------
class Layer:
    def __init__(self, act="linear"):
        self.shape = (0, 0)
        self.act = act
        #TODO: declare mapping of your activation function here (or later in child class if you like)
        ###>>> start of solution
        
        ###<<< end of solution
        pass

    def initialize(self, prevLayer):
        pass

    def forward(self, x, isTraining):
        pass

#------------------------------------------------------------------------------
#   InputLayer class
#------------------------------------------------------------------------------
class InputLayer(Layer):
    def __init__(self, nX):
        super().__init__(act="linear")

        self.nX = nX

    def initialize(self, prevLayer):
        self.shape = (self.nX, 1)

    def forward(self, x):
        return x
    
#------------------------------------------------------------------------------
#   Basic Dense Layer class
#------------------------------------------------------------------------------
class DenseLayer(Layer):
    def __init__(self, nUnits, act="linear"):
        super().__init__(act)
        ###>>> start of solution
        
        self.neurons = []
        self.numOfNeurons = nUnits
        
        ###<<< end of solution
        pass

    def initialize(self, prevLayer):
        ###>>> start of solution
        #initialize all neurons
        self.shape = (self.numOfNeurons, prevLayer.shape[0])
        for x in range(self.numOfNeurons):
            neuron = self.Neuron(self.act)
            neuron.initialize(prevLayer)
            self.neurons.append(neuron)
        
        ###<<< end of solution
        pass

    def forward(self, X):
        ###>>> start of solution
        calculations = []
        
        for n in self.neurons:
            calculations.append(n.forward(X).tolist()[0]) # pretoze forward vrati np.array([[data]])

        return np.array(calculations)
        ###<<< end of solution
        pass
    
    class Neuron:
        """Single Neuron used in the Dense Layer"""
        def __init__(self, act="linear"):
            ###>>> start of solution
            self.act = act
            ###<<< end of solution
            pass
        def initialize(self, prevLayer):
            ###>>> start of solution
            self.weights = np.random.rand(1,prevLayer.shape[0]) AxB x BxC => AxC ===> aktualnej
                                          #1 ->pretoze to robime po riadkoch
                                          #prevLayer.shape[0] -> vahy X data, a teraz zistujeme vahy (data), takze potrebujeme
                                          # YYxZZ s tym ze ZZ musi byt ako data ZZxFF (a data su velkost X matice previous dat)
            ###<<< end of solution
            pass
        def forward(self, X):
            ###>>> start of solution
            temp = np.dot(self.weights,X)
            _func = CreateActivationFunction(self.act)
            
            return _func(temp)
            ###<<< end of solution
            pass<jupyter_output><empty_output><jupyter_text>ActivationsImplement three standard activation functions (Linear, ReLU, Sigmoid), which you will use in your implementation of dense layer for logical neurons.<jupyter_code>#------------------------------------------------------------------------------
#   ActivationFunction class
#------------------------------------------------------------------------------
class ActivationFunction:
    def __init__(self):
        pass

    def __call__(self, Z):
        pass

#------------------------------------------------------------------------------
#   LinearActivationFunction class
#------------------------------------------------------------------------------
class LinearActivationFunction(ActivationFunction):
    def __call__(self, Z):
        ###>>> start of solution
        return (2*Z + 0)
        ###<<< end of solution
        pass

    def derivate(self, Z):
        # do not need to implement now
        pass
    
#------------------------------------------------------------------------------
#   SigmoidActivationFunction class
#------------------------------------------------------------------------------
class SigmoidActivationFunction(ActivationFunction):
    def __call__(self, Z):
        ###>>> start of solution
        return (1/(1+np.exp(-Z)))
        ###<<< end of solution
        pass

    def derivate(self, Z):
        # do not need to implement now
        pass

#------------------------------------------------------------------------------
#   RELUActivationFunction class
#------------------------------------------------------------------------------
class RELUActivationFunction(ActivationFunction):
    def __call__(self, Z):
        ###>>> start of solution
        return (np.maximum(0,Z))
        ###<<< end of solution
        pass

    def derivate(self, Z):
        # do not need to implement now
        pass
    
# Activation mapping
    
MAP_ACTIVATION_FUCTIONS = {
    "linear": LinearActivationFunction,
    "relu": RELUActivationFunction,
    "sigmoid": SigmoidActivationFunction
}

def CreateActivationFunction(kind):
    if (kind in MAP_ACTIVATION_FUCTIONS):
        return MAP_ACTIVATION_FUCTIONS[kind]()
    raise ValueError(kind, "Unknown activation function {0}".format(kind))<jupyter_output><empty_output><jupyter_text>Model classThis is the basic class which holds all of your layers and encapsulate functionality to predict results from your input as a forward pass through all the layers after you create your model and initialize all the layers.<jupyter_code>#------------------------------------------------------------------------------
#   Model class
#------------------------------------------------------------------------------
class Model:
    def __init__(self):
        self.layers = []

    def addLayer(self,  layer):
        self.layers.append(layer)

    def initialize(self, layers):
        
        ###>>> start of solution
#        inputLayer = InputLayer(16)
#        inputLayer.initialize({})
        
        denseLayer1 = DenseLayer(2, self.typeOfActivation)
        denseLayer1.initialize(inputLayer)
        self.addLayer(denseLayer1)

        
#        self.addLayer(inputLayer)
        
#        for l in range(numOfDenseLayers):
#            denseLayer = DenseLayer(2, actionFuctions[l],)
        
#        denseLayer1 = DenseLayer(2, self.typeOfActivation)
#        denseLayer1.initialize(inputLayer)
#        self.addLayer(denseLayer1)
        
#        denseLayer2 = DenseLayer(2, self.typeOfActivation)
#        denseLayer2.initialize(denseLayer1)
#        self.addLayer(denseLayer2)
        
#        denseLayer3 = DenseLayer(2, self.typeOfActivation)
#        denseLayer3.initialize(denseLayer2)
#        self.addLayer(denseLayer3)
        
#        denseLayer4 = DenseLayer(2, self.typeOfActivation)
#        denseLayer4.initialize(denseLayer3)
#        self.addLayer(denseLayer4)
        ###<<< end of solution
        pass
    
    def predict(self, X):
        ###>>> start of solution
        
        for x in self.layers:
            X = x.forward(X)
            
        return X    
        ###<<< end of solution
        pass<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.20409418379898084, 'fraction_numerical': 0.07151642945000879, 'mean_word_length': 1.8030041861610442, 'pattern_counts': {'":': 3, '<': 68, '<?xml version=': 0, '>': 73, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '516646', 'license': 'BSD-2-Clause', 'n_tokens_mistral': 3730, 'n_tokens_neox': 3244, 'n_words': 974, 'path': 'week_2/Task_1_neuron_inner_class.ipynb', 'repo_name': 'xzalabai/neural_networks_at_fiit', 'size': '11382'}