<commit_before># Multi-Layer Perceptron

This tutorial is based on [http://deeplearning.net/tutorial/mlp.html](http://deeplearning.net/tutorial/mlp.html) but uses Latte to implement the code examples.  

A Multi-Layer Perceptron can be described as a logistic regression classifier where the input is first transformed using a learnt non-linear transformation $\Phi$.  This intermediate layer performing the transformation is referred to as a **hidden layer**.  One hidden layer is sufficient to make MLPs a **universal approximator** [[1][1]].  

## The Model
An MLP with a single hidden layer can be represented graphically as follows:

![Single Hidden Layer MLP](http://deeplearning.net/tutorial/_images/mlp.png)

To understand this representation, we'll first define the properties of a singular neuron.  A **Neuron**, depicted in the figure as a circle, computes an output (typically called an activation) as a function of its inputs.  In this figure, an input is depicted as a directed edge flowing into a **Neuron**.  For an MLP, the function to compute the output of a **Neuron** begins with a weighted sum of the inputs.  Formally, if we describe the input as a vector of values $x$ and a vector of weights $w$, this operation can be written as $w \cdot x$ (dot product).  Next, we will shift this value by a learned bias $b$, then apply an activation function $s$ such as $tanh$, $sigmoid$ or $relu$.  The entire computation is written as $s(w \cdot x + b)$.  The values of $w$ and $b$ will be learnt using back-propogation.

## Defining a WeightedNeuron
Defining a **WeightedNeuron** begins with a subtype of the abstract `Neuron` type.  The `Neuron` type contains 4 default fields:
- `value`    -- contains the output value of the neuron
- `∇`        -- contains the gradient of the neuron
- `inputs`   -- a vector of vectors of input values.
- `∇inputs`  -- a vector of gradients for connected neurons

For our **WeightedNeuron** we will define the following additional fields:

- `weights`  -- a vector of learned weights
- `∇weights` -- a vector of gradients for the weights
- `bias`     -- the bias value
- `∇bias`    -- the gradient for the bias value

```julia
@neuron type WeightedNeuron
    weights  :: DenseArray{Float32}
    ∇weights :: DenseArray{Float32}

    bias     :: DenseArray{Float32}
    ∇bias    :: DenseArray{Float32}
end
```

!!! note
    We do not define the default fields as using the @neuron macro for the type definition will specify them for us.  Furthermore the macro defines a constructor function that automatically initializes the default fields.

Next we define the forward computation for the neuron:
```julia<commit_msg>Minor spacing fix<commit_after># Multi-Layer Perceptron

This tutorial is based on [http://deeplearning.net/tutorial/mlp.html](http://deeplearning.net/tutorial/mlp.html) but uses Latte to implement the code examples.  

A Multi-Layer Perceptron can be described as a logistic regression classifier where the input is first transformed using a learnt non-linear transformation $\Phi$.  This intermediate layer performing the transformation is referred to as a **hidden layer**.  One hidden layer is sufficient to make MLPs a **universal approximator** [[1][1]].  

## The Model
An MLP with a single hidden layer can be represented graphically as follows:

![Single Hidden Layer MLP](http://deeplearning.net/tutorial/_images/mlp.png)

To understand this representation, we'll first define the properties of a singular neuron.  A **Neuron**, depicted in the figure as a circle, computes an output (typically called an activation) as a function of its inputs.  In this figure, an input is depicted as a directed edge flowing into a **Neuron**.  For an MLP, the function to compute the output of a **Neuron** begins with a weighted sum of the inputs.  Formally, if we describe the input as a vector of values $x$ and a vector of weights $w$, this operation can be written as $w \cdot x$ (dot product).  Next, we will shift this value by a learned bias $b$, then apply an activation function $s$ such as $tanh$, $sigmoid$ or $relu$.  The entire computation is written as $s(w \cdot x + b)$.  The values of $w$ and $b$ will be learnt using back-propogation.

## Defining a WeightedNeuron
Defining a **WeightedNeuron** begins with a subtype of the abstract `Neuron` type.  The `Neuron` type contains 4 default fields:

- `value`    -- contains the output value of the neuron
- `∇`        -- contains the gradient of the neuron
- `inputs`   -- a vector of vectors of input values.
- `∇inputs`  -- a vector of gradients for connected neurons

For our **WeightedNeuron** we will define the following additional fields:

- `weights`  -- a vector of learned weights
- `∇weights` -- a vector of gradients for the weights
- `bias`     -- the bias value
- `∇bias`    -- the gradient for the bias value

```julia
@neuron type WeightedNeuron
    weights  :: DenseArray{Float32}
    ∇weights :: DenseArray{Float32}

    bias     :: DenseArray{Float32}
    ∇bias    :: DenseArray{Float32}
end
```

!!! note
    We do not define the default fields as using the @neuron macro for the type definition will specify them for us.  Furthermore the macro defines a constructor function that automatically initializes the default fields.

Next we define the forward computation for the neuron:
```julia
{'fraction_non_alphanumeric': 0.08313049990638458, 'fraction_numerical': 0.0041190788241902265, 'mean_word_length': 3.9371534195933457, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2927948', 'n_tokens_mistral': 1516, 'n_tokens_neox': 1414, 'n_words': 747}