<commit_before>% <NAME>
% Deep learning and other machine learning cool stuff
% 1 Jan 2014

# Introduction
  The aim of the project.

# Background

## History of machine learning and progresses
 Properly define a machine learning model?

## Artificial neural networks

  Artifical neural networks (ANNs) are machine learning models biologically inspired from the animal brain in the hope that they can be used to reverse engineer how animals learn to perfrom certain tasks.

  Even though ANNs are inspired by biological neural networks, they are far from being biologically realistic, as the building block of an ANN is an extremely simplifcated version of the neuron.

  Despite this drawback, artificial neural nets have proven to be succesful in a variety of applications, such as image and speech recognition.

  In the following, when we refer to a neural network we will be meaning artificial neural network (ANN).


### Definition and structure

 Cite some semnficant results and their decay
 Quote Hinton in the reasons that neural nets were abandoned (lack of CPU, too much backprop, not enough labeled data)

 Why deep nets did not work before. Talk about how with the pre trained nets you have the guarantte to get better results
  than you got beore

 neural nets have thirved since Hinton has (quote here) has shown that restricted boltzmann machines can be combined together and be used both as generative and discriminative models.

### Activation functions

### Sigmoid

### Tanh

## Learning in neural networks
## Backpropagation and gradient descent methods
  introduction bla bla

  uote this
  http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf

### Error function
  Talk about it's importance in backprop and mention MSE and Gaussian assumption

### Computing the error derivative with respect to the weights
  TODO: put the derivation in here
  Add the derivation with the  bias as well

### Using the error derivative
  Once we know how to compute the error derivatives with respect to the weights of the network, multiple issues arise:
    * When should we update the weights
    * By how much we should update the weights

#### Gradient descent

#### Types of learning
  * online
     Update the weights after each traning case. This means that the error function changes for each update, and hence the weight updates do not necessarily agree, and one training case can condratict another traning case
  * full-batch
    Use the entire data set to compute the error function (which is them sum of all the errors on the individual cases) and then do the weight updates by using the gradient with respect to this error function (the gradient with respect to the weights will be the sum of the individiual error derivatives for each training case). The issue with this is that when we start off, we have very bad weights, and it will take significant amount of time until we improve the weights. A better approach is to use some traning case at the begining to asjust the weights to be sensitive, and then continue updating the weights. This leads to mini-batch learning.
  * mini-batch
    Combine the above two approaches: update after a small sample of traning cases. This will make the weights oscilate less then online learning and will solve the issues of batch learning as well.

#### Parameters for backpropagation
  * Learning rate
      During learning, monitor the error on a validation set.
      If the error is steadily decreasing, then increase the learning rate.
      If the error is increasing, then decrease the learning rate.

      Towards the end of learning, decrease the learning rate. You can do that once the
      error stops decreasing steadily. This helps removing the fluctuations in the values of the weights between mini batches, and helps towards keeping a steady set of weights for the final ones.


  * momentum? dropout etc here (not sure if here: momentum should be here though)
    Nesterov method for momentum

    Quote this:
      http://www.cs.toronto.edu/~hinton/absps/georgerectified.pdf
      Note that in the dropout section it mentions to other papers that showed it is efficient so
      just limit yourself to quoting them and describe droput.

  TODO: quote bishop here as well

### Issues with backpropagation
  Write the wiki things

### Efficiency of backprop
  Quote bishop 5.3.3

## Overfitting: add this to what I used?
  The aim of a neural network is to learn the regularities from the mapping to input to output. (TODO: nice picture here ). Due to the limited amount of data presented to the network, there will also be accidental regularities found in the traning cases. Those accidental regularities result in **sampling error**, potientally making the network not generalize well to unseen testing data. It is impossible for a network to distinguish between real regularities that we aim to learn and the accidental regularities in the data it sees. This is an important issue that arises when using machine learning technquie, and it is generally referred to as overfitting.

  TODO: overfitting example!!! (Hinton gives the poligon one, I should give a different one)


  How to stop overfitting:
    * Cross validation
    * Weight decay
    * Weight sharing (maybe do not mention this)
    * Early stopping
    * Model averaging
    * Dropout (mention this)
    * Generative pre traning (discussed later in the report)


# Restricted Boltzman machines


## Hopfield networks (MAYBE NOT DO THIS? it is kind of unrelated)
  Add nice pics.<commit_msg>report<commit_after>% <NAME>
% Deep learning and other machine learning cool stuff
% 1 Jan 2014

# Introduction
  The aim of the project.

# Background

## History of machine learning and progresses
 Properly define a machine learning model?

## Artificial neural networks

  Artifical neural networks (ANNs) are machine learning models biologically inspired from the animal brain in the hope that they can be used to reverse engineer how animals learn to perfrom certain tasks.

  Even though ANNs are inspired by biological neural networks, they are far from being biologically realistic, as the building block of an ANN is an extremely simplifcated version of the neuron.

  Despite this drawback, artificial neural nets have proven to be succesful in a variety of applications, such as image and speech recognition.

  In the following, when we refer to a neural network we will be meaning artificial neural network (ANN).


### Definition and structure

 Cite some semnficant results and their decay
 Quote Hinton in the reasons that neural nets were abandoned (lack of CPU, too much backprop, not enough labeled data)

 Why deep nets did not work before. Talk about how with the pre trained nets you have the guarantte to get better results
  than you got beore

 neural nets have thirved since Hinton has (quote here) has shown that restricted boltzmann machines can be combined together and be used both as generative and discriminative models.

### Activation functions

### Sigmoid

### Tanh

## Learning in neural networks
## Backpropagation and gradient descent methods
  introduction bla bla

  uote this
  http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf

### Error function
  Talk about it's importance in backprop and mention MSE and Gaussian assumption

### Computing the error derivative with respect to the weights
  TODO: put the derivation in here
  Add the derivation with the  bias as well

### Using the error derivative
  Once we know how to compute the error derivatives with respect to the weights of the network, multiple issues arise:
    * When should we update the weights
    * By how much we should update the weights

#### Gradient descent

#### Types of learning
  * online
     Update the weights after each traning case. This means that the error function changes for each update, and hence the weight updates do not necessarily agree, and one training case can condratict another traning case
  * full-batch
    Use the entire data set to compute the error function (which is them sum of all the errors on the individual cases) and then do the weight updates by using the gradient with respect to this error function (the gradient with respect to the weights will be the sum of the individiual error derivatives for each training case). The issue with this is that when we start off, we have very bad weights, and it will take significant amount of time until we improve the weights. A better approach is to use some traning case at the begining to asjust the weights to be sensitive, and then continue updating the weights. This leads to mini-batch learning.
  * mini-batch
    Combine the above two approaches: update after a small sample of traning cases. This will make the weights oscilate less then online learning and will solve the issues of batch learning as well.

#### Parameters for backpropagation
  * Learning rate
      During learning, monitor the error on a validation set.
      If the error is steadily decreasing, then increase the learning rate.
      If the error is increasing, then decrease the learning rate.

      Towards the end of learning, decrease the learning rate. You can do that once the
      error stops decreasing steadily. This helps removing the fluctuations in the values of the weights between mini batches, and helps towards keeping a steady set of weights for the final ones.


  * momentum? dropout etc here (not sure if here: momentum should be here though)
    Nesterov method for momentum

    Quote this:
      http://www.cs.toronto.edu/~hinton/absps/georgerectified.pdf
      Note that in the dropout section it mentions to other papers that showed it is efficient so
      just limit yourself to quoting them and describe droput.

    Say how it is inspired by evolution and reproduction, first minutes of this talk
    https://www.youtube.com/watch?v=DleXA5ADG78

  TODO: quote bishop here as well

### Issues with backpropagation
  Write the wiki things

### Efficiency of backprop
  Quote bishop 5.3.3

## Overfitting: add this to what I used?
  The aim of a neural network is to learn the regularities from the mapping to input to output. (TODO: nice picture here ). Due to the limited amount of data presented to the network, there will also be accidental regularities found in the traning cases. Those accidental regularities result in **sampling error**, potientally making the network not generalize well to unseen testing data. It is impossible for a network to distinguish between real regularities that we aim to learn and the accidental regularities in the data it sees. This is an important issue that arises when using machine learning technquie, and it is generally referred to as overfitting.

  TODO: overfitting example!!! (Hinton gives the poligon one, I should give a different one)


  How to stop overfitting:
    * Cross validation
    * Weight decay
    * Weight sharing (maybe do not mention this)
    * Early stopping
    * Model averaging
    * Dropout (mention this)
    * Generative pre traning (discussed later in the report)


# Restricted Boltzman machines


## Hopfield networks (MAYBE NOT DO THIS? it is kind of unrelated)
  Add nice pics.
{'fraction_non_alphanumeric': 0.036686601803088456, 'fraction_numerical': 0.0020530215120949744, 'mean_word_length': 3.940035273368607, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 1, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3346557', 'n_tokens_mistral': 2728, 'n_tokens_neox': 2569, 'n_words': 1674}