<commit_before>﻿Architectures
   Feedforward
      Branching and Serialization
   Recurrent
      Gated Recurrent Unit
      Long Short Term Memory
      Echo State Networks
      Bi-Directional RNN
   Recursive
      Matrix Vector Recursive Neural Netowrk
      Recursive Neural Tensor Network
   Convolutional
   Restricted Boltzmann Machines
   Deep Belief Nets
   Autoencoders
   Word2vec
   Dynamic Memory Networks
   Deep Reinforcement Learning
   Generative Adversarial Networks

Activation Functions
   sigmoid
   tanh
   Rectified Linear Units
      Leaky RLUs
   Exponential Linear Units
   maxout
   softmax

Cost Functions
   squared error
   cross entropy
   max margin
   
Gradient Descent
   stochastic
   minibatch
   batch
   momentum
   rprop and rmsprop
   adagrad
   adadelta
   adam
   
   Newton's Method
   Limited Memory Broyden–Fletcher–Goldfarb–Shanno Algorithm
   Hessian Free Optimization and Conjugate Gradients

Learning Rate
   constant
   decay
   bold driver
   annealing
   adaptative learning rate for each parameter
   
Preventing Overfitting
   Add More Data
      Bootstrapping, Patch Selection (for images), and Synthetic Data
   Use a Simpler Model
   Early Stopping based on Validation Set Accuracy
   LaGrange Constraints on Input Weight Vector Length per Node
   L1/L2 Regularization
   Add Gaussian Noise to Intpus or Weights
   Stochastic Binary Activation Units with Regular Sigmoid Backprop
   Maximum A Posterior (Bayesian) Weight Decay
      MacKay's Method - Output Error Variance and Layer Weight Variance
   Full Bayesian Learning
      Grid points that represent possible parameter space
      Distribution = Grid Point | Data * Output | Data and Grid Point
      Markov Chain Monte Carlo w/ Random Movement or MiniBatches for evaluating grid points
   Average Different Models (e.g. Boosting/Bagging)
   Dropout Regularization
   
Optimization Tricks
   Input Feature Mean/Std Normalization
   Xavier Weight Initialization Based on Fan-In/Fan-Out
   Orthogonal Weight Initialization
   Batch Normalization
   Residual Deep Learning
   Stochastic Depth Networks
   Competitive Normalization and Max Pooling for Convolutional NN
   Gradient Clipping for Recurrent NN
   Generatively pre-train deep nets using RBM layers
   Distributed Computing and GPU Optimization
  
Other Applications
   Feature Reduction and Selection
   Predicting Input Feature Similarity By Comparing Learned Feature Layers

NN Libraries and Tools
   theano
      lasagne
      keras
   tensorflow<commit_msg>added note about identity matrix initialization<commit_after>﻿Architectures
   Feedforward
      Branching and Serialization
   Recurrent
      Gated Recurrent Unit
      Long Short Term Memory
      Echo State Networks
      Bi-Directional RNN
   Recursive
      Matrix Vector Recursive Neural Netowrk
      Recursive Neural Tensor Network
   Convolutional
   Restricted Boltzmann Machines
   Deep Belief Nets
   Autoencoders
   Word2vec
   Dynamic Memory Networks
   Deep Reinforcement Learning
   Generative Adversarial Networks

Activation Functions
   sigmoid
   tanh
   Rectified Linear Units
      Leaky RLUs
   Exponential Linear Units
   maxout
   softmax

Cost Functions
   squared error
   cross entropy
   max margin
   
Gradient Descent
   stochastic
   minibatch
   batch
   momentum
   rprop and rmsprop
   adagrad
   adadelta
   adam
   
   Newton's Method
   Limited Memory Broyden–Fletcher–Goldfarb–Shanno Algorithm
   Hessian Free Optimization and Conjugate Gradients

Learning Rate
   constant
   decay
   bold driver
   annealing
   adaptative learning rate for each parameter
   
Preventing Overfitting
   Add More Data
      Bootstrapping, Patch Selection (for images), and Synthetic Data
   Use a Simpler Model
   Early Stopping based on Validation Set Accuracy
   LaGrange Constraints on Input Weight Vector Length per Node
   L1/L2 Regularization
   Add Gaussian Noise to Intpus or Weights
   Stochastic Binary Activation Units with Regular Sigmoid Backprop
   Maximum A Posterior (Bayesian) Weight Decay
      MacKay's Method - Output Error Variance and Layer Weight Variance
   Full Bayesian Learning
      Grid points that represent possible parameter space
      Distribution = Grid Point | Data * Output | Data and Grid Point
      Markov Chain Monte Carlo w/ Random Movement or MiniBatches for evaluating grid points
   Average Different Models (e.g. Boosting/Bagging)
   Dropout Regularization
   
Optimization Tricks
   Input Feature Mean/Std Normalization
   Xavier Weight Initialization Based on Fan-In/Fan-Out
   Orthogonal Weight Initialization or Identity Matrix Initialization
   Batch Normalization
   Residual Deep Learning
   Stochastic Depth Networks
   Competitive Normalization and Max Pooling for Convolutional NN
   Gradient Clipping for Recurrent NN
   Generatively pre-train deep nets using RBM layers
   Distributed Computing and GPU Optimization
  
Other Applications
   Feature Reduction and Selection
   Predicting Input Feature Similarity By Comparing Learned Feature Layers

NN Libraries and Tools
   theano
      lasagne
      keras
   tensorflow
{'fraction_non_alphanumeric': 0.012845465161541456, 'fraction_numerical': 0.0011677695601401323, 'mean_word_length': 3.3040201005025125, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2436719', 'n_tokens_mistral': 1433, 'n_tokens_neox': 1304, 'n_words': 568}