<commit_before>.. _neural_network:

====================================
Neural network models (unsupervised)
====================================

.. currentmodule:: sklearn.neural_network


.. _rbm:

Restricted Boltzmann Machines
=============================

Restricted Boltzmann Machines (RBM) are unsupervised nonlinear feature learners
based on a probabilistic model.  The features extracted by an RBM give good
results when fed into a linear classifier such as a linear SVM or perceptron.

The training method is based on an approximation, for efficiency.  It prevents
the representations from straying far from the input data, which makes them
capture interesting regularities, but makes the model less useful for small
datasets, and usually not useful for density estimation.

Deep neural networks that are notoriously difficult to train from scratch can
be simplified by initializing each layer's weights with the weights of an RBM.

The makes assumptions regarding the distribution of inputs.  At the moment,
scikit-learn only provides :class:`BernoulliRBM`, which assumes the input are
either binary values or values between 0 and 1, each encoding the probability
that the specific feature would be turned on.

Graphical model and parametrization
-----------------------------------

The graphical model of an RBM is a fully-connected bipartite graph.

.. image:: ../images/rbm_graph.png
   :align: center

The nodes are random variables whose states depend on the state of the other
nodes they are connected to.  The model is therefore parameterized by the
weights of the connections, as well as one intercept (bias) term for each
visible and hidden unit, ommited from the image for simplicity.

The energy function measures the quality of a joint assignment:

.. math:: 

   E(\mathbf{v}, \mathbf{h}) = \sum_i \sum_j w_{ij}v_ih_j + \sum_i b_iv_i
     + \sum_j c_jh_j

In the formula above, :math:`\mathbf{b}` and :math:`\mathbf{c}` are the
intercept vectors for the visible and hidden layers, respectively. The
joint probability of the model is defined in terms of the energy:

.. math::<commit_msg>DOC add plot and example to docs<commit_after>.. _neural_network:

====================================
Neural network models (unsupervised)
====================================

.. currentmodule:: sklearn.neural_network


.. _rbm:

Restricted Boltzmann Machines
=============================

Restricted Boltzmann Machines (RBM) are unsupervised nonlinear feature learners
based on a probabilistic model.  The features extracted by an RBM give good
results when fed into a linear classifier such as a linear SVM or perceptron.

The training method is based on an approximation, for efficiency.  It prevents
the representations from straying far from the input data, which makes them
capture interesting regularities, but makes the model less useful for small
datasets, and usually not useful for density estimation.

Deep neural networks that are notoriously difficult to train from scratch can
be simplified by initializing each layer's weights with the weights of an RBM.

The makes assumptions regarding the distribution of inputs.  At the moment,
scikit-learn only provides :class:`BernoulliRBM`, which assumes the input are
either binary values or values between 0 and 1, each encoding the probability
that the specific feature would be turned on.

.. figure:: ../auto_examples/images/plot_rbm_logistic_classification_1.png
   :target: ../auto_examples/plot_rbm_logistic_classification.html
   :align: center
   :scale: 100%

.. topic:: Examples:

   * :ref:`example_plot_rbm_logistic_classification.py`


Graphical model and parametrization
-----------------------------------

The graphical model of an RBM is a fully-connected bipartite graph.

.. image:: ../images/rbm_graph.png
   :align: center

The nodes are random variables whose states depend on the state of the other
nodes they are connected to.  The model is therefore parameterized by the
weights of the connections, as well as one intercept (bias) term for each
visible and hidden unit, ommited from the image for simplicity.

The energy function measures the quality of a joint assignment:

.. math:: 

   E(\mathbf{v}, \mathbf{h}) = \sum_i \sum_j w_{ij}v_ih_j + \sum_i b_iv_i
     + \sum_j c_jh_j

In the formula above, :math:`\mathbf{b}` and :math:`\mathbf{c}` are the
intercept vectors for the visible and hidden layers, respectively. The
joint probability of the model is defined in terms of the energy:

.. math::
{'fraction_non_alphanumeric': 0.11301065719360569, 'fraction_numerical': 0.0017761989342806395, 'mean_word_length': 4.331360946745562, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '903748', 'n_tokens_mistral': 1200, 'n_tokens_neox': 1112, 'n_words': 570}