<commit_before>Datasets
========

MNIST Dataset
+++++++++++++


 `MNIST <http://yann.lecun.com/exdb/mnist>`_ dataset consists of handwritten 
 digit images and it is divided in 60 000 examples for the training set and 
 10 000 examples for testing. All examples have been size-normalized and 
 centered in a fixed size image of 28 x 28 pixels. In the original dataset 
 each pixel of the image is represented by a value between 0 and 255, where 
 0 is black, 255 is  white and anything in between is a different shade of grey. 


 Here are some examples of MNIST digits: |5|, |0|, |4|, |1|, |9|, |2|

 .. |5| image:: /images/mnist_0.png
 .. |0| image:: /images/mnist_1.png
 .. |4| image:: /images/mnist_2.png
 .. |1| image:: /images/mnist_3.png
 .. |9| image:: /images/mnist_4.png
 .. |2| image:: /images/mnist_5.png
 
 For convenience we pickled the dataset to make it easier to use in python.
 The pickled file represents a tuple of 3 lists : the training set, the 
 validation set and the testing set. Each element of any of the three lists 
 represents a minibatch of 10 examples. Such an elemnt is a tuple composed 
 of the list of 10 images and the list of class labels for each of the
 images. An image is represented as numpy array of 784 (28 x 28) float
 values between 0 and 1 ( 0 stands for black, 1 for white). The labels
 are numbers between 0 and 9 indicating which digit the image
 represents. Loading and accessing the dataset in the python can be done as 
 follows: 

 .. code-block:: python<commit_msg>removed / in front of paths for images<commit_after>Datasets
========

MNIST Dataset
+++++++++++++


 `MNIST <http://yann.lecun.com/exdb/mnist>`_ dataset consists of handwritten 
 digit images and it is divided in 60 000 examples for the training set and 
 10 000 examples for testing. All examples have been size-normalized and 
 centered in a fixed size image of 28 x 28 pixels. In the original dataset 
 each pixel of the image is represented by a value between 0 and 255, where 
 0 is black, 255 is  white and anything in between is a different shade of grey. 


 Here are some examples of MNIST digits: |5|, |0|, |4|, |1|, |9|, |2|

 .. |5| image:: images/mnist_0.png
 .. |0| image:: images/mnist_1.png
 .. |4| image:: images/mnist_2.png
 .. |1| image:: images/mnist_3.png
 .. |9| image:: images/mnist_4.png
 .. |2| image:: images/mnist_5.png
 
 For convenience we pickled the dataset to make it easier to use in python.
 The pickled file represents a tuple of 3 lists : the training set, the 
 validation set and the testing set. Each element of any of the three lists 
 represents a minibatch of 10 examples. Such an elemnt is a tuple composed 
 of the list of 10 images and the list of class labels for each of the
 images. An image is represented as numpy array of 784 (28 x 28) float
 values between 0 and 1 ( 0 stands for black, 1 for white). The labels
 are numbers between 0 and 9 indicating which digit the image
 represents. Loading and accessing the dataset in the python can be done as 
 follows: 

 .. code-block:: python
{'fraction_non_alphanumeric': 0.08466819221967964, 'fraction_numerical': 0.03792088917947042, 'mean_word_length': 3.4671532846715327, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '181317', 'n_tokens_mistral': 980, 'n_tokens_neox': 854, 'n_words': 479}