<commit_before># Helper scripts to download the original test-data

download_mnist.py
Downloads the original handwritten mnist digits from Yan LeCun's website at:
[http://yann.lecun.com/exdb/mnist/]
The data is transformed into a single [https://support.hdfgroup.org/HDF5/](HDF5 database) file.

## Requirements:
* HDF5 OS drivers (Arch linux: hdf5-cpp-fortran, Mac homebrew: )
* Python HDF5 support: python-h5py [http://docs.h5py.org/en/latest/index.html]

## Download required data:
Execute:
  python download_mnist.py

Currently only tested with Python 3.x

## Result:
mnist.h5 in directory datasets

Then use
  mnisttest <path-to-data>/mnist.h5<commit_msg>formatting<commit_after># Helper scripts to download the original test-data

## MNIST: download_mnist.py
Downloads the original handwritten mnist digits from [Yan LeCun's website](http://yann.lecun.com/exdb/mnist/)
The data is transformed into a single [HDF5 database](https://support.hdfgroup.org/HDF5/) file.

## Requirements:
* HDF5 OS drivers (Arch linux: hdf5-cpp-fortran, Mac homebrew: )
* Python HDF5 support: [python-h5py](http://docs.h5py.org/en/latest/index.html)

## Download required data:
```
python download_mnist.py
```
Currently only tested with Python 3.x

## Result:
```mnist.h5``` in directory ```datasets```.

Then use:
```
mnisttest <path-to-database>/mnist.h5
```
{'fraction_non_alphanumeric': 0.12862453531598514, 'fraction_numerical': 0.01486988847583643, 'mean_word_length': 4.877729257641922, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5172165', 'n_tokens_mistral': 483, 'n_tokens_neox': 462, 'n_words': 128}