<commit_before># patternizer

Introduction
____________

Hey y'all, so this is my repo for learning about machine learning. Everything is written in c++ using only the standard
library. I know, I know, there are fast libraries out there, but the point of this exercise was not to build something
as fast as possible, but rather to learn, and what better way than to do it from scratch? With that in mind, there are
additional classes associated with each of the machine learning types for support.

With that jargon out of the way, here is a current list of machine learning things I have built.

1. <a href="#perceptron">Perceptron</a>
2. Simple Neural Network (Basic cost function, 1 hidden layer, prebuilt to read MINST data).

As I continue teaching myself, this list will grow, so enjoy!



<a id="perceptron">Perceptron</a>
__________

Being the among the oldest flavors of machine learning, I've implemented one of my one.
It works like you'd expect. Train it by giving an input and a target output, and it will adjust its
weight until it gets where it should be.

In main.cpp, there is a function demonstrating how the class works called perceptronDemo.



Simple Neural Network
_____________________

Moving up the chain, I've implemented a simple neural network. It uses the Quadratic Cost Function ((target - output)^2)/2,
allows you to define a learning rate, and to define the number of inputs, hidden layer neurons, and outputs. It's pretty slick...
I've managed to get over 90% accuracy on the MINST set within 30 training epochs.

In order to make this work, I've created a Matrix library. Unfortunately, it does not use the hardware to speed things up (GPU anyone?),
but that may come with time. Also, there is a class to convert the MINST data from big to little endian and store it.

In main.cpp, you will find main.cpp implementing, training, and testing a SimpleNeuralNetwork instance.



Future
______

The next step will be making the "Simple" Neural Network more complex. This includes adding regularization techniques,
having the option of different cost functions including the cross-entropy cost function, and adding momentum to the gradient
descent.

Once these additional things are added, I believe I will be home free to continue onwards to a "deep" Network.

Other areas to be pursued:
Genetic algorithms -- Finds probable optimizations on weird data sets
                   (Most value if can take 1 pound, 500 different things with different weights and values...)

Reinforcement Learning -- Agents learn in environment meaning you don't need initial training data

Generative Learning -- The ability to "copy". Given a bunch of hand written 5's, output your own 5.

Networks with Memory -- Traditional NN "forgets" previous learning when switched to a new task. Can you stop that?
                     This includes: long-short term memory networks, elastic weight consolidation, and progressive neural networks

Less Data, Smaller Models -- Can simpler NN's learn from complex ones to reduce the training time? Transfer training...

Environment Creation -- We can't really just let a robot ai run wild... create a virtual environment and have ai learn there...




<commit_msg>md prettier<commit_after># patternizer

## Introduction
____________

Hey y'all, so this is my repo for learning about machine learning. Everything is written in c++ using only the standard
library. I know, I know, there are fast libraries out there, but the point of this exercise was not to build something
as fast as possible, but rather to learn, and what better way than to do it from scratch? With that in mind, there are
additional classes associated with each of the machine learning types for support.

With that jargon out of the way, here is a current list of machine learning things I have built.

1. <a href="#perceptron">Perceptron</a>
2. <a href="simpleNN>Simple Neural Network</a> (Basic cost function, 1 hidden layer, prebuilt to read MINST data).
3. <a href="future"> Future </a>

As I continue teaching myself, this list will grow, so enjoy!



## <a id="perceptron">Perceptron</a>
__________

Being the among the oldest flavors of machine learning, I've implemented one of my one.
It works like you'd expect. Train it by giving an input and a target output, and it will adjust its
weight until it gets where it should be.

In main.cpp, there is a function demonstrating how the class works called perceptronDemo.



## <a id="simpleNN">Simple Neural Network</a>
_____________________

Moving up the chain, I've implemented a simple neural network. It uses the Quadratic Cost Function ((target - output)^2)/2,
allows you to define a learning rate, and to define the number of inputs, hidden layer neurons, and outputs. It's pretty slick...
I've managed to get over 90% accuracy on the MINST set within 30 training epochs.

In order to make this work, I've created a Matrix library. Unfortunately, it does not use the hardware to speed things up (GPU anyone?),
but that may come with time. Also, there is a class to convert the MINST data from big to little endian and store it.

In main.cpp, you will find main.cpp implementing, training, and testing a SimpleNeuralNetwork instance.



## <a id="future">Future</a>
______

The next step will be making the "Simple" Neural Network more complex. This includes adding regularization techniques,
having the option of different cost functions including the cross-entropy cost function, and adding momentum to the gradient
descent.

Once these additional things are added, I believe I will be home free to continue onwards to a "deep" Network.

Other areas to be pursued:
Genetic algorithms -- Finds probable optimizations on weird data sets
                   (Most value if can take 1 pound, 500 different things with different weights and values...)

Reinforcement Learning -- Agents learn in environment meaning you don't need initial training data

Generative Learning -- The ability to "copy". Given a bunch of hand written 5's, output your own 5.

Networks with Memory -- Traditional NN "forgets" previous learning when switched to a new task. Can you stop that?
                     This includes: long-short term memory networks, elastic weight consolidation, and progressive neural networks

Less Data, Smaller Models -- Can simpler NN's learn from complex ones to reduce the training time? Transfer training...

Environment Creation -- We can't really just let a robot ai run wild... create a virtual environment and have ai learn there...





{'fraction_non_alphanumeric': 0.0532843362425356, 'fraction_numerical': 0.00474659317103047, 'mean_word_length': 4.067494181536074, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 19, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1101231', 'n_tokens_mistral': 1652, 'n_tokens_neox': 1524, 'n_words': 995}