<commit_before>Experiments in text classification, inspired by reading an [introduction to
Naive Bayes](https://monkeylearn.com/blog/practical-explanation-naive-bayes-classifier/).

My focus was on learning the fundamental algorithms rather than building a good
general purpose classification library. There are tools out there that are
suitable for real world use - this is not one of them.

## Usage

Basic naive bayes classification:

    require 'classifier'

    classifier = Classifier::NaiveBayes.new("spam", "ham")
    classifier.train("spam", "cheap", "viagra")
    classifier.train("ham", "cheap", "fruit")
    puts classifier.classify("fruit"),inspect

To test the performance of a classifier, first build a CSV data file like this:

    category1,feature1,feature2
    category2,feature1,feature3

Then run the test harness:

    ruby -I lib bin/classify-nb input.csv

To run multiple files through the test harness:

    find . -name "input*.csv" | xargs -L1 ruby -I lib bin/classify-nb

## Specs

There are a number of specs that demonstrate intended behaviour. You can run them via rspec:

    bundle exec rspec --format doc

## TODO

* Adjust Classifier::NaiveBayes to allow a store to be passed in
* Build a demonstration postgres store
* expand README with more examples
  * text classification: tokenisation, stemming, stop words, etc
  * feature selection
  * writing a custom store
* consider adding a way for documents to be removed from a store
* should we accept training data with no features?
* should we attempt to classify with no features?<commit_msg>add TODO<commit_after>Experiments in text classification, inspired by reading an [introduction to
Naive Bayes](https://monkeylearn.com/blog/practical-explanation-naive-bayes-classifier/).

My focus was on learning the fundamental algorithms rather than building a good
general purpose classification library. There are tools out there that are
suitable for real world use - this is not one of them.

## Usage

Basic naive bayes classification:

    require 'classifier'

    classifier = Classifier::NaiveBayes.new("spam", "ham")
    classifier.train("spam", "cheap", "viagra")
    classifier.train("ham", "cheap", "fruit")
    puts classifier.classify("fruit"),inspect

To test the performance of a classifier, first build a CSV data file like this:

    category1,feature1,feature2
    category2,feature1,feature3

Then run the test harness:

    ruby -I lib bin/classify-nb input.csv

To run multiple files through the test harness:

    find . -name "input*.csv" | xargs -L1 ruby -I lib bin/classify-nb

## Specs

There are a number of specs that demonstrate intended behaviour. You can run them via rspec:

    bundle exec rspec --format doc

## TODO

* Adjust Classifier::NaiveBayes to allow a store to be passed in
* Build a demonstration postgres store
* expand README with more examples
  * text classification: tokenisation, stemming, stop words, etc
  * feature selection
  * writing a custom store
* consider adding a way for documents to be removed from a store
* should we accept training data with no features?
* should we attempt to classify with no features?
* move caching to a class that's external to stores
{'fraction_non_alphanumeric': 0.07234175241658872, 'fraction_numerical': 0.004365450576863112, 'mean_word_length': 4.207792207792208, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2432603', 'n_tokens_mistral': 908, 'n_tokens_neox': 855, 'n_words': 395}