<commit_before>## Overview
This is a fun proof-of-concept Naive Bayes classifier built using Elasticsearch aggregations.  

Naive Bayes classifiers work based on term and document frequencies differences between class labels.  Elasticsearch is a search engine which is designed to handle term/doc frequencies and aggregating across features (such as labels), so this seemed like a natural fit.

Most of the code is plumbing...the actual classification only requires two different aggregation queries.

### Dataset
This project uses a subtree-split of the [Reuters-21578-JSON dataset](https://github.com/fergiemcdowall/reuters-21578-json).  This is the classic Reuters dataset which has been jsonified.

Note: the newer Reuters RCV1 would have been a better candidate for classification, but I did not feel like writing an XML parser for it :)

### Scripts

- `importReuters.php` : imports the Reuters dataset into Elasticsearch.  Data will go into `/reuters/train` and `/reuters/test`
- `testReuters.php` : begins the classification on the test data
- `/src/ElasticBayes/` : the NaiveBayes classification implementation using Elasticsearch

### Running this example locally

```shell
git clone https://github.com/polyfractal/ElasticBayes.git
curl -s http://getcomposer.org/installer | php
php composer.phar install
php importReuters.php
php testReuters.php
```

## Training
Training the classifier is simple.  Here are the steps:

1. Create an index with an appropriate mapping
2. Index the documents from your corpus
3. There is no step 3

Naive Bayes classifiers are "trained" by obtaining term and document frequencies for each label.  This is typically done by tokenizing your input, normalizing the tokens, then building an in-memory hash which holds all the data.

Elasticsearch does this natively out of the box.  Just create an analyzer and index documents.  Voila!  Trained Naive Bayes classifier

## Testing
Once the clssifier has been "trained", we can use it to predict the classes of new documents.  Rather than walk through all the code, I am going to show the salient Elasticsearch queries and explain their purpose.  The rest of the code should be trivial to understand, since it is just logistical plumbing.

#### Total label counts
This query obtains the set of labels and their counts.

```php
$params = [
    'index' => 'reuters',
    'type' => 'train',
    'search_type' => 'count',
    'body' => [
        'aggs' => [
            'counts' => [
                'terms' => [
                    'field' => $this->field,
                    'size' => 200
                ]
            ]
        ]
    ]
];
$results = $this->client->search($params);
```
<commit_msg>Add blurb about test fields/labels<commit_after>## Overview
This is a fun proof-of-concept Naive Bayes classifier built using Elasticsearch aggregations.  

Naive Bayes classifiers work based on term and document frequencies differences between class labels.  Elasticsearch is a search engine which is designed to handle term/doc frequencies and aggregating across features (such as labels), so this seemed like a natural fit.

Most of the code is plumbing...the actual classification only requires two different aggregation queries.

### Dataset
This project uses a subtree-split of the [Reuters-21578-JSON dataset](https://github.com/fergiemcdowall/reuters-21578-json).  This is the classic Reuters dataset which has been jsonified.

Note: the newer Reuters RCV1 would have been a better candidate for classification, but I did not feel like writing an XML parser for it :)

### Scripts

- `importReuters.php` : imports the Reuters dataset into Elasticsearch.  Data will go into `/reuters/train` and `/reuters/test`
- `testReuters.php` : begins the classification on the test data
- `/src/ElasticBayes/` : the NaiveBayes classification implementation using Elasticsearch

### Running this example locally

```shell
git clone https://github.com/polyfractal/ElasticBayes.git
curl -s http://getcomposer.org/installer | php
php composer.phar install
php importReuters.php
php testReuters.php
```

## Training
Training the classifier is simple.  Here are the steps:

1. Create an index with an appropriate mapping
2. Index the documents from your corpus
3. There is no step 3

Naive Bayes classifiers are "trained" by obtaining term and document frequencies for each label.  This is typically done by tokenizing your input, normalizing the tokens, then building an in-memory hash which holds all the data.

Elasticsearch does this natively out of the box.  Just create an analyzer and index documents.  Voila!  Trained Naive Bayes classifier

## Testing
Once the clssifier has been "trained", we can use it to predict the classes of new documents.  Rather than walk through all the code, I am going to show the salient Elasticsearch queries and explain their purpose.  The rest of the code should be trivial to understand, since it is just logistical plumbing.

There are three fields and two sets of labels that you can potentially test on:

**Labels**
- `topics`
- `places`

**Input Fields**
- `body`
- `title`
- `combined` - both `body` and `title` concatenated into a single field

#### Total label counts
This query obtains the set of labels and their counts.

```php
$params = [
    'index' => 'reuters',
    'type' => 'train',
    'search_type' => 'count',
    'body' => [
        'aggs' => [
            'counts' => [
                'terms' => [
                    'field' => $this->field,
                    'size' => 200
                ]
            ]
        ]
    ]
];
$results = $this->client->search($params);
```

{'fraction_non_alphanumeric': 0.07945840014252628, 'fraction_numerical': 0.006413682522715126, 'mean_word_length': 3.523771152296535, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 27, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6282233', 'n_tokens_mistral': 1560, 'n_tokens_neox': 1454, 'n_words': 702}