<commit_before>Search results clustering for ElasticSearch
===========================================

This clustering plugin adds on-the-fly text clustering capability
to an ElasticSearch node. Clustering algorithms from the Carrot2
project (open source) or proprietary Lingo3G algorithm from
Carrot Search can be used for actual clustering implementation.


Installation
------------

In order to install a stable version of the plugin, 
run ElasticSearch's `plugin` utility (remember to pick the
ES-compatible version of the plugin from the table below!).

    bin/plugin install org.carrot2/elasticsearch-carrot2/2.2.1

To install from sources (master branch), run:

    mvn clean package
    
and then install with:

    bin/plugin install file:(plugin)/target/releases/*.zip

From ES 2.2.0 the installer will request confirmation concerning
extended security permissions. You have to accept it (and live with
it).


Usage guide
-----------

Once installed, restart ElasticSearch and point your browser to:
<http://localhost:9200/_plugin/elasticsearch-carrot2/>
(or wherever your ES node is deployed). That file contains
some sample data and query examples.

Alternatively, the plugin's folder contains `_site/curl/` sub-folder
which in turn contains sample `curl` scripts that inject data into
ES and query the clustering plugin.


Versions and compatibility
--------------------------

Recommended compatibility chart (matching versions of ES, Carrot2, 
and optionally Lingo3G). (+) means it'll probably work with newer
releases (we test against latest version from that branch). 

Starting with ES 2.0, the plugin is compiled against *an exact* version of ES
and *will not work* with any other version. The numbering of the plugin
will always correspond to the numbering of ES to easily identify
the version of ES the plugin will work with.

    ------------------------------------------------------------------
    | Clustering Plugin, ES (matching versions)  | Carrot2 | Lingo3G |
    ------------------------------------------------------------------
    | (master, unreleased)                       | 3.12.0  | 1.13.0  |
    | 2.2.1                                      | 3.12.0  | 1.13.0  |
    | 2.2.0                                      | 3.11.0  | 1.12.3  |
    | 2.1.0 -> 2.1.2                             | 3.11.0  | 1.12.3  |
    | 2.0.0 -> 2.0.2                             | 3.11.0  | 1.12.3  |
    ------------------------------------------------------------------

    ------------------------------------------------------------------
    | Clustering Plugin | Elasticsearch          | Carrot2 | Lingo3G |
    ------------------------------------------------------------------<commit_msg>Compatibility chart.<commit_after>Search results clustering for ElasticSearch
===========================================

This clustering plugin adds on-the-fly text clustering capability
to an ElasticSearch node. Clustering algorithms from the Carrot2
project (open source) or proprietary Lingo3G algorithm from
Carrot Search can be used for actual clustering implementation.


Installation
------------

In order to install a stable version of the plugin, 
run ElasticSearch's `plugin` utility (remember to pick the
ES-compatible version of the plugin from the table below!).

    bin/plugin install org.carrot2/elasticsearch-carrot2/2.3.0

To install from sources (master branch), run:

    mvn clean package
    
and then install with:

    bin/plugin install file:(plugin)/target/releases/*.zip

From ES 2.2.0 the installer will request confirmation concerning
extended security permissions. You have to accept it (and live with
it).


Usage guide
-----------

Once installed, restart ElasticSearch and point your browser to:
<http://localhost:9200/_plugin/elasticsearch-carrot2/>
(or wherever your ES node is deployed). That file contains
some sample data and query examples.

Alternatively, the plugin's folder contains `_site/curl/` sub-folder
which in turn contains sample `curl` scripts that inject data into
ES and query the clustering plugin.


Versions and compatibility
--------------------------

Recommended compatibility chart (matching versions of ES, Carrot2, 
and optionally Lingo3G). (+) means it'll probably work with newer
releases (we test against latest version from that branch). 

Starting with ES 2.0, the plugin is compiled against *an exact* version of ES
and *will not work* with any other version. The numbering of the plugin
will always correspond to the numbering of ES to easily identify
the version of ES the plugin will work with.

    ------------------------------------------------------------------
    | Clustering Plugin, ES (matching versions)  | Carrot2 | Lingo3G |
    ------------------------------------------------------------------
    | (master, unreleased)                       | 3.12.0  | 1.13.0  |
    | 2.3.0                                      | 3.12.0  | 1.13.0  |
    | 2.2.1                                      | 3.12.0  | 1.13.0  |
    | 2.2.0                                      | 3.11.0  | 1.12.3  |
    | 2.1.0 -> 2.1.2                             | 3.11.0  | 1.12.3  |
    | 2.0.0 -> 2.0.2                             | 3.11.0  | 1.12.3  |
    ------------------------------------------------------------------

    ------------------------------------------------------------------
    | Clustering Plugin | Elasticsearch          | Carrot2 | Lingo3G |
    ------------------------------------------------------------------
{'fraction_non_alphanumeric': 0.21405228758169934, 'fraction_numerical': 0.03140885984023239, 'mean_word_length': 1.6321070234113713, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1132991', 'n_tokens_mistral': 1460, 'n_tokens_neox': 1293, 'n_words': 536}