<commit_before>Elasticsearch Word Delimiter 2 Filter
=====================================

The WordDelimiterFilter2 analysis plugin provides with an extension over the mainstream WordDelimiterFilter.

Installation
-----------

Simply run at the root of your ElasticSearch v0.20.2+ installation:

	bin/plugin -install com.yakaz.elasticsearch.plugins/elasticsearch-analysis-worddelimiter2/1.2.0

This will download the plugin from the Central Maven Repository.

For older versions of ElasticSearch, you can still use the longer:

	bin/plugin -url http://oss.sonatype.org/content/repositories/releases/com/yakaz/elasticsearch/plugins/elasticsearch-analysis-worddelimiter2/1.0.1/elasticsearch-analysis-worddelimiter2-1.0.1.zip install elasticsearch-analysis-worddelimiter2

In order to declare this plugin as a dependency, add the following to your `pom.xml`:

```xml
<dependency>
    <groupId>com.yakaz.elasticsearch.plugins</groupId>
    <artifactId>elasticsearch-analysis-worddelimiter2</artifactId>
    <version>1.2.0</version>
</dependency>
```

Version matrix:

	┌─────────────────────────────────┬──────────────────────┐
	│ WordDelimiter 2 Analysis Plugin │ ElasticSearch        │
	├─────────────────────────────────┼──────────────────────┤
	│ 1.2.x                           │ 1.0.0.RC1 ─► (1.4.2) │
	├─────────────────────────────────┼──────────────────────┤
	│ 1.1.x                           │ 0.90 ─► (0.90.11)    │
	├─────────────────────────────────┼──────────────────────┤
	│ 1.0.x                           │ 0.19 ─► 0.20         │
	└─────────────────────────────────┴──────────────────────┘

Description
-----------

Lucene 4's WordDelimiterFilter is exposed since ElasticSearch v0.17.0.
This plugin exposes an extension over this filter, packaged as an ElasticSearch 0.19.0+ plugin.

See [Lucene WordDelimiterFilter JavaDoc][WDFJavadoc] for more information about the base functionality.

Added features
--------------
<commit_msg>v1.2.0 works with ES 1.4.3<commit_after>Elasticsearch Word Delimiter 2 Filter
=====================================

The WordDelimiterFilter2 analysis plugin provides with an extension over the mainstream WordDelimiterFilter.

Installation
-----------

Simply run at the root of your ElasticSearch v0.20.2+ installation:

	bin/plugin -install com.yakaz.elasticsearch.plugins/elasticsearch-analysis-worddelimiter2/1.2.0

This will download the plugin from the Central Maven Repository.

For older versions of ElasticSearch, you can still use the longer:

	bin/plugin -url http://oss.sonatype.org/content/repositories/releases/com/yakaz/elasticsearch/plugins/elasticsearch-analysis-worddelimiter2/1.0.1/elasticsearch-analysis-worddelimiter2-1.0.1.zip install elasticsearch-analysis-worddelimiter2

In order to declare this plugin as a dependency, add the following to your `pom.xml`:

```xml
<dependency>
    <groupId>com.yakaz.elasticsearch.plugins</groupId>
    <artifactId>elasticsearch-analysis-worddelimiter2</artifactId>
    <version>1.2.0</version>
</dependency>
```

Version matrix:

	┌─────────────────────────────────┬──────────────────────┐
	│ WordDelimiter 2 Analysis Plugin │ ElasticSearch        │
	├─────────────────────────────────┼──────────────────────┤
	│ 1.2.x                           │ 1.0.0.RC1 ─► (1.4.3) │
	├─────────────────────────────────┼──────────────────────┤
	│ 1.1.x                           │ 0.90 ─► (0.90.11)    │
	├─────────────────────────────────┼──────────────────────┤
	│ 1.0.x                           │ 0.19 ─► 0.20         │
	└─────────────────────────────────┴──────────────────────┘

Description
-----------

Lucene 4's WordDelimiterFilter is exposed since ElasticSearch v0.17.0.
This plugin exposes an extension over this filter, packaged as an ElasticSearch 0.19.0+ plugin.

See [Lucene WordDelimiterFilter JavaDoc][WDFJavadoc] for more information about the base functionality.

Added features
--------------

{'fraction_non_alphanumeric': 0.25966709346991035, 'fraction_numerical': 0.032266325224071704, 'mean_word_length': 3.792638036809816, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 19, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 4}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4565248', 'n_tokens_mistral': 1403, 'n_tokens_neox': 1181, 'n_words': 304}