<commit_before># ElasticSearch Approx Plugin

A plugin for [ElasticSearch](http://www.elasticsearch.org/) to use approximate
methods for certain queries, to greatly reduce memory usage and network traffic.

Approximate counting is performed via the excellent probabilistic data
structures from [stream-lib](https://github.com/clearspring/stream-lib).

This work is inspired in part by
[elasticsearch-ls-plugins](https://github.com/lovelysystems/elasticsearch-ls-plugins)
by Lovely Systems -- some of their code has been reused here.

**NB We have not yet updated the plugin to work with ElasticSearch 0.20+.**

## Distinct Date Histogram

Counts the total numbers of terms and _unique_ terms in a given field, for each
interval in a date range.

For example:

    curl -XPOST "http://localhost:9200/pop1/_search?search_type=count&pretty=true" -d'{
        "query": {
            "match_all" : {}
        },
            "facets" : {
                "histo" : {
                    "distinct_date_histogram" : {
                        "key_field" : "datetime",
                        "value_field" : "userid",
                        "max_exact_per_shard" : 100,
                        "interval" : "hour"
                    }
                }
            }
    }'

Returns something like:

    {
      "took" : 3966,
      "timed_out" : false,
      "_shards" : {
        "total" : 5,
        "successful" : 5,
        "failed" : 0
      },
      "hits" : {
        "total" : 24131101,
        "max_score" : 0.0,
        "hits" : [ ]
      },
      "facets" : {
        "histo" : {
          "_type" : "distinct_date_histogram",
          "entries" : [ {
            "time" : 1333580400000,
            "count" : 2133,
            "distinct_count" : 1515
          }, {
            "time" : 1333584000000,
            "count" : 191991,
            "distinct_count" : 12407
          }, {
              ...
          }, {
            "time" : 1333753200000,
            "count" : 238022,
            "distinct_count" : 26092
          } ]
        }
      }
    }

The facet works initially by keeping a hashset of all terms encountered, but
when this grows too large, it falls back to an approximate counting method.

The `max_exact_per_shard` parameter controls how many distinct values will be
gathered for a given bucket from a single shard, before migrating to the
approximate count instead. This can be used to keep memory usage under control.
It defaults to 1000.

The approximate counting method is hard-coded to HyperLogLog method with a
relative standard deviation of 0.0025, which uses about 80KB of memory per
bucket per shard.

An obvious extension would be to give users the option to control the algorithm
used, and its parameters, so they can tune these based on the expected
cardinality of the field in question, and the desired memory usage. (Pull
requests gratefully received.)

Note that the counts are based on terms, so if the `value_field` is tokenized,
the result won't indicate the number of distinct _values_ in that field, but
rather, the number of distinct tokens (post-analysis).

## Term list facet

This is a simple facet to quickly retrieve an unsorted term list for a field,
if you don't care about counts or ordering etc. It allows you to set a max_per_shard
cutoff similar to the previous facet.

For example:

    curl -XPOST "http://localhost:9200/pop1/_search?search_type=count&pretty=true" -d'{
        "query": {
            "match_all" : {}
        },
        "facets" : {
            "term_list_facet" : {
                "term_list" : {
                    "key_field" : "txt1",
                    "max_per_shard" : 100
                }
            }
        }
    }'

Returns something like:

    {
      "took" : 45,
      "timed_out" : false,
      "_shards" : {
        "total" : 3,
        "successful" : 3,
        "failed" : 0
      },
      "hits" : {
        "total" : 132,
        "max_score" : 1.0,
        "hits" : [ ]
      },
      "facets" : {
        "term_list_facet" : {
          "_type" : "term_list",
          "entries" : [ "trdq", "amrqkke", "ebztmm", "pja", "qobmepbor", "bxpoh", "krsm", "kpgz", "hotodwfq", "qbpxxlfin", "lsnosgx", "qyznyhrqcu", "poekzt", "qbsmks", "adbazy", "swnjdvziqh", "eqabkxb", "xdz", "jlg", "scn", "jdn" ]
        }
      }
    }

There is also a `read_from_cache` option. If set to true, the values will be
read from ElasticSearch's field data cache, instead of directly from the Lucene
index on disk. Whether this is faster or not depends on whether the data is
already cached -- if it has to be read and cached first, it will be slower. But
it's hard to make any absolute recommendations due to OS disk caching etc., so
the best advice is to experiment.

This facet doesn't actually use anything clever like appropximate counting --
it's not really approximate in the same sense as the previous one -- but we
thought you might find it useful.

## Building and testing

It's all done via Maven, so just `mvn test` to build the plugin and run the
tests. Amongst other things, they check that the distinct counts are within a
tolerance of 1% of the expected values.

If you get errors like `OutOfMemoryError[Direct buffer memory]]]` or other
weird errors from ElasticSearch, then you may need to raise the amount of
memory you allocate to the mvn process. Try using `-Xmx1G` in the `MAVEN_OPTS`
variable. (If you're an Eclipse user, put `-Xmx1G` in the VM Arguments box of
the Arguments tab in Run Configurations for that test.)

Yes, this is quite a lot of memory for unit tests, but the tests use several
iterations of randomly generated data of increasing size, in order to verify
the accuracy of the approximate counts. The final run puts over a million
distinct values in each bucket. For the same reason, the tests take several
minutes to run.

### Important note
<commit_msg>filtering is not supported in direct mode<commit_after># ElasticSearch Approx Plugin

A plugin for [ElasticSearch](http://www.elasticsearch.org/) to use approximate
methods for certain queries, to greatly reduce memory usage and network traffic.

Approximate counting is performed via the excellent probabilistic data
structures from [stream-lib](https://github.com/clearspring/stream-lib).

This work is inspired in part by
[elasticsearch-ls-plugins](https://github.com/lovelysystems/elasticsearch-ls-plugins)
by Lovely Systems -- some of their code has been reused here.

**NB We have not yet updated the plugin to work with ElasticSearch 0.20+.**

## Distinct Date Histogram

Counts the total numbers of terms and _unique_ terms in a given field, for each
interval in a date range.

For example:

    curl -XPOST "http://localhost:9200/pop1/_search?search_type=count&pretty=true" -d'{
        "query": {
            "match_all" : {}
        },
            "facets" : {
                "histo" : {
                    "distinct_date_histogram" : {
                        "key_field" : "datetime",
                        "value_field" : "userid",
                        "max_exact_per_shard" : 100,
                        "interval" : "hour"
                    }
                }
            }
    }'

Returns something like:

    {
      "took" : 3966,
      "timed_out" : false,
      "_shards" : {
        "total" : 5,
        "successful" : 5,
        "failed" : 0
      },
      "hits" : {
        "total" : 24131101,
        "max_score" : 0.0,
        "hits" : [ ]
      },
      "facets" : {
        "histo" : {
          "_type" : "distinct_date_histogram",
          "entries" : [ {
            "time" : 1333580400000,
            "count" : 2133,
            "distinct_count" : 1515
          }, {
            "time" : 1333584000000,
            "count" : 191991,
            "distinct_count" : 12407
          }, {
              ...
          }, {
            "time" : 1333753200000,
            "count" : 238022,
            "distinct_count" : 26092
          } ]
        }
      }
    }

The facet works initially by keeping a hashset of all terms encountered, but
when this grows too large, it falls back to an approximate counting method.

The `max_exact_per_shard` parameter controls how many distinct values will be
gathered for a given bucket from a single shard, before migrating to the
approximate count instead. This can be used to keep memory usage under control.
It defaults to 1000.

The approximate counting method is hard-coded to HyperLogLog method with a
relative standard deviation of 0.0025, which uses about 80KB of memory per
bucket per shard.

An obvious extension would be to give users the option to control the algorithm
used, and its parameters, so they can tune these based on the expected
cardinality of the field in question, and the desired memory usage. (Pull
requests gratefully received.)

Note that the counts are based on terms, so if the `value_field` is tokenized,
the result won't indicate the number of distinct _values_ in that field, but
rather, the number of distinct tokens (post-analysis).

## Term list facet

This is a simple facet to quickly retrieve an unsorted term list for a field,
if you don't care about counts or ordering etc. It allows you to set a max_per_shard
cutoff similar to the previous facet.

For example:

    curl -XPOST "http://localhost:9200/pop1/_search?search_type=count&pretty=true" -d'{
        "query": {
            "match_all" : {}
        },
        "facets" : {
            "term_list_facet" : {
                "term_list" : {
                    "key_field" : "txt1",
                    "max_per_shard" : 100
                }
            }
        }
    }'

Returns something like:

    {
      "took" : 45,
      "timed_out" : false,
      "_shards" : {
        "total" : 3,
        "successful" : 3,
        "failed" : 0
      },
      "hits" : {
        "total" : 132,
        "max_score" : 1.0,
        "hits" : [ ]
      },
      "facets" : {
        "term_list_facet" : {
          "_type" : "term_list",
          "entries" : [ "trdq", "amrqkke", "ebztmm", "pja", "qobmepbor", "bxpoh", "krsm", "kpgz", "hotodwfq", "qbpxxlfin", "lsnosgx", "qyznyhrqcu", "poekzt", "qbsmks", "adbazy", "swnjdvziqh", "eqabkxb", "xdz", "jlg", "scn", "jdn" ]
        }
      }
    }

There is also a `read_from_cache` option. If set to true, the values will be
read from ElasticSearch's field data cache, instead of directly from the Lucene
index on disk. Whether this is faster or not depends on whether the data is
already cached -- if it has to be read and cached first, it will be slower. But
it's hard to make any absolute recommendations due to OS disk caching etc., so
the best advice is to experiment.

One other important note is that when reading directly from the Lucene index, filtering is not supported.	


This facet doesn't actually use anything clever like appropximate counting --
it's not really approximate in the same sense as the previous one -- but we
thought you might find it useful.



## Building and testing

It's all done via Maven, so just `mvn test` to build the plugin and run the
tests. Amongst other things, they check that the distinct counts are within a
tolerance of 1% of the expected values.

If you get errors like `OutOfMemoryError[Direct buffer memory]]]` or other
weird errors from ElasticSearch, then you may need to raise the amount of
memory you allocate to the mvn process. Try using `-Xmx1G` in the `MAVEN_OPTS`
variable. (If you're an Eclipse user, put `-Xmx1G` in the VM Arguments box of
the Arguments tab in Run Configurations for that test.)

Yes, this is quite a lot of memory for unit tests, but the tests use several
iterations of randomly generated data of increasing size, in order to verify
the accuracy of the approximate counts. The final run puts over a million
distinct values in each bucket. For the same reason, the tests take several
minutes to run.

### Important note

{'fraction_non_alphanumeric': 0.08845539392920206, 'fraction_numerical': 0.021861599260068948, 'mean_word_length': 2.4119334480780266, 'pattern_counts': {'":': 4, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 4, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2782439', 'n_tokens_mistral': 3713, 'n_tokens_neox': 3394, 'n_words': 1393}