<commit_before>================
Searching with S
================

.. contents::
   :local:


Overview
========

ElasticUtils makes querying and filtering and collecting facets from
ElasticSearch simple.

For example::

   q = (S().filter(product='firefox')
           .filter(version='4.0', platform='all')
           .facet(products={'field': 'product', 'global': True})
           .facet(versions={'field': 'version'})
           .facet(platforms={'field': 'platform'})
           .facet(types={'field': 'type'})
           .doctypes('addon')
           .indexes('addon_index')
           .query(title='Example'))


The ElasticSearch REST API curl would look like this::

    $ curl -XGET 'http://localhost:9200/addon_index/addon/_search' -d '{
    'query': {'term': {'title': 'Example'}},
    'filter': {'and': [{'term': {'product': 'firefox'}},
                       {'term': {'platform': 'all'}},
                       {'term': {'version': '4.0'}}]},
    'facets': {
       'platforms': {
           'facet_filter': {
               'and': [
                   {'term': {'product': 'firefox'}},
                   {'term': {'platform': 'all'}},
                   {'term': {'version': '4.0'}}]},
           'field': 'platform'},
       'products': {
           'facet_filter': {
               'and': [
                   {'term': {'product': 'firefox'}},
                   {'term': {'platform': 'all'}},
                   {'term': {'version': '4.0'}}]},
           'field': 'product',
           'global': True},
       'types': {
           'facet_filter': {
               'and': [
                   {'term': {'product': 'firefox'}},
                   {'term': {'platform': 'all'}},
                   {'term': {'version': '4.0'}}]},
           'field': 'type'},
       'versions': {
           'facet_filter': {
               'and': [
                   {'term': {'product': 'firefox'}},
                   {'term': {'platform': 'all'}},
                   {'term': {'version': '4.0'}}]},
           'field': 'version'}}
    }
    '

That's it!

For the rest of this chapter, when we translate ElasticUtils queries
to their equivalent ElasticSearch REST API, we're going to use a
shorthand and only talk about the body of the request which we'll call
the `ElasticSearch JSON`.


.. seealso::

   http://www.elasticsearch.org/guide/reference/api/
     ElasticSearch docs on api

   http://www.elasticsearch.org/guide/reference/api/search/
     ElasticSearch docs on search api

   http://curl.haxx.se/
     Documentation on curl


All about S
===========

Basic untyped S
---------------

`S` is the class that you instantiate to create a search. For example::

    searcher = S()


`S` has a bunch of methods that all return a new `S` with additional
accumulated search criteria.

For example::

   s1 = S()

   s2 = s1.query(content__text='tabs')

   s3 = s2.filter(awesome=True)

   s4 = s2.filter(awesome=False)

`s1`, `s2`, and `s3` are all different `S` objects. `s1` is a match
all.

`s2` has a query.

`s3` has everything in `s2` plus a ``awesome=True`` filter.

`s4` has everything in `s2` with a ``awesome=False`` filter.

When you create an `S` with no type, it's called an "untyped S". If
you don't specify ``.values_dict`` or ``.values_list``, then your
search results are in the form of a sequence of `DefaultMappingType`
instances. More about this in :ref:`queries-mapping-type`.


Typed S
-------

You can also construct a `typed S` which is an `S` with a
`MappingType` subclass. For example::

    from elasticutils import MappingType, S

    class MyMappingType(MappingType):
        @classmethod
        def get_index(cls):
            return 'sumo_index'

        @classmethod
        def get_mapping_type_name(cls):
            return 'mymappingtype'


    results = S(MyMappingType).query(title__text='plugins')


``results`` will be an iterable of `MyMappingType` instances---one for
each search result.


Match All
=========

By default ``S()`` with no filters or queries specified will do a
``match_all`` query in ElasticSearch.

.. seealso::

   http://www.elasticsearch.org/guide/reference/query-dsl/match-all-query.html
     ElasticSearch match_all documentation


Queries vs. Filters
===================

A search can contain multiple queries and multiple filters. The two
things are very different.

A filter determines whether a document is in the results set or
not. If you do a term filter on whether field `foo` has value `bar`,
then the result set ONLY has documents where `foo` has value `bar`.
Filters are fast and filter results are cached in ElasticSearch.

A query affects the score for a document. If you do a term query on
whether field `foo` has value `bar`, then the result set will score
documents where the query holds true higher than documents where the
query does not hold true. Queries are slower than filters and
query results are not cached in ElasticSearch.

The other place where this affects things is when you specify
facets. See :ref:`queries-chapter-facets-section` for details.<commit_msg>Tweak the filter caching text further.<commit_after>================
Searching with S
================

.. contents::
   :local:


Overview
========

ElasticUtils makes querying and filtering and collecting facets from
ElasticSearch simple.

For example::

   q = (S().filter(product='firefox')
           .filter(version='4.0', platform='all')
           .facet(products={'field': 'product', 'global': True})
           .facet(versions={'field': 'version'})
           .facet(platforms={'field': 'platform'})
           .facet(types={'field': 'type'})
           .doctypes('addon')
           .indexes('addon_index')
           .query(title='Example'))


The ElasticSearch REST API curl would look like this::

    $ curl -XGET 'http://localhost:9200/addon_index/addon/_search' -d '{
    'query': {'term': {'title': 'Example'}},
    'filter': {'and': [{'term': {'product': 'firefox'}},
                       {'term': {'platform': 'all'}},
                       {'term': {'version': '4.0'}}]},
    'facets': {
       'platforms': {
           'facet_filter': {
               'and': [
                   {'term': {'product': 'firefox'}},
                   {'term': {'platform': 'all'}},
                   {'term': {'version': '4.0'}}]},
           'field': 'platform'},
       'products': {
           'facet_filter': {
               'and': [
                   {'term': {'product': 'firefox'}},
                   {'term': {'platform': 'all'}},
                   {'term': {'version': '4.0'}}]},
           'field': 'product',
           'global': True},
       'types': {
           'facet_filter': {
               'and': [
                   {'term': {'product': 'firefox'}},
                   {'term': {'platform': 'all'}},
                   {'term': {'version': '4.0'}}]},
           'field': 'type'},
       'versions': {
           'facet_filter': {
               'and': [
                   {'term': {'product': 'firefox'}},
                   {'term': {'platform': 'all'}},
                   {'term': {'version': '4.0'}}]},
           'field': 'version'}}
    }
    '

That's it!

For the rest of this chapter, when we translate ElasticUtils queries
to their equivalent ElasticSearch REST API, we're going to use a
shorthand and only talk about the body of the request which we'll call
the `ElasticSearch JSON`.


.. seealso::

   http://www.elasticsearch.org/guide/reference/api/
     ElasticSearch docs on api

   http://www.elasticsearch.org/guide/reference/api/search/
     ElasticSearch docs on search api

   http://curl.haxx.se/
     Documentation on curl


All about S
===========

Basic untyped S
---------------

`S` is the class that you instantiate to create a search. For example::

    searcher = S()


`S` has a bunch of methods that all return a new `S` with additional
accumulated search criteria.

For example::

   s1 = S()

   s2 = s1.query(content__text='tabs')

   s3 = s2.filter(awesome=True)

   s4 = s2.filter(awesome=False)

`s1`, `s2`, and `s3` are all different `S` objects. `s1` is a match
all.

`s2` has a query.

`s3` has everything in `s2` plus a ``awesome=True`` filter.

`s4` has everything in `s2` with a ``awesome=False`` filter.

When you create an `S` with no type, it's called an "untyped S". If
you don't specify ``.values_dict`` or ``.values_list``, then your
search results are in the form of a sequence of `DefaultMappingType`
instances. More about this in :ref:`queries-mapping-type`.


Typed S
-------

You can also construct a `typed S` which is an `S` with a
`MappingType` subclass. For example::

    from elasticutils import MappingType, S

    class MyMappingType(MappingType):
        @classmethod
        def get_index(cls):
            return 'sumo_index'

        @classmethod
        def get_mapping_type_name(cls):
            return 'mymappingtype'


    results = S(MyMappingType).query(title__text='plugins')


``results`` will be an iterable of `MyMappingType` instances---one for
each search result.


Match All
=========

By default ``S()`` with no filters or queries specified will do a
``match_all`` query in ElasticSearch.

.. seealso::

   http://www.elasticsearch.org/guide/reference/query-dsl/match-all-query.html
     ElasticSearch match_all documentation


Queries vs. Filters
===================

A search can contain multiple queries and multiple filters. The two
things are very different.

A filter determines whether a document is in the results set or
not. If you do a term filter on whether field `foo` has value `bar`,
then the result set ONLY has documents where `foo` has value `bar`.
Filters are fast and filter results are cached in ElasticSearch when
appropriate.

A query affects the score for a document. If you do a term query on
whether field `foo` has value `bar`, then the result set will score
documents where the query holds true higher than documents where the
query does not hold true. Queries are slower than filters and
query results are not cached in ElasticSearch.

The other place where this affects things is when you specify
facets. See :ref:`queries-chapter-facets-section` for details.


.. seealso::

   http://www.elasticsearch.org/guide/reference/query-dsl/
     ElasticSearch Filters and Caching notes
{'fraction_non_alphanumeric': 0.15267249757045676, 'fraction_numerical': 0.006219630709426628, 'mean_word_length': 2.5745050364709967, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 7, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1825166', 'n_tokens_mistral': 3074, 'n_tokens_neox': 2955, 'n_words': 1041}