<commit_before>[[fulltext-search]]
= Fulltext Indexing and Searching with Elasticsearch

== Overview

When enabled, the Elasticsearch integration will, by default, forward all content to an Elasticsearch cluster for fulltext indexing.

== Annotation-based Configuration

Spring Content Elasticsearch requires a `RestHighLevelClient` bean that is used as the connection to your Elasticsearch
cluster.

Elasticsearch can be enabled with the following Java Config.

.Enabling Spring Content Elasticsearch with Java Config
====
[source, java]
----
@Configuration
@EnableElasticsearchFulltextIndexing        <1>
@EnableFilesystemStores                     <2>
public static class ApplicationConfig {

                                            <3>
    public RestHighLevelClient client() {
        return new RestHighLevelClient(RestClient.builder(new HttpHost("localhost", 9200, "http")));
    }

}
----
1. Specify the `@EnableElasticsearchFulltextIndexing` annotation in an `@Configuration` class
2. Spring Content Elasticsearch works with any Spring Content Store module
3. Ensure a `RestHighLevelClient` bean is instantiated somewhere within your `@Configuration`
====

== Spring Boot Configuration

Alternatively, you can use the Spring Boot Starter `spring-content-elasticsearch-boot-starter`.

When using this method of configuration the `@EnableElasticsearchFulltextIndexing` annotation can be omitted as it will
be added for you.  As will a RestHighLevelClient client bean configured to connect to localhost.

The following configuration properties (prefix spring.content.elasticsearch) are supported.

[cols="2*", options="header"]
|=========
| Property | Description
| autoindex | Whether, or not, to enable autoindexing to index content as it is added
|=========

== Making Stores Searchable

With fulltext-indexing enabled, Store interfaces can be made `Searchable`.  See
<<content-repositories.search,Searchable Stores>> for more information on how to do this.

== Custom Indexing

By default when you `@EnableElasticsearchFulltextIndexing` a store event handler is registered that intercepts content
being added to a Store and sends that content to your Elasticsearch cluster for full-text indexing.  This is usually
all you need.  However, sometimes you may need more control over when documents are indexed.  For these cases you can
use the `IndexService` bean directly in your code to index (or unindex) content as required.

When performing custom indexing it is usual to turn of the auto-indexing feature but specifying
`spring.content.elasticsearch.autoindex=false` in your application properties.

== Text Extraction

For images and other media, it also possible to configure the elasticsearch integration to perform text extraction and
send that instead of the image content to Elasticsearch.

This requires two stages of configuration:

1. Add one or more renderers to the application context.  These renderers are used to perform the text extraction.  To be
used for text extraction a renderer *must* produce `text/plain` content but can consume any suitable mimetype.   When
content matching its `consume` mime type is added to a Store the renderer will invoked to extract text and this text
will be sent to the Elasticsearch for fulltext indexing in place of the original content.

.Adding a renderer to perform text extraction
====
[source, java]
----
@Configuration
@EnableElasticsearchFulltextIndexing
@EnableFilesystemStores
public static class ApplicationConfig {

    public RestHighLevelClient client() {
        return new RestHighLevelClient(RestClient.builder(new HttpHost("localhost", 9200, "http")));
    }
    
    @Bean
    public RenditionProvider jpgTextExtractor() {
        return new RenditionProvider() {
            
            @Override
            public String consumes() {
                return "image/jpg";                     // can be any mime-type
            }

            @Override
            public String[] produces() {
                return new String[] {"text/plain"};     // must be 'text/plain'
            }

            @Override
            public InputStream convert(InputStream fromInputSource, String toMimeType) {
                ...implementation...
            }
        }
    }
}
----
====

2. Make the Store Renderable

.Making the Store interface Renderable
====
[source, java]
----
public interface DocumentStore extends ContentStore<Document, UUID> implements Searchable<Document>, Renderable<Document> {
}
====<commit_msg>Add elasticsearch docs for custom attriibutes, filter queries and search result types<commit_after>[[fulltext-search]]
= Fulltext Indexing and Searching with Elasticsearch

== Overview

When enabled, the Elasticsearch integration will, by default, forward all content to an Elasticsearch cluster for fulltext indexing.

== Annotation-based Configuration

Spring Content Elasticsearch requires a `RestHighLevelClient` bean that is used as the connection to your Elasticsearch
cluster.

Elasticsearch can be enabled with the following Java Config.

.Enabling Spring Content Elasticsearch with Java Config
====
[source, java]
----
@Configuration
@EnableElasticsearchFulltextIndexing        <1>
@EnableFilesystemStores                     <2>
public static class ApplicationConfig {

                                            <3>
    public RestHighLevelClient client() {
        return new RestHighLevelClient(RestClient.builder(new HttpHost("localhost", 9200, "http")));
    }

}
----
1. Specify the `@EnableElasticsearchFulltextIndexing` annotation in an `@Configuration` class
2. Spring Content Elasticsearch works with any Spring Content Store module
3. Ensure a `RestHighLevelClient` bean is instantiated somewhere within your `@Configuration`
====

== Spring Boot Configuration

Alternatively, you can use the Spring Boot Starter `spring-content-elasticsearch-boot-starter`.

When using this method of configuration the `@EnableElasticsearchFulltextIndexing` annotation can be omitted as it will
be added for you.  As will a RestHighLevelClient client bean configured to connect to localhost.

The following configuration properties (prefix spring.content.elasticsearch) are supported.

[cols="2*", options="header"]
|=========
| Property | Description
| autoindex | Whether, or not, to enable autoindexing to index content as it is added
|=========

== Making Stores Searchable

With fulltext-indexing enabled, Store interfaces can be made `Searchable`.  See
<<content-repositories.search,Searchable Stores>> for more information on how to do this.

== Custom Indexing

By default when you `@EnableElasticsearchFulltextIndexing` a store event handler is registered that intercepts content
being added to a Store and sends that content to your Elasticsearch cluster for full-text indexing.  This is usually
all you need.  However, sometimes you may need more control over when documents are indexed.  For these cases you can
use the `IndexService` bean directly in your code to index (or unindex) content as required.

When performing custom indexing it is usual to turn of the auto-indexing feature but specifying
`spring.content.elasticsearch.autoindex=false` in your application properties.

== Text Extraction

For images and other media, it also possible to configure the elasticsearch integration to perform text extraction and
send that instead of the image content to Elasticsearch.

This requires two stages of configuration:

1. Add one or more renderers to the application context.  These renderers are used to perform the text extraction.  To be
used for text extraction a renderer *must* produce `text/plain` content but can consume any suitable mime type.   When
content matching its `consume` mime type is added to a Store the renderer will be invoked to extract text and this extracted text
will then be sent to the Elasticsearch for fulltext indexing in place of the original content.

.Adding a renderer to perform text extraction
====
[source, java]
----
@Configuration
@EnableElasticsearchFulltextIndexing
@EnableFilesystemStores
public static class ApplicationConfig {

    public RestHighLevelClient client() {
        return new RestHighLevelClient(RestClient.builder(new HttpHost("localhost", 9200, "http")));
    }
    
    @Bean
    public RenditionProvider jpgTextExtractor() {
        return new RenditionProvider() {
            
            @Override
            public String consumes() {
                return "image/jpg";                     // can be any mime-type
            }

            @Override
            public String[] produces() {
                return new String[] {"text/plain"};     // must be 'text/plain'
            }

            @Override
            public InputStream convert(InputStream fromInputSource, String toMimeType) {
                ...implementation...
            }
        }
    }
}
----
====

2. Make the Store Renderable as this will be used internally to extract the text

.Making the Store interface Renderable
====
[source, java]
----
public interface DocumentStore extends ContentStore<Document, UUID> implements Searchable<Document>, Renderable<Document> {
}
====

== Custom Attributes and Filtering Queries

By default Spring Content Elasticsearch indexes content only.  However, it is common to synchronize additional 
attributes from the primary domain model that can then be used for filtering full-text queries or for efficiently populating 
search results (removing the need to perform subsequent queries against the primary domain model).

To synchronize additional attributes when content is indexed add a bean that implements `AttributeProvider` to your 
application's configuration:

====
[source, java]
----
    @Bean
    public AttributeProvider<Document> attributeProvider() {
        return new AttributeProvider<Document>() {

            @Override
            public Map<String, String> synchronize(Document entity) {

                Map<String, String> attrs = new HashMap<>();
                attrs.put("title", entity.getTitle());
                attrs.put("author", entity.getAuthor());
                return attrs;
            }
        };
    }
====

To customize the query that gets executed when a Store's Searchable method is invoked add a `FilterQueryProvider` bean to your 
application's configuration:

====
[source, java]
----
    @Bean
    public FilterQueryProvider fqProvider() {
        return new FilterQueryProvider() {

            @Override
            public String[] filterQueries(Class<?> entity) {

                return new String[] {"author:<EMAIL>"};
            }
        };
    }
----
NOTE: this bean is often a request scoped bean or has an implementation based on a thread local variable in order to build and return 
filter queries based on the current execution context. 
====

== Search Return Types

`Searchable` is a generic type allowing you to specify the return type of the result set.  The simplest option is to 
type this interface to String in which case result sets will be collections of content IDs.

You can also type the interface to your own custom class.  Several annotations are available allowing you to tailor full-text search
results to your specific needs:

- @ContentId; extracts the content ID of the content from your search results
- @Highlight; extracts highlighted snippets from your search results so you can show users where the query matches are
- Attribute; extracts the specified attribute from your search results (must be synchronized using an `AttributeProvider`)
{'fraction_non_alphanumeric': 0.06892067620286085, 'fraction_numerical': 0.0029475509319462504, 'mean_word_length': 3.5363743609909557, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2404172', 'n_tokens_mistral': 2764, 'n_tokens_neox': 2696, 'n_words': 1276}