<commit_before>[[fulltext-search]]
= Fulltext Indexing & Searching with Solr

== Overview

When enabled, the Solr integration will forward all content to an Solr cluster for fulltext indexing.

== Dependencies

Spring Content Solr requires a `SolrClient` bean that is used as the connection to the Solr
cluster that you wish to use for indexing.

Solr is therefore enabled with following depedencies:

====
[source, xml]
----
	<dependency>
		<groupId>org.apache.solr</groupId>
		<artifactId>solr-solrj</artifactId>
		<version>7.2.1</version>                <1>
		<exclusions>
			<exclusion>
				<groupId>org.codehaus.woodstox</groupId>
				<artifactId>wstx-asl</artifactId>
			</exclusion>
			<exclusion>
				<artifactId>log4j</artifactId>
				<groupId>log4j</groupId>
			</exclusion>
		</exclusions>
	</dependency>
----
1. Specify the version that matches your Solr server.  NB: as Spring Content certifies against the latest version, the
version must be compatible with this.
====

And the following Java Config:

.Enabling Spring Content Solr with Java Config
====
[source, java]
----
@Configuration
@EnableFullTextSolrIndexing                 <1>
@EnableFilesystemStores                     <2>
public static class ApplicationConfig {

    @Bean                                   <3>
    public SolrClient solrClient() {
        return new HttpSolrClient.Builder("http://localhost:8983/solr/solr").build();
    }

}
----
1. Specify the `@EnableFullTextSolrIndexing` annotation in an `@Configuration` class
2. Spring Content Solr works with any Spring Content Store Module
3. Ensure a `(SolrJ) SolrClient` `@Bean` is instantiated somewhere within your `@Configuration`
====

== Configuring

If required, credentials can be specified using the following configuration properties.

[cols="2*", options="header"]
|=========
| Property | Description
| solr.username | Solr user
| solr.password | <PASSWORD>'s password
|=========

== Making Stores Searchable

With fulltext-indexing enabled, Store interfaces can be made `Searchable`.  See
<<content-repositories.search,Searchable Stores>> for more information on how to do this.

== Custom Indexing

When you `@EnableFullTextSolrIndexing` a store event handler is registered that intercepts content being added to a
Store and sends that content to your Solr instance for full-text indexing.  This is usually all you need.  However,
sometimes you may need more control over when documents are indexed.  For these cases you can use the `IndexService`
bean directly in your code to index (or unindex) content as required.

== Custom Attributes and Filtering Queries

By default Spring Content Solr indexes content only.  However, it is common to synchronize additional 
attributes from the primary domain model that can then be used for filtering full-text queries or for efficiently populating 
search results (removing the need to perform subsequent queries against the primary domain model).

To synchronize additional attributes when content is indexed add a bean that implements `AttributeProvider` to your 
application's configuration:

====
[source, java]
----
    @Bean
    public AttributeProvider<Document> attributeProvider() {
        return new AttributeProvider<Document>() {

            @Override
            public Map<String, String> synchronize(Document entity) {

                Map<String, String> attrs = new HashMap<>();
                attrs.put("title", entity.getTitle());
                attrs.put("author", entity.getAuthor());
                return attrs;
            }
        };
    }
====

To customize the query that gets executed when a Store's Searchable method is invoked add a `FilterQueryProvider` bean to your 
application's configuration:

====
[source, java]
----
    @Bean
    public FilterQueryProvider fqProvider() {
        return new FilterQueryProvider() {

            @Override
            public String[] filterQueries(Class<?> entity) {

                return new String[] {"author:<EMAIL>"};
            }
        };
    }
----
NOTE: this bean is often a request scoped bean or has an implementation based on a thread local variable in order to build and return 
filter queries based on the current execution context. 
====

== Search Return Types

`Searchable` is a generic type allowing you to specify the return type of the result set.  The simplest option is to 
type this interface to String in which case result sets will be collections of content IDs.

You can also type the interface to your own custom class.  Several annotations are available allowing you to tailor full-text search
results to your specific needs:

- @ContentId; extracts the content ID of the content from your search results
- @Highlight; extracts highlighted snippets from your search results so you can show users where the query matches are
- Attribute; extracts the specified attribute from your search results (must be synchronized using an `AttributeProvider`)<commit_msg>Update solr docs<commit_after>[[fulltext-search]]
= Fulltext Indexing & Searching with Solr

== Overview

When enabled, the Solr integration will forward all content to an Solr cluster for fulltext indexing.

== Dependencies

Spring Content Solr requires a `SolrClient` bean that is used as the connection to the Solr
cluster that you wish to use for indexing.

Solr is therefore enabled with following depedencies:

====
[source, xml]
----
	<dependency>
		<groupId>org.apache.solr</groupId>
		<artifactId>solr-solrj</artifactId>
		<version>7.2.1</version>                <1>
		<exclusions>
			<exclusion>
				<groupId>org.codehaus.woodstox</groupId>
				<artifactId>wstx-asl</artifactId>
			</exclusion>
			<exclusion>
				<artifactId>log4j</artifactId>
				<groupId>log4j</groupId>
			</exclusion>
		</exclusions>
	</dependency>
----
1. Specify the version that matches your Solr server.  NB: as Spring Content certifies against the latest version, the
version must be compatible with this.
====

And the following Java Config:

.Enabling Spring Content Solr with Java Config
====
[source, java]
----
@Configuration
@EnableFullTextSolrIndexing                 <1>
@EnableFilesystemStores                     <2>
public static class ApplicationConfig {

    @Bean                                   <3>
    public SolrClient solrClient() {
        return new HttpSolrClient.Builder("http://localhost:8983/solr/solr").build();
    }

}
----
1. Specify the `@EnableFullTextSolrIndexing` annotation in an `@Configuration` class
2. Spring Content Solr works with any Spring Content Storage Module
3. Ensure a `(SolrJ) SolrClient` `@Bean` is instantiated somewhere within your `@Configuration`
====

== When your Solr cluster is secured with Basic Authentication 

If required, credentials can be specified using the following configuration properties.

[cols="2*", options="header"]
|=========
| Property | Description
| solr.username | Solr user
| solr.password | Solr user's password
|=========

== Making Stores Searchable

With fulltext-indexing enabled, Store interfaces can be made `Searchable`.  See
<<content-repositories.search,Searchable Stores>> for more information on how to do this.

== Taking control of the indexing process

When you `@EnableFullTextSolrIndexing` a store event handler is registered that intercepts content being added to a
Store and sends that content to your Solr instance for full-text indexing.  This is usually all you need.  However,
sometimes you may need more control over when documents are indexed.  For these cases you can use the `IndexService`
bean directly in your code to index (or unindex) content as required.

== Indexing additional attributes

By default Spring Content Solr indexes content only.  However, it is common to index additional 
attributes (from the primary domain model) that can then be used for filtering subsequent full-text queries 
or for efficiently populating search results (removing the need to perform additional queries against the 
primary domain model).

Add a bean that implements `AttributeProvider` to your application's configuration:

====
[source, java]
----
    @Bean
    public AttributeProvider<Document> attributeProvider() {
        return new AttributeProvider<Document>() {

            @Override
            public Map<String, String> synchronize(Document entity) {

                Map<String, String> attrs = new HashMap<>();
                attrs.put("title", entity.getTitle());
                attrs.put("author", entity.getAuthor());
                return attrs;
            }
        };
    }
====

Whenever content is indexed this bean will be consulted providing an opportunity to provide the set additional attributes that 
you would like to be indexed.

To customize the query that gets executed when a Store's Searchable method is invoked add a `FilterQueryProvider` bean to your 
application's configuration:

====
[source, java]
----
    @Bean
    public FilterQueryProvider fqProvider() {
        return new FilterQueryProvider() {

            @Override
            public String[] filterQueries(Class<?> entity) {

                return new String[] {"author:<EMAIL>"};
            }
        };
    }
----
NOTE: this bean is often a request scoped bean or has an implementation based on a thread local variable in order to build and return 
filter queries based on the current execution context. 
====

When fulltext queries are performed this bean will be consulted providing an opportunity to supplement the query
with your own filters.  These filters are often based on the additional attributes that have been indexed.    

== Search Return Types

`Searchable` is a generic type allowing you to specify the return type of the result set.  

The simplest option is to type this interface to String in which case the result set will be a collection of the content IDs
of content matching the search terms provided.

You can also type the interface to your own custom class.  Several annotations are available allowing you to tailor full-text search
results to your specific needs:

- @ContentId; extracts the content ID of the content from your search results
- @Highlight; extracts highlighted snippets from your search results so you can show users where the query matches are.  Note
that highlighters usually analyze the stored field value so you may have to ensure the field that your content is indexed into
(usually `\_text_`) is also a stored field.
- Attribute; extracts the specified attribute from your search results (must be synchronized using an `AttributeProvider`)

The following example show a searchable `ContentStore` returning a custom type `SearchResult`.

====
[source, java]
----
    public interface DocumentStore 
        extends ContentStore<Document, String>, Searchable<SearchResult> {
    }

    @Getter
    @Setter
    public static class SearchResult {

        @ContentId
        private String contentId;

        @Highlight
        private String highlight;

        @Attribute(name = "email")
        private String email;
    }
----
====
{'fraction_non_alphanumeric': 0.0796267777878431, 'fraction_numerical': 0.003261164960594257, 'mean_word_length': 3.4878048780487805, 'pattern_counts': {'":': 0, '<': 76, '<?xml version=': 0, '>': 76, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1077711', 'n_tokens_mistral': 2926, 'n_tokens_neox': 2709, 'n_words': 1254}