<commit_before><!--
   Licensed to the Apache Software Foundation (ASF) under one or more
   contributor license agreements.  See the NOTICE file distributed with
   this work for additional information regarding copyright ownership.
   The ASF licenses this file to You under the Apache License, Version 2.0
   (the "License"); you may not use this file except in compliance with
   the License.  You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

   Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.
  -->
  
# <a name="indexing"></a> Indexing

* [Indexing](#indexing)
    * [Overview](#overview)
        * [New in 1.6](#new-1.6)
    * [Indexing Flow](#indexing-flow)
        * [Index Definitions](#index-defnitions)
            * [oak:index node ](#oak-index-nodes)
        * [Synchronous Indexing](#sync-indexing)
        * [Asynchronous Indexing](#async-indexing)
            * [Checkpoint](#checkpoint)
            * [Indexing Lane](#indexing-lane)
            * [Clustered Setup](#cluster)
                * [Indexing Lease](#async-index-lease)
            * [Indexing Lag](#async-index-lag)
            * [Setup](#async-index-setup)
            * [Async Indexing MBean](#async-index-mbean)
            * [Isolating Corrupt Indexes](#corrupt-index-handling)
        * [Near Real Time Indexing](#nrt-indexing)
            * [NRT Indexing Modes](#nrt-indexing-modes)
                * [nrt](#nrt-indexing-mode-nrt)
                * [sync](#nrt-indexing-mode-sync)
            * [Cluster Setup](#nrt-indexing-cluster-setup)
            * [Configuration](#nrt-indexing-config)
    * [Reindexing](#reindexing)            
  
## <a name="overview"></a> Overview
  
For queries to perform well, Oak supports indexing of content that is stored in the repository. 
Indexing works on comparing different versions of the node data
(technically, `Diff` between the base `NodeState` and the modified `NodeState`). 
There are indexing modes that define
how comparing is performed, and when the index content gets updated:
  
1. Synchronous Indexing
2. Asynchronous Indexing
3. Near Real Time (NRT) Indexing

Indexing makes use of [Commit Editors](../architecture/nodestate.html#commit-editors). 
Some of the editors are of type `IndexEditor`, which are responsible for updating index content 
based on changes in main content. 
Currently, Oak has following in built editors:

1. PropertyIndexEditor
2. ReferenceEditor
3. LuceneIndexEditor
4. SolrIndexEditor

### <a name="new-1.6"></a> New in 1.6

* [Near Real Time (NRT) Indexing](#nrt-indexing)
* [Multiple Async indexers setup via OSGi config](#async-index-setup)
* [Isolating Corrupt Indexes](#corrupt-index-handling)

## <a name="indexing-flow"></a> Indexing Flow

The `IndexEditor` is invoked as part of a commit (`Session.save()`), 
or as part of the asynchronous "diff" process. 
For both cases, at some stage "diff" is performed between the _before_ and the _after_ state, 
and passed to `IndexUpdate`, which is responsible for invoking the `IndexEditor`
based on the _discovered_ index definitions.

### <a name="index-defnitions"></a> Index Definitions

Index definitions are nodes of type `oak:QueryIndexDefinition`
which are stored under a special node named `oak:index`.
As part of diff traversal, at each level `IndexUpdate` looks for `oak:index` nodes. 
Below is the canonical index definition structure:

    /oak:index/indexName
      - jcr:primaryType = "oak:QueryIndexDefinition"
      - type (string) mandatory
      - async (string) multiple
      - reindex (boolean)
      
The index definitions nodes have following properties:

1. `type` - It determines the _type_ of index. Based on the `type`, 
    `IndexUpdate` looks for an `IndexEditor` of the given 
    type from the registered `IndexEditorProvider`. 
    For out-of-the-box Oak setup, it can have one of the following values
    * `reference` -  Configured with the out-of-the-box setup
    * `counter` - Configured with the out-of-the-box setup
    * `property`
    * `lucene`
    * `solr`
2. `async` - This determines if the index is to be updated synchronously or asynchronously. 
    It can have following values:
    * `sync` - The default value. It indicates that index is meant to be updated as part of each commit.
    * `nrt`  - Indicates that index is a [near real time](#nrt-indexing) index. 
    * `async` - Indicates that index is to be updated asynchronously. 
       In such a case, this value is used to determine
       the [indexing lane](#indexing-lane)
    * Any other value which ends in `async`. 
3. `reindex` - If set to `true`, reindexing is performed for that index. 
    After reindexing is done, the property value is set to `false`.
    Refer to [reindexing](#reindexing) for more details.
    
Based on the above two properties, the `IndexUpdate` creates an `IndexEditor` instances 
as it traverses the "diff", and registers them with itself, passing on the callbacks for various changes.

#### <a name="oak-index-nodes"></a>oak:index node 

Indexing logic supports placing `oak:index` nodes at any path. 
Depending on the location, such indexes only index content which are present under those paths. <commit_msg>OAK-5946 - Document indexing flow<commit_after><!--
   Licensed to the Apache Software Foundation (ASF) under one or more
   contributor license agreements.  See the NOTICE file distributed with
   this work for additional information regarding copyright ownership.
   The ASF licenses this file to You under the Apache License, Version 2.0
   (the "License"); you may not use this file except in compliance with
   the License.  You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

   Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.
  -->
  
# <a name="indexing"></a> Indexing

* [Indexing](#indexing)
    * [Overview](#overview)
        * [New in 1.6](#new-1.6)
    * [Indexing Flow](#indexing-flow)
        * [Index Definitions](#index-defnitions)
            * [Index Definition Location](#oak-index-nodes)
        * [Synchronous Indexing](#sync-indexing)
        * [Asynchronous Indexing](#async-indexing)
            * [Checkpoint](#checkpoint)
            * [Indexing Lane](#indexing-lane)
            * [Clustered Setup](#cluster)
                * [Indexing Lease](#async-index-lease)
            * [Indexing Lag](#async-index-lag)
            * [Setup](#async-index-setup)
            * [Async Indexing MBean](#async-index-mbean)
            * [Isolating Corrupt Indexes](#corrupt-index-handling)
        * [Near Real Time Indexing](#nrt-indexing)
            * [NRT Indexing Modes](#nrt-indexing-modes)
                * [nrt](#nrt-indexing-mode-nrt)
                * [sync](#nrt-indexing-mode-sync)
            * [Cluster Setup](#nrt-indexing-cluster-setup)
            * [Configuration](#nrt-indexing-config)
    * [Reindexing](#reindexing)            
  
## <a name="overview"></a> Overview
  
For queries to perform well, Oak supports indexing of content that is stored in the repository. 
Indexing works on comparing different versions of the node data
(technically, `Diff` between the base `NodeState` and the modified `NodeState`). 
There are indexing modes that define
how comparing is performed, and when the index content gets updated:
  
1. Synchronous Indexing
2. Asynchronous Indexing
3. Near Real Time (NRT) Indexing

Indexing makes use of [Commit Editors](../architecture/nodestate.html#commit-editors). 
Some of the editors are of type `IndexEditor`, which are responsible for updating index content 
based on changes in main content. 
Currently, Oak has following in built editors:

1. PropertyIndexEditor
2. ReferenceEditor
3. LuceneIndexEditor
4. SolrIndexEditor

### <a name="new-1.6"></a> New in 1.6

* [Near Real Time (NRT) Indexing](#nrt-indexing)
* [Multiple Async indexers setup via OSGi config](#async-index-setup)
* [Isolating Corrupt Indexes](#corrupt-index-handling)

## <a name="indexing-flow"></a> Indexing Flow

The `IndexEditor` is invoked as part of a commit (`Session.save()`), 
or as part of the asynchronous "diff" process. 
For both cases, at some stage "diff" is performed between the _before_ and the _after_ state, 
and passed to `IndexUpdate`, which is responsible for invoking the `IndexEditor`
based on the _discovered_ index definitions.

### <a name="index-defnitions"></a> Index Definitions

Index definitions are nodes of type `oak:QueryIndexDefinition`
which are stored under a special node named `oak:index`.
As part of diff traversal, at each level `IndexUpdate` looks for `oak:index` nodes. 
Below is the canonical index definition structure:

    /oak:index/indexName
      - jcr:primaryType = "oak:QueryIndexDefinition"
      - type (string) mandatory
      - async (string) multiple
      - reindex (boolean)
      
The index definitions nodes have following properties:

1. `type` - It determines the _type_ of index. Based on the `type`, 
    `IndexUpdate` looks for an `IndexEditor` of the given 
    type from the registered `IndexEditorProvider`. 
    For out-of-the-box Oak setup, it can have one of the following values
    * `reference` -  Configured with the out-of-the-box setup
    * `counter` - Configured with the out-of-the-box setup
    * `property`
    * `lucene`
    * `solr`
2. `async` - This determines if the index is to be updated synchronously or asynchronously. 
    It can have following values:
    * `sync` - The default value. It indicates that index is meant to be updated as part of each commit.
    * `nrt`  - Indicates that index is a [near real time](#nrt-indexing) index. 
    * `async` - Indicates that index is to be updated asynchronously. 
       In such a case, this value is used to determine
       the [indexing lane](#indexing-lane)
    * Any other value which ends in `async`. 
3. `reindex` - If set to `true`, reindexing is performed for that index. 
    After reindexing is done, the property value is set to `false`.
    Refer to [reindexing](#reindexing) for more details.
    
Based on the above two properties, the `IndexUpdate` creates an `IndexEditor` instances 
as it traverses the "diff", and registers them with itself, passing on the callbacks for various changes.

#### <a name="oak-index-nodes"></a> Index Definition Location

Indexing logic supports placing `oak:index` nodes at any path. 
Depending on the location, such indexes only index content which are present under those paths. 
{'fraction_non_alphanumeric': 0.09455405036821529, 'fraction_numerical': 0.004364033093917629, 'mean_word_length': 3.3616177636796194, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2879266', 'n_tokens_mistral': 3175, 'n_tokens_neox': 3038, 'n_words': 1239}