<commit_before>[[search-aggregations-bucket-significanttext-aggregation]]
=== Significant Text Aggregation

experimental[]

An aggregation that returns interesting or unusual occurrences of free-text terms in a set.
It is like the <<search-aggregations-bucket-significantterms-aggregation,significant terms>> aggregation but differs in that:

* It is specifically designed for use on type `text` fields
* It does not require field data or doc-values
* It re-analyzes text content on-the-fly meaning it can also filter duplicate sections of
noisy text that otherwise tend to skew statistics.

WARNING: Re-analyzing _large_ result sets will require a lot of time and memory. It is recommended that the significant_text
         aggregation is used as a child of either the <<search-aggregations-bucket-sampler-aggregation,sampler>> or 
         <<search-aggregations-bucket-diversified-sampler-aggregation,diversified sampler>> aggregation to limit the analysis
         to a _small_ selection of top-matching documents e.g. 200. This will typically improve speed, memory use and quality of
         results.

.Example use cases:
* Suggesting "H5N1" when users search for "bird flu" to help expand queries
* Suggesting keywords relating to stock symbol $ATI for use in an automated news classifier

In these cases the words being selected are not simply the most popular terms in results. The most popular words tend to be
very boring (_and, of, the, we, I, they_ ...).
The significant words are the ones that have undergone a significant change in popularity measured between a _foreground_ and _background_ set.
If the term "H5N1" only exists in 5 documents in a 10 million document index and yet is found in 4 of the 100 documents that make up a user's search results
that is significant and probably very relevant to their search. 5/10,000,000 vs 4/100 is a big swing in frequency.

experimental[The `significant_text` aggregation is new and may change in non-backwards compatible ways if we add further text-analysis features e.g. phrase detection]

==== Basic use

In the typical use case, the _foreground_ set of interest is a selection of the top-matching search results for a query 
and the _background_set used for statistical comparisons is the index or indices from which the results were gathered.
<commit_msg>Docs - remove experimental marker from significant_text aggregation (#34598)<commit_after>[[search-aggregations-bucket-significanttext-aggregation]]
=== Significant Text Aggregation

An aggregation that returns interesting or unusual occurrences of free-text terms in a set.
It is like the <<search-aggregations-bucket-significantterms-aggregation,significant terms>> aggregation but differs in that:

* It is specifically designed for use on type `text` fields
* It does not require field data or doc-values
* It re-analyzes text content on-the-fly meaning it can also filter duplicate sections of
noisy text that otherwise tend to skew statistics.

WARNING: Re-analyzing _large_ result sets will require a lot of time and memory. It is recommended that the significant_text
         aggregation is used as a child of either the <<search-aggregations-bucket-sampler-aggregation,sampler>> or 
         <<search-aggregations-bucket-diversified-sampler-aggregation,diversified sampler>> aggregation to limit the analysis
         to a _small_ selection of top-matching documents e.g. 200. This will typically improve speed, memory use and quality of
         results.

.Example use cases:
* Suggesting "H5N1" when users search for "bird flu" to help expand queries
* Suggesting keywords relating to stock symbol $ATI for use in an automated news classifier

In these cases the words being selected are not simply the most popular terms in results. The most popular words tend to be
very boring (_and, of, the, we, I, they_ ...).
The significant words are the ones that have undergone a significant change in popularity measured between a _foreground_ and _background_ set.
If the term "H5N1" only exists in 5 documents in a 10 million document index and yet is found in 4 of the 100 documents that make up a user's search results
that is significant and probably very relevant to their search. 5/10,000,000 vs 4/100 is a big swing in frequency.

==== Basic use

In the typical use case, the _foreground_ set of interest is a selection of the top-matching search results for a query 
and the _background_set used for statistical comparisons is the index or indices from which the results were gathered.

{'fraction_non_alphanumeric': 0.04821942048219421, 'fraction_numerical': 0.013050210130502102, 'mean_word_length': 4.461352657004831, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '17921', 'n_tokens_mistral': 1183, 'n_tokens_neox': 1075, 'n_words': 625}