<commit_before>[[analysis-pattern_replace-tokenfilter]]
=== Pattern replace token filter
++++
<titleabbrev>Pattern replace</titleabbrev>
++++

Uses a regular expression to match and replace token substrings.

The `pattern_replace` filter uses
https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html[Java's
regular expression syntax]. By default, the filter replaces matching
substrings with an empty substring (`""`).

Regular expressions cannot be anchored to the
beginning or end of a token. Replacement substrings can use Java's
https://docs.oracle.com/javase/8/docs/api/java/util/regex/Matcher.html#appendReplacement-java.lang.StringBuffer-java.lang.String-[`$g` syntax] to reference capture groups
from the original token text.

[WARNING]
====
A poorly-written regular expression may run slowly or return a
StackOverflowError, causing the node running the expression to exit suddenly.
<commit_msg>[DOCS] Fix pattern replace token filter intro (#64189)<commit_after>[[analysis-pattern_replace-tokenfilter]]
=== Pattern replace token filter
++++
<titleabbrev>Pattern replace</titleabbrev>
++++

Uses a regular expression to match and replace token substrings.

The `pattern_replace` filter uses
https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html[Java's
regular expression syntax]. By default, the filter replaces matching
substrings with an empty substring (`""`).

Replacement substrings can use Java's
https://docs.oracle.com/javase/8/docs/api/java/util/regex/Matcher.html#appendReplacement-java.lang.StringBuffer-java.lang.String-[`$g` syntax] to reference capture groups
from the original token text.

[WARNING]
====
A poorly-written regular expression may run slowly or return a
StackOverflowError, causing the node running the expression to exit suddenly.

{'fraction_non_alphanumeric': 0.10555555555555556, 'fraction_numerical': 0.005, 'mean_word_length': 5.67037037037037, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7152502', 'n_tokens_mistral': 510, 'n_tokens_neox': 479, 'n_words': 167}