<commit_before><html>
<body>
The highlight package contains classes to provide "keyword in context" features
typically used to highlight search terms in the text of results pages. <br>
The Highlighter class is the central component and can be used to extract the
most interesting sections of a piece of text and highlight them, with the help of
Fragmenter, FragmentScorer and Formatter classes.
<h2>Example Usage</h2>

<pre>
		IndexSearcher searcher = new IndexSearcher(ramDir);
		Query query = QueryParser.parse("Kenne*", FIELD_NAME, analyzer);
		query=query.rewrite(reader); //required to expand search terms
		Hits hits = searcher.search(query);

		Highlighter highlighter =new Highlighter(this,new QueryScorer(query));<commit_msg>Clarification of Javadocs - code was missing Formatter implementation<commit_after><html>
<body>
The highlight package contains classes to provide "keyword in context" features
typically used to highlight search terms in the text of results pages. <br>
The Highlighter class is the central component and can be used to extract the
most interesting sections of a piece of text and highlight them, with the help of
Fragmenter, FragmentScorer and Formatter classes.
<h2>Example Usage</h2>

<pre>
		IndexSearcher searcher = new IndexSearcher(ramDir);
		Query query = QueryParser.parse("Kenne*", FIELD_NAME, analyzer);
		query=query.rewrite(reader); //required to expand search terms
		Hits hits = searcher.search(query);

		Highlighter highlighter =new Highlighter(new QueryScorer(query));
{'fraction_non_alphanumeric': 0.07241606319947334, 'fraction_numerical': 0.0026333113890717576, 'mean_word_length': 5.725663716814159, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '424030', 'n_tokens_mistral': 428, 'n_tokens_neox': 414, 'n_words': 178}