<commit_before>Lucene contrib change Log

======================= Trunk (not yet released) =======================
  
Build

 * LUCENE-2845: Moved contrib/benchmark to modules.

New Features

 * LUCENE-2604: Added RegexpQuery support to contrib/queryparser.
   (<NAME>, <NAME>)

 * LUCENE-2373: Added a Codec implementation that works with append-only
   filesystems (such as e.g. Hadoop DFS). SegmentInfos writing/reading
   code is refactored to support append-only FS, and to allow for future
   customization of per-segment information. (<NAME>)

 * LUCENE-2479: Added ability to provide a sort comparator for spelling suggestions along
   with two implementations.  The existing comparator (score, then frequency) is the default (Grant Ingersoll)

 * LUCENE-2608: Added the ability to specify the accuracy at method time in the SpellChecker.  The per class
   method is also still available.  (<NAME>)
    
 * LUCENE-2507: Added DirectSpellChecker, which retrieves correction candidates directly 
   from the term dictionary using levenshtein automata.  (<NAME>)

 * LUCENE-2836: Add FieldCacheRewriteMethod, which rewrites MultiTermQueries
   using the FieldCache's TermsEnum.  (<NAME>)
  
API Changes

 * LUCENE-2606: Changed RegexCapabilities interface to fix thread 
   safety, serialization, and performance problems. If you have
   written a custom RegexCapabilities it will need to be updated
   to the new API.  (<NAME>, <NAME>)

 * LUCENE-2638 MakeHighFreqTerms.TermStats public to make it more useful
   for API use. (<NAME>)

 * LUCENE-2912: The field-specific hashmaps in SweetSpotSimilarity were removed.
   Instead, use SimilarityProvider to return different SweetSpotSimilaritys
   for different fields, this way all parameters (such as TF factors) can be 
   customized on a per-field basis.  (<NAME>)
   
Bug Fixes

 * LUCENE-3045: fixed QueryNodeImpl.containsTag(String key) that was
   not lowercasing the key before checking for the tag (Adriano Crestani)

======================= Lucene 3.x (not yet released) =======================

Changes in runtime behavior

 * LUCENE-3086: ItalianAnalyzer now uses ElisionFilter with a set of Italian
   contractions by default.  (<NAME>)

Bug Fixes

 * LUCENE-3045: fixed QueryNodeImpl.containsTag(String key) that was
   not lowercasing the key before checking for the tag (Adriano Crestani)

 * LUCENE-3026: SmartChineseAnalyzer's WordTokenFilter threw NullPointerException
   on sentences longer than 32,767 characters.  (wangzhenghang via <NAME>)
   
 * LUCENE-2939: Highlighter should try and use maxDocCharsToAnalyze in 
   WeightedSpanTermExtractor when adding a new field to MemoryIndex as well as 
   when using CachingTokenStream. This can be a significant performance bug for
   large documents. (<NAME>)

 * LUCENE-3043: GermanStemmer threw IndexOutOfBoundsException if it encountered
   a zero-length token.  (<NAME>)
   
 * LUCENE-3044: ThaiWordFilter didn't reset its cached state correctly, this only
   caused a problem if you consumed a tokenstream, then reused it, added different
   attributes to it, and consumed it again.  (<NAME>, <NAME>)

New Features

 * LUCENE-3016: Add analyzer for Latvian.  (<NAME>)

 * LUCENE-1421: create new grouping contrib module, enabling search
   results to be grouped by a single-valued indexed field.  This
   module was factored out of Solr's grouping implementation, but
   it cannot group by function queries nor arbitrary queries.  (Mike
   McCandless)
<commit_msg>LUCENE-3098: add missing CHANGES entry<commit_after>Lucene contrib change Log

======================= Trunk (not yet released) =======================
  
Build

 * LUCENE-2845: Moved contrib/benchmark to modules.

New Features

 * LUCENE-2604: Added RegexpQuery support to contrib/queryparser.
   (<NAME>, <NAME>)

 * LUCENE-2373: Added a Codec implementation that works with append-only
   filesystems (such as e.g. Hadoop DFS). SegmentInfos writing/reading
   code is refactored to support append-only FS, and to allow for future
   customization of per-segment information. (<NAME>)

 * LUCENE-2479: Added ability to provide a sort comparator for spelling suggestions along
   with two implementations.  The existing comparator (score, then frequency) is the default (Grant Ingersoll)

 * LUCENE-2608: Added the ability to specify the accuracy at method time in the SpellChecker.  The per class
   method is also still available.  (<NAME>)
    
 * LUCENE-2507: Added DirectSpellChecker, which retrieves correction candidates directly 
   from the term dictionary using levenshtein automata.  (<NAME>)

 * LUCENE-2836: Add FieldCacheRewriteMethod, which rewrites MultiTermQueries
   using the FieldCache's TermsEnum.  (<NAME>)
  
API Changes

 * LUCENE-2606: Changed RegexCapabilities interface to fix thread 
   safety, serialization, and performance problems. If you have
   written a custom RegexCapabilities it will need to be updated
   to the new API.  (<NAME>, <NAME>)

 * LUCENE-2638 MakeHighFreqTerms.TermStats public to make it more useful
   for API use. (<NAME>)

 * LUCENE-2912: The field-specific hashmaps in SweetSpotSimilarity were removed.
   Instead, use SimilarityProvider to return different SweetSpotSimilaritys
   for different fields, this way all parameters (such as TF factors) can be 
   customized on a per-field basis.  (<NAME>)
   
Bug Fixes

 * LUCENE-3045: fixed QueryNodeImpl.containsTag(String key) that was
   not lowercasing the key before checking for the tag (Ad<NAME>)

======================= Lucene 3.x (not yet released) =======================

Changes in runtime behavior

 * LUCENE-3086: ItalianAnalyzer now uses ElisionFilter with a set of Italian
   contractions by default.  (<NAME>)

Bug Fixes

 * LUCENE-3045: fixed QueryNodeImpl.containsTag(String key) that was
   not lowercasing the key before checking for the tag (<NAME>)

 * LUCENE-3026: SmartChineseAnalyzer's WordTokenFilter threw NullPointerException
   on sentences longer than 32,767 characters.  (wangzhenghang via <NAME>)
   
 * LUCENE-2939: Highlighter should try and use maxDocCharsToAnalyze in 
   WeightedSpanTermExtractor when adding a new field to MemoryIndex as well as 
   when using CachingTokenStream. This can be a significant performance bug for
   large documents. (<NAME>)

 * LUCENE-3043: GermanStemmer threw IndexOutOfBoundsException if it encountered
   a zero-length token.  (<NAME>)
   
 * LUCENE-3044: ThaiWordFilter didn't reset its cached state correctly, this only
   caused a problem if you consumed a tokenstream, then reused it, added different
   attributes to it, and consumed it again.  (<NAME>, <NAME>)

New Features

 * LUCENE-3016: Add analyzer for Latvian.  (<NAME>)

 * LUCENE-1421: create new grouping contrib module, enabling search
   results to be grouped by a single-valued indexed field.  This
   module was factored out of Solr's grouping implementation, but
   it cannot group by function queries nor arbitrary queries.  (Mike
   McCandless)

 * LUCENE-3098: add AllGroupsCollector, to collect all unique groups
   (but in unspecified order).  (<NAME> via Mike
   McCandless)

{'fraction_non_alphanumeric': 0.08547368421052631, 'fraction_numerical': 0.024140350877192983, 'mean_word_length': 4.262924667651403, 'pattern_counts': {'":': 0, '<': 40, '<?xml version=': 0, '>': 40, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1176399', 'n_tokens_mistral': 2234, 'n_tokens_neox': 1936, 'n_words': 840}