<commit_before>Lucene Change Log

$Id$

Trunk (not yet released)

Changes in runtime behavior

 1. 's' and 't' have been removed from the list of default stopwords
    in StopAnalyzer (also used in by StandardAnalyzer). Having e.g. 's'
    as a stopword meant that 's-class' led to the same results as 'class'.
    Note that this problem still exists for 'a', e.g. in 'a-class' as
    'a' continues to be a stopword.
    (<NAME>)

 2. LUCENE-478: Updated the list of Unicode code point ranges for CJK
    (now split into CJ and K) in StandardAnalyzer.  (<NAME> and
    <NAME> via Otis Gospodnetic)

 3. Modified some CJK Unicode code point ranges in StandardTokenizer.jj,
    and added a few more of them to increase CJK character coverage.
    Also documented some of the ranges.
    (Otis Gospodnetic)

 4. LUCENE-489: Add support for leading wildcard characters (*, ?) to
    QueryParser.  Default is to disallow them, as before.
    (<NAME> via Otis Gospodnetic)

 5. LUCENE-703: QueryParser changed to default to use of ConstantScoreRangeQuery
    for range queries. Added useOldRangeQuery property to QueryParser to allow
    selection of old RangeQuery class if required.
    (<NAME>)

 6. LUCENE-543: WildcardQuery now performs a TermQuery if the provided term
    does not contain a wildcard character (? or *), when previously a
    StringIndexOutOfBoundsException was thrown.
    (<NAME> via <NAME>)

 7. LUCENE-726: Removed the use of deprecated doc.fields() method and
    Enumeration.
    (<NAME> via Otis Gospodnetic)

 8. LUCENE-436: Removed finalize() in TermInfosReader and SegmentReader,
    and added a call to enumerators.remove() in TermInfosReader.close().
    The finalize() overrides were added to help with a pre-1.4.2 JVM bug
    that has since been fixed, plus we no longer support pre-1.4.2 JVMs.
    (Otis Gospodnetic)

 9. LUCENE-771: The default location of the write lock is now the
    index directory, and is named simply "write.lock" (without a big
    digest prefix).  The system properties "org.apache.lucene.lockDir"
    nor "java.io.tmpdir" are no longer used as the global directory
    for storing lock files, and the LOCK_DIR field of FSDirectory is
    now deprecated.  (Mike McCandless)

New features

 1. LUCENE-503: New ThaiAnalyzer and ThaiWordFilter in contrib/analyzers
    (<NAME> via <NAME>)

 2. LUCENE-545: New FieldSelector API and associated changes to
    IndexReader and implementations.  New Fieldable interface for use
    with the lazy field loading mechanism.  (<NAME> and Chuck
    Williams via Grant Ingersoll)

 3. LUCENE-676: Move Solr's PrefixFilter to Lucene core. (Yura
    Smolsky, Y<NAME>)

 4. LUCENE-678: Added NativeFSLockFactory, which implements locking
    using OS native locking (via java.nio.*).  (<NAME> via
    Yonik Seeley)

 5. LUCENE-544: Added the ability to specify different boosts for
    different fields when using MultiFieldQueryParser (<NAME>
    via Otis Gospodnetic)

 6. LUCENE-528: New IndexWriter.addIndexesNoOptimize() that doesn't
    optimize the index when adding new segments, only performing
    merges as needed.  (<NAME> via Yonik Seeley)

 7. LUCENE-573: QueryParser now allows backslash escaping in
    quoted terms and phrases. (<NAME> via Yonik Seeley)

 8. LUCENE-716: QueryParser now allows specification of unicode
    characters in terms via a unicode escape of the form \uXXXX
    (<NAME> via Yonik Seeley)

 9. LUCENE-709: Added RAMDirectory.sizeInBytes(), IndexWriter.ramSizeInBytes()
    and IndexWriter.flushRamSegments(), allowing applications to
    control the amount of memory used to buffer documents.
    (<NAME> via Yonik Seeley)

10. LUCENE-723: QueryParser now parses *:* as MatchAllDocsQuery
    (Yonik Seeley)

11. LUCENE-741: Command-line utility for modifying or removing norms
    on fields in an existing index.  This is mostly based on LUCENE-496
    and lives in contrib/miscellaneous.
    (<NAME>, <NAME>)

12. LUCENE-759: Added NGramTokenizer and EdgeNGramTokenizer classes and
    their passing unit tests.
    (Otis Gospodnetic)

13. LUCENE-565: Added NewIndexModifier (subclass of IndexWriter) to
    more efficiently handle updating documents (the "delete then add"
    use case).  This is intended to be an eventual replacement for the
    existing IndexModifier.  Added IndexWriter.flush() (renamed from
    flushRamSegments()) to flush all pending updates (held in RAM), to
    the Directory.  (<NAME> via <NAME>)

API Changes

 1. LUCENE-438: Remove "final" from Token, implement Cloneable, allow
    changing of termText via setTermText().  (Yonik Seeley)

 2. org.apache.lucene.analysis.nl.WordlistLoader has been deprecated
    and is supposed to be replaced with the WordlistLoader class in
    package org.apache.lucene.analysis (<NAME>)

 3. LUCENE-609: Revert return type of Document.getField(s) to Field
    for backward compatibility, added new Document.getFieldable(s)
    for access to new lazy loaded fields. (Yonik Seeley)

 4. LUCENE-608: Document.fields() has been deprecated and a new method
    Document.getFields() has been added that returns a List instead of
    an Enumeration (<NAME>)

 5. LUCENE-605: New Explanation.isMatch() method and new ComplexExplanation
    subclass allows explain methods to produce Explanations which model
    "matching" independent of having a positive value.
    (<NAME>)

 6. LUCENE-621: New static methods IndexWriter.setDefaultWriteLockTimeout
    and IndexWriter.setDefaultCommitLockTimeout for overriding default
    timeout values for all future instances of IndexWriter (as well
    as for any other classes that may refrence the static values,
    ie: IndexReader).
    (<NAME> via <NAME>)

 7. LUCENE-638: FSDirectory.list() now only returns the directory's<commit_msg>Forgot changes.txt updates for 762<commit_after>Lucene Change Log

$Id$

Trunk (not yet released)

Changes in runtime behavior

 1. 's' and 't' have been removed from the list of default stopwords
    in StopAnalyzer (also used in by StandardAnalyzer). Having e.g. 's'
    as a stopword meant that 's-class' led to the same results as 'class'.
    Note that this problem still exists for 'a', e.g. in 'a-class' as
    'a' continues to be a stopword.
    (<NAME>)

 2. LUCENE-478: Updated the list of Unicode code point ranges for CJK
    (now split into CJ and K) in StandardAnalyzer.  (<NAME> and
    <NAME> via Otis Gospodnetic)

 3. Modified some CJK Unicode code point ranges in StandardTokenizer.jj,
    and added a few more of them to increase CJK character coverage.
    Also documented some of the ranges.
    (Otis Gospodnetic)

 4. LUCENE-489: Add support for leading wildcard characters (*, ?) to
    QueryParser.  Default is to disallow them, as before.
    (<NAME> via Otis Gospodnetic)

 5. LUCENE-703: QueryParser changed to default to use of ConstantScoreRangeQuery
    for range queries. Added useOldRangeQuery property to QueryParser to allow
    selection of old RangeQuery class if required.
    (<NAME>)

 6. LUCENE-543: WildcardQuery now performs a TermQuery if the provided term
    does not contain a wildcard character (? or *), when previously a
    StringIndexOutOfBoundsException was thrown.
    (<NAME> via <NAME>)

 7. LUCENE-726: Removed the use of deprecated doc.fields() method and
    Enumeration.
    (<NAME> via Otis Gospodnetic)

 8. LUCENE-436: Removed finalize() in TermInfosReader and SegmentReader,
    and added a call to enumerators.remove() in TermInfosReader.close().
    The finalize() overrides were added to help with a pre-1.4.2 JVM bug
    that has since been fixed, plus we no longer support pre-1.4.2 JVMs.
    (Otis Gospodnetic)

 9. LUCENE-771: The default location of the write lock is now the
    index directory, and is named simply "write.lock" (without a big
    digest prefix).  The system properties "org.apache.lucene.lockDir"
    nor "java.io.tmpdir" are no longer used as the global directory
    for storing lock files, and the LOCK_DIR field of FSDirectory is
    now deprecated.  (<NAME>)

New features

 1. LUCENE-503: New ThaiAnalyzer and ThaiWordFilter in contrib/analyzers
    (<NAME> via <NAME>)

 2. LUCENE-545: New FieldSelector API and associated changes to
    IndexReader and implementations.  New Fieldable interface for use
    with the lazy field loading mechanism.  (<NAME> and Chuck
    Williams via <NAME>)

 3. LUCENE-676: Move Solr's PrefixFilter to Lucene core. (Yura
    Smolsky, Yonik Seeley)

 4. LUCENE-678: Added NativeFSLockFactory, which implements locking
    using OS native locking (via java.nio.*).  (<NAME> via
    Yonik Seeley)

 5. LUCENE-544: Added the ability to specify different boosts for
    different fields when using MultiFieldQueryParser (<NAME>
    via Otis Gospodnetic)

 6. LUCENE-528: New IndexWriter.addIndexesNoOptimize() that doesn't
    optimize the index when adding new segments, only performing
    merges as needed.  (<NAME> via Yonik Seeley)

 7. LUCENE-573: QueryParser now allows backslash escaping in
    quoted terms and phrases. (<NAME> via Yonik Seeley)

 8. LUCENE-716: QueryParser now allows specification of unicode
    characters in terms via a unicode escape of the form \uXXXX
    (<NAME> via Yonik Seeley)

 9. LUCENE-709: Added RAMDirectory.sizeInBytes(), IndexWriter.ramSizeInBytes()
    and IndexWriter.flushRamSegments(), allowing applications to
    control the amount of memory used to buffer documents.
    (<NAME> via Yonik Seeley)

10. LUCENE-723: QueryParser now parses *:* as MatchAllDocsQuery
    (Yonik Seeley)

11. LUCENE-741: Command-line utility for modifying or removing norms
    on fields in an existing index.  This is mostly based on LUCENE-496
    and lives in contrib/miscellaneous.
    (<NAME>, <NAME>)

12. LUCENE-759: Added NGramTokenizer and EdgeNGramTokenizer classes and
    their passing unit tests.
    (Otis Gospodnetic)

13. LUCENE-565: Added NewIndexModifier (subclass of IndexWriter) to
    more efficiently handle updating documents (the "delete then add"
    use case).  This is intended to be an eventual replacement for the
    existing IndexModifier.  Added IndexWriter.flush() (renamed from
    flushRamSegments()) to flush all pending updates (held in RAM), to
    the Directory.  (<NAME> via Mike McCandless)

14. LUCENE-762: Added in SIZE and SIZE_AND_BREAK FieldSelectorResult options
    which allow one to retrieve the size of a field without retrieving the actual field.
    (<NAME> via <NAME>)

API Changes

 1. LUCENE-438: Remove "final" from Token, implement Cloneable, allow
    changing of termText via setTermText().  (Yonik Seeley)

 2. org.apache.lucene.analysis.nl.WordlistLoader has been deprecated
    and is supposed to be replaced with the WordlistLoader class in
    package org.apache.lucene.analysis (Dan<NAME>)

 3. LUCENE-609: Revert return type of Document.getField(s) to Field
    for backward compatibility, added new Document.getFieldable(s)
    for access to new lazy loaded fields. (<NAME>)

 4. LUCENE-608: Document.fields() has been deprecated and a new method
    Document.getFields() has been added that returns a List instead of
    an Enumeration (<NAME>)

 5. LUCENE-605: New Explanation.isMatch() method and new ComplexExplanation
    subclass allows explain methods to produce Explanations which model
    "matching" independent of having a positive value.
    (<NAME>)

 6. LUCENE-621: New static methods IndexWriter.setDefaultWriteLockTimeout
    and IndexWriter.setDefaultCommitLockTimeout for overriding default
    timeout values for all future instances of IndexWriter (as well
    as for any other classes that may refrence the static values,
    ie: IndexReader).
    (<NAME> via <NAME>)

 7. LUCENE-638: FSDirectory.list() now only returns the directory's
{'fraction_non_alphanumeric': 0.0689801820192226, 'fraction_numerical': 0.0210938164497746, 'mean_word_length': 3.662172878667724, 'pattern_counts': {'":': 0, '<': 60, '<?xml version=': 0, '>': 60, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5026749', 'n_tokens_mistral': 3782, 'n_tokens_neox': 3358, 'n_words': 1503}