<commit_before>.. -*- rst -*-

:orphan:

.. highlightlang:: none

News
====

.. _release-5-0-1:

Release 5.0.1 - 2015/03/29
--------------------------

Improvements
^^^^^^^^^^^^

* [:doc:`/reference/commands/logical_range_filter`] Supported filter
  and sort.
* Supported range search by multiple column index.
* Added API :doc:`/reference/api/overview` document for users who want
  to use Groonga as library.
* [incompatible] Changed internal type of ``_score`` to floating point
  number from 32bit integer number. This is incompatible change for DB
  API users. This *isn't* incompatible change for query API users. It
  means that users who just use :doc:`/reference/commands/select`
  aren't affected. Use the following code that works with both older
  and newer Groonga:

  .. code-block:: c

     grn_obj *score;
     double score_value;

     if (score->header.domain == GRN_DB_FLOAT) {
       score_value = GRN_FLOAT_VALUE(score);
     } else {
       score_value = (double)GRN_INT32_VALUE_FLOAT_VALUE(score);
     }

* [:doc:`/reference/commands/select`] Added more strict check for
  invalid drilldown parameter.
* Added :c:func:`grn_ctx_get_all_tables()`. [Suggested by Masatoshi
  Teruya]
* ??? Supported sort by index value.
* Supported to customize score function. See :doc:`/reference/scorer`
  for details.
* Added the following predicates that check :c:type:`grn_obj` type to
  DB API:

  * :c:func:`grn_obj_is_table()`
  * :c:func:`grn_obj_is_proc_proc()`
  * :c:func:`grn_obj_is_function_proc()`
  * :c:func:`grn_obj_is_selector_proc()`
  * :c:func:`grn_obj_is_scorer_proc()`

* [experimental] Supported skipping posting list when searching
  popular term and rare term at the same time. It will improve
  performance. Set ``GRN_II_CURSOR_SET_MIN_ENABLE`` environment
  variable to ``1`` to enable the feature. The feature is disabled by
  default.
* [doc] Added :doc:`/reference/functions/in_values` document.
* [doc] Added :doc:`/reference/commands/logical_count` document.
* [mruby] Implemented custom ``#inspect`` method. Is is useful for
  debugging.
* Added :doc:`/reference/scorers/scorer_tf_at_most` scorer. It
  limits not to exceed specified score regardless of term frequency.
* [mruby] Supported estimating size for index search.
* Added the following functions to estimate size by index:

  * :c:func:`grn_expr_estimate_size()`
  * :c:func:`grn_ii_estimate_size_for_query()`
  * :c:func:`grn_ii_estimate_size_for_lexicon_cursor()`

* Added missing :ref:`normalizer-auto` availability check. [GitHub#283]
  [Reported by Tasuku SUENAGA]
* Dropped Visual Studio 2010 support.
* [experimental][mecab] Supported chunked tokenization. This feature
  is a workaround for MeCab's "too long sentense" error.  Specify
  ``yes`` to ``GRN_MECAB_CHUNKED_TOKENIZE_ENABLED`` environment
  variable to enable it. By this configuration, Groonga splits a long
  text (8192 bytes over text by default) into small chunks and passes
  each chunk to MeCab. By this process, the above error isn't
  occurred. Additionally, you can customize chunk threshold bytes by
  ``GRN_MECAB_CHUNK_SIZE_THRESHOLD`` environment variable. Note that
  ``,``, ``.``, ``!``, ``?``, ``U+3001 IDEOGRAPHIC COMMA``, ``U+3002
  IDEOGRAPHIC FULL STOP``, ``U+FF01 FULLWIDTH EXCLAMATION MARK`` and
  ``U+FF1F FULLWIDTH QUESTION MARK`` are treated as chunk delimiter
  characters.
* Supported ``--pid-file`` in server mode of
  :doc:`/reference/executables/groonga`.
* [groonga-httpd] Supported graceful stop to clean Groonga. It doesn't
  terminate the open connections immediately.
* Supported REGEXP. Use ``COLUMN_NAME:~REGEXP`` in query syntax,
  ``TARGET @~ REGEXP`` in script syntax.
* [experimental] Added :doc:`/reference/commands/plugin_unregister`
  command.
* [http] Supported new line and "," for chunk separator in POST
  data. It decreases buffer size, as a result, it reduces load time
  without these separators.
* Added ``TokenRegexp`` tokenizer. See
  :doc:`/reference/regular_expression` for details.
* [doc] Added :doc:`/reference/tokenizers` document.
* Improved POSIX.2 compatibility by using ``.`` as bash's "source"
  command replacement. [GitHub#317] [Patch by Jun Kuriyama]
* [windows] Changed to default IO version 1. It reduces disk usage on<commit_msg>doc: use "regular expression"<commit_after>.. -*- rst -*-

:orphan:

.. highlightlang:: none

News
====

.. _release-5-0-1:

Release 5.0.1 - 2015/03/29
--------------------------

Improvements
^^^^^^^^^^^^

* [:doc:`/reference/commands/logical_range_filter`] Supported filter
  and sort.
* Supported range search by multiple column index.
* Added API :doc:`/reference/api/overview` document for users who want
  to use Groonga as library.
* [incompatible] Changed internal type of ``_score`` to floating point
  number from 32bit integer number. This is incompatible change for DB
  API users. This *isn't* incompatible change for query API users. It
  means that users who just use :doc:`/reference/commands/select`
  aren't affected. Use the following code that works with both older
  and newer Groonga:

  .. code-block:: c

     grn_obj *score;
     double score_value;

     if (score->header.domain == GRN_DB_FLOAT) {
       score_value = GRN_FLOAT_VALUE(score);
     } else {
       score_value = (double)GRN_INT32_VALUE_FLOAT_VALUE(score);
     }

* [:doc:`/reference/commands/select`] Added more strict check for
  invalid drilldown parameter.
* Added :c:func:`grn_ctx_get_all_tables()`. [Suggested by Masatoshi
  Teruya]
* ??? Supported sort by index value.
* Supported to customize score function. See :doc:`/reference/scorer`
  for details.
* Added the following predicates that check :c:type:`grn_obj` type to
  DB API:

  * :c:func:`grn_obj_is_table()`
  * :c:func:`grn_obj_is_proc_proc()`
  * :c:func:`grn_obj_is_function_proc()`
  * :c:func:`grn_obj_is_selector_proc()`
  * :c:func:`grn_obj_is_scorer_proc()`

* [experimental] Supported skipping posting list when searching
  popular term and rare term at the same time. It will improve
  performance. Set ``GRN_II_CURSOR_SET_MIN_ENABLE`` environment
  variable to ``1`` to enable the feature. The feature is disabled by
  default.
* [doc] Added :doc:`/reference/functions/in_values` document.
* [doc] Added :doc:`/reference/commands/logical_count` document.
* [mruby] Implemented custom ``#inspect`` method. Is is useful for
  debugging.
* Added :doc:`/reference/scorers/scorer_tf_at_most` scorer. It
  limits not to exceed specified score regardless of term frequency.
* [mruby] Supported estimating size for index search.
* Added the following functions to estimate size by index:

  * :c:func:`grn_expr_estimate_size()`
  * :c:func:`grn_ii_estimate_size_for_query()`
  * :c:func:`grn_ii_estimate_size_for_lexicon_cursor()`

* Added missing :ref:`normalizer-auto` availability check. [GitHub#283]
  [Reported by <NAME>]
* Dropped Visual Studio 2010 support.
* [experimental][mecab] Supported chunked tokenization. This feature
  is a workaround for MeCab's "too long sentense" error.  Specify
  ``yes`` to ``GRN_MECAB_CHUNKED_TOKENIZE_ENABLED`` environment
  variable to enable it. By this configuration, Groonga splits a long
  text (8192 bytes over text by default) into small chunks and passes
  each chunk to MeCab. By this process, the above error isn't
  occurred. Additionally, you can customize chunk threshold bytes by
  ``GRN_MECAB_CHUNK_SIZE_THRESHOLD`` environment variable. Note that
  ``,``, ``.``, ``!``, ``?``, ``U+3001 IDEOGRAPHIC COMMA``, ``U+3002
  IDEOGRAPHIC FULL STOP``, ``U+FF01 FULLWIDTH EXCLAMATION MARK`` and
  ``U+FF1F FULLWIDTH QUESTION MARK`` are treated as chunk delimiter
  characters.
* Supported ``--pid-file`` in server mode of
  :doc:`/reference/executables/groonga`.
* [groonga-httpd] Supported graceful stop to clean Groonga. It doesn't
  terminate the open connections immediately.
* Supported regular expression. Use ``COLUMN_NAME:~REGEXP`` in query syntax,
  ``TARGET @~ REGEXP`` in script syntax.
* [experimental] Added :doc:`/reference/commands/plugin_unregister`
  command.
* [http] Supported new line and "," for chunk separator in POST
  data. It decreases buffer size, as a result, it reduces load time
  without these separators.
* Added ``TokenRegexp`` tokenizer. See
  :doc:`/reference/regular_expression` for details.
* [doc] Added :doc:`/reference/tokenizers` document.
* Improved POSIX.2 compatibility by using ``.`` as bash's "source"
  command replacement. [GitHub#317] [Patch by <NAME>yama]
* [windows] Changed to default IO version 1. It reduces disk usage on
{'fraction_non_alphanumeric': 0.120088868101029, 'fraction_numerical': 0.010757717492984098, 'mean_word_length': 4.472168905950096, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4510564', 'n_tokens_mistral': 2882, 'n_tokens_neox': 2716, 'n_words': 879}