<commit_before>* Riak 1.0.0 Release Notes

NOTE: These release notes should be considered a work in progress
until the final 1.0.0 release is made.

** Major Features and Improvements for Riak
*** 2i
    Secondary Indexes (2I) makes it easier to find your data in
    Riak.  With 2i, an application can index an object under an arbitrary
    number of field/value pairs specified at write time.  Later, the
    application can query the index, pulling back a list of keys that
    match a given value or fall within a given range of values.  This
    simple mechanism enables applications to model alternate keys,
    one-to-many relationships, and many-to-many relationships in
    Riak.  Indexing is atomic and real-time.  Querying is integrated with
    MapReduce.  In this release, the query feature set is minimal: only
    exact match and range queries are supported, and only on a single
    field.  In addition, the system must be configured to use the
    =riak_kv_eleveldb_backend=.  Future releases will bring support for more
    complex queries and additional backends, depending upon user feedback.


*** Backend Refactoring
   There are 3 main areas of change with regard to the backend refactoring.
**** Changes to the suite of available backends
     - Added: eleveldb backend
     -  Merged: ets and cache backend have been consolidated into a single 
        memory backend.
     -  Removed: dets, filesystem (fs), and gb_trees backends
**** Standardized API
     - All of the backend modules now share a common API.
     - Vnode code has been greatly simplified.
**** Asynchronous folding
     - The vnode no longer has to be blocked by fold operations.
     - A node for a backend capable of asynchronous folds starts a worker 
       pool to handle fold requests.
     - Asynchronous folds are done using read-only snapshots of the data.
     - Bitcask, eleveldb, memory, and multi backends support asynchronous folding.
     - Asynchronous folding may be disabled by adding ={async_folds, false}=
       in the =riak_kv= section of the =app.config= file. 


*** Lager
    Lager is Riak’s new logging library.  It aims to alleviate some of the
    shortcomings in Erlang/OTP’s bundled logging library; specifically
    more readable log messages, support for external logfile rotation,
    more granular log levels and runtime changes.


*** LevelDB
    Riak now has support for LevelDB, a fast key-value storage library
    written at Google that provides an ordered mapping from string keys to
    string values.  This backend provides a viable alternative to the
    innostore backend, with better performance.  LevelDB has an append-only
    design that provides high-performance writes.  Unlike Bitcask, however,
    LevelDB keeps keys in sorted order and uses a multi-level cache to
    keep portions of the keyspace in memory.  This design allows LevelDB to
    effectively deal with very large keyspaces, but at the cost of
    additional seeks on read.


*** Pipe
    Riak Pipe is the new plumbing for Riak's MapReduce system.  It improves
    upon the legacy system in many ways, including built-in backpressure
    to keep phases from overloading others downstream, and a design shaped
    to make much better use of the parallel-processing resources available
    in the cluster.  Riak Pipe is also a general queue-based parallel
    processing system, which new applications may find is a more natural
    starting point than bare Riak Core.   Riak Pipe is the default plumbing
    for MapReduce queries on new clusters.   To move old clusters onto Riak
    Pipe MapReduce, add ={mapred_system, pipe}= to the =riak_kv= section
    of your node’s =app.config=.   To force Riak to use the legacy system for
    MapReduce, set ={mapred_system, legacy}= in the =riak_kv= section of
    =app.config= instead.


*** URL encoding changes over REST API
    Riak now decode URL encoded bucket/keys over the REST API, rather than
    prior behavior which was to decode links but not decode
    buckets/keys.  The default cluster behavior is configurable in
    =riak_core= section of =app.config=: ={http_url_decoding, on}= provides the
    new behavior (decode everything), missing/anything else provides the
    current behavior.

    The module =riak_kv_encoding_migrate.erl= is also provided to help
    migrate existing encoded buckets/keys to their decoded equivalents


*** Riak Clustering Overhaul
    The clustering subsystem of Riak has been overhauled for 1.0, addressing a 
    variety of bugs and limitations.
    - When reassigning a partition to a new node, the partition data from
      the existing owner is first transferred to the new owner before
      officially changing the ownership in the ring. This fixes a bug
      where 404s could appear while ownership was being changed.
    - Adding/removing nodes to a cluster is now more robust, and it is no 
      longer necessary to check for ring convergence (=riak-admin ringready=)
      between adding/removing nodes. Adding multiple nodes all at once should 
      "just work".  
    - Handoff related to changing node owners can now occur while a cluster is 
      under load, therefore allowing a Riak cluster to scale up and down during
      load / normal operation.
    - Various other clustering bug/fixes. See the fixed bug list for details.    
**** Notes
     - =riak-admin join= has new semantics.  The command is now a one-way
       operation that joins a single node to cluster.  The node that the
       command is executed under should be the desired joining node, and
       the target of the command should be a member of the desired target
       cluster.  The new command requires the joining node to be a singleton
       (1-node) cluster.
     - =riak-admin leave= is now the only safe way to remove a node from a
       cluster.  The leave command ensures that the exiting node will
       handoff all its partitions before leaving the cluster.  It should be
       executed by the node intended to leave.
     - =riak-admin remove= is now changed to a force-remove, where a node
       is immediately removed from the cluster without waiting on
       handoff.  This is designed for cases where a node is unrecoverable
       and for which handoff does not make sense.
     - The new cluster changes require all nodes to be up and reachable in
       order for new members to be integrated into the cluster and for the
       data to be rebalanced.  During brief node outages, the new protocol
       will wait until all nodes are eventually back online and continue
       automatically.  If it is known that a node will be offline for an
       extended period, the new =riak-admin down= command can be used to
       mark a node as offline and the cluster will then resume integrating
       nodes and performing ring rebalances.  Nodes marked as down will
       automatically rejoin and reintegrate into the cluster when they come
       back online.


*** Get/Put/Delete Improvements
    The way that Riak versions and updates objects has been
    overhauled.  ClientIds are no longer used when updating objects, the
    server handles all versioning using a vector clock id per-vnode.
    
    New clusters are configured with the new vclock behavior turned on.
    If you are performing a rolling upgrade of an existing cluster, once
    all nodes have been upgraded the =app.config= needs to be updated to add
    ={vnode_vclocks, true}=.
    
    Puts are now coordinated in the same way as on the original Dynamo
    system.  Requests must be handled by a node in the preference list
    (primary or fallback) for that bucket/key.   Nodes will automatically
    forward to a valid node when necessary and increment the coord_redirs
    stats counter.  The put is initially written to the local vnode before
    forwarding to the remote vnodes.   This ensures that the updated vclock
    for the riak object will replace the existing value or create siblings
    in partitioning/failure scenario where the same client can see both
    sides.
    
    Error proofing for the failure scenarios has made it so that clients
    no longer have to be well behaved.   If =allow_mult= is set =true=, every
    time you create a new object and put over an existing one it will
    create a sibling.  Vector clocks should now be much smaller in size as
    only a few vclock ids are now updated.  This should resolve a number of
    issues due to vclock pruning causing siblings.
    
    Gets have been changed to return more information during failure.
    Prior to 1.0 there were cases where Riak returned not found if not
    enough valid responses were returned.  The case of not enough responses
    has been changed to an error instead reported as 503 over HTTP or as
    ={error, {r_val_unsatistfied, R, NumResponses}}= for Erlang/PBC clients.

    New options have been added to the get requests for handling
    notfound.  Prior to 1.0 only successful reads were counted towards R
    and there was some logic to try and fail early rather than wait until
    the request timed out if not enough replies were received
    (basic_quorum).  This meant when a node went down and another node
    didn’t response you would get a not found response that triggered a
    read repair and then if you retrieved the object again it would be
    present.
    
    Now that other enhancements have been made (delete and asynchronous
    improvements to the vnodes) we can change notfounds to be counted
    towards R and disable the basic_quorum logic by setting bucket
    properties to =[{notfound_ok, true}, {basic_quorum, false}]= and reduce
    the number of cases where notfound is returned on first request when
    an object could be.


*** Search

**** Integration into Riak

Prior to the 1.0 release if you wanted a Riak cluster with search
capability you needed to install the Riak Search package.  As of 1.0
this functionality is now included with the standard and enterprise
Riak packages.  By default this functionality is turned off but
enabling it is a simple matter of changing the =enabled= flag to
=true= in the =riak_search= section of the =app.config= file.

**** Data Center Replication Support

Multi-datacenter replication that comes with Riak EDS now fully
supports Search.  Now, not only will the standard KV data be
replicated but also any indexes created by Search.  To be clear, this
includes all indexes no matter how they were created; whether by the
Search bucket hook, =search-cmd index=, or the Solr-like interface.

**** Removal of Java Support

Prior to 1.0 Riak Search provided the ability to interface with the
standard Lucene analyzers or even other customer analyzers written in
Java.  While this certainly can be useful it added extra complexity to
both the code and the running system.  After consulting with our
clients and community it was determined that removing Java support<commit_msg>Updated 1.0.0 release notes with delete/restore changes.<commit_after>* Riak 1.0.0 Release Notes

NOTE: These release notes should be considered a work in progress
until the final 1.0.0 release is made.

** Major Features and Improvements for Riak
*** 2i
    Secondary Indexes (2I) makes it easier to find your data in
    Riak.  With 2i, an application can index an object under an arbitrary
    number of field/value pairs specified at write time.  Later, the
    application can query the index, pulling back a list of keys that
    match a given value or fall within a given range of values.  This
    simple mechanism enables applications to model alternate keys,
    one-to-many relationships, and many-to-many relationships in
    Riak.  Indexing is atomic and real-time.  Querying is integrated with
    MapReduce.  In this release, the query feature set is minimal: only
    exact match and range queries are supported, and only on a single
    field.  In addition, the system must be configured to use the
    =riak_kv_eleveldb_backend=.  Future releases will bring support for more
    complex queries and additional backends, depending upon user feedback.


*** Backend Refactoring
   There are 3 main areas of change with regard to the backend refactoring.
**** Changes to the suite of available backends
     - Added: eleveldb backend
     -  Merged: ets and cache backend have been consolidated into a single 
        memory backend.
     -  Removed: dets, filesystem (fs), and gb_trees backends
**** Standardized API
     - All of the backend modules now share a common API.
     - Vnode code has been greatly simplified.
**** Asynchronous folding
     - The vnode no longer has to be blocked by fold operations.
     - A node for a backend capable of asynchronous folds starts a worker 
       pool to handle fold requests.
     - Asynchronous folds are done using read-only snapshots of the data.
     - Bitcask, eleveldb, memory, and multi backends support asynchronous folding.
     - Asynchronous folding may be disabled by adding ={async_folds, false}=
       in the =riak_kv= section of the =app.config= file. 


*** Lager
    Lager is Riak’s new logging library.  It aims to alleviate some of the
    shortcomings in Erlang/OTP’s bundled logging library; specifically
    more readable log messages, support for external logfile rotation,
    more granular log levels and runtime changes.


*** LevelDB
    Riak now has support for LevelDB, a fast key-value storage library
    written at Google that provides an ordered mapping from string keys to
    string values.  This backend provides a viable alternative to the
    innostore backend, with better performance.  LevelDB has an append-only
    design that provides high-performance writes.  Unlike Bitcask, however,
    LevelDB keeps keys in sorted order and uses a multi-level cache to
    keep portions of the keyspace in memory.  This design allows LevelDB to
    effectively deal with very large keyspaces, but at the cost of
    additional seeks on read.


*** Pipe
    Riak Pipe is the new plumbing for Riak's MapReduce system.  It improves
    upon the legacy system in many ways, including built-in backpressure
    to keep phases from overloading others downstream, and a design shaped
    to make much better use of the parallel-processing resources available
    in the cluster.  Riak Pipe is also a general queue-based parallel
    processing system, which new applications may find is a more natural
    starting point than bare Riak Core.   Riak Pipe is the default plumbing
    for MapReduce queries on new clusters.   To move old clusters onto Riak
    Pipe MapReduce, add ={mapred_system, pipe}= to the =riak_kv= section
    of your node’s =app.config=.   To force Riak to use the legacy system for
    MapReduce, set ={mapred_system, legacy}= in the =riak_kv= section of
    =app.config= instead.


*** URL encoding changes over REST API
    Riak now decode URL encoded bucket/keys over the REST API, rather than
    prior behavior which was to decode links but not decode
    buckets/keys.  The default cluster behavior is configurable in
    =riak_core= section of =app.config=: ={http_url_decoding, on}= provides the
    new behavior (decode everything), missing/anything else provides the
    current behavior.

    The module =riak_kv_encoding_migrate.erl= is also provided to help
    migrate existing encoded buckets/keys to their decoded equivalents


*** Riak Clustering Overhaul
    The clustering subsystem of Riak has been overhauled for 1.0, addressing a 
    variety of bugs and limitations.
    - When reassigning a partition to a new node, the partition data from
      the existing owner is first transferred to the new owner before
      officially changing the ownership in the ring. This fixes a bug
      where 404s could appear while ownership was being changed.
    - Adding/removing nodes to a cluster is now more robust, and it is no 
      longer necessary to check for ring convergence (=riak-admin ringready=)
      between adding/removing nodes. Adding multiple nodes all at once should 
      "just work".  
    - Handoff related to changing node owners can now occur while a cluster is 
      under load, therefore allowing a Riak cluster to scale up and down during
      load / normal operation.
    - Various other clustering bug/fixes. See the fixed bug list for details.    
**** Notes
     - =riak-admin join= has new semantics.  The command is now a one-way
       operation that joins a single node to cluster.  The node that the
       command is executed under should be the desired joining node, and
       the target of the command should be a member of the desired target
       cluster.  The new command requires the joining node to be a singleton
       (1-node) cluster.
     - =riak-admin leave= is now the only safe way to remove a node from a
       cluster.  The leave command ensures that the exiting node will
       handoff all its partitions before leaving the cluster.  It should be
       executed by the node intended to leave.
     - =riak-admin remove= is now changed to a force-remove, where a node
       is immediately removed from the cluster without waiting on
       handoff.  This is designed for cases where a node is unrecoverable
       and for which handoff does not make sense.
     - The new cluster changes require all nodes to be up and reachable in
       order for new members to be integrated into the cluster and for the
       data to be rebalanced.  During brief node outages, the new protocol
       will wait until all nodes are eventually back online and continue
       automatically.  If it is known that a node will be offline for an
       extended period, the new =riak-admin down= command can be used to
       mark a node as offline and the cluster will then resume integrating
       nodes and performing ring rebalances.  Nodes marked as down will
       automatically rejoin and reintegrate into the cluster when they come
       back online.


*** Get/Put Improvements
    The way that Riak versions and updates objects has been
    overhauled.  ClientIds are no longer used when updating objects, the
    server handles all versioning using a vector clock id per-vnode.
    
    New clusters are configured with the new vclock behavior turned on.
    If you are performing a rolling upgrade of an existing cluster, once
    all nodes have been upgraded the =app.config= needs to be updated to add
    ={vnode_vclocks, true}=.
    
    Puts are now coordinated in the same way as on the original Dynamo
    system.  Requests must be handled by a node in the preference list
    (primary or fallback) for that bucket/key.   Nodes will automatically
    forward to a valid node when necessary and increment the coord_redirs
    stats counter.  The put is initially written to the local vnode before
    forwarding to the remote vnodes.   This ensures that the updated vclock
    for the riak object will replace the existing value or create siblings
    in partitioning/failure scenario where the same client can see both
    sides.
    
    Error proofing for the failure scenarios has made it so that clients
    no longer have to be well behaved.   If =allow_mult= is set =true=, every
    time you create a new object and put over an existing one it will
    create a sibling.  Vector clocks should now be much smaller in size as
    only a few vclock ids are now updated.  This should resolve a number of
    issues due to vclock pruning causing siblings.
    
    Gets have been changed to return more information during failure.
    Prior to 1.0 there were cases where Riak returned not found if not
    enough valid responses were returned.  The case of not enough responses
    has been changed to an error instead reported as 503 over HTTP or as
    ={error, {r_val_unsatistfied, R, NumResponses}}= for Erlang/PBC clients.

    New options have been added to the get requests for handling
    notfound.  Prior to 1.0 only successful reads were counted towards R
    and there was some logic to try and fail early rather than wait until
    the request timed out if not enough replies were received
    (basic_quorum).  This meant when a node went down and another node
    didn’t response you would get a not found response that triggered a
    read repair and then if you retrieved the object again it would be
    present.
    
    Now that other enhancements have been made (delete and asynchronous
    improvements to the vnodes) we can change notfounds to be counted
    towards R and disable the basic_quorum logic by setting bucket
    properties to =[{notfound_ok, true}, {basic_quorum, false}]= and reduce
    the number of cases where notfound is returned on first request when
    an object could be.

*** Delete Changes

    The changes to the vector clocks make it very important that the tombstones
    written by deletes are removed from all vnodes.  In 0.14.2 the tombstone
    was removed as it was confirmed that all vnodes had the tombstone stored.
    For 1.0.0 this has been changed to delay the removal by a short period of
    time (default 3s) and is aborted if the object is updated.  The behavior
    is configurable by setting {delete_mode, Mode} in the riak_kv secion of
    app.config and can be set to the following

      keep - tombstones are kept forever

      immediate - tombstones are removed without delay - 0.14.2 behavior.

      NNNN - delay in microseconds to check for changes before removing tombstone.
             The default is 3000 for 3s.

    The riak_client, HTTP and PBC interfaces have been modified to return vclock 
    information for deleted objects.  riak_client:get accepts a deletedvclock
    option which changes a deleted object read from {error, notfound} to
    {error, {deleted, VClock}}.

    The HTTP interface returns X-Riak-Vclock with 404s now.  The PBC interface
    can request the vclock setting the deletedvclock option on get requests.

    Clients that have not been updated to take advantage of the new information
    may create siblings with tombstones if they create a new object over one
    deleted recently enough the tombstone still exists.

*** Backup/Restore Changes

    Restore has been changed to restore the exact objects that were backed up.
    This means that if they have been updated since the backup, or deleted
    recently enough that the tombstone has not been removed, then the backed
    up object will not be restored.  Waiting until the tombstones are removed
    should enable the objects to be restored (however if delete_mode=keep this will
    never happen).

    In 0.14.2 restoring an object would have updted the vclock with a random
    client id and created a sibling, and if allow_mult=false the two resolved
    by the last updated time.

*** Search

**** Integration into Riak

Prior to the 1.0 release if you wanted a Riak cluster with search
capability you needed to install the Riak Search package.  As of 1.0
this functionality is now included with the standard and enterprise
Riak packages.  By default this functionality is turned off but
enabling it is a simple matter of changing the =enabled= flag to
=true= in the =riak_search= section of the =app.config= file.

**** Data Center Replication Support

Multi-datacenter replication that comes with Riak EDS now fully
supports Search.  Now, not only will the standard KV data be
replicated but also any indexes created by Search.  To be clear, this
includes all indexes no matter how they were created; whether by the
Search bucket hook, =search-cmd index=, or the Solr-like interface.

**** Removal of Java Support

Prior to 1.0 Riak Search provided the ability to interface with the
standard Lucene analyzers or even other customer analyzers written in
Java.  While this certainly can be useful it added extra complexity to
both the code and the running system.  After consulting with our
clients and community it was determined that removing Java support
{'fraction_non_alphanumeric': 0.034649543426593835, 'fraction_numerical': 0.00354417712546387, 'mean_word_length': 3.2464589235127477, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4179684', 'n_tokens_mistral': 6123, 'n_tokens_neox': 5839, 'n_words': 3529}