<commit_before>/*
Package lmdbsync provides advanced synchronization for LMDB environments at the
cost of performance.  The package provides a drop-in replacement for *lmdb.Env
that can be used in situations where the database may be resized or where the
flag lmdb.NoLock is used.

Bypassing an Env's methods to access the underlying lmdb.Env is not safe.  The
severity of such usage depends such behavior should be strictly avoided as it
may produce undefined behavior from the LMDB C library.

Resizing the environment

The Env type synchronizes all calls to Env.SetMapSize so that it may, with some
caveats, be safely called in the presence of concurrent transactions after an
environment has been opened.  All running transactions must complete before the
method will be called on the underlying lmdb.Env.

If an open transaction depends on a change in map size then the Env will
deadlock and block all future transactions.  When using a Handler to
automatically resize the map this implies the restriction that transactions
must terminate independently of the creation and termination of other
transactions to avoid deadlock

In the simplest example, a view transaction that attempts an update on the
underlying Env will deadlock the environment if the map is full and a Handler
attempts to resize the map so the update may be retried.

	env.View(func(txn *lmdb.Txn) (err error) {
		v, err := txn.Get(db, key)
		if err != nil {
			return err
		}
		err = env.Update(func(txn *lmdb.Txn) (err error) { // deadlock on lmdb.MapFull!
			txn.Put(dbi, key, append(v, b...))
		})
		return err
	}

The update should instead be prepared inside the view and then executed
following its termination.  This removes the implicit dependence of the view on
calls to Env.SetMapSize().

	var v []byte
	env.View(func(txn *lmdb.Txn) (err error) {
		// RawRead isn't used because the value will be used outside the
		// transaction.
		v, err = txn.Get(db, key)
		if err != nil {
			return err
		}
		return nil
	}
	if err != nil {
		// ...
	}
	err = env.Update(func(txn *lmdb.Txn) (err error) { // no deadlock, even if env is resized!
		txn.Put(dbi, key, append(v, b...))
	})

The developers of LMDB officially recommend against applications changing the
memory map size for an open database.  It requires careful synchronization by
all processes accessing the database file.  And, a large memory map will not
affect disk usage on operating systems that support sparse files (e.g. Linux,
not OS X).

See mdb_env_set_mapsize.

MapResized

When multiple processes access and resize an environment it is not uncommon to
encounter a MapResized error which requires synchronized action before
continuing normal operations.

Using the Handler interface provided by the package MapResizedHandler can be
used to automatically resize an enviornment when a lmdb.MapResized error is
encountered.  However, usage of the MapResizedHandler puts important caveats on
how one can safely work with transactions.  See the function documentation for
more detailed information.

When other processes may change an environment's map size it is extremely
important to ensure that transactions terminate independent of all other
transactions.  The MapResized error may be returned at the beginning of any
transaction.

See mdb_txn_begin and MDB_MAP_RESIZED.

MapFull

Similar to the MapResizedHandler the MapFullHandler will automatically resize
the map and retry transactions when a MapFull error is encountered.  Usage of
the MapFullHandler puts important caveats on how one can safely work with
transactions.  See the function documentation for more detailed information.

The caveats on transactions are lessened if lmdb.MapFull is the only error
being handled (when multi-processing is not a concern).  The only requirement
then is that view transactions not depend on the termination of updates
transactions.

See mdb_env_set_mapsize and MDB_MAP_FULL.

NoLock

When the lmdb.NoLock flag is set on an environment Env handles all transaction
synchronization using Go structures and is an experimental feature.  It is
unclear what benefits this provides.

Usage of lmdb.NoLock requires that update transactions acquire an exclusive
lock on the environment.  In such cases it is required that view transactions
execute independently of update transactions, a requirement more strict than
that from handling MapFull.

See mdb_env_open and MDB_NOLOCK.
*/
package lmdbsync

import (<commit_msg>lmdbsync: clean up godoc header<commit_after>/*
Package lmdbsync provides advanced synchronization for LMDB environments at the
cost of performance.  The package provides a drop-in replacement for *lmdb.Env
that can be used in situations where the database may be resized or where the
flag lmdb.NoLock is used.

Bypassing an Env's methods to access the underlying lmdb.Env is not safe.  The
severity of such usage depends such behavior should be strictly avoided as it
may produce undefined behavior from the LMDB C library.

Resizing the environment

The Env type synchronizes all calls to Env.SetMapSize so that it may, with some
caveats, be safely called in the presence of concurrent transactions after an
environment has been opened.  All running transactions must complete before the
method will be called on the underlying lmdb.Env.

If an open transaction depends on a call to Env.SetMapSize then the Env will
deadlock and block all future transactions.  When using a Handler to
automatically call Env.SetMapSize this implies the restriction that
transactions must terminate independently of the creation/termination of other
transactions to avoid deadlock.

In the simplest example, a view transaction that attempts an update on the
underlying Env will deadlock the environment if the map is full and a Handler
attempts to resize the map so the update may be retried.

	env.View(func(txn *lmdb.Txn) (err error) {
		v, err := txn.Get(db, key)
		if err != nil {
			return err
		}
		err = env.Update(func(txn *lmdb.Txn) (err error) { // deadlock on lmdb.MapFull!
			txn.Put(dbi, key, append(v, b...))
		})
		return err
	}

The update should instead be prepared inside the view and then executed
following its termination.  This removes the implicit dependence of the view on
calls to Env.SetMapSize().

	var v []byte
	env.View(func(txn *lmdb.Txn) (err error) {
		// RawRead isn't used because the value will be used outside the
		// transaction.
		v, err = txn.Get(db, key)
		if err != nil {
			return err
		}
		return nil
	}
	if err != nil {
		// ...
	}
	err = env.Update(func(txn *lmdb.Txn) (err error) { // no deadlock, even if env is resized!
		txn.Put(dbi, key, append(v, b...))
	})

The developers of LMDB officially recommend against applications changing the
memory map size for an open database.  It requires careful synchronization by
all processes accessing the database file.  And, a large memory map will not
affect disk usage on operating systems that support sparse files (e.g. Linux,
not OS X).

See mdb_env_set_mapsize.

MapFull

The MapFullHandler function configures an Env to automatically call increase
the map size with Env.SetMapSize and retry transactions when a lmdb.MapFull
error prevents an update from being committed.

Because updates may need to execute multiple times in the presence of
lmdb.MapFull it is important to make sure their TxnOp functions are idempotent
and do not cause unwanted additive change to the program state.

See mdb_txn_commit and MDB_MAP_FULL.

MapResized

When multiple processes access and resize an environment it is not uncommon to
encounter a MapResized error which prevents the TxnOp from being executed and
requires a synchronized call to Env.SetMapSize before continuing normal
operation.

The MapResizedHandler function configures an Env to automatically adopt a new
map size when a lmdb.MapResized error is encountered and retry execution of the
TxnOp.

See mdb_txn_begin and MDB_MAP_RESIZED.

NoLock

When the lmdb.NoLock flag is set on an environment Env handles all transaction
synchronization using Go structures and is an experimental feature.  It is
unclear what benefits this provides.

Usage of lmdb.NoLock requires that update transactions acquire an exclusive
lock on the environment.  In such cases it is required that view transactions
execute independently of update transactions, a requirement more strict than
that from handling MapFull.

See mdb_env_open and MDB_NOLOCK.
*/
package lmdbsync

import (
{'fraction_non_alphanumeric': 0.04265908282971916, 'fraction_numerical': 0.0, 'mean_word_length': 4.301507537688442, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2192959', 'n_tokens_mistral': 2345, 'n_tokens_neox': 2235, 'n_words': 1211}