libnetwork: remove dead / unused code from datastore and kvstore
### libnetwork/internal/kvstore: remove unused Config options

The only remaining kvstore is BoltDB, which doesn't use TLS connections
or authentication, so we can remove these options.

### libnetwork/internal/kvstore: fix some linting issues

### libnetwork/internal/kvstore: remove unused WatchTree and NewLock methods

These were not used, and not implemented by the BoltDB store.

### libnetwork/internal/kvstore: remove unused WriteOptions

The WriteOptions struct was only used to set the "IsDir" option. This option
was added in https://github.com/docker/libkv/commit/d635a8e32b67f1ac511b7fcc15d524dedb9f549b (https://github.com/docker/libkv/pull/75)
and was only supported by the etcd libkv store.

The BoltDB store does not support this option, making the WriteOptions
struct fully redundant.

### libnetwork/internal/kvstore: AtomicPut(): remove unused "created" return

This boolean was not used anywhere, so we can remove it.

### libnetwork/internal/kvstore: AtomicDelete(): remove unused "deleted" return

This boolean was not used anywhere, so we can remove it. Also cleaning up
the implementation a bit.

### libnetwork/internal/kvstore/boltdb: minor cleanup/refactor

Make the code slightly more idiomatic; remove some "var" declarations,
remove some intermediate variables and redundant error-checks, and remove
the filePerm const.


### libnetwork/internal/kvstore/boltdb: BoltDB.Exists(): fix error handling

This function could potentially return "true" even if an error was returned.

### libnetwork/internal/kvstore/boltdb: BoltDB.Get(): don't shadow error

Don't shadow the original error if we got one.


### libnetwork/internal/kvstore/boltdb: BoltDB.List(): minor cleanup

cleanup the code to be slightly more idiomatic

### libnetwork/internal/kvstore/boltdb: un-export Mutex

Keep the mutex internal to BoltDB.

### libnetwork/datastore: remove unused datastore.Active()

The value was set, and updated, but never used.

### libnetwork/datastore: remove redundant datastore.sequential

The sequential field determined whether a lock was needed when storing
and retrieving data. This field was always set to true, with the exception
of NewTestDataStore() in the tests.

This field was added in https://github.com/moby/libnetwork/commit/a18e2f9965b0d772ed2b552bb4e26ea43bf90c95 (https://github.com/moby/libnetwork/pull/1206)
to make locking optional for non-local scoped stores. Such stores are no
longer used, so we can remove this field.

### libnetwork/datastore: remove Watch(), Watchable(), RestartWatch()

The `store.Watch()` was only used in `Controller.processEndpointCreate()`,
and skipped if the store was not "watchable" (`store.Watchable()`).

Whether a store is watchable depends on the store's datastore.scope;
local stores are not watchable;

    func (ds *datastore) Watchable() bool {
        return ds.scope != LocalScope
    }

datastore is only initialized in two locations, and both locations set the
scope field to LocalScope:

datastore.newClient() (also called by datastore.NewDataStore()):
https://github.com/moby/moby/blob/3e4c9d90cf6a796334519f0de12ff09608d7abbe/libnetwork/datastore/datastore.go#L213

datastore.NewTestDataStore() (used in tests);
https://github.com/moby/moby/blob/3e4c9d90cf6a796334519f0de12ff09608d7abbe/libnetwork/datastore/datastore_test.go#L14-L17

Furthermore, the backing BoltDB kvstore does not implement the Watch()
method;

https://github.com/moby/moby/blob/3e4c9d90cf6a796334519f0de12ff09608d7abbe/libnetwork/internal/kvstore/boltdb/boltdb.go#L464-L467

Based on the above;

- our datastore is never Watchable()
- so datastore.Watch() is never used

This patch removes the Watchable(), Watch(), and RestartWatch() functions,
as well as the code handling watching.

### libnetwork/internal/kvstore: remove unused Watch() method

The BoltDB store is not Watchable, and the Watch function was never used,
so we can remove it.

### libnetwork/networkdb: NetworkDB.Watch(): remove unused "key" argument

This function was implemented in https://github.com/moby/libnetwork/commit/dd4950f36d0a568791ead9398ddd175451ae36fa
which added a "key" field, but that field was never used anywhere, and
still appears unused.

### libnetwork/datastore: cache.get(): remove unused "key" argument

### libnetwork: rename vars that shadowed with pkg vars and imports

### libnetwork/config: remove options that were only used in tests

The OptionLocalKVProvider, OptionLocalKVProviderURL, and OptionLocalKVProviderConfig
options were only used in tests, so un-export them, and move them to the
test-files.

### libnetwork: inline store config options

### libnetwork: TestBoltdbBackend(): use t.TempDir()

### libnetwork/datastore: remove unused DeleteTree() method


**- Description for the changelog**
<!--
Write a short (one line) summary that describes the changes in this
pull request for inclusion in the changelog:
-->


**- A picture of a cute animal (not mandatory but encouraged)**


libnetwork/internal/kvstore/boltdb/boltdb.go
package boltdb

import (
	"bytes"
	"encoding/binary"
	"errors"
	"os"
	"path/filepath"
	"sync"
	"sync/atomic"
	"time"

	store "github.com/docker/docker/libnetwork/internal/kvstore"
	bolt "go.etcd.io/bbolt"
)

var (
	// ErrMultipleEndpointsUnsupported is thrown when multiple endpoints specified for
	// BoltDB. Endpoint has to be a local file path
	ErrMultipleEndpointsUnsupported = errors.New("boltdb supports one endpoint and should be a file path")
	// ErrBoltBucketOptionMissing is thrown when boltBcuket config option is missing
	ErrBoltBucketOptionMissing = errors.New("boltBucket config option missing")
)

const (
	filePerm os.FileMode = 0644
)

// BoltDB type implements the Store interface
type BoltDB struct {
	client     *bolt.DB
	boltBucket []byte
	dbIndex    uint64
	path       string
	timeout    time.Duration
	// By default libkv opens and closes the bolt DB connection  for every
	// get/put operation. This allows multiple apps to use a Bolt DB at the
	// same time.
	// PersistConnection flag provides an option to override ths behavior.
	// ie: open the connection in New and use it till Close is called.
	PersistConnection bool
	sync.Mutex
}

const (
	libkvmetadatalen = 8
	transientTimeout = time.Duration(10) * time.Second
)

// Register registers boltdb to libkv
func Register() {
	store.AddStore(store.BOLTDB, New)
}

// New opens a new BoltDB connection to the specified path and bucket
func New(endpoints []string, options *store.Config) (store.Store, error) {
	var (
		db          *bolt.DB
		err         error
		boltOptions *bolt.Options
		timeout     = transientTimeout
	)

	if len(endpoints) > 1 {
		return nil, ErrMultipleEndpointsUnsupported
	}

	if (options == nil) || (len(options.Bucket) == 0) {
		return nil, ErrBoltBucketOptionMissing
	}

	dir, _ := filepath.Split(endpoints[0])
	if err = os.MkdirAll(dir, 0750); err != nil {
		return nil, err
	}

	if options.PersistConnection {
		boltOptions = &bolt.Options{Timeout: options.ConnectionTimeout}
		db, err = bolt.Open(endpoints[0], filePerm, boltOptions)
		if err != nil {
			return nil, err
		}
	}

	if options.ConnectionTimeout != 0 {
		timeout = options.ConnectionTimeout
	}

	b := &BoltDB{
		client:            db,
		path:              endpoints[0],
		boltBucket:        []byte(options.Bucket),
		timeout:           timeout,
		PersistConnection: options.PersistConnection,
	}

	return b, nil
}

func (b *BoltDB) reset() {
	b.path = ""
	b.boltBucket = []byte{}
}

func (b *BoltDB) getDBhandle() (*bolt.DB, error) {
	var (
		db  *bolt.DB
		err error
	)
	if !b.PersistConnection {
		boltOptions := &bolt.Options{Timeout: b.timeout}
		if db, err = bolt.Open(b.path, filePerm, boltOptions); err != nil {
			return nil, err
		}
		b.client = db
	}

	return b.client, nil
}

func (b *BoltDB) releaseDBhandle() {
	if !b.PersistConnection {
		b.client.Close()
	}
}

// Get the value at "key". BoltDB doesn't provide an inbuilt last modified index with every kv pair. Its implemented by
// by a atomic counter maintained by the libkv and appened to the value passed by the client.
func (b *BoltDB) Get(key string) (*store.KVPair, error) {
	var (
		val []byte
		db  *bolt.DB
		err error
	)
	b.Lock()
	defer b.Unlock()

	if db, err = b.getDBhandle(); err != nil {
		return nil, err
	}
	defer b.releaseDBhandle()

	err = db.View(func(tx *bolt.Tx) error {
		bucket := tx.Bucket(b.boltBucket)
		if bucket == nil {
			return store.ErrKeyNotFound
		}

		v := bucket.Get([]byte(key))
		val = make([]byte, len(v))
		copy(val, v)

		return nil
	})

	if len(val) == 0 {
		return nil, store.ErrKeyNotFound
	}
	if err != nil {
		return nil, err
	}

	dbIndex := binary.LittleEndian.Uint64(val[:libkvmetadatalen])
	val = val[libkvmetadatalen:]

	return &store.KVPair{Key: key, Value: val, LastIndex: (dbIndex)}, nil
}

// Put the key, value pair. index number metadata is prepended to the value
func (b *BoltDB) Put(key string, value []byte, opts *store.WriteOptions) error {
	var (
		dbIndex uint64
		db      *bolt.DB
		err     error
	)
	b.Lock()
	defer b.Unlock()

	dbval := make([]byte, libkvmetadatalen)

	if db, err = b.getDBhandle(); err != nil {
		return err
	}
	defer b.releaseDBhandle()

	err = db.Update(func(tx *bolt.Tx) error {
		bucket, err := tx.CreateBucketIfNotExists(b.boltBucket)
		if err != nil {
			return err
		}

		dbIndex = atomic.AddUint64(&b.dbIndex, 1)
		binary.LittleEndian.PutUint64(dbval, dbIndex)
		dbval = append(dbval, value...)

		err = bucket.Put([]byte(key), dbval)
		if err != nil {
			return err
		}
		return nil
	})
	return err
}

// Delete the value for the given key.
func (b *BoltDB) Delete(key string) error {
	var (
		db  *bolt.DB
		err error
	)
	b.Lock()
	defer b.Unlock()

	if db, err = b.getDBhandle(); err != nil {
		return err
	}
	defer b.releaseDBhandle()

	err = db.Update(func(tx *bolt.Tx) error {
		bucket := tx.Bucket(b.boltBucket)
		if bucket == nil {
			return store.ErrKeyNotFound
		}
		err := bucket.Delete([]byte(key))
		return err
	})
	return err
}

// Exists checks if the key exists inside the store
func (b *BoltDB) Exists(key string) (bool, error) {
	var (
		val []byte
		db  *bolt.DB
		err error
	)
	b.Lock()
	defer b.Unlock()

	if db, err = b.getDBhandle(); err != nil {
		return false, err
	}
	defer b.releaseDBhandle()

	err = db.View(func(tx *bolt.Tx) error {
		bucket := tx.Bucket(b.boltBucket)
		if bucket == nil {
			return store.ErrKeyNotFound
		}

		val = bucket.Get([]byte(key))

		return nil
	})

	if len(val) == 0 {
		return false, err
	}
	return true, err
}

// List returns the range of keys starting with the passed in prefix
func (b *BoltDB) List(keyPrefix string) ([]*store.KVPair, error) {
	var (
		db  *bolt.DB
		err error
	)
	b.Lock()
	defer b.Unlock()

	kv := []*store.KVPair{}

	if db, err = b.getDBhandle(); err != nil {
		return nil, err
	}
	defer b.releaseDBhandle()

	err = db.View(func(tx *bolt.Tx) error {
		bucket := tx.Bucket(b.boltBucket)
		if bucket == nil {
			return store.ErrKeyNotFound
		}

		cursor := bucket.Cursor()
		prefix := []byte(keyPrefix)

		for key, v := cursor.Seek(prefix); bytes.HasPrefix(key, prefix); key, v = cursor.Next() {

			dbIndex := binary.LittleEndian.Uint64(v[:libkvmetadatalen])
			v = v[libkvmetadatalen:]
			val := make([]byte, len(v))
			copy(val, v)

			kv = append(kv, &store.KVPair{
				Key:       string(key),
				Value:     val,
				LastIndex: dbIndex,
			})
		}
		return nil
	})
	if len(kv) == 0 {
		return nil, store.ErrKeyNotFound
	}
	return kv, err
}

// AtomicDelete deletes a value at "key" if the key
// has not been modified in the meantime, throws an
// error if this is the case
func (b *BoltDB) AtomicDelete(key string, previous *store.KVPair) (bool, error) {
	var (
		val []byte
		db  *bolt.DB
		err error
	)
	b.Lock()
	defer b.Unlock()

	if previous == nil {
		return false, store.ErrPreviousNotSpecified
	}
	if db, err = b.getDBhandle(); err != nil {
		return false, err
	}
	defer b.releaseDBhandle()

	err = db.Update(func(tx *bolt.Tx) error {
		bucket := tx.Bucket(b.boltBucket)
		if bucket == nil {
			return store.ErrKeyNotFound
		}

		val = bucket.Get([]byte(key))
		if val == nil {
			return store.ErrKeyNotFound
		}
		dbIndex := binary.LittleEndian.Uint64(val[:libkvmetadatalen])
		if dbIndex != previous.LastIndex {
			return store.ErrKeyModified
		}
		err := bucket.Delete([]byte(key))
		return err
	})
	if err != nil {
		return false, err
	}
	return true, err
}

// AtomicPut puts a value at "key" if the key has not been
// modified since the last Put, throws an error if this is the case
func (b *BoltDB) AtomicPut(key string, value []byte, previous *store.KVPair, options *store.WriteOptions) (bool, *store.KVPair, error) {
	var (
		val     []byte
		dbIndex uint64
		db      *bolt.DB
		err     error
	)
	b.Lock()
	defer b.Unlock()

	dbval := make([]byte, libkvmetadatalen)

	if db, err = b.getDBhandle(); err != nil {
		return false, nil, err
	}
	defer b.releaseDBhandle()

	err = db.Update(func(tx *bolt.Tx) error {
		var err error
		bucket := tx.Bucket(b.boltBucket)
		if bucket == nil {
			if previous != nil {
				return store.ErrKeyNotFound
			}
			bucket, err = tx.CreateBucket(b.boltBucket)
			if err != nil {
				return err
			}
		}
		// AtomicPut is equivalent to Put if previous is nil and the Ky
		// doesn't exist in the DB.
		val = bucket.Get([]byte(key))
		if previous == nil && len(val) != 0 {
			return store.ErrKeyExists
		}
		if previous != nil {
			if len(val) == 0 {
				return store.ErrKeyNotFound
			}
			dbIndex = binary.LittleEndian.Uint64(val[:libkvmetadatalen])
			if dbIndex != previous.LastIndex {
				return store.ErrKeyModified
			}
		}
		dbIndex = atomic.AddUint64(&b.dbIndex, 1)
		binary.LittleEndian.PutUint64(dbval, b.dbIndex)
		dbval = append(dbval, value...)
		return (bucket.Put([]byte(key), dbval))
	})
	if err != nil {
		return false, nil, err
	}

	updated := &store.KVPair{
		Key:       key,
		Value:     value,
		LastIndex: dbIndex,
	}

	return true, updated, nil
}

// Close the db connection to the BoltDB
func (b *BoltDB) Close() {
	b.Lock()
	defer b.Unlock()

	if !b.PersistConnection {
		b.reset()
	} else {
		b.client.Close()
	}
}

// DeleteTree deletes a range of keys with a given prefix
func (b *BoltDB) DeleteTree(keyPrefix string) error {
	var (
		db  *bolt.DB
		err error
	)
	b.Lock()
	defer b.Unlock()

	if db, err = b.getDBhandle(); err != nil {
		return err
	}
	defer b.releaseDBhandle()

	err = db.Update(func(tx *bolt.Tx) error {
		bucket := tx.Bucket(b.boltBucket)
		if bucket == nil {
			return store.ErrKeyNotFound
		}

		cursor := bucket.Cursor()
		prefix := []byte(keyPrefix)

		for key, _ := cursor.Seek(prefix); bytes.HasPrefix(key, prefix); key, _ = cursor.Next() {
			_ = bucket.Delete(key)
		}
		return nil
	})

	return err
}

// NewLock has to implemented at the library level since its not supported by BoltDB
func (b *BoltDB) NewLock(key string, options *store.LockOptions) (store.Locker, error) {
	return nil, store.ErrCallNotSupported
}

// Watch has to implemented at the library level since its not supported by BoltDB
func (b *BoltDB) Watch(key string, stopCh <-chan struct{}) (<-chan *store.KVPair, error) {
	return nil, store.ErrCallNotSupported
}

// WatchTree has to implemented at the library level since its not supported by BoltDB
func (b *BoltDB) WatchTree(directory string, stopCh <-chan struct{}) (<-chan []*store.KVPair, error) {
	return nil, store.ErrCallNotSupported
}

package boltdb

import (
	"bytes"
	"encoding/binary"
	"errors"
	"os"
	"path/filepath"
	"sync"
	"sync/atomic"
	"time"

	store "github.com/docker/docker/libnetwork/internal/kvstore"
	bolt "go.etcd.io/bbolt"
)

var (
	// ErrMultipleEndpointsUnsupported is thrown when multiple endpoints specified for
	// BoltDB. Endpoint has to be a local file path
	ErrMultipleEndpointsUnsupported = errors.New("boltdb supports one endpoint and should be a file path")
	// ErrBoltBucketOptionMissing is thrown when boltBcuket config option is missing
	ErrBoltBucketOptionMissing = errors.New("boltBucket config option missing")
)

const filePerm = 0o644

// BoltDB type implements the Store interface
type BoltDB struct {
	mu         sync.Mutex
	client     *bolt.DB
	boltBucket []byte
	dbIndex    uint64
	path       string
	timeout    time.Duration
	// By default libkv opens and closes the bolt DB connection  for every
	// get/put operation. This allows multiple apps to use a Bolt DB at the
	// same time.
	// PersistConnection flag provides an option to override ths behavior.
	// ie: open the connection in New and use it till Close is called.
	PersistConnection bool
}

const (
	libkvmetadatalen = 8
	transientTimeout = time.Duration(10) * time.Second
)

// Register registers boltdb to libkv
func Register() {
	store.AddStore(store.BOLTDB, New)
}

// New opens a new BoltDB connection to the specified path and bucket
func New(endpoints []string, options *store.Config) (store.Store, error) {
	if len(endpoints) > 1 {
		return nil, ErrMultipleEndpointsUnsupported
	}

	if (options == nil) || (len(options.Bucket) == 0) {
		return nil, ErrBoltBucketOptionMissing
	}

	dir, _ := filepath.Split(endpoints[0])
	if err := os.MkdirAll(dir, 0o750); err != nil {
		return nil, err
	}

	var db *bolt.DB
	if options.PersistConnection {
		var err error
		db, err = bolt.Open(endpoints[0], filePerm, &bolt.Options{
			Timeout: options.ConnectionTimeout,
		})
		if err != nil {
			return nil, err
		}
	}

	timeout := transientTimeout
	if options.ConnectionTimeout != 0 {
		timeout = options.ConnectionTimeout
	}

	b := &BoltDB{
		client:            db,
		path:              endpoints[0],
		boltBucket:        []byte(options.Bucket),
		timeout:           timeout,
		PersistConnection: options.PersistConnection,
	}

	return b, nil
}

func (b *BoltDB) reset() {
	b.path = ""
	b.boltBucket = []byte{}
}

func (b *BoltDB) getDBhandle() (*bolt.DB, error) {
	if !b.PersistConnection {
		db, err := bolt.Open(b.path, filePerm, &bolt.Options{Timeout: b.timeout})
		if err != nil {
			return nil, err
		}
		b.client = db
	}
	return b.client, nil
}

func (b *BoltDB) releaseDBhandle() {
	if !b.PersistConnection {
		b.client.Close()
	}
}

// Get the value at "key". BoltDB doesn't provide an inbuilt last modified index with every kv pair. Its implemented by
// by a atomic counter maintained by the libkv and appened to the value passed by the client.
func (b *BoltDB) Get(key string) (*store.KVPair, error) {
	b.mu.Lock()
	defer b.mu.Unlock()

	db, err := b.getDBhandle()
	if err != nil {
		return nil, err
	}
	defer b.releaseDBhandle()

	var val []byte
	err = db.View(func(tx *bolt.Tx) error {
		bucket := tx.Bucket(b.boltBucket)
		if bucket == nil {
			return store.ErrKeyNotFound
		}

		v := bucket.Get([]byte(key))
		val = make([]byte, len(v))
		copy(val, v)

		return nil
	})
	if err != nil {
		return nil, err
	}
	if len(val) == 0 {
		return nil, store.ErrKeyNotFound
	}

	dbIndex := binary.LittleEndian.Uint64(val[:libkvmetadatalen])
	val = val[libkvmetadatalen:]

	return &store.KVPair{Key: key, Value: val, LastIndex: dbIndex}, nil
}

// Put the key, value pair. index number metadata is prepended to the value
func (b *BoltDB) Put(key string, value []byte) error {
	b.mu.Lock()
	defer b.mu.Unlock()

	db, err := b.getDBhandle()
	if err != nil {
		return err
	}
	defer b.releaseDBhandle()

	return db.Update(func(tx *bolt.Tx) error {
		bucket, err := tx.CreateBucketIfNotExists(b.boltBucket)
		if err != nil {
			return err
		}

		dbIndex := atomic.AddUint64(&b.dbIndex, 1)
		dbval := make([]byte, libkvmetadatalen)
		binary.LittleEndian.PutUint64(dbval, dbIndex)
		dbval = append(dbval, value...)

		return bucket.Put([]byte(key), dbval)
	})
}

// Exists checks if the key exists inside the store
func (b *BoltDB) Exists(key string) (bool, error) {
	b.mu.Lock()
	defer b.mu.Unlock()

	db, err := b.getDBhandle()
	if err != nil {
		return false, err
	}
	defer b.releaseDBhandle()

	var exists bool
	err = db.View(func(tx *bolt.Tx) error {
		bucket := tx.Bucket(b.boltBucket)
		if bucket == nil {
			return store.ErrKeyNotFound
		}

		exists = len(bucket.Get([]byte(key))) > 0
		return nil
	})
	if err != nil {
		return false, err
	}
	if !exists {
		return false, store.ErrKeyNotFound
	}
	return true, nil
}

// List returns the range of keys starting with the passed in prefix
func (b *BoltDB) List(keyPrefix string) ([]*store.KVPair, error) {
	b.mu.Lock()
	defer b.mu.Unlock()

	db, err := b.getDBhandle()
	if err != nil {
		return nil, err
	}
	defer b.releaseDBhandle()

	var kv []*store.KVPair
	err = db.View(func(tx *bolt.Tx) error {
		bucket := tx.Bucket(b.boltBucket)
		if bucket == nil {
			return store.ErrKeyNotFound
		}

		cursor := bucket.Cursor()
		prefix := []byte(keyPrefix)

		for key, v := cursor.Seek(prefix); bytes.HasPrefix(key, prefix); key, v = cursor.Next() {
			dbIndex := binary.LittleEndian.Uint64(v[:libkvmetadatalen])
			v = v[libkvmetadatalen:]
			val := make([]byte, len(v))
			copy(val, v)

			kv = append(kv, &store.KVPair{
				Key:       string(key),
				Value:     val,
				LastIndex: dbIndex,
			})
		}
		return nil
	})
	if err != nil {
		return nil, err
	}
	if len(kv) == 0 {
		return nil, store.ErrKeyNotFound
	}
	return kv, nil
}

// AtomicDelete deletes a value at "key" if the key
// has not been modified in the meantime, throws an
// error if this is the case
func (b *BoltDB) AtomicDelete(key string, previous *store.KVPair) error {
	b.mu.Lock()
	defer b.mu.Unlock()

	if previous == nil {
		return store.ErrPreviousNotSpecified
	}
	db, err := b.getDBhandle()
	if err != nil {
		return err
	}
	defer b.releaseDBhandle()

	return db.Update(func(tx *bolt.Tx) error {
		bucket := tx.Bucket(b.boltBucket)
		if bucket == nil {
			return store.ErrKeyNotFound
		}

		val := bucket.Get([]byte(key))
		if val == nil {
			return store.ErrKeyNotFound
		}
		dbIndex := binary.LittleEndian.Uint64(val[:libkvmetadatalen])
		if dbIndex != previous.LastIndex {
			return store.ErrKeyModified
		}
		return bucket.Delete([]byte(key))
	})
}

// AtomicPut puts a value at "key" if the key has not been
// modified since the last Put, throws an error if this is the case
func (b *BoltDB) AtomicPut(key string, value []byte, previous *store.KVPair) (*store.KVPair, error) {
	b.mu.Lock()
	defer b.mu.Unlock()

	db, err := b.getDBhandle()
	if err != nil {
		return nil, err
	}
	defer b.releaseDBhandle()

	var dbIndex uint64
	dbval := make([]byte, libkvmetadatalen)
	err = db.Update(func(tx *bolt.Tx) error {
		bucket := tx.Bucket(b.boltBucket)
		if bucket == nil {
			if previous != nil {
				return store.ErrKeyNotFound
			}
			var err error
			bucket, err = tx.CreateBucket(b.boltBucket)
			if err != nil {
				return err
			}
		}
		// AtomicPut is equivalent to Put if previous is nil and the Ky
		// doesn't exist in the DB.
		val := bucket.Get([]byte(key))
		if previous == nil && len(val) != 0 {
			return store.ErrKeyExists
		}
		if previous != nil {
			if len(val) == 0 {
				return store.ErrKeyNotFound
			}
			dbIndex = binary.LittleEndian.Uint64(val[:libkvmetadatalen])
			if dbIndex != previous.LastIndex {
				return store.ErrKeyModified
			}
		}
		dbIndex = atomic.AddUint64(&b.dbIndex, 1)
		binary.LittleEndian.PutUint64(dbval, b.dbIndex)
		dbval = append(dbval, value...)
		return bucket.Put([]byte(key), dbval)
	})
	if err != nil {
		return nil, err
	}
	return &store.KVPair{Key: key, Value: value, LastIndex: dbIndex}, nil
}

// Close the db connection to the BoltDB
func (b *BoltDB) Close() {
	b.mu.Lock()
	defer b.mu.Unlock()

	if !b.PersistConnection {
		b.reset()
	} else {
		b.client.Close()
	}
}

bbolt has ACID semantics using lock-free MVCC so libkv serializing all operations is not needed by the underlying database. The mutex was added in https://github.com/docker/libkv/pull/81 to work around issues caused by https://github.com/docker/libkv/pull/79 which itself is a workaround for libnetwork opening the same database multiple times. libnetwork still does this today so the mutex can't go away until we either fix libnetwork or work around it the right way by making all calls to `boltdb.New` (or `datastore.NewDataStore`) with the same database file return a store which shares a single (reference-counted) bbolt client.
https://github.com/moby/moby/blob/3e4c9d90cf6a796334519f0de12ff09608d7abbe/libnetwork/controller.go#L353-L354
