<commit_before>---
authors: <NAME> <<EMAIL>>
state: draft
discussion: https://github.com/joyent/rfd/issues/130
---

<!--
    This Source Code Form is subject to the terms of the Mozilla Public
    License, v. 2.0. If a copy of the MPL was not distributed with this
    file, You can obtain one at http://mozilla.org/MPL/2.0/.
-->

<!--
    Copyright (c) 2019, Joyent, Inc.
-->

# RFD 168 Bootstrapping a Manta Buckets deployment

## Overview

This RFD covers the steps required to bootstrap a manta buckets deployment. This
includes specifying the buckets manatee shards, creating the initial hash ring
for the buckets system, and assigning virtual nodes to the buckets shards. It
also covers the changes to the current manta deployment scripts to enable the
creation of manta buckets shards that are distinct from other manta shard types
and the creation of a hash ring image that is distinct from the hash ring image
used by a standard manta deployment.

## Prerequisites

The manta buckets system is designed to be deployed in an existing manta
system. The manta `webapi` tier has been built to serve both standard manta
directory-based requests as well as manta buckets requests. From a
[SAPI](https://github.com/joyent/sdc-sapi) point-of-view the buckets system
appears to fit under the umbrella of a `manta` application rather than being a
completely separate application. Operating under this assumption there are a few necessary
changes in order to support the addition of the buckets system into the manta
application.

The first required change is to add a new shard type to manta. Shards are
currently managed with the
[`manta-shardadm`](https://github.com/joyent/sdc-manta/blob/master/cmd/manta-shardadm.js)
script. Currently, there are three shard types: `index`, `storage`, and `job` (a.k.a
`marlin`). The `index` shard type houses the object metadata for manta's
directory-based storage option. It is not possible to reuse the `index` shard
type for buckets metadata for a few reasons. First, the version of PostgreSQL being targeted for
buckets is not the same as the current default version used by the `index`
shards. Second, the number of metadata shards required for a buckets deployment
may need to evolve differently from that of directory-based manta. The proposal
is to use a new `buckets` shard type for the manta buckets metadata and change the
`manta-shardadm` script to be aware of this new type.

The next required change is to be able to generate a mapping of virtual nodes to
the `buckets` shards. This is done for production manta deployments using the [`manta-create-topology.sh`](https://github.com/joyent/sdc-manta/blob/master/bin/manta-create-topology.sh)
script. The proposal is to create a similar script for buckets,
`manta-create-buckets-topology.sh`. The script would have the same function, but
would hopefully reduce the chance of [harmful side effects](https://github.com/joyent/sdc-manta/blob/master/bin/manta-create-topology.sh#L61-L63)
that could result from adding a buckets option to the existing
`manta-create-topology.sh` script.

## Bootstrapping process

This section will operate under the assumption the work to meet the
prerequisites outlined in the previous section has been completed.

The first step is to use `manta-shardadm` to create the set of initial shards
for the buckets deployment. The invocation of `manta-shardadm` might look as
follows:

```
manta-shardadm set -b "1.boray 2.boray"
```

The next step is to generate a consistent hash ring that maps a set of virtual
nodes (vnodes) to the shards specified in the previous step. Use the
`manta-buckets-create-topology.sh` script for this. Provide the script with the
number of vnodes to initialize the hash ring with and it will generate a ring
image, upload it to the image API server, and then update the SAPI metadata for
manta with the image UUID under the `BUCKETS_HASH_RING_IMAGE` key as well as a
`BUCKETS_HASH_RING_DIGEST` key representing the SHA256 digest of the ring
file. This information will be used by the manta buckets component [`electric-boray`](https://github.com/joyent/electric-boray).

The composition of the resulting image generated by
`manta-buckets-create-topology.sh` will be different from that of the original
`manta-create-topology.sh` script. Whereas `manta-create-topology` uses
`node-fash` to create a leveldb database `manta-buckets-create-topology.sh` will
use the memory backend option provided by `node-fash` and serialize the
resulting ring to `STDOUT`. The serialized ring data will be further transformed
to remove unnecessary data and add other useful data (*e.g.* the
`vnodeToPnodeMap_` that is not directly exposed by `node-fash` when the ring is
serialized) and then captured in a file.

[`Electric-moray`](https://github.com/joyent/electric-moray) only pulls in the hash
ring image file when its zone is created and
configured. `Electric-boray` in contrast will check with SAPI to verify it is using the correct image every time
the service starts. `Electric-boray` will attempt to refresh its hash ring data
if the image id reported by SAPI is different **and** the SAPI value for
`BUCKETS_HASH_RING_UPDATE` is set to `true` or if the SHA256 digest of the
local ring file does not match the recorded value of the ring file
digest. Eventually it may be the case that `electric-boray` queries SAPI about
the ring information while running as part of a way to enable ring changes
without service restarts or degradation, but that will not be the case
initially. `Electric-boray` loads the ring data into memory when it starts and
does not interact with the ring file data afterwards.

Once these steps are completed then the full set of manta buckets zones may be
deployed. This set include `electric-boray` service zones and `boray` service
zones. Additionally, the `webapi` service zones must be updated to a version
that has support for manta buckets.<commit_msg>Update RFD 168 based on feedback<commit_after>---
authors: <NAME> <<EMAIL>>
state: draft
discussion: https://github.com/joyent/rfd/issues/130
---

<!--
    This Source Code Form is subject to the terms of the Mozilla Public
    License, v. 2.0. If a copy of the MPL was not distributed with this
    file, You can obtain one at http://mozilla.org/MPL/2.0/.
-->

<!--
    Copyright (c) 2019, Joyent, Inc.
-->

# RFD 168 Bootstrapping a Manta Buckets deployment

## Overview

This RFD covers the steps required to bootstrap a manta buckets deployment. This
includes specifying the buckets manatee shards, creating the initial hash ring
for the buckets system, and assigning virtual nodes to the buckets shards. It
also covers the changes to the current manta deployment scripts to enable the
creation of manta buckets shards that are distinct from other manta shard types
and the creation of a hash ring image that is distinct from the hash ring image
used by a standard manta deployment.

## Prerequisites

The manta buckets system is designed to be deployed in an existing manta
system. The manta `webapi` tier has been built to serve both standard manta
directory-based requests as well as manta buckets requests. From a
[SAPI](https://github.com/joyent/sdc-sapi) point-of-view the buckets system
appears to fit under the umbrella of a `manta` application rather than being a
completely separate application. Operating under this assumption there are a few necessary
changes in order to support the addition of the buckets system into the manta
application.

The first required change is to add a new shard type to manta. Shards are
currently managed with the
[`manta-shardadm`](https://github.com/joyent/sdc-manta/blob/master/cmd/manta-shardadm.js)
script. Currently, there are three shard types: `index`, `storage`, and `job` (a.k.a
`marlin`). The `index` shard type houses the object metadata for manta's
directory-based storage option. It is not possible to reuse the `index` shard
type for buckets metadata for a few reasons. First, the version of PostgreSQL being targeted for
buckets is not the same as the current default version used by the `index`
shards. Second, the number of metadata shards required for a buckets deployment
may need to evolve differently from that of directory-based manta. The proposal
is to use a new `buckets` shard type for the manta buckets metadata and change the
`manta-shardadm` script to be aware of this new type.

The next required change is to be able to generate a mapping of virtual nodes to
the `buckets` shards. This is done for production manta deployments using the [`manta-create-topology.sh`](https://github.com/joyent/sdc-manta/blob/master/bin/manta-create-topology.sh)
script. The proposal is to create a new program that provides the functionality
of `manta-create-toplogy.sh` and also provides the ability generate the mapping
of virtual nodes needed for `buckets` shards. The `manta-create-toplogy.sh`
script will be removed.

## Bootstrapping process

This section will operate under the assumption the work to meet the
prerequisites outlined in the previous section has been completed.

It is also assumed that the shards being specified as part of the manta buckets
system have been created. A new `buckets-postgres` sharded service is being
added to manta that allows for buckets shards to be created using the
`manta-adm` tooling.

The first step is to use `manta-shardadm` to create the set of initial shards
for the buckets deployment. The invocation of `manta-shardadm` might look as
follows:

```
manta-shardadm set -b "1.boray 2.boray"
```

Aside from adding the shard information to the SAPI manta application data this
command should also attempt a simple sanity check on each shard. This would
include verifying that each specified shard exists and that each one is an instance of
the `buckets-postgres` service.

The next step is to generate a consistent hash ring that maps a set of virtual
nodes (vnodes) to the shards specified in the previous step. The program that
supersedes `manta-create-topology.sh` script should be used for this. Provide
the script with the number of vnodes to initialize the hash ring with and it
will generate a ring image, upload it to the image API server, and then update
the SAPI metadata for manta with the image UUID under the
`BUCKETS_HASH_RING_IMAGE` key as well as a `BUCKETS_HASH_RING_DIGEST` key
representing the SHA256 digest of the ring file. This information will be used
by the manta buckets component
[`electric-boray`](https://github.com/joyent/electric-boray).

The composition of the image for a buckets deployment will be
different from the images created for the normal manta system. Whereas
`manta-create-topology` uses `node-fash` to create a leveldb database, the hash
ring for the buckets system will be generated using the memory backend option
provided by `node-fash`. The ring data will be serialized and then further
transformed to remove unnecessary data and add other useful data (*e.g.* the
`vnodeToPnodeMap_` that is not directly exposed by `node-fash` when the ring is
serialized) and then captured in a file. This file will be captured in the image
that is uploaded and later retrieved by electric-boray.

Once these steps are completed then the full set of manta buckets zones may be
deployed and used. This set include `electric-boray` service zones and `boray`
service zones. Additionally, the `webapi` service zones must be updated to a
version that has support for manta buckets.
{'fraction_non_alphanumeric': 0.057569668908884424, 'fraction_numerical': 0.004193238403075041, 'mean_word_length': 4.48538572113081, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 12, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6018355', 'n_tokens_mistral': 3223, 'n_tokens_neox': 3046, 'n_words': 1672}