<commit_before># Raft implementation

SwarmKit uses the Raft consensus protocol to synchronize state between manager
nodes and support high availability. The lowest level portions of this are
provided by the `github.com/coreos/etcd/raft` package. SwarmKit's
`github.com/docker/swarmkit/manager/state/raft` package builds a complete
solution on top of this, adding things like saving and loading state on disk,
an RPC layer so nodes can pass Raft messages over a network, and dynamic cluster
membership.

## A quick review of Raft

The details of the Raft protocol are outside the scope of this document, but
it's well worth reviewing the [raft paper](https://raft.github.io/raft.pdf).

Essentially, Raft gives us two things. It provides the mechanism to elect a
leader, which serves as the arbiter or all consensus decisions. It also provides
a distributed log that we can append entries to, subject to the leader's
approval. The distributed log is the basic building block for agreeing on and
distributing state. Once an entry in the log becomes *committed*, it becomes an
immutable part of the log that will survive any future leader elections and
changes to the cluster. We can think of a committed log entry as piece of state
that the cluster has reached agreement on.

## Role of the leader

The leader has special responsibilities in the Raft protocol, but we also assign
it special functions in SwarmKit outside the context of Raft. For example, the
scheduler, orchestrators, dispatcher, and CA run on the leader node. This is not
a design requirement, but simplifies things somewhat. If these components ran in
a distributed fashion, we would need some mechanism to resolve conflicts between
writes made by different nodes. Limiting decision-making to the leader avoids
the need for this, since we can be certain that there is at most one leader at
any time. The leader is also guaranteed to have the most up-to-date data in its
store, so it is best positioned to make decisions.

The basic rule is that anything which writes to the Raft-backed data store needs
to run on the leader. If a follower node tries to write to the data store, the
write will fail. Writes will also fail on a node that starts out as the leader
but loses its leadership position before the write finishes.

## Raft IDs vs. node IDs

Nodes in SwarmKit are identified by alphanumeric strings, but `etcd/raft` uses
integers to identify Raft nodes. Thus, managers have two distinct IDs. The Raft
IDs are assigned dynamically when a node joins the Raft consensus group. A node
could potentially leave the Raft consensus group (through demotion), then later
get promoted and rejoin under a different Raft ID. In this case, the node ID
would stay the same, because it's a cryptographically-verifiable property of the
node's certificate, but the Raft ID is assigned arbitrarily and would change.

It's important to note that a Raft ID can't be reused after a node that was
using the ID leaves the consensus group. These Raft IDs of nodes that are no
longer part of the cluster are saved in a list to make sure they aren't reused.
If a node with a Raft ID on this list tries to use Raft RPCs, other nodes won't
honor these requests.

## Logs and snapshots

There are two sets of files on disk that provide persistent state for Raft.
There is a set of WAL (write-ahead log files). These store a series of log
entries and Raft metadata, such as the current term, index, and committed index.
WAL files are automatically rotated when they reach a certain size.

To avoid having to retain every entry in the history of the log, snapshots
serialize a view of the state at a particular point in time. After a snapshot
gets taken, logs that predate the snapshot are no longer necessary, because the
snapshot captures all the information that's needed from the log up to that
point. The number of old snapshots and WALs to retain is configurable.

In SwarmKit's usage, WALs mostly contain protobuf-serialized data store<commit_msg>Updated raft design. (#2353)<commit_after># Raft implementation

SwarmKit uses the Raft consensus protocol to synchronize state between manager
nodes and support high availability. The lowest level portions of this are
provided by the `github.com/coreos/etcd/raft` package. SwarmKit's
`github.com/docker/swarmkit/manager/state/raft` package builds a complete
solution on top of this, adding things like saving and loading state on disk,
an RPC layer so nodes can pass Raft messages over a network, and dynamic cluster
membership.

## A quick review of Raft

The details of the Raft protocol are outside the scope of this document, but
it's well worth reviewing the [raft paper](https://raft.github.io/raft.pdf).

Essentially, Raft gives us two things. It provides the mechanism to elect a
leader, which serves as the arbiter or all consensus decisions. It also provides
a distributed log that we can append entries to, subject to the leader's
approval. The distributed log is the basic building block for agreeing on and
distributing state. Once an entry in the log becomes *committed*, it becomes an
immutable part of the log that will survive any future leader elections and
changes to the cluster. We can think of a committed log entry as piece of state
that the cluster has reached agreement on.

## Role of the leader

The leader has special responsibilities in the Raft protocol, but we also assign
it special functions in SwarmKit outside the context of Raft. For example, the
scheduler, orchestrators, dispatcher, and CA run on the leader node. This is not
a design requirement, but simplifies things somewhat. If these components ran in
a distributed fashion, we would need some mechanism to resolve conflicts between
writes made by different nodes. Limiting decision-making to the leader avoids
the need for this, since we can be certain that there is at most one leader at
any time. The leader is also guaranteed to have the most up-to-date data in its
store, so it is best positioned to make decisions.

The basic rule is that anything which writes to the Raft-backed data store needs
to run on the leader. If a follower node tries to write to the data store, the
write will fail. Writes will also fail on a node that starts out as the leader
but loses its leadership position before the write finishes.

## Raft IDs vs. node IDs

Nodes in SwarmKit are identified by alphanumeric strings, but `etcd/raft` uses
integers to identify Raft nodes. Thus, managers have two distinct IDs. The Raft
IDs are assigned dynamically when a node joins the Raft consensus group. A node
could potentially leave the Raft consensus group (through demotion), then later
get promoted and rejoin under a different Raft ID. In this case, the node ID
would stay the same, because it's a cryptographically-verifiable property of the
node's certificate, but the Raft ID is assigned arbitrarily and would change.

It's important to note that a Raft ID can't be reused after a node that was
using the ID leaves the consensus group. These Raft IDs of nodes that are no
longer part of the cluster are saved (persisted on disk) in a list (a blacklist,
if you will) to make sure they aren't reused. If a node with a Raft ID on this list
tries to use Raft RPCs, other nodes won't honor these requests. etcd/raft doesn't allow
reuse of raft Id, which is likely done to avoid ambiguity.

The blacklist of demoted/removed nodes is used to restrict these nodes from
communicating and affecting cluster state. A membership list is also persisted,
however this does not restrict communication between nodes.
This is done to favor stability (and availability, by enabling faster return to
non-degraded state) over consistency, by allowing newly added nodes (which may not
have propagated to all the raft group members) to join and communicate with the group
even though the membership list may not consistent at the point in time (but eventually
will be). In case of node demotion/removal from the group, the affected node may be able
to communicate with the other members until the change is fully propagated.

## Logs and snapshots

There are two sets of files on disk that provide persistent state for Raft.
There is a set of WAL (write-ahead log files). These store a series of log
entries and Raft metadata, such as the current term, index, and committed index.
WAL files are automatically rotated when they reach a certain size.

To avoid having to retain every entry in the history of the log, snapshots
serialize a view of the state at a particular point in time. After a snapshot
gets taken, logs that predate the snapshot are no longer necessary, because the
snapshot captures all the information that's needed from the log up to that
point. The number of old snapshots and WALs to retain is configurable.

In SwarmKit's usage, WALs mostly contain protobuf-serialized data store
{'fraction_non_alphanumeric': 0.03297199638663053, 'fraction_numerical': 0.00045167118337850043, 'mean_word_length': 4.450461538461538, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5124259', 'n_tokens_mistral': 2205, 'n_tokens_neox': 2117, 'n_words': 1460}