<commit_before>## Messaging

### Background

The messaging layer takes care of the direct peer to peer transfer of segment
batches, acks, segment completion and segment retries to the relevant virtual
peers. The messaging layer design document can be found at
[design/messaging.md](../design/messaging.md).

### Messaging Implementations

The Onyx messaging implementation is pluggable and alternative implementations
can be selected via the `:onyx.messaging/impl` [peer-config](peer-config.md#onyxmessagingimpl).

#### Aeron Messaging

Owing to [Aeron's](https://github.com/real-logic/Aeron) high throughput and low
latency, Aeron is the default Onyx messaging implementation. There are a few
relevant considerations when using the Aeron implementation.

##### Subscription (Connection) Multiplexing

One issue when scaling Onyx to a many node cluster, is that every virtual peer
may require a communications channel to any other virtual peer. As a result, a
naive implementation will require up to m<sup>2</sup> connections over the
cluster, where m is the number of virtual peers. By sharing Aeron subscribers
between virtual peers on a node, this can be reduced to n<sup>2</sup>
connections, where n is the number of nodes. This reduces the amount of
overhead required to maintain connections between peers, allowing the
cluster to scale better as the number of nodes to increase.

It is worth noting that Aeron subscribers (receivers) must also generally
perform deserialization.  Therefore, subscribers may become CPU bound by the
amount of deserializaton work that needs to be performed. In order to reduce
this effect, we allow multiple subscribers to be created per node (i.e. peer
group). This can be tuned via `:onyx.messaging.aeron/subscriber-count` in
[peer-config](peer-config.md#onyxmessagingaeronsubscriber-count). As increasing the number of subscribers, may
lead back to an undesirable growth in the number of connections between nodes,
each node will only choose one subscription to communicate through. These are
chosen via a hash of the combined IPs of the nodes, in order to consistently
spread the use of subscribers over the cluster.

Clusters which perform a large proportion of the time serializing should
consider increasing the subscriber count. As a general guide, `# cores = #
virtual peers + # subscribers`.

##### Connection Short Circuiting

When virtual peers are co-located on the same node, messaging will bypass the
use of Aeron and directly communicate the message without any use of the
network and without any serialization. Therefore, performance benchmarks<commit_msg>Better, clearer description.<commit_after>## Messaging

### Background

The messaging layer takes care of the direct peer to peer transfer of segment
batches, acks, segment completion and segment retries to the relevant virtual
peers. The messaging layer design document can be found at
[design/messaging.md](../design/messaging.md).

### Messaging Implementations

The Onyx messaging implementation is pluggable and alternative implementations
can be selected via the `:onyx.messaging/impl` [peer-config](peer-config.md#onyxmessagingimpl).

#### Aeron Messaging

Owing to [Aeron's](https://github.com/real-logic/Aeron) high throughput and low
latency, Aeron is the default Onyx messaging implementation. There are a few
relevant considerations when using the Aeron implementation.

##### Subscription (Connection) Multiplexing

One issue when scaling Onyx to a many node cluster, is that every virtual peer
may require a communications channel to any other virtual peer. As a result, a
naive implementation will require up to m<sup>2</sup> connections over the
cluster, where m is the number of virtual peers. By sharing Aeron subscribers
between virtual peers on a node, this can be reduced to n<sup>2</sup>
connections, where n is the number of nodes. This reduces the amount of
overhead required to maintain connections between peers, allowing the
cluster to scale better as the number of nodes to increase.

It is worth noting that Aeron subscribers (receivers) must also generally
perform deserialization.  Therefore, subscribers may become CPU bound by the
amount of deserializaton work that needs to be performed. In order to reduce
this effect, multiple subscribers can be instantiated per node.  This can be
tuned via `:onyx.messaging.aeron/subscriber-count` in
[peer-config](peer-config.md#onyxmessagingaeronsubscriber-count). As increasing
the number of subscribers may lead back to an undesirable growth in the number
of connections between nodes, each node will only choose one subscription to
communicate through. The choice of subscriber is calculated via a hash of the combined IPs of the
communicating nodes, in order to consistently spread the use of subscribers over the cluster.

Clusters which perform a large proportion of the time serializing should
consider increasing the subscriber count. As a general guide, `# cores = #
virtual peers + # subscribers`.

##### Connection Short Circuiting

When virtual peers are co-located on the same node, messaging will bypass the
use of Aeron and directly communicate the message without any use of the
network and without any serialization. Therefore, performance benchmarks
{'fraction_non_alphanumeric': 0.052, 'fraction_numerical': 0.0007619047619047619, 'mean_word_length': 5.12004662004662, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1975906', 'n_tokens_mistral': 1337, 'n_tokens_neox': 1235, 'n_words': 714}