<commit_before>Protocol specification
======================

The protocol uses two ports, the control port 29192 and the data port 29292.
The data port is used for uploading/downloading large files (pictures,
videos, etc).  Appart from the initial exchange, everything is encrypted using
sodium's crypto_box.

All messages are prepended by a u16 unencrypted size field.  This size is
offset by 1 (as in, a value of 5 would mean that the message is 6 bytes long).
The size excludes the size of the authentication tag (crypto_box_MACBYTES).
Note that the translation from encoded size to actual size is uniform, so you
must add crypto_box_box_MACBYTES even if the message is not encrypted for the
control channel, and the size of the normal header for the data channel. For
messages on the data channel, the size also excludes the size of the header.
Messages on the control channel are capped to a maximum size of 1024 (so
encoded as 1023).

Encryption Negotiation
----------------------

    struct ClientHel {
        u16 version;
        u64 nonce1;
        u8 public_key[32];
    };

The ClientHel message is the first thing that the client must send on creating
the connection.

The version field contains the current version of the protocol. This version is
version 0.

Nonces are random 8 byte values. The final computed nonce for a message is
nonce1 followed by nonce2 followed by a message counter, where the client uses
even numbers and the server uses odd numbers. e.g. The third message the client
has sent this session has a counter of 5.

The client transmits it's public key. This serves not only as a means of
encryption, it also identifies the user. The client is expected to already know
the server's public key.

    struct ServerHel {
        u8 error = 0;
        u64 nonce2;
        // begin encryption
        u8 session_token[32];
        u32 color;
        u8 avatar_sha256[32];
        u8 name[];
    };

The ServerHel is sent when the server receives a ClientHel, and the server has
accepted the connection. Now that the client has access to the other half of
the nonce, it can do encryption, and so the session token is encrypted. The
session token is required to open a data connection later. All further
communication on this channel is encrypted. The name and color specify the
saved values for the user's preferences. Additionally, the SHA256 of the avatar
file is provided so that the client can decide to request it from the server if
it is changed.

    struct ErrorBadVersion {
        u8 error = 1;
        u16 min_version;
        u16 max_version;
    };

If the provided version is not acceptable, ErrorBadVersion is sent instead of
ServerHel, providing the permitted range of versions. The min and max are
inclusive. The server disconnects immediatly after sending this error message
(and indeed after all error messages).

     struct ErrorUnknownUser {
         u8 error = 2;
     };

If the server does not know of the user, this is sent, and as mentioned above
the server disconnects.

Additionally, the server may disconnect immediately on a protocol violation
without sending a message.

    struct ClientDataHello {
        u8 session_token[32];
    };

At this point the client must open a data connection and send the
ClientDataHello. If the session token is invalid, the server closes the
connection. Otherwise it transmits a single zero byte as an ACK. All further
communication on the data channel is encrypted.

    struct ClientLo {
        u8 peer_public_key[32];
    };

Once the data connection is set up, the second part of the hello is sent. The
peer key indicates who the client wants to talk to.

    struct ServerLo {
        u8 error = 0;
        u32 color;
        u8 avatar_sha256[32];
        u8 name[];
    };

Once the server is ready, it sends the settings for the peer.

     struct ErrorUnknownPeer {
         u8 error = 1;
     };

If the server does not know the peer, it sends an error and disconnects.

Message Type Overview
---------------------

Type Client      Server
---- ------      ------
   0             Ack
   1             DAck
   2 TextMessage TextMessage
   3 PartMessage PartMessage
   4 RequestPart
   5 Replay
   6 Exclusive   Exclusive
   7 Seen        Seen
   8 Typing      Typing
   9 Name        Name
  10 Color       Color
  11 Avatar      Avatar

Sending Messages
----------------

    struct AckServer {
        u8 type = 0;
        u63 message_id;
        u63 timestamp;
    };

This is sent by the server in response to a message.  The id is a unique number
identifying the message. It is incremented for every message. The timestamp is
a millisecond offset from the Unix Epoch representing the time the server
received the message. The type `u63` indicates that the type is 64 bits wide,
but the top bit must be clear.

    struct DAckServer {
        u8 type = 1;
        u63 message_id;
        u63 part;
    };

This is sent by the server in response to fully receiving a message part (in
the case of BigText, the whole message. For something like Photos, each
photo is a separate part).

    struct TextMessageClient {
        u8 type = 2;
        u8 message[];
    };

Send a text message. The message contents are UTF-8 encoded. The message may
not have any leading or trailing whitespace. The message size is capped to
1006, so that the server can deliver it to the peer without violating the 1024
byte total message length.

    struct PartMessageClient {
        u8 type = 3;
        u8 part_type;
        u32 part_sizes[];
    };

Send a message whose data is large enough to require being sent over the data
channel. The number of parts is capped to 251 as per the same restriction as
for TextMessage. Most parts may not appear more than once per message. Parts
must not be sent before receiving acknowledgement of the message from the
server. There are currently two part types defined:

Send a large text message (part_type = 0). The message data is encoded the same
way as for a normal message.

Send photos (part_type = 1). The photos are encoded as JPEGs. This part may
appear multiple times in a message, though it may not be mixed with other part
types.

Receiving Messages
------------------

    struct TextMessageServer {
        u8 type = 2;
        u8 sender;
        u63 message_id;
        u63 timestamp;
        u8 message[];
    };

Sent by the server to indicate a received message. The sender field is zero in
the case of a message from the peer, and one for messages sent by the user on a
different connection or in a past session.

    struct PartMessageServer {
        u8 type = 3;
        u8 sender;
        u63 message_id;
        u63 timestamp;
        u8 part_type;
        u32 part_sizes[];
    };

Sent by the server to indicate a received part message. Unlike with sending
messages, the client must explicitly request the parts.
<commit_msg>Improve formatting of protocol spec<commit_after>Protocol specification
======================

The protocol uses two ports, the control port 29192 and the data port 29292.
The data port is used for uploading/downloading large files (pictures,
videos, etc).  Appart from the initial exchange, everything is encrypted using
sodium's `crypto_box`.

All messages are prepended by a u16 unencrypted size field.  This size is
offset by 1 (as in, a value of 5 would mean that the message is 6 bytes long).
The size excludes the size of the authentication tag (`crypto_box_MACBYTES`).
Note that the translation from encoded size to actual size is uniform, so you
must add `crypto_box_MACBYTES` even if the message is not encrypted for the
control channel, and the size of the normal header for the data channel. For
messages on the data channel, the size also excludes the size of the header.
Messages on the control channel are capped to a maximum size of 1024 (so
encoded as 1023).

Encryption Negotiation
----------------------

    struct HelClient {
        u16 version;
        u64 nonce1;
        u8 public_key[32];
    };

The `HelClient` message is the first thing that the client must send on
creating the connection.

The version field contains the current version of the protocol. This version is
version 0.

Nonces are random 8 byte values. The final computed nonce for a message is
nonce1 followed by nonce2 followed by a message counter, where the client uses
even numbers and the server uses odd numbers. e.g. The third message the client
has sent this session has a counter of 5.

The client transmits it's public key. This serves not only as a means of
encryption, it also identifies the user. The client is expected to already know
the server's public key.

    struct HelServer {
        u8 error = 0;
        u64 nonce2;
        // begin encryption
        u8 session_token[32];
        u32 color;
        u8 avatar_sha256[32];
        u8 name[];
    };

The `HelServer` is sent when the server receives a `HelClient`, and the server
has accepted the connection. Now that the client has access to the other half
of the nonce, it can do encryption, and so the session token is encrypted. The
session token is required to open a data connection later. All further
communication on this channel is encrypted. The name and color specify the
saved values for the user's preferences. Additionally, the SHA256 of the avatar
file is provided so that the client can decide to request it from the server if
it is changed.

    struct ErrorBadVersion {
        u8 error = 1;
        u16 min_version;
        u16 max_version;
    };

If the provided version is not acceptable, `ErrorBadVersion` is sent instead of
`HelServer`, providing the permitted range of versions. The min and max are
inclusive. The server disconnects immediatly after sending this error message
(and indeed after all error messages).

     struct ErrorUnknownUser {
         u8 error = 2;
     };

If the server does not know of the user, this is sent, and as mentioned above
the server disconnects.

Additionally, the server may disconnect immediately on a protocol violation
without sending a message.

    struct DataHelloClient {
        u8 session_token[32];
    };

At this point the client must open a data connection and send the
`DataHelloClient`. If the session token is invalid, the server closes the
connection. Otherwise it transmits a single zero byte as an ACK. All further
communication on the data channel is encrypted.

    struct LoClient {
        u8 peer_public_key[32];
    };

Once the data connection is set up, the second part of the hello is sent. The
peer key indicates who the client wants to talk to.

    struct LoServer {
        u8 error = 0;
        u32 color;
        u8 avatar_sha256[32];
        u8 name[];
    };

Once the server is ready, it sends the settings for the peer.

     struct ErrorUnknownPeer {
         u8 error = 1;
     };

If the server does not know the peer, it sends an error and disconnects.

Message Type Overview
---------------------

Type Client      Server
---- ------      ------
   0             Ack
   1             DAck
   2 TextMessage TextMessage
   3 PartMessage PartMessage
   4 RequestPart
   5 Replay
   6 Exclusive   Exclusive
   7 Seen        Seen
   8 Typing      Typing
   9 Name        Name
  10 Color       Color
  11 Avatar      Avatar

Sending Messages
----------------

    struct AckServer {
        u8 type = 0;
        u63 message_id;
        u63 timestamp;
    };

This is sent by the server in response to a message.  The id is a unique number
identifying the message. It is incremented for every message. The timestamp is
a millisecond offset from the Unix Epoch representing the time the server
received the message. The type `u63` indicates that the type is 64 bits wide,
but the top bit must be clear.

    struct DAckServer {
        u8 type = 1;
        u63 message_id;
        u63 part;
    };

This is sent by the server in response to fully receiving a message part (in
the case of `BigText`, the whole message. For something like Photos, each photo
is a separate part).

    struct TextMessageClient {
        u8 type = 2;
        u8 message[];
    };

Send a text message. The message contents are UTF-8 encoded. The message may
not have any leading or trailing whitespace. The message size is capped to
1006, so that the server can deliver it to the peer without violating the 1024
byte total message length.

    struct PartMessageClient {
        u8 type = 3;
        u8 part_type;
        u32 part_sizes[];
    };

Send a message whose data is large enough to require being sent over the data
channel. The number of parts is capped to 251 as per the same restriction as
for `TextMessage`. Most parts may not appear more than once per message. Parts
must not be sent before receiving acknowledgement of the message from the
server. There are currently two part types defined:

Send a large text message (`part_type = 0`). The message data is encoded the same
way as for a normal message.

Send photos (`part_type = 1`). The photos are encoded as JPEGs. This part may
appear multiple times in a message, though it may not be mixed with other part
types.

Receiving Messages
------------------

    struct TextMessageServer {
        u8 type = 2;
        u8 sender;
        u63 message_id;
        u63 timestamp;
        u8 message[];
    };

Sent by the server to indicate a received message. The sender field is zero in
the case of a message from the peer, and one for messages sent by the user on a
different connection or in a past session.

    struct PartMessageServer {
        u8 type = 3;
        u8 sender;
        u63 message_id;
        u63 timestamp;
        u8 part_type;
        u32 part_sizes[];
    };

Sent by the server to indicate a received part message. Unlike with sending
messages, the client must explicitly request the parts.

{'fraction_non_alphanumeric': 0.05388180764774044, 'fraction_numerical': 0.022161066048667438, 'mean_word_length': 2.776045939294504, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4569897', 'n_tokens_mistral': 3764, 'n_tokens_neox': 3488, 'n_words': 2024}