<commit_before>Encoding
####################

This section describes how serialized prophy messages look like.

Prophy message wire format features no field or type tags,
no submessage delimiters and no integer packing.

Prophy ensures that each field in message is aligned.
This allows to manipulate message directly in serialized buffer
and contributes to encoding speed.

Numeric types
====================

Prophy encodes integral and floating point numbers according
to their size and endianness. Enums are treated as 32-bit unsigned numbers.

==================  =======================  =======================
Number 42 encodes   as little endian         as big endian
==================  =======================  =======================
u8/i8               2a                       2a
u16/i16             2a 00                    00 2a
u32/i32             2a 00 00 00              00 00 00 2a
u64/i64             2a 00 00 00 00 00 00 00  00 00 00 00 00 00 00 2a
float               00 00 28 42              42 28 00 00
double              00 00 00 00 00 00 45 40  40 45 00 00 00 00 00 00
enum                2a 00 00 00              00 00 00 2a
==================  =======================  =======================

Signed types treat most significant bit as sign bit following U2 encoding rules.

.. note ::
    All following examples will have numbers encoded
    in little endian for the sake of brevity.

Array
==========

Fixed array
------------

Dynamic array
--------------

Limited array
---------------

Greedy array
--------------

Struct array
===============

A sequence of fields which get serialized in order. Following struct::

    struct X
    {
        u16 x;
        u16 y;
    };

with fields set to 1 and 2 will yield::
<commit_msg>arrays encoding described<commit_after>Encoding
####################

This section describes how serialized prophy messages look like.

Prophy message wire format features no field or type tags,
no submessage delimiters and no integer packing.

Prophy ensures that each field in message is aligned.
This allows to manipulate message directly in serialized buffer
and contributes to encoding speed.

Numeric types
====================

Prophy encodes integral and floating point numbers according
to their size and endianness. Enums are treated as 32-bit unsigned numbers.

==================  =======================  =======================
Number 42 encodes   as little endian         as big endian
==================  =======================  =======================
u8/i8               2a                       2a
u16/i16             2a 00                    00 2a
u32/i32             2a 00 00 00              00 00 00 2a
u64/i64             2a 00 00 00 00 00 00 00  00 00 00 00 00 00 00 2a
float               00 00 28 42              42 28 00 00
double              00 00 00 00 00 00 45 40  40 45 00 00 00 00 00 00
enum                2a 00 00 00              00 00 00 2a
==================  =======================  =======================

Signed types treat most significant bit as sign bit following U2 encoding rules.

.. note ::
    All following examples will have numbers encoded
    in little endian for the sake of brevity.

Array
==========

Array is a sequence of elements of the same type.
Elements may be numeric types, structs or unions.

.. note ::
    Dynamic struct may not be held in fixed or limited array.

.. note ::
    Unlimited struct may not be held in any array.

There are so much as 4 different types of arrays in Prophy.

Fixed array
------------

Fixed array has fixed number of elements, hence fixed size on wire. This array::

    u16 x[4];

with elements set to 1, 2, 3 and 4 would encode as::

    01 00 02 00 03 00 04 00

Dynamic array
--------------

Dynamic array has varying number of elements
counted by 32-bit unsigned delimiter. This one::

    u16 x<>;

with 2 elements set to 1 and 2 encodes as::

    02 00 00 00 01 00 02 00

Limited array
---------------

A combination of fixed and dynamic one.
Delimited by an element counter, has fixed size on wire.
This requires it to have an upper limit. Such array::

    u16 x<4>;

with 2 elements set to 1 and 2 encodes as::

    02 00 00 00 01 00 02 00 00 00 00 00

Greedy array
--------------

Variable element array without element counter. This one::

    u16 x<...>;

with 2 elements set to 1 and 2 encodes as::

    01 00 02 00

.. note ::
    Greedy array can be used only in the last field of struct.
    Such struct may also be only the last field of any other struct.

Bytes
---------

Bytes field is an array of bytes, which can be handled by codec
in more effective way than array of u8. Wire-format, however,
is the same.

Struct
============

A sequence of fields which get serialized in order. Following struct::

    struct X
    {
        u16 x;
        u16 y;
    };

with fields set to 1 and 2 will yield::

{'fraction_non_alphanumeric': 0.1556739041376485, 'fraction_numerical': 0.07558377714051619, 'mean_word_length': 2.3285616905248805, 'pattern_counts': {'":': 0, '<': 6, '<?xml version=': 0, '>': 6, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5240646', 'n_tokens_mistral': 1573, 'n_tokens_neox': 1178, 'n_words': 683}