<commit_before>Vectors
=======

Joedb stores each column of a table in a ``std::vector`` container. This allows
very efficient data manipulation. In particular, reading or writing a large
column of a primitive type (float or int) can be performed as a single call to
the operating system function that reads a file to a buffer or write a buffer
to a file.

In order to keep performance optimal, it is recommended to not delete records
in the middle of a table. As soon has the table contains holes, many of the
vector optimizations don't work well any more.

Allocating and Using Vectors
----------------------------

Code generated by the joedb compiler offers convenient functions to allocate a
set of consecutive rows, and access them like a vector.

For instance, when using a database defined by this schema:

.. code-block:: none

    create_table float
    add_field float value float32

It is possible to allocate and manipulate a vector of floats like this:

.. code-block:: c++

     const size_t n = 5;
     auto v = db.new_vector_of_float(n);
     for (size_t i = 0; i < n; i++)
      db.set_value(v[i], 0.1f * float(i));

``v`` is a usual reference, and can be stored in a field of type ``references float``.<commit_msg>vector documentation<commit_after>Vectors
=======

Joedb stores each column of a table in a ``std::vector`` container. This allows
very efficient data manipulation. In particular, reading or writing a large
column of a primitive type (float or int) can be performed as a single call to
the operating system function that reads a file to a buffer or write a buffer
to a file.

In order to keep performance optimal, it is recommended to not delete records
in the middle of a table. As soon has the table contains holes, many of the
vector optimizations don't work well any more.

Code generated by the joedb compiler offers convenient functions to allocate a
set of consecutive rows, and access them like a vector.

For instance, when using a database defined by this schema:

.. code-block:: none

    create_table float
    add_field float value float32

It is possible to allocate and manipulate a vector of floats like this:

.. code-block:: c++

     const size_t n = 5;
     auto v = db.new_vector_of_float(n);
     for (size_t i = 0; i < n; i++)
      db.set_value(v[i], 0.1f * float(i));

``v`` is a usual reference, and can be stored in a field of type ``references float``.

The loop above will write vector values one by one. In order to efficiently
write the whole vector in a single operation, the for loop above can be
replaced by:

.. code-block:: c++

    {
     auto value = db.update_vector_of_value(v, n);
     for (size_t i = 0; i < n; i++)
      value[i] = 0.1f * float(i);
    }

The destructor of the ``value`` object will write the data to the joedb file.
{'fraction_non_alphanumeric': 0.0807829181494662, 'fraction_numerical': 0.005338078291814947, 'mean_word_length': 3.4689984101748808, 'pattern_counts': {'":': 0, '<': 6, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4000434', 'n_tokens_mistral': 798, 'n_tokens_neox': 770, 'n_words': 430}