<commit_before>*********
Functions
*********

BQL provides a number of built-in functions that are described in this chapter.
Function names and meaning of parameters have been heavily inspired by `PostgreSQL <http://www.postgresql.org/docs/9.5/static/functions.html>`_.
However, be aware that the accepted and returned types may differ as there is no simple mapping between BQL and SQL data types.

Numeric Functions
=================

General Functions
-----------------

The table below shows some common mathematical functions that can be used in BQL.

+----------------------------------+------------------------------+---------------+------------------------------------------------------+-------------------------------+---------------+
| Function                         | Input Type                   | Return Type   | Description                                          | Example                       | Result        |
+==================================+==============================+===============+======================================================+===============================+===============+
| ``abs(x)``                       | ``int`` or ``float``         | same as input | absolute value                                       | ``abs(-17.4)``                | ``17.4``      |
+----------------------------------+------------------------------+---------------+------------------------------------------------------+-------------------------------+---------------+
| ``cbrt(x)``                      | ``int`` or ``float``         | ``float``     | cube root                                            | ``cbrt(27.0)``, |br|          | ``3.0``, |br| |
|                                  |                              |               |                                                      | ``cbrt(-3)``                  | ``-1.442...`` |
+----------------------------------+------------------------------+---------------+------------------------------------------------------+-------------------------------+---------------+
| ``ceil(x)``                      | ``int`` or ``float``         | same as input | smallest integer not less than argument              | ``ceil(1.3)``, |br|           | ``2.0``, |br| |
|                                  |                              |               |                                                      | ``ceil(-1.7)``                | ``-1.0``      |
+----------------------------------+------------------------------+---------------+------------------------------------------------------+-------------------------------+---------------+
| ``degrees(x)``                   | ``int`` or ``float``         | ``float``     | radians to degrees                                   | ``degrees(3.14159...)``       | ``180.0``     |
+----------------------------------+------------------------------+---------------+------------------------------------------------------+-------------------------------+---------------+
| ``div(y, x)``                    | 2 x ``int`` or 2 x ``float`` | same as input | integer quotient of ``y``/``x``                      | ``div(9, 4)``, |br|           | ``2``, |br|   |
|                                  |                              |               |                                                      | ``div(9.3, 4.5)``             | ``2.0``       |
+----------------------------------+------------------------------+---------------+------------------------------------------------------+-------------------------------+---------------+
| ``exp(x)``                       | ``int`` or ``float``         | ``float``     | exponential                                          | ``exp(1.0)``                  | ``2.7182...`` |
+----------------------------------+------------------------------+---------------+------------------------------------------------------+-------------------------------+---------------+
| ``floor(x)``                     | ``int`` or ``float``         | same as input | largest integer not greater than argument            | ``floor(1.3)``, |br|          | ``1.0``, |br| |
|                                  |                              |               |                                                      | ``floor(-1.7)``               | ``-2.0``      |
+----------------------------------+------------------------------+---------------+------------------------------------------------------+-------------------------------+---------------+
| ``ln(x)``                        | ``int`` or ``float``         | ``float``     | natural logarithm                                    | ``ln(2)``                     | ``0.6931...`` |
+----------------------------------+------------------------------+---------------+------------------------------------------------------+-------------------------------+---------------+
| ``log(x)``                       | ``int`` or ``float``         | ``float``     | base 10 logarithm                                    | ``log(100)``                  | ``2.0``       |
+----------------------------------+------------------------------+---------------+------------------------------------------------------+-------------------------------+---------------+
| ``log(b, x)``                    | 2 x ``int`` or 2 x ``float`` | ``float``     | logarithm to base b                                  | ``log(2.5, 6.25)``, |br|      | ``2.0``, |br| |
|                                  |                              |               |                                                      | ``log(2, 8)``                 | ``3.0``       |
+----------------------------------+------------------------------+---------------+------------------------------------------------------+-------------------------------+---------------+
| ``mod(y, x)``                    | 2 x ``int`` or 2 x ``float`` | same as input | remainder of ``y``/``x``                             | ``mod(9, 4)``, |br|           | ``1``, |br|   |
|                                  |                              |               |                                                      | ``mod(9.3, 4.5)``             | ``0.3``       |
+----------------------------------+------------------------------+---------------+------------------------------------------------------+-------------------------------+---------------+
| ``pi()``                         | none                         | ``float``     | "π" constant                                         | ``pi()``                      | ``3.1415...`` |
+----------------------------------+------------------------------+---------------+------------------------------------------------------+-------------------------------+---------------+
| ``power(a, b)``                  | 2 x ``int`` or 2 x ``float`` | ``float``     | ``a`` raised to the power of ``b``                   | ``power(9.0, 3.0)``, |br|     | ``729.0``,    |
|                                  |                              |               |                                                      | ``power(2, -1)``              | |br| ``0.5``  |
+----------------------------------+------------------------------+---------------+------------------------------------------------------+-------------------------------+---------------+
| ``radians(x)``                   | ``int`` or ``float``         | ``float``     | degrees to radians                                   | ``radians(180)``              | ``3.1415...`` |
+----------------------------------+------------------------------+---------------+------------------------------------------------------+-------------------------------+---------------+
| ``round(x)``                     | ``int`` or ``float``         | same as input | round to nearest integer                             | ``round(1.3)``, |br|          | ``1.0``, |br| |
|                                  |                              |               |                                                      | ``round(-1.7)``               | ``-2.0``      |
+----------------------------------+------------------------------+---------------+------------------------------------------------------+-------------------------------+---------------+
| ``sign(x)``                      | ``int`` or ``float``         | ``int``       | sign of the argument (-1, 0, +1)                     | ``sign(-8.4)``                | ``-1``        |
+----------------------------------+------------------------------+---------------+------------------------------------------------------+-------------------------------+---------------+
| ``sqrt(x)``                      | ``int`` or ``float``         | ``float``     | square root                                          | ``sqrt(2)``                   | ``1.4142...`` |
+----------------------------------+------------------------------+---------------+------------------------------------------------------+-------------------------------+---------------+
| ``trunc(x)``                     | ``int`` or ``float``         | same as input | truncate toward zero                                 | ``trunc(1.3)``, |br|          | ``1.0``, |br| |
|                                  |                              |               |                                                      | ``trunc(-1.7)``               | ``-1.0``      |
+----------------------------------+------------------------------+---------------+------------------------------------------------------+-------------------------------+---------------+

.. todo::

   fix the table width of the function below in HTML

+----------------------------------+------------------------------+---------------+------------------------------------------------------+-------------------------------+---------------+
| Function                         | Input Type                   | Return Type   | Description                                          | Example                       | Result        |
+==================================+==============================+===============+======================================================+===============================+===============+
| ``width_bucket(x, l, r, count)`` | ``x``, ``l``, ``r``:         | ``int``       | return the bucket number to which ``x`` would be     | ``width_bucket(5, 0, 10, 5)`` | ``3``         |
|                                  | ``int`` or ``float``,        |               | assigned in a histogram having ``count`` equal-width |                               |               |
|                                  | ``count``: ``int``           |               | buckets spanning the range ``l`` to ``r``; returns   |                               |               |
|                                  |                              |               | ``0`` or ``count+1`` for an input outside the range  |                               |               |
+----------------------------------+------------------------------+---------------+------------------------------------------------------+-------------------------------+---------------+

If a given parameter is outside the mathematically valid range for that function (e.g., ``sqrt(-2)``, ``log(0)``, ``div(2.0, 0.0)``) and the return type is ``float``, then ``NaN`` is returned.
However, if the return type is ``int`` (e.g., ``div(2, 0)``), there is no ``NaN`` option and an error will occur instead.


Pseudo-Random Functions
-----------------------

The table below shows functions for generating pseudo-random numbers.

+----------------+------------+-------------+-----------------------------------------------------------------------+
| Function       | Input Type | Return Type | Description                                                           |
+================+============+=============+=======================================================================+
| ``random()``   | none       | ``float``   | random value in the range :math:`0.0 <= x < 1.0`                      |
+----------------+------------+-------------+-----------------------------------------------------------------------+
| ``setseed(x)`` | ``float``  | ``null``    | set seed (:math:`-1.0 <= x <= 1.0`) for subsequent ``random()`` calls |
+----------------+------------+-------------+-----------------------------------------------------------------------+

The characteristics of the values returned by ``random()`` are equal to those from `the Go rand module <https://golang.org/pkg/math/rand/>`_.
It is not suitable for cryptographic applications.


Trigonometric Functions
-----------------------

Finally, the table below shows the available trigonometric functions.
All trigonometric functions take arguments and return values of type ``float``.
Trigonometric functions arguments are expressed in radians.
Inverse functions return values are expressed in radians.


+-------------+-----------------+
| Function    | Description     |
+=============+=================+
| ``acos(x)`` | inverse cosine  |
+-------------+-----------------+
| ``asin(x)`` | inverse sine    |
+-------------+-----------------+
| ``atan(x)`` | inverse tangent |
+-------------+-----------------+
| ``cos(x)``  | cosine          |
+-------------+-----------------+
| ``cot(x)``  | cotangent       |
+-------------+-----------------+
| ``sin(x)``  | sine            |
+-------------+-----------------+
| ``tan(x)``  | tangent         |
+-------------+-----------------+

<commit_msg>add string functions (as table)<commit_after>*********
Functions
*********

BQL provides a number of built-in functions that are described in this chapter.
Function names and meaning of parameters have been heavily inspired by `PostgreSQL <http://www.postgresql.org/docs/9.5/static/functions.html>`_.
However, be aware that the accepted and returned types may differ as there is no simple mapping between BQL and SQL data types.

Numeric Functions
=================

General Functions
-----------------

The table below shows some common mathematical functions that can be used in BQL.

+----------------------------------+------------------------------+---------------+------------------------------------------------------+-------------------------------+---------------+
| Function                         | Input Type                   | Return Type   | Description                                          | Example                       | Result        |
+==================================+==============================+===============+======================================================+===============================+===============+
| ``abs(x)``                       | ``int`` or ``float``         | same as input | absolute value                                       | ``abs(-17.4)``                | ``17.4``      |
+----------------------------------+------------------------------+---------------+------------------------------------------------------+-------------------------------+---------------+
| ``cbrt(x)``                      | ``int`` or ``float``         | ``float``     | cube root                                            | ``cbrt(27.0)``, |br|          | ``3.0``, |br| |
|                                  |                              |               |                                                      | ``cbrt(-3)``                  | ``-1.442...`` |
+----------------------------------+------------------------------+---------------+------------------------------------------------------+-------------------------------+---------------+
| ``ceil(x)``                      | ``int`` or ``float``         | same as input | smallest integer not less than argument              | ``ceil(1.3)``, |br|           | ``2.0``, |br| |
|                                  |                              |               |                                                      | ``ceil(-1.7)``                | ``-1.0``      |
+----------------------------------+------------------------------+---------------+------------------------------------------------------+-------------------------------+---------------+
| ``degrees(x)``                   | ``int`` or ``float``         | ``float``     | radians to degrees                                   | ``degrees(3.14159...)``       | ``180.0``     |
+----------------------------------+------------------------------+---------------+------------------------------------------------------+-------------------------------+---------------+
| ``div(y, x)``                    | 2 x ``int`` or 2 x ``float`` | same as input | integer quotient of ``y``/``x``                      | ``div(9, 4)``, |br|           | ``2``, |br|   |
|                                  |                              |               |                                                      | ``div(9.3, 4.5)``             | ``2.0``       |
+----------------------------------+------------------------------+---------------+------------------------------------------------------+-------------------------------+---------------+
| ``exp(x)``                       | ``int`` or ``float``         | ``float``     | exponential                                          | ``exp(1.0)``                  | ``2.7182...`` |
+----------------------------------+------------------------------+---------------+------------------------------------------------------+-------------------------------+---------------+
| ``floor(x)``                     | ``int`` or ``float``         | same as input | largest integer not greater than argument            | ``floor(1.3)``, |br|          | ``1.0``, |br| |
|                                  |                              |               |                                                      | ``floor(-1.7)``               | ``-2.0``      |
+----------------------------------+------------------------------+---------------+------------------------------------------------------+-------------------------------+---------------+
| ``ln(x)``                        | ``int`` or ``float``         | ``float``     | natural logarithm                                    | ``ln(2)``                     | ``0.6931...`` |
+----------------------------------+------------------------------+---------------+------------------------------------------------------+-------------------------------+---------------+
| ``log(x)``                       | ``int`` or ``float``         | ``float``     | base 10 logarithm                                    | ``log(100)``                  | ``2.0``       |
+----------------------------------+------------------------------+---------------+------------------------------------------------------+-------------------------------+---------------+
| ``log(b, x)``                    | 2 x ``int`` or 2 x ``float`` | ``float``     | logarithm to base b                                  | ``log(2.5, 6.25)``, |br|      | ``2.0``, |br| |
|                                  |                              |               |                                                      | ``log(2, 8)``                 | ``3.0``       |
+----------------------------------+------------------------------+---------------+------------------------------------------------------+-------------------------------+---------------+
| ``mod(y, x)``                    | 2 x ``int`` or 2 x ``float`` | same as input | remainder of ``y``/``x``                             | ``mod(9, 4)``, |br|           | ``1``, |br|   |
|                                  |                              |               |                                                      | ``mod(9.3, 4.5)``             | ``0.3``       |
+----------------------------------+------------------------------+---------------+------------------------------------------------------+-------------------------------+---------------+
| ``pi()``                         | none                         | ``float``     | "π" constant                                         | ``pi()``                      | ``3.1415...`` |
+----------------------------------+------------------------------+---------------+------------------------------------------------------+-------------------------------+---------------+
| ``power(a, b)``                  | 2 x ``int`` or 2 x ``float`` | ``float``     | ``a`` raised to the power of ``b``                   | ``power(9.0, 3.0)``, |br|     | ``729.0``,    |
|                                  |                              |               |                                                      | ``power(2, -1)``              | |br| ``0.5``  |
+----------------------------------+------------------------------+---------------+------------------------------------------------------+-------------------------------+---------------+
| ``radians(x)``                   | ``int`` or ``float``         | ``float``     | degrees to radians                                   | ``radians(180)``              | ``3.1415...`` |
+----------------------------------+------------------------------+---------------+------------------------------------------------------+-------------------------------+---------------+
| ``round(x)``                     | ``int`` or ``float``         | same as input | round to nearest integer                             | ``round(1.3)``, |br|          | ``1.0``, |br| |
|                                  |                              |               |                                                      | ``round(-1.7)``               | ``-2.0``      |
+----------------------------------+------------------------------+---------------+------------------------------------------------------+-------------------------------+---------------+
| ``sign(x)``                      | ``int`` or ``float``         | ``int``       | sign of the argument (-1, 0, +1)                     | ``sign(-8.4)``                | ``-1``        |
+----------------------------------+------------------------------+---------------+------------------------------------------------------+-------------------------------+---------------+
| ``sqrt(x)``                      | ``int`` or ``float``         | ``float``     | square root                                          | ``sqrt(2)``                   | ``1.4142...`` |
+----------------------------------+------------------------------+---------------+------------------------------------------------------+-------------------------------+---------------+
| ``trunc(x)``                     | ``int`` or ``float``         | same as input | truncate toward zero                                 | ``trunc(1.3)``, |br|          | ``1.0``, |br| |
|                                  |                              |               |                                                      | ``trunc(-1.7)``               | ``-1.0``      |
+----------------------------------+------------------------------+---------------+------------------------------------------------------+-------------------------------+---------------+

.. todo::

   fix the table width of the function below in HTML

+----------------------------------+------------------------------+---------------+------------------------------------------------------+-------------------------------+---------------+
| Function                         | Input Type                   | Return Type   | Description                                          | Example                       | Result        |
+==================================+==============================+===============+======================================================+===============================+===============+
| ``width_bucket(x, l, r, count)`` | ``x``, ``l``, ``r``:         | ``int``       | return the bucket number to which ``x`` would be     | ``width_bucket(5, 0, 10, 5)`` | ``3``         |
|                                  | ``int`` or ``float``,        |               | assigned in a histogram having ``count`` equal-width |                               |               |
|                                  | ``count``: ``int``           |               | buckets spanning the range ``l`` to ``r``; returns   |                               |               |
|                                  |                              |               | ``0`` or ``count+1`` for an input outside the range  |                               |               |
+----------------------------------+------------------------------+---------------+------------------------------------------------------+-------------------------------+---------------+

If a given parameter is outside the mathematically valid range for that function (e.g., ``sqrt(-2)``, ``log(0)``, ``div(2.0, 0.0)``) and the return type is ``float``, then ``NaN`` is returned.
However, if the return type is ``int`` (e.g., ``div(2, 0)``), there is no ``NaN`` option and an error will occur instead.


Pseudo-Random Functions
-----------------------

The table below shows functions for generating pseudo-random numbers.

+----------------+------------+-------------+-----------------------------------------------------------------------+
| Function       | Input Type | Return Type | Description                                                           |
+================+============+=============+=======================================================================+
| ``random()``   | none       | ``float``   | random value in the range :math:`0.0 <= x < 1.0`                      |
+----------------+------------+-------------+-----------------------------------------------------------------------+
| ``setseed(x)`` | ``float``  | ``null``    | set seed (:math:`-1.0 <= x <= 1.0`) for subsequent ``random()`` calls |
+----------------+------------+-------------+-----------------------------------------------------------------------+

The characteristics of the values returned by ``random()`` are equal to those from `the Go rand module <https://golang.org/pkg/math/rand/>`_.
It is not suitable for cryptographic applications.


Trigonometric Functions
-----------------------

Finally, the table below shows the available trigonometric functions.
All trigonometric functions take arguments and return values of type ``float``.
Trigonometric functions arguments are expressed in radians.
Inverse functions return values are expressed in radians.


+-------------+-----------------+
| Function    | Description     |
+=============+=================+
| ``acos(x)`` | inverse cosine  |
+-------------+-----------------+
| ``asin(x)`` | inverse sine    |
+-------------+-----------------+
| ``atan(x)`` | inverse tangent |
+-------------+-----------------+
| ``cos(x)``  | cosine          |
+-------------+-----------------+
| ``cot(x)``  | cotangent       |
+-------------+-----------------+
| ``sin(x)``  | sine            |
+-------------+-----------------+
| ``tan(x)``  | tangent         |
+-------------+-----------------+


String Functions
================

The table below shows some common functions for strings that can be used in BQL.

+-------------------------------+-----------------------------+-------------+--------------------------------------------------------------------------+---------------------------------------+------------------------------------------------------------------------+
| Function                      | Input Type                  | Return Type | Description                                                              | Example                               | Result                                                                 |
+===============================+=============================+=============+==========================================================================+=======================================+========================================================================+
| ``bit_length(s)``             | ``string``                  | ``int``     | Number of bits in string                                                 | ``bit_length('über')``                | ``40``                                                                 |
+-------------------------------+-----------------------------+-------------+--------------------------------------------------------------------------+---------------------------------------+------------------------------------------------------------------------+
| ``btrim(s)``                  | ``string``                  | ``string``  | Remove whitespace from the start and end of ``s``                        | ``btrim('  trim  ')``                 | ``'trim'``                                                             |
+-------------------------------+-----------------------------+-------------+--------------------------------------------------------------------------+---------------------------------------+------------------------------------------------------------------------+
| ``btrim(s, chars)``           | 2 x ``string``              | ``string``  | Remove the longest string consisting only of characters in ``chars``     | ``btrim('xyxtrimyyx', 'xy')``         | ``'trim'``                                                             |
|                               |                             |             | from the start and end of ``s``                                          |                                       |                                                                        |
+-------------------------------+-----------------------------+-------------+--------------------------------------------------------------------------+---------------------------------------+------------------------------------------------------------------------+
| ``char_length(s)``            | ``string``                  | ``int``     | Number of characters in ``s``                                            | ``char_length('über')``               | ``4``                                                                  |
+-------------------------------+-----------------------------+-------------+--------------------------------------------------------------------------+---------------------------------------+------------------------------------------------------------------------+
| ``concat(s [, ...])``         | *n* x ``string``            | ``string``  | Concatenate the text representations of all the arguments.               | ``concat('abc', NULL, '22')``         | ``'abc22'``                                                            |
|                               |                             |             | NULL arguments are ignored.                                              |                                       |                                                                        |
+-------------------------------+-----------------------------+-------------+--------------------------------------------------------------------------+---------------------------------------+------------------------------------------------------------------------+
| ``concat_ws(sep, s [, ...])`` | (1 + *n*) x ``string``      | ``string``  | Concatenate all but the first argument with separators.                  | ``concat_ws('-', 'abc', NULL, '22')`` | ``'abc-22'``                                                           |
|                               |                             |             | The first argument is used as the separator string.                      |                                       |                                                                        |
|                               |                             |             | NULL arguments are ignored.                                              |                                       |                                                                        |
+-------------------------------+-----------------------------+-------------+--------------------------------------------------------------------------+---------------------------------------+------------------------------------------------------------------------+
| ``format(s, [x, ...])``       | ``string``, *n* x any       | ``string``  | Format arguments according to a format string.                           | ``format('%s-%d', 'abc', 22)``        | ``'abc-22'``                                                           |
|                               |                             |             | This function is similar to the Go function ``fmt.Sprintf``.             |                                       |                                                                        |
+-------------------------------+-----------------------------+-------------+--------------------------------------------------------------------------+---------------------------------------+------------------------------------------------------------------------+
| ``lower(s)``                  | ``string``                  | ``string``  | Convert ``s`` to lower case                                              | ``lower('ÜBer')``                     | ``'über'``                                                             |
+-------------------------------+-----------------------------+-------------+--------------------------------------------------------------------------+---------------------------------------+------------------------------------------------------------------------+
| ``ltrim(s)``                  | ``string``                  | ``string``  | Remove whitespace from the start of ``s``                                | ``ltrim('  trim  ')``                 | ``'trim  '``                                                           |
+-------------------------------+-----------------------------+-------------+--------------------------------------------------------------------------+---------------------------------------+------------------------------------------------------------------------+
| ``ltrim(s, chars)``           | 2 x ``string``              | ``string``  | Remove the longest string consisting only of characters in ``chars``     | ``ltrim('xyxtrimyyx', 'xy')``         | ``trimyyx``                                                            |
|                               |                             |             | from the start of ``s``                                                  |                                       |                                                                        |
+-------------------------------+-----------------------------+-------------+--------------------------------------------------------------------------+---------------------------------------+------------------------------------------------------------------------+
| ``md5(s)``                    | ``string``                  | ``string``  | Calculates the MD5 hash of ``s``, returning the result in hexadecimal    | ``md5('abc')``                        | ``'900150983cd24fb0d6963f7d28e17f72'``                                 |
+-------------------------------+-----------------------------+-------------+--------------------------------------------------------------------------+---------------------------------------+------------------------------------------------------------------------+
| ``octet_length(s)``           | ``string``                  | ``int``     | Number of bytes in ``s``                                                 | ``octet_length('über')``              | ``5``                                                                  |
+-------------------------------+-----------------------------+-------------+--------------------------------------------------------------------------+---------------------------------------+------------------------------------------------------------------------+
| ``overlay(s, r, from)``       | 2 x ``string``, ``int``     | ``string``  | Replace substring                                                        | ``overlay('Txxxxas', 'hom', 2)``      | ``'Thomxas'``                                                          |
+-------------------------------+-----------------------------+-------------+--------------------------------------------------------------------------+---------------------------------------+------------------------------------------------------------------------+
| ``overlay(s, r, from, for)``  | 2 x ``string``, 2 x ``int`` | ``string``  | Replace substring                                                        | ``overlay('Txxxxas', 'hom', 2, 4)``   | ``'Thomas'``                                                           |
+-------------------------------+-----------------------------+-------------+--------------------------------------------------------------------------+---------------------------------------+------------------------------------------------------------------------+
| ``rtrim(s)``                  | ``string``                  | ``string``  | Remove whitespace from the end of ``s``                                  | ``rtrim('  trim  ')``                 | ``'  trim'``                                                           |
+-------------------------------+-----------------------------+-------------+--------------------------------------------------------------------------+---------------------------------------+------------------------------------------------------------------------+
| ``rtrim(s, chars)``           | 2 x ``string``              | ``string``  | Remove the longest string consisting only of characters in ``chars``     | ``rtrim('xyxtrimyyx', 'xy')``         | ``xyxtrim``                                                            |
|                               |                             |             | from the end of ``s``                                                    |                                       |                                                                        |
+-------------------------------+-----------------------------+-------------+--------------------------------------------------------------------------+---------------------------------------+------------------------------------------------------------------------+
| ``sha1(s)``                   | ``string``                  | ``string``  | Calculates the SHA1 hash of ``s``, returning the result in hexadecimal   | ``sha1('abc')``                       | ``'a9993e364706816aba3e25717850c26c9cd0d89d'``                         |
+-------------------------------+-----------------------------+-------------+--------------------------------------------------------------------------+---------------------------------------+------------------------------------------------------------------------+
| ``sha256(s)``                 | ``string``                  | ``string``  | Calculates the SHA256 hash of ``s``, returning the result in hexadecimal | ``sha256('abc')``                     | ``'ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad'`` |
+-------------------------------+-----------------------------+-------------+--------------------------------------------------------------------------+---------------------------------------+------------------------------------------------------------------------+
| ``strpos(s, t)``              | 2 x ``string``              | ``int``     | Location of specified substring ``t`` in ``s``                           | ``strpos('high', 'ig')``              | ``2``                                                                  |
+-------------------------------+-----------------------------+-------------+--------------------------------------------------------------------------+---------------------------------------+------------------------------------------------------------------------+
| ``substring(s, r)``           | 2 x ``string``              | ``string``  | Extract substring matching regular expression ``r`` from ``s``.          | ``substring('Thomas', '...$')``       | ``'mas'``                                                              |
|                               |                             |             | See Go ``regexp`` package for details of matching.                       |                                       |                                                                        |
+-------------------------------+-----------------------------+-------------+--------------------------------------------------------------------------+---------------------------------------+------------------------------------------------------------------------+
| ``substring(s, from)``        | ``string``, ``int``         | ``string``  | Extract substring                                                        | ``substring('Thomas', 2)``            | ``'homas'``                                                            |
+-------------------------------+-----------------------------+-------------+--------------------------------------------------------------------------+---------------------------------------+------------------------------------------------------------------------+
| ``substring(s, from, for)``   | ``string``, 2 x ``int``     | ``string``  | Extract substring                                                        | ``substring('Thomas', 2, 3)``         | ``'hom'``                                                              |
+-------------------------------+-----------------------------+-------------+--------------------------------------------------------------------------+---------------------------------------+------------------------------------------------------------------------+
| ``upper(s)``                  | ``string``                  | ``string``  | Convert ``s`` to upper case                                              | ``upper('ÜBer')``                     | ``'ÜBER'``                                                             |
+-------------------------------+-----------------------------+-------------+--------------------------------------------------------------------------+---------------------------------------+------------------------------------------------------------------------+

{'fraction_non_alphanumeric': 0.48825423889362507, 'fraction_numerical': 0.011722080136402387, 'mean_word_length': 0.443529090038969, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 7, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 319, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6926190', 'n_tokens_mistral': 6851, 'n_tokens_neox': 6133, 'n_words': 1585}