<commit_before>4.1 Timeseries & Logs
=====================

EventQL is an excellent fit to store high volumes of timeseries-structured data
like metric, tracking data, logfiles, sensor measurements or other
machine-generated data.

### High Volume Timeseries Data

To get the best performance when handling large arrival rates of new events,
consider these settings.

#### 1. Use a datetime field as the partitioning key

You should use a datetime field as the first part of the primary key for time
series tables. Since primary keys are automatically unique, the usual pattern
when setting up a timeseries table is to use a primary key that consists of the
event time and a unique event id.

#### 2. Set the partition size hint

EventQL will dynamically re-partition the table as you add more data to keep
each partition in the 500MB to 1GB range. You can optionally give EventQL a hint
about the amount of writes you expect. Specifying this hint will minimize the
number of re-partitioning operations and allow you to ramp up the insert rate
much quicker.

To set the hint you have to specifty a `finite_partition_size` when creating a
table.

    CREATE TABLE high_volume_logging_data (
      time            DATETIME,
      event_id        STRING,
      ...
      PRIMARY KEY(time, event_id)
    ) WITH finite_partition_size = 600000000;

Use this formula to calculate the value:

    $finite_partition_size = (750MB / $expected_new_data_per_day) * 86400000000

So, for example, if you expect around 10GB of new data a day, `7200000000` would be
a good value. If you expect 1000GB of new data a day, `60000000` is a good value.
If you expect 10TB of new data a day, set the finite partition size to `10000000`.

If your estimation is off, it will not cause any problems. You can update the
hint at any time. To read more about the finite partition size setting check out
the [Partitioning page] (../../tables/partitioning/).

#### 3. Enable the async_split option

Another recommended setting for handling high volumes of inserts is enabling the
"enable_async_split" option. This allows a splitting partition to immediately
redirect all inserts to the new targets once it has started the split.
<commit_msg>formatting<commit_after>4.1 Timeseries & Logs
=====================

EventQL is an excellent fit to store high volumes of timeseries-structured data
like metric, tracking data, logfiles, sensor measurements or other
machine-generated data.

### High Volume Timeseries Data

To get the best performance when handling large arrival rates of new events,
consider these settings.

#### 1. Use a datetime field as the partitioning key

You should use a datetime field as the first part of the primary key for time
series tables. Since primary keys are automatically unique, the usual pattern
when setting up a timeseries table is to use a primary key that consists of the
event time and a unique event id.

#### 2. Set the partition size hint

EventQL will dynamically re-partition the table as you add more data to keep
each partition in the 500MB to 1GB range. You can optionally give EventQL a hint
about the amount of writes you expect. Specifying this hint will minimize the
number of re-partitioning operations and allow you to ramp up the insert rate
much quicker.

To set the hint you have to specifty a `finite_partition_size` when creating a
table.

    CREATE TABLE high_volume_logging_data (
      time            DATETIME,
      event_id        STRING,
      ...
      PRIMARY KEY(time, event_id)
    ) WITH finite_partition_size = 600000000;

Use this formula to calculate the value:

    $finite_partition_size = (750MB / $expected_new_data_per_day) * 86400000000

So, for example, if you expect around 10GB of new data a day, `7200000000` would be
a good value. If you expect 1000GB of new data a day, `60000000` is a good value.
If you expect 10TB of new data a day, set the finite partition size to `10000000`.

If your estimation is off, it will not cause any problems. You can update the
hint at any time. To read more about the finite partition size setting check out
the [Partitioning page] (../../tables/partitioning/).

#### 3. Enable the async_split option

Another recommended setting for handling high volumes of inserts is enabling the
`enable_async_split` switch. This allows a splitting partition to immediately
redirect all inserts to the new targets once it has started the split.

{'fraction_non_alphanumeric': 0.05295315682281059, 'fraction_numerical': 0.02987101154107264, 'mean_word_length': 3.9607182940516275, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 10, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4323455', 'n_tokens_mistral': 1236, 'n_tokens_neox': 1087, 'n_words': 654}