<commit_before>8.1 Architecture
================

This page starts by giving a brief "10,000 feet" overview of the general
architecture of EventQL and then gives links to more detailed information about
the respective subsystems.

The primary audience for this page are EventQL developers and users that want to
learn about the internals of EventQL.

### Data Model

EventQL is a distributed database by design and is deployed onto a number of
machines called a "cluster". A cluster consists of many, equally privileged,
EventQL server instances that connect to a coordination service, like ZooKeeper.

The core unit of data storage in EventQL are tables and rows. On the data, it
supports the usual database operations, i.e. inserting, updating and querying
rows.

Each table has a strict schema and a mandatory primary key. This primary key is
used to automatically split a table into many partitions of roughly 500MB.
Each partition is then stored on N servers in the cluster.

Partioning is fully transparent to the user -- from a user perspective
interacting with an EventQL cluter feels just like interacting with an ordinary
SQL database.

Each copy of a partition is stored on the respective server's disk as a log
structured merge tree of "columnar storage tables" (cstables). A cstable is a
container file that stores many rows of a given schema in column-oriented layout.

### Queries

Clients connect to an EventQL cluster to create and manage tables, insert and
update rows into tables and execute queries. When a client wants to executes a
SQL (or MapReduce) query on the data it can sent the query to any server in the
cluster.

To execute a query, a server will first identify all tables referenced from the
query and then, for each table, identify the partitions that need to be scanned
to answer the query (taking into account WHERE restrictions on the primary key).
It will then rewrite the query into shards (in a simple query, one shard per
partition) so that as much work as possible can be performed local to the data. 
Then, all shards of the query are executed in parallel on the servers
that store the respective partitions, minimizing data transfers. On each shard,
the subset of the data that is required to answer the query (i.e. only the
actually referenced columns in the table) is loaded from the cstables on disk.
Finally, all shard results are merged and sent back to the client.

### Consistency & Durability

EventQL is _not_ intendend for transactional workloads but still gives some
consistency guarantees:
<commit_msg>segment files<commit_after>8.1 Architecture
================

This page starts by giving a brief "10,000 feet" overview of the general
architecture of EventQL and then gives links to more detailed information about
the respective subsystems.

The primary audience for this page are EventQL developers and users that want to
learn about the internals of EventQL.

### Data Model

EventQL is a distributed database by design and is deployed onto a number of
machines called a "cluster". A cluster consists of many, equally privileged,
EventQL server instances that connect to a coordination service, like ZooKeeper.

The core unit of data storage in EventQL are tables and rows. On the data, it
supports the usual database operations, i.e. inserting, updating and querying
rows.

Each table has a strict schema and a mandatory primary key. This primary key is
used to automatically split a table into many partitions of roughly 500MB.
Each partition is then stored on N servers in the cluster.

Partioning is fully transparent to the user -- from a user perspective
interacting with an EventQL cluter feels just like interacting with an ordinary
SQL database.

Each copy of a partition is stored on the respective server's disk as a log
structured merge tree of segment files. Each segment file is encoded using the
cstable file format. CSTable is a container file format that stores many rows
of a given schema in column-oriented layout.

### Queries

Clients connect to an EventQL cluster to create and manage tables, insert and
update rows into tables and execute queries. When a client wants to executes a
SQL (or MapReduce) query on the data it can sent the query to any server in the
cluster.

To execute a query, a server will first identify all tables referenced from the
query and then, for each table, identify the partitions that need to be scanned
to answer the query (taking into account WHERE restrictions on the primary key).
It will then rewrite the query into shards (in a simple query, one shard per
partition) so that as much work as possible can be performed local to the data. 
Then, all shards of the query are executed in parallel on the servers
that store the respective partitions, minimizing data transfers. On each shard,
the subset of the data that is required to answer the query (i.e. only the
actually referenced columns in the table) is loaded from the cstables on disk.
Finally, all shard results are merged and sent back to the client.

### Consistency & Durability

EventQL is _not_ intendend for transactional workloads but still gives some
consistency guarantees:

{'fraction_non_alphanumeric': 0.0343289371605896, 'fraction_numerical': 0.003878975950349108, 'mean_word_length': 4.527331189710611, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2948632', 'n_tokens_mistral': 1256, 'n_tokens_neox': 1185, 'n_words': 837}