<commit_before># Virtual Schema API Documentation

## Table of Contents
- [Introduction](#introduction)
- [Requests and Responses](#requests-and-responses)
  - [Create Virtual Schema](#create-virtual-schema)
  - [Refresh](#refresh)
  - [Set Properties](#set-properties)
  - [Drop Virtual Schema](#drop-virtual-schema)
  - [Get Capabilities](#get-capabilities)
  - [Pushdown](#pushdown)
- [Embedded Commonly Used Json Elements](#embedded-commonly-used-json-elements)
  - [Schema Metadata Info](#schema-metadata-info)
  - [Schema Metadata](#schema-metadata)
- [Expressions](#expressions)
  - [Table](#table)
  - [Join](#join)
  - [Column Lookup](#column-lookup)
  - [Literal](#literal)
  - [Predicates](#predicates)
  - [Scalar Functions](#scalar-functions)
  - [Aggregate Functions](#aggregate-functions)

## Introduction

There are the following request and response types:

| Type                        | Called ...     |
| :-------------------------- | :------------- |
| **Create Virtual Schema**   | ... for each `CREATE VIRTUAL SCHEMA ...` statement |
| **Refresh**                 | ... for each `ALTER VIRTUAL SCHEMA ... REFRESH ...` statement. |
| **Set Properties**          | ... for each `ALTER VIRTUAL SCHEMA ... SET ...` statement. |
| **Drop Virtual Schema**     | ... for each `DROP VIRTUAL SCHEMA ...` statement. |
| **Get Capabilities**        | ... whenever a virtual table is queried in a `SELECT` statement. |
| **Pushdown**                | ... whenever a virtual table is queried in a `SELECT` statement. |

We describe each of the types in the following sections.

**Please note:** To keep the documentation concise we defined the elements which are commonly in separate sections below, e.g. `schemaMetadataInfo` and `schemaMetadata`.

## Requests and Responses

### Create Virtual Schema

Informs the Adapter about the request to create a Virtual Schema, and asks the Adapter for the metadata (tables and columns).

The Adapter is allowed to throw an Exception if the user missed to provide mandatory properties or in case of any other problems (e.g. connectivity).

**Request:**

```json
{
    "type": "createVirtualSchema",
    "schemaMetadataInfo": {
        ...
    }
}
```

**Response:**

```json
{
    "type": "createVirtualSchema",
    "schemaMetadata": {
        ...
    }
}
```

Notes
* `schemaMetadata` is mandatory. However, it is allowed to contain no tables.


### Refresh

Request to refresh the metadata for the whole Virtual Schema, or for specified tables.

**Request:**

```json
{
    "type": "refresh",
    "schemaMetadataInfo": {
        ...
    },
    "requestedTables": ["T1", "T2"]
}
```

Notes
* `requestedTables` is optional. If existing, only the specified tables shall be refreshed. The specified tables do not have to exist, it just tell Adapter to update these tables (which might be changed, deleted, added, or non-existing).

**Response:**

```json
{
    "type": "refresh",
    "schemaMetadata": {
        ...
    },
    "requestedTables": ["T1", "T2"]
}
```

Notes
* `schemaMetadata` is optional. It can be skipped if the adapter does not want to refresh (e.g. because he detected that there is no change).
* `questedTables` must exist if and only if the element existed in the request. The values must be the same as in the request (to make sure that Adapter only refreshed these tables).

### Set Properties

Request to set properties. The Adapter can decide whether he needs to send back new metadata. The Adapter is allowed to throw an Exception if the user provided invalid properties or in case of any other problems (e.g. connectivity).

**Request:**

```json
{
    "type": "setProperties",
    "schemaMetadataInfo": {
        ...
    },<commit_msg>Fixed typo<commit_after># Virtual Schema API Documentation

## Table of Contents
- [Introduction](#introduction)
- [Requests and Responses](#requests-and-responses)
  - [Create Virtual Schema](#create-virtual-schema)
  - [Refresh](#refresh)
  - [Set Properties](#set-properties)
  - [Drop Virtual Schema](#drop-virtual-schema)
  - [Get Capabilities](#get-capabilities)
  - [Pushdown](#pushdown)
- [Embedded Commonly Used Json Elements](#embedded-commonly-used-json-elements)
  - [Schema Metadata Info](#schema-metadata-info)
  - [Schema Metadata](#schema-metadata)
- [Expressions](#expressions)
  - [Table](#table)
  - [Join](#join)
  - [Column Lookup](#column-lookup)
  - [Literal](#literal)
  - [Predicates](#predicates)
  - [Scalar Functions](#scalar-functions)
  - [Aggregate Functions](#aggregate-functions)

## Introduction

There are the following request and response types:

| Type                        | Called ...     |
| :-------------------------- | :------------- |
| **Create Virtual Schema**   | ... for each `CREATE VIRTUAL SCHEMA ...` statement |
| **Refresh**                 | ... for each `ALTER VIRTUAL SCHEMA ... REFRESH ...` statement. |
| **Set Properties**          | ... for each `ALTER VIRTUAL SCHEMA ... SET ...` statement. |
| **Drop Virtual Schema**     | ... for each `DROP VIRTUAL SCHEMA ...` statement. |
| **Get Capabilities**        | ... whenever a virtual table is queried in a `SELECT` statement. |
| **Pushdown**                | ... whenever a virtual table is queried in a `SELECT` statement. |

We describe each of the types in the following sections.

**Please note:** To keep the documentation concise we defined the elements which are commonly in separate sections below, e.g. `schemaMetadataInfo` and `schemaMetadata`.

## Requests and Responses

### Create Virtual Schema

Informs the Adapter about the request to create a Virtual Schema, and asks the Adapter for the metadata (tables and columns).

The Adapter is allowed to throw an Exception if the user missed to provide mandatory properties or in case of any other problems (e.g. connectivity).

**Request:**

```json
{
    "type": "createVirtualSchema",
    "schemaMetadataInfo": {
        ...
    }
}
```

**Response:**

```json
{
    "type": "createVirtualSchema",
    "schemaMetadata": {
        ...
    }
}
```

Notes
* `schemaMetadata` is mandatory. However, it is allowed to contain no tables.


### Refresh

Request to refresh the metadata for the whole Virtual Schema, or for specified tables.

**Request:**

```json
{
    "type": "refresh",
    "schemaMetadataInfo": {
        ...
    },
    "requestedTables": ["T1", "T2"]
}
```

Notes
* `requestedTables` is optional. If existing, only the specified tables shall be refreshed. The specified tables do not have to exist, it just tell Adapter to update these tables (which might be changed, deleted, added, or non-existing).

**Response:**

```json
{
    "type": "refresh",
    "schemaMetadata": {
        ...
    },
    "requestedTables": ["T1", "T2"]
}
```

Notes
* `schemaMetadata` is optional. It can be skipped if the adapter does not want to refresh (e.g. because he detected that there is no change).
* `requestedTables` must exist if and only if the element existed in the request. The values must be the same as in the request (to make sure that Adapter only refreshed these tables).

### Set Properties

Request to set properties. The Adapter can decide whether he needs to send back new metadata. The Adapter is allowed to throw an Exception if the user provided invalid properties or in case of any other problems (e.g. connectivity).

**Request:**

```json
{
    "type": "setProperties",
    "schemaMetadataInfo": {
        ...
    },
{'fraction_non_alphanumeric': 0.1397979797979798, 'fraction_numerical': 0.0010774410774410774, 'mean_word_length': 3.1532438478747205, 'pattern_counts': {'":': 24, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4573527', 'n_tokens_mistral': 2123, 'n_tokens_neox': 2014, 'n_words': 783}