<commit_before># Charlatan

[![Build Status](https://travis-ci.org/BatchLabs/charlatan.svg?branch=master)](https://travis-ci.org/BatchLabs/charlatan)

**Charlatan** is a query engine for lists or streams of records in different
formats. It natively supports CSV and JSON formats but can easily be extended
to others.

It supports an SQL-like query language that is defined below. Queries are
applied to records to extract values depending on zero or more criteria.

## Query Syntax

```
SELECT <fields> FROM <source> [ WHERE <value> ] [ STARTING AT <index> ]
```

- `<fields>` is a list of comma-separated field names. Each field name must
  exist in the source. When reading CSV files, the field names are the column
  names, while when reading JSON they represent keys.
- `<source>` is the filename from which the data is read. The API is agnostique
  on this and one can implement support for any source type.
- `<value>` is a SQL-like value, which can be either a constant (e.g.
  `WHERE 1`), a field (e.g. `WHERE archived`) or any operation using comparison
  operators (`=`, `!=`, `<`, `<=`, `>`, `>=`, `AND`, `OR`) and optionally
  parentheses (e.g. `WHERE (foo > 2) AND (bar = "yo")`). The parser allows to
  use `&&` instead of `AND` and `||` instead of `OR`. It also support inclusive
  range tests, like `WHERE age BETWEEN 20 AND 30`.
- `<index>` can be used to skip the first N records.

Constant values include strings, integers, floats, booleans and the `null`
value.
<commit_msg>'LIMIT' docs<commit_after># Charlatan

[![Build Status](https://travis-ci.org/BatchLabs/charlatan.svg?branch=master)](https://travis-ci.org/BatchLabs/charlatan)

**Charlatan** is a query engine for lists or streams of records in different
formats. It natively supports CSV and JSON formats but can easily be extended
to others.

It supports an SQL-like query language that is defined below. Queries are
applied to records to extract values depending on zero or more criteria.

## Query Syntax

```
SELECT <fields> FROM <source> [ WHERE <value> ] [ STARTING AT <index> ] [ LIMIT <count> ]
```

- `<fields>` is a list of comma-separated field names. Each field name must
  exist in the source. When reading CSV files, the field names are the column
  names, while when reading JSON they represent keys.
- `<source>` is the filename from which the data is read. The API is agnostique
  on this and one can implement support for any source type.
- `<value>` is a SQL-like value, which can be either a constant (e.g.
  `WHERE 1`), a field (e.g. `WHERE archived`) or any operation using comparison
  operators (`=`, `!=`, `<`, `<=`, `>`, `>=`, `AND`, `OR`) and optionally
  parentheses (e.g. `WHERE (foo > 2) AND (bar = "yo")`). The parser allows to
  use `&&` instead of `AND` and `||` instead of `OR`. It also support inclusive
  range tests, like `WHERE age BETWEEN 20 AND 30`.
- `STARTING AT <index>` can be used to skip the first N records.
- `LIMIT <count>` can be used to keep only the first N matched records.

Constant values include strings, integers, floats, booleans and the `null`
value.

{'fraction_non_alphanumeric': 0.11719002913564261, 'fraction_numerical': 0.0038847523470378763, 'mean_word_length': 4.065573770491803, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 27, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1580649', 'n_tokens_mistral': 963, 'n_tokens_neox': 911, 'n_words': 436}