<commit_before>MANUAL
======
Argos has 2 individual programs, indexer and searcher.

Indexer
-------

Indexer can build or update index from source file.

Indexer has following command line arguments:

* `-c|--config`, a config XML file, the format is described in the section 'Config.XML'. Indexer will try to create a new index if this argument exists, otherwise indexer will try to open an existing index at `--index-dir`.
* `-d|--index-dir`, the directory to store index, the directory must *not* exist if there is a config XML specified.
* `-i|--input-file`, input file name, indexer reads input data from stdin if the file name is `-` or this argument doesn't appear.

### Config.XML ###

The XML file defines index schema, which has root name `<config>`, and a child node `<fields>`.

In node `<fields>`, one or more fields can be defined by adding `<field>` node with following attributes:

* `name`, required, the name of the field, the name must starts with `[A-Za-z_]`, and followed by `[A-Za-z_0-9]*`, no other characters allowed. Names are case-sensitive.
* `type`, required, the type of the field, can be:
    + `integer`, a signed interger number.
    + `float`, a signed 4 bytes float number.
    + `double`, a signed 8 bytes double number.
    + `geolocation`, a pair of 4 bytes float numbers, first is latitude, then langitude.
    + `string`, a string with UTF-8 encoding. NOTE: Argos doesn't enforce character encoding, but indexing may fail if the string is not correctly encoded.
* `bytes`, optional, only usable when `type="integer"`, can be 1, 2, 4, or 8, default is 8.
* `multi`, optional, only usable when type is not `"string"`, can be `true` or `false`, the field is multi-value when `multi="true"`.
* `store`, optional, can be `"true"` or `"false"`, if `store="false"`, the field itself is not stored in the index and the field value cannot be retrieved, default is `"true"`.
* `index`, optional, can be `"true"` or `"false"`, the field will be indexed if `index="true"`. Different type of fields will be indexed differently.
* `namespace`, optional, only usable when `index="true"`, 2 or more fields can use same namespace, which means the HAS() will search terms in all of them. By default each field has different namespace; Fields with `namespace="-"` can be searched via term query.
* `analyzer`, optional, only usable when `index="true"` and `type="string"`, indicates how indexer break text into terms, can be:
    + `token`, seperate text by space, without more processes.
    + `text`, replace all punctuations with space; seperate text by space; convert all Latin letters to lower case; remove accent marks; and break CJK text into single ideograms.
    + `bigram`, same as above, with CJK text does not only generate single ideograms, but also BiGrams.

### Input File Format ###

The input file is basically CSV file, each line contains a single document.

NOTE: The first field of each document *must* be the primary key, which is implicit and *not* in config.xml file.

There are some exceptions from standard CSV format:

* Multi-value field has format `[v1,v2...]`, no quotes, empty field is `[]`.
* geolocation field has format `<lat,long>`, no quotes.

Indexer simply uses `std::getline()` to read from input file, so the maxium length of each document depends on the compiler and stdc++ lib. It should be ok to handle lines over 1M bytes on most systems.


Searcher
--------

Searcher has HTTP interface that supports GET method, POST is not supported.

Maximium request size is 64KB, this is hardcoded and can only be changed by re-compilation.

Searcher supports multiple indices at same time, each index has their own name, and operations to the index starts with prefix '/index_name'.

Searcher has following command-line arguments:

* `-d|--index-dir`, index directory, can be used multiple times to support multiple indices.
* `-a|--listen-addr`, listening address, default to `0.0.0.0`.
* `-p|--listen-port`, listening port, default to `8765`.
* `-r|--doc-root`, root directory for static contents, including XSLT files, default to `.`.

    You don't need it if you're not connecting to Argos with browsers.

* `-t|--threads`, number of threads to serve HTTP request, default to `3`.

    Argos threading model is effective enough so you should not use too many threads. A reasonable value is the number of CPU.

* `-l|--log-config`, log configuration file to be used, which is a [log4cplus](log4cplus.sf.net) log properties file.

    All logs are printed to stdout if the argument is missing.

### Query ###

URL prefix is `/index_name/query`, following parameters are supported:

* `m`, match condition
* `f`, filter expression
* `s`, sorting criteria
* `nr`, number of results to return
* `sk`, number of top results to skip, after sorting
* `fl`, returned fields
* `h`, histograms
* `fmt`, return format

#### Match condition ####

Match condition can be combination of following operators:

* `"text"`, return documents contains `"text"` in `namespace="-"`, this condition will be expanded into a serial of terms to match, using "text" analyzer.
* `AND(c1,c2,...cn)`, return documents match *all* conditions
* `OR(c1,c2,...cn)`, return documents match *any* conditions
* `HAS(field,"text)`, return documents have `field` match `"text"`, text is expanded with the analyzer specified for field, in config.xml
* `INFO([field,]* "text")`, this is a special query, it matches any documents, in addition, it retrieves match info for field and text, which can be used later in filter and sorting criteria.

Operators are case-sensitive.

#### Filter ####

Filter is a single expression, each document returned in match condition will be checked against with the expression, and all document with expression value 0 will be discarded.

Expression syntax is in prefix form, following functions are supported:

* Named constants, include `E`, `PI`, and `NOW`, `NOW` is a double number of seconds between current time and the epoch (1970-1-1).
* Math functions, include `ADD`, `SUB`, `MUL`, and `DIV`.
* Comparison, include `LT`, `LE`, `GT`, `GT`, `EQ`, `NE`, returns `0` or `1`.
* Logical functions, include `AND`, `OR`, `NOT`, `XOR`, these functions operate bit-wise for integral oprands, otherwise logical.
* Condition function, `IF(condition,true_clause,false_clause)` returns value of `true_clause` when `condition` evaluates to non-zero, otherwise `false_clause`.
* `LEN(v)`, returns number elements of array or length of string.
* `AT(array,idx)`, returns `array[idx]`
* `FIND(array,element)`, returns the index of element in array
* `DIST(geoloc1,geoloc2)`, returns spherical distance between geoloc1 and geoloc2 in meter, using radius of Earth.
* `RANGE(v,anchor1,anchor2,...anchorn)`, return `n` if `v` in `[anchorn, anchorn+1)`
* Document specific operators, start with '@'
    + `@DOCCOUNT`, returns number of documents in the index
    + `@DL(field)`, return number of terms in the field for current document, this value depends on the analyzer the field uses
    + `@HAS(field,"text")`, this operator is used with `INFO(field,"text")` in match condition, returns 0 or 1 if the field contains `"text"`. NOTE: Without INFO part in match condition, this operator may return 0 even if text is actually in the field

#### Sorting Criteria ####

Argos supports multiple sorting criteria, each criterion is `expression,[ASC|DESC]`, the expression is used to calculate the sorting key, supports same operators as in filter. ASC to sort in ascending order, DESC to descending.

Addition sorting criterion is used when former sorting keys are equal.

<commit_msg>Update manual for MINDIST operator<commit_after>MANUAL
======
Argos has 2 individual programs, indexer and searcher.

Indexer
-------

Indexer can build or update index from source file.

Indexer has following command line arguments:

* `-c|--config`, a config XML file, the format is described in the section 'Config.XML'. Indexer will try to create a new index if this argument exists, otherwise indexer will try to open an existing index at `--index-dir`.
* `-d|--index-dir`, the directory to store index, the directory must *not* exist if there is a config XML specified.
* `-i|--input-file`, input file name, indexer reads input data from stdin if the file name is `-` or this argument doesn't appear.

### Config.XML ###

The XML file defines index schema, which has root name `<config>`, and a child node `<fields>`.

In node `<fields>`, one or more fields can be defined by adding `<field>` node with following attributes:

* `name`, required, the name of the field, the name must starts with `[A-Za-z_]`, and followed by `[A-Za-z_0-9]*`, no other characters allowed. Names are case-sensitive.
* `type`, required, the type of the field, can be:
    + `integer`, a signed interger number.
    + `float`, a signed 4 bytes float number.
    + `double`, a signed 8 bytes double number.
    + `geolocation`, a pair of 4 bytes float numbers, first is latitude, then langitude.
    + `string`, a string with UTF-8 encoding. NOTE: Argos doesn't enforce character encoding, but indexing may fail if the string is not correctly encoded.
* `bytes`, optional, only usable when `type="integer"`, can be 1, 2, 4, or 8, default is 8.
* `multi`, optional, only usable when type is not `"string"`, can be `true` or `false`, the field is multi-value when `multi="true"`.
* `store`, optional, can be `"true"` or `"false"`, if `store="false"`, the field itself is not stored in the index and the field value cannot be retrieved, default is `"true"`.
* `index`, optional, can be `"true"` or `"false"`, the field will be indexed if `index="true"`. Different type of fields will be indexed differently.
* `namespace`, optional, only usable when `index="true"`, 2 or more fields can use same namespace, which means the HAS() will search terms in all of them. By default each field has different namespace; Fields with `namespace="-"` can be searched via term query.
* `analyzer`, optional, only usable when `index="true"` and `type="string"`, indicates how indexer break text into terms, can be:
    + `token`, seperate text by space, without more processes.
    + `text`, replace all punctuations with space; seperate text by space; convert all Latin letters to lower case; remove accent marks; and break CJK text into single ideograms.
    + `bigram`, same as above, with CJK text does not only generate single ideograms, but also BiGrams.

### Input File Format ###

The input file is basically CSV file, each line contains a single document.

NOTE: The first field of each document *must* be the primary key, which is implicit and *not* in config.xml file.

There are some exceptions from standard CSV format:

* Multi-value field has format `[v1,v2...]`, no quotes, empty field is `[]`.
* geolocation field has format `<lat,long>`, no quotes.

Indexer simply uses `std::getline()` to read from input file, so the maxium length of each document depends on the compiler and stdc++ lib. It should be ok to handle lines over 1M bytes on most systems.


Searcher
--------

Searcher has HTTP interface that supports GET method, POST is not supported.

Maximium request size is 64KB, this is hardcoded and can only be changed by re-compilation.

Searcher supports multiple indices at same time, each index has their own name, and operations to the index starts with prefix '/index_name'.

Searcher has following command-line arguments:

* `-d|--index-dir`, index directory, can be used multiple times to support multiple indices.
* `-a|--listen-addr`, listening address, default to `0.0.0.0`.
* `-p|--listen-port`, listening port, default to `8765`.
* `-r|--doc-root`, root directory for static contents, including XSLT files, default to `.`.

    You don't need it if you're not connecting to Argos with browsers.

* `-t|--threads`, number of threads to serve HTTP request, default to `3`.

    Argos threading model is effective enough so you should not use too many threads. A reasonable value is the number of CPU.

* `-l|--log-config`, log configuration file to be used, which is a [log4cplus](log4cplus.sf.net) log properties file.

    All logs are printed to stdout if the argument is missing.

### Query ###

URL prefix is `/index_name/query`, following parameters are supported:

* `m`, match condition
* `f`, filter expression
* `s`, sorting criteria
* `nr`, number of results to return
* `sk`, number of top results to skip, after sorting
* `fl`, returned fields
* `h`, histograms
* `fmt`, return format

#### Match condition ####

Match condition can be combination of following operators:

* `"text"`, return documents contains `"text"` in `namespace="-"`, this condition will be expanded into a serial of terms to match, using "text" analyzer.
* `AND(c1,c2,...cn)`, return documents match *all* conditions
* `OR(c1,c2,...cn)`, return documents match *any* conditions
* `HAS(field,"text)`, return documents have `field` match `"text"`, text is expanded with the analyzer specified for field, in config.xml
* `INFO([field,]* "text")`, this is a special query, it matches any documents, in addition, it retrieves match info for field and text, which can be used later in filter and sorting criteria.

Operators are case-sensitive.

#### Filter ####

Filter is a single expression, each document returned in match condition will be checked against with the expression, and all document with expression value 0 will be discarded.

Expression syntax is in prefix form, following functions are supported:

* Named constants, include `E`, `PI`, and `NOW`, `NOW` is a double number of seconds between current time and the epoch (1970-1-1).
* Math functions, include `ADD`, `SUB`, `MUL`, and `DIV`.
* Comparison, include `LT`, `LE`, `GT`, `GT`, `EQ`, `NE`, returns `0` or `1`.
* Logical functions, include `AND`, `OR`, `NOT`, `XOR`, these functions operate bit-wise for integral oprands, otherwise logical.
* Condition function, `IF(condition,true_clause,false_clause)` returns value of `true_clause` when `condition` evaluates to non-zero, otherwise `false_clause`.
* `LEN(v)`, returns number elements of array or length of string.
* `AT(array,idx)`, returns `array[idx]`
* `FIND(array,element)`, returns the index of element in array
* `DIST(geoloc1,geoloc2)`, returns spherical distance between geoloc1 and geoloc2 in meter, using radius of Earth.
* `MINDIST(geoloc1,[geoloc2,...])`, returns the nearest spherical distance between geoloc1 and all geolocations in the second parameter, which must be an array of gelocations.
* `RANGE(v,anchor1,anchor2,...anchorn)`, return `n` if `v` in `[anchorn, anchorn+1)`
* Document specific operators, start with '@'
    + `@DOCCOUNT`, returns number of documents in the index
    + `@DL(field)`, return number of terms in the field for current document, this value depends on the analyzer the field uses
    + `@HAS(field,"text")`, this operator is used with `INFO(field,"text")` in match condition, returns 0 or 1 if the field contains `"text"`. NOTE: Without INFO part in match condition, this operator may return 0 even if text is actually in the field

#### Sorting Criteria ####

Argos supports multiple sorting criteria, each criterion is `expression,[ASC|DESC]`, the expression is used to calculate the sorting key, supports same operators as in filter. ASC to sort in ascending order, DESC to descending.

Addition sorting criterion is used when former sorting keys are equal.


{'fraction_non_alphanumeric': 0.10892243892956652, 'fraction_numerical': 0.006933195101406078, 'mean_word_length': 4.426863572433192, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 4}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3557151', 'n_tokens_mistral': 4360, 'n_tokens_neox': 4187, 'n_words': 2190}