<commit_before><!-- START doctoc generated TOC please keep comment here to allow auto update -->
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
**Table of Contents**  *generated with [DocToc](https://github.com/thlorenz/doctoc)*

- [API](#api)
  - [Initialisation](#initialisation)
    - [Make an index](#make-an-index)
    - [Access the API](#access-the-api)
  - [Altering the index](#altering-the-index)
    - [DELETE](#delete)
    - [PUT](#put)
  - [Composable querying](#composable-querying)
    - [AND](#and)
    - [DOCUMENTS](#documents)
    - [NOT](#not)
    - [OR](#or)
  - [Searching](#searching)
    - [GET](#get)
    - [SEARCH](#search)
  - [Tokenisation](#tokenisation)
    - [DICTIONARY](#dictionary)
  - [Aggregation](#aggregation)
    - [BUCKET](#bucket)
    - [BUCKETFILTER](#bucketfilter)
    - [DISTINCT](#distinct)
  - [Accessing the underlying index](#accessing-the-underlying-index)
    - [INDEX](#index)
- [FAQ](#faq)
  - [How do I get my data into search-index?](#how-do-i-get-my-data-into-search-index)
    - [Replicate an index](#replicate-an-index)
    - [Create a new index](#create-a-new-index)
  - [What is the difference between AND, GET and SEARCH?](#what-is-the-difference-between-and-get-and-search)
  - [How do I get out entire documents and not just document IDs?](#how-do-i-get-out-entire-documents-and-not-just-document-ids)
  - [How do I search on specific fields?](#how-do-i-search-on-specific-fields)
  - [How do I compose queries?](#how-do-i-compose-queries)
  - [How do I perform a simple aggregation on a field?](#how-do-i-perform-a-simple-aggregation-on-a-field)
    - [Get a list of unique values for a field](#get-a-list-of-unique-values-for-a-field)
    - [Get a set of document ids per unique field value](#get-a-set-of-document-ids-per-unique-field-value)
    - [Get counts per unique field value](#get-counts-per-unique-field-value)
    - [Define custom "buckets"](#define-custom-buckets)
    - [Combine an aggregation with a search](#combine-an-aggregation-with-a-search)
  - [How do I make a simple typeahead / autosuggest / matcher](#how-do-i-make-a-simple-typeahead--autosuggest--matcher)

<!-- END doctoc generated TOC please keep comment here to allow auto update -->

# API

## Initialisation
### Make an index

`si([options[, callback]])`

```javascript
import si from 'search-index'

// creates a DB called "myDB" using levelDB (node.js), or indexedDB (browser)
const db = si({ name: 'myDB' })
```

In some cases you will want to start operating on the database
instentaneously. In these cases you can wait for the callback:

```javascript
import si from 'search-index'

// creates a DB called "myDB" using levelDB (node.js), or indexedDB (browser)
si({ name: 'myDB' }, (err, db) => {
  // db is guaranteed to be open and available
})
```

### Access the API

Using one of the methods above you should now have an index called
`db` (or another name of your choosing). If you want to call, say
`GET` you could do either

```javascript
db.SEARCH('searchterm')
```

or

```javascript
const { SEARCH, GET } = idx
SEARCH('searchterm')
GET('searchterm')
```


## Altering the index
### DELETE

`DELETE([ ...Promise ]).then(result)`

Deletes all objects by ID

### PUT

`PUT([ ...Promise ]).then(result)`

Add objects to database


## Composable querying
### AND

`AND([ ...Promise ]).then(result)`

Boolean AND. Return IDs of objects that have prop.A AND prop.B

### DOCUMENTS

`DOCUMENTS([ ...id ]).then(result)`

Get documents by ID

### NOT

`NOT([ ...Promise ]).then(result)`

Where A and B are sets, `NOT` Returns the ids of objects that are
present in A, but not in B.

### OR

`OR([ ...Promise ]).then(result)`

Return ids of objects that are in one or more of the query clauses


## Searching
### GET

`GET(property).then(result)`

`GET` returns all object ids for objects that contain the given
property, aggregated by object id.

For example get all names between `h` and `l`:

```javascript
GET({ gte: 'h', lte: 'l' }).then(result)
```

Or to get all objects that have a `name` property that begins with 'h'

```javascript
GET('h').then(result)
```

### SEARCH

`SEARCH([ ...Promise ]).then(result)`

Search the database and get documents back. 

```javascript
  SEARCH(
    OR('bananas', 'different'),  // search clauses can be nested promises
    'coolness'                       // or strings (defaults to GET)
  ).then(result)
```

## Tokenisation
### DICTIONARY

`DICTIONARY(options).then(result)`

Options:

* gte : greater than or equal to
* lte : less than or equal to

Gets an array of tokens stored in the index. Usefull for i.e. auto-complete or auto-suggest scenarios.

Examples on usage:

```javascript
// get all tokens in the index
DICTIONARY().then( /* array of tokens */ )

// get all tokens in the body.text field
DICTIONARY('body.text').then( /* array of tokens */ )

// get tokens in the body.text field that starts with 'cool'
DICTIONARY('body.text.cool').then( /* array of tokens */ )

// you can also use gte/lte ("greater/less than or equal")
DICTIONARY({
  gte: 'body.text.a',
  lte: 'body.text.g'
}).then( /* array of tokens */ )
```

## Aggregation
### BUCKET

`BUCKET([ ...Promise ]).then(result)`

Return IDs of objects that match the query

### BUCKETFILTER

`BUCKETFILTER([ ...bucket ], filter query).then(result)`

The first argument is an array of buckets, the second is an expression
that filters each bucket

### DISTINCT

`DISTINCT(options).then(result)`

`DISTINCT` returns every value in the db that is greater than equal
to `gte` and less than or equal to `lte` (sorted alphabetically)

For example- get all names between `h` and `l`:

```javascript
DISTINCT({ gte: 'h', lte: 'l' }).then(result)
```

## Accessing the underlying index
### INDEX

`INDEX`<commit_msg>added strap lines<commit_after><!-- START doctoc generated TOC please keep comment here to allow auto update -->
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
**Table of Contents**  *generated with [DocToc](https://github.com/thlorenz/doctoc)*

- [API](#api)
  - [Initialisation](#initialisation)
    - [Make an index](#make-an-index)
    - [Access the API](#access-the-api)
  - [Altering the index](#altering-the-index)
    - [DELETE](#delete)
    - [PUT](#put)
  - [Composable querying](#composable-querying)
    - [AND](#and)
    - [DOCUMENTS](#documents)
    - [NOT](#not)
    - [OR](#or)
  - [Searching](#searching)
    - [GET](#get)
    - [SEARCH](#search)
  - [Tokenisation](#tokenisation)
    - [DICTIONARY](#dictionary)
  - [Aggregation](#aggregation)
    - [BUCKET](#bucket)
    - [BUCKETFILTER](#bucketfilter)
    - [DISTINCT](#distinct)
  - [Accessing the underlying index](#accessing-the-underlying-index)
    - [INDEX](#index)
- [FAQ](#faq)
  - [How do I get my data into search-index?](#how-do-i-get-my-data-into-search-index)
    - [Replicate an index](#replicate-an-index)
    - [Create a new index](#create-a-new-index)
  - [What is the difference between AND, GET and SEARCH?](#what-is-the-difference-between-and-get-and-search)
  - [How do I get out entire documents and not just document IDs?](#how-do-i-get-out-entire-documents-and-not-just-document-ids)
  - [How do I search on specific fields?](#how-do-i-search-on-specific-fields)
  - [How do I compose queries?](#how-do-i-compose-queries)
  - [How do I perform a simple aggregation on a field?](#how-do-i-perform-a-simple-aggregation-on-a-field)
    - [Get a list of unique values for a field](#get-a-list-of-unique-values-for-a-field)
    - [Get a set of document ids per unique field value](#get-a-set-of-document-ids-per-unique-field-value)
    - [Get counts per unique field value](#get-counts-per-unique-field-value)
    - [Define custom "buckets"](#define-custom-buckets)
    - [Combine an aggregation with a search](#combine-an-aggregation-with-a-search)
  - [How do I make a simple typeahead / autosuggest / matcher](#how-do-i-make-a-simple-typeahead--autosuggest--matcher)

<!-- END doctoc generated TOC please keep comment here to allow auto update -->

# API

## Initialisation

### Make an index

`si([options[, callback]])`

```javascript
import si from 'search-index'

// creates a DB called "myDB" using levelDB (node.js), or indexedDB (browser)
const db = si({ name: 'myDB' })
```

In some cases you will want to start operating on the database
instentaneously. In these cases you can wait for the callback:

```javascript
import si from 'search-index'

// creates a DB called "myDB" using levelDB (node.js), or indexedDB (browser)
si({ name: 'myDB' }, (err, db) => {
  // db is guaranteed to be open and available
})
```

### Access the API

Using one of the methods above you should now have an index called
`db` (or another name of your choosing). If you want to call, say
`GET` you could do either

```javascript
db.SEARCH('searchterm')
```

or

```javascript
const { SEARCH, GET } = idx
SEARCH('searchterm')
GET('searchterm')
```


## Altering the index

***Add, update or delete data from the index***

### DELETE

`DELETE([ ...Promise ]).then(result)`

Deletes all objects by ID

### PUT

`PUT([ ...Promise ]).then(result)`

Add objects to database


## Composable querying

***These query functions can be mixed together in any combination to make powerful and expressive queries***

### AND

`AND([ ...Promise ]).then(result)`

Boolean AND. Return IDs of objects that have prop.A AND prop.B

### DOCUMENTS

`DOCUMENTS([ ...id ]).then(result)`

Get documents by ID

### NOT

`NOT([ ...Promise ]).then(result)`

Where A and B are sets, `NOT` Returns the ids of objects that are
present in A, but not in B.

### OR

`OR([ ...Promise ]).then(result)`

Return ids of objects that are in one or more of the query clauses


## Searching

***These functions allow you to search in your corpus for keywords***

### GET

`GET(property).then(result)`

`GET` returns all object ids for objects that contain the given
property, aggregated by object id.

For example get all names between `h` and `l`:

```javascript
GET({ gte: 'h', lte: 'l' }).then(result)
```

Or to get all objects that have a `name` property that begins with 'h'

```javascript
GET('h').then(result)
```

### SEARCH

`SEARCH([ ...Promise ]).then(result)`

Search the database and get documents back. 

```javascript
  SEARCH(
    OR('bananas', 'different'),  // search clauses can be nested promises
    'coolness'                       // or strings (defaults to GET)
  ).then(result)
```

## Tokenisation

***Tokenisation allows you to create functionality based on the set of tokens that is in the index such as autosuggest or word clouds***

### DICTIONARY

`DICTIONARY(options).then(result)`

Options:

* gte : greater than or equal to
* lte : less than or equal to

Gets an array of tokens stored in the index. Usefull for i.e. auto-complete or auto-suggest scenarios.

Examples on usage:

```javascript
// get all tokens in the index
DICTIONARY().then( /* array of tokens */ )

// get all tokens in the body.text field
DICTIONARY('body.text').then( /* array of tokens */ )

// get tokens in the body.text field that starts with 'cool'
DICTIONARY('body.text.cool').then( /* array of tokens */ )

// you can also use gte/lte ("greater/less than or equal")
DICTIONARY({
  gte: 'body.text.a',
  lte: 'body.text.g'
}).then( /* array of tokens */ )
```

## Aggregation

***You can use the aggregation functions to categorise the index data, normally for the purposes of website navigation or dividing data up into segments***

### BUCKET

`BUCKET([ ...Promise ]).then(result)`

Return IDs of objects that match the query

### BUCKETFILTER

`BUCKETFILTER([ ...bucket ], filter query).then(result)`

The first argument is an array of buckets, the second is an expression
that filters each bucket

### DISTINCT

`DISTINCT(options).then(result)`

`DISTINCT` returns every value in the db that is greater than equal
to `gte` and less than or equal to `lte` (sorted alphabetically)

For example- get all names between `h` and `l`:

```javascript
DISTINCT({ gte: 'h', lte: 'l' }).then(result)
```

## Accessing the underlying index
### INDEX

`INDEX`
{'fraction_non_alphanumeric': 0.15046659509455776, 'fraction_numerical': 0.0, 'mean_word_length': 3.704739704739705, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 11, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1395365', 'n_tokens_mistral': 4092, 'n_tokens_neox': 3870, 'n_words': 1355}