<commit_before># Levi

Streaming full-text search for Node.js and browsers. Using LevelDB as storage backend.

[![Build Status](https://travis-ci.org/cshum/levi.svg?branch=master)](https://travis-ci.org/cshum/levi)

```
npm install levi
```

Full-text search using TF-IDF and cosine similarity. 
Stream based query mechanism plus query-time field boost options. 
Provided with configurable text processing pipeline: Tokenizer, Porter Stemmer and Stopwords filter.

Levi is built on [LevelUP](https://github.com/Level/levelup) - a fast, asynchronous, 
[transactional](https://github.com/cshum/level-transactions/) storage interface.
By default, it uses [LevelDB](https://github.com/Level/leveldown) on Node.js and [IndexedDB](https://github.com/maxogden/level.js) on browser. 
Also works with a variety of LevelDOWN compatible backends.

In addition, Levi provides relevancy scoring for live changing data using [TF-ICF](http://cda.ornl.gov/publications/ICMLA06.pdf) - a TF-IDF approximation based on existing corpus.
Such scoring matches comparably close to TF-IDF when existing corpus is sufficiently large,
with significantly better performance O(N) instead of O(N^2).

## API

### levi(path, [options])
### levi(sublevel, [options])

Create a new Levi instance with [LevelUP](https://github.com/Level/levelup#ctor) database path,

```js
var levi = require('levi')

// passing db location
var lv = levi('db') 
.use(levi.tokenizer())
.use(levi.stemmer())
.use(levi.stopword())

```
alternatively passing a section of [SublevelUP](https://github.com/cshum/sublevelup).

```js
var sublevel = require('sublevelup')
var db = sublevel(levelup('db'))

// passing db section `search`
var lv = levi(db.sublevel('search'))
.use(levi.tokenizer())
.use(levi.stemmer())
.use(levi.stopword())

```

Text processing pipeline `levi.tokenizer()`, `levi.stemmer()`, `levi.stopword()` are required for indexing.
These are exposed as [ginga](https://github.com/cshum/ginga) plugins so that they can be swapped for different language configurations.

### .put(key, value, [options], [callback])

Index document identified by `key`. `value` can be object or string.
Use object fields for `value` if you want field boost options for search.

```js
// string as value
lv.put('a', 'Lorem Ipsum is simply dummy text.', function (err) { ... })

// object fields as value<commit_msg>clean up levl() exampel<commit_after># Levi

Streaming full-text search for Node.js and browsers. Using LevelDB as storage backend.

[![Build Status](https://travis-ci.org/cshum/levi.svg?branch=master)](https://travis-ci.org/cshum/levi)

```
npm install levi
```

Full-text search using TF-IDF and cosine similarity. 
Stream based query mechanism plus query-time field boost options. 
Provided with configurable text processing pipeline: Tokenizer, Porter Stemmer and Stopwords filter.

Levi is built on [LevelUP](https://github.com/Level/levelup) - a fast, asynchronous, 
[transactional](https://github.com/cshum/level-transactions/) storage interface.
By default, it uses [LevelDB](https://github.com/Level/leveldown) on Node.js and [IndexedDB](https://github.com/maxogden/level.js) on browser. 
Also works with a variety of LevelDOWN compatible backends.

In addition, Levi provides relevancy scoring for live changing data using [TF-ICF](http://cda.ornl.gov/publications/ICMLA06.pdf) - a TF-IDF approximation based on existing corpus.
Such scoring matches comparably close to TF-IDF when existing corpus is sufficiently large,
with significantly better performance O(N) instead of O(N^2).

## API

### levi(path, [options])
### levi(sublevel, [options])

Create a new Levi instance with [LevelUP](https://github.com/Level/levelup#ctor) database path,
or a section of [SublevelUP](https://github.com/cshum/sublevelup).

```js
var levi = require('levi')

// levi instance of database path `db`
var lv = levi('db') 
.use(levi.tokenizer())
.use(levi.stemmer())
.use(levi.stopword())

```

Text processing pipeline `levi.tokenizer()`, `levi.stemmer()`, `levi.stopword()` are required for indexing.
These are exposed as [ginga](https://github.com/cshum/ginga) plugins so that they can be swapped for different language configurations.

### .put(key, value, [options], [callback])

Index document identified by `key`. `value` can be object or string.
Use object fields for `value` if you want field boost options for search.

```js
// string as value
lv.put('a', 'Lorem Ipsum is simply dummy text.', function (err) { ... })

// object fields as value
{'fraction_non_alphanumeric': 0.1341030195381883, 'fraction_numerical': 0.0013321492007104796, 'mean_word_length': 4.835492227979275, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 18, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3413034', 'n_tokens_mistral': 1441, 'n_tokens_neox': 1394, 'n_words': 465}