<commit_before>Overview
========

This is a simple, pure-Go, full text indexing and search library.

I made it for use on small to medium websites, although there is nothing web-specific about it's API or operation.

Cdb (http://github.com/jbarham/go-cdb) is used to perform the indexing and lookups.

Status
------

This project is experimental.  Breaking changes very well may occur.

Usage
------

First, you must create an index.  Like this:

	import "github.com/bradleypeabody/fulltext"

	// create new index with temp dir (usually "" is fine)
	idx, err := fulltext.NewIndexer(""); if err != nil { panic(err) }
	defer idx.Close()

	// for each document you want to add, you do something like this:
	doc := fulltext.IndexDoc{
		Id: []byte(uuid), // unique identifier (the path to a webpage works...)
		StoreValue: []byte(title), // bytes you want to be able to retrieve from search results
		IndexValue: []byte(data), // bytes you want to be split into words and indexed<commit_msg>made note about go get cdb<commit_after>Overview
========

This is a simple, pure-Go, full text indexing and search library.

I made it for use on small to medium websites, although there is nothing web-specific about it's API or operation.

Cdb (http://github.com/jbarham/go-cdb) is used to perform the indexing and lookups.

Status
------

This project is experimental.  Breaking changes very well may occur.

Notes on Building
--------

fulltext requires CDB:

 go get github.com/jbarham/go-cdb

Usage
------

First, you must create an index.  Like this:

	import "github.com/bradleypeabody/fulltext"

	// create new index with temp dir (usually "" is fine)
	idx, err := fulltext.NewIndexer(""); if err != nil { panic(err) }
	defer idx.Close()

	// for each document you want to add, you do something like this:
	doc := fulltext.IndexDoc{
		Id: []byte(uuid), // unique identifier (the path to a webpage works...)
		StoreValue: []byte(title), // bytes you want to be able to retrieve from search results
		IndexValue: []byte(data), // bytes you want to be split into words and indexed
{'fraction_non_alphanumeric': 0.11438223938223938, 'fraction_numerical': 0.0, 'mean_word_length': 3.5262008733624453, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '579942', 'n_tokens_mistral': 624, 'n_tokens_neox': 596, 'n_words': 275}