<commit_before><h1>WARNING</h1>

0.5 is not yet released and may contain bugs and regressions. However, it is shaping up nicely so I recommend you try it first. 0.4 remains available if you hit insurmountable problems; but please take time to file a ticket at github if you do.

<h1>Version Compatibility</h1>

<table>
<tr><th>CouchDB</th><th>couchdb-lucene</th></tr>
<tr><td>0.9.1, 0.10</td><td>0.4</td></tr>
<tr><td>0.11 (not yet released)</td><td>0.4-maint (0.4 with patch for trunk compatibility)</td></tr>
<tr><td>0.10+</td><td>0.5 (not yet released)</td></tr>
</table>

<h1>Issue Tracking</h1>

Issue tracking at <a href="http://github.com/rnewson/couchdb-lucene/issues">github</a>.

<h1>Minimum System Requirements</h1>

Java 1.5 (or above) is required; the Sun version is recommended as it's regularly tested against.

<h1>Build couchdb-lucene</h1>

<ol>
<li>Install Maven 2.
<li>checkout repository
<li>type 'mvn'
<li>configure couchdb (see below)
</ol>

You will now have a zip file in the target/ directory. This contains all the couchdb-lucene code, dependencies, startup scripts and configuration files to run couchdb-lucene.

<h1>Configure CouchDB</h1>

<pre>
[couchdb]
os_process_timeout=60000 ; increase the timeout from 5 seconds.

[external]
fti=/path/to/python /path/to/couchdb-lucene/tools/couchdb-external-hook.py

[httpd_db_handlers]
_fti = {couch_httpd_external, handle_external_req, <<"fti">>}
</pre>

<h2>Hook options</h2>

<table>
<tr><th>Option</th><th>Meaning</th><th>Default Value</th></tr>
<tr><td>--remote-host</td><td>The hostname of the couchdb-lucene server</td><td>localhost</td></tr>
<tr><td>--remote-port</td><td>The port of the couchdb-lucene server</td><td>5985</td></tr>
<tr><td>--local-key</td><td>The key for the local couchdb instance as known to the couchdb-lucene server</td><td>local</td></tr>
</table>

<h1>Configure couchdb-lucene</h1>

couchdb-lucene runs in a single, standalone JVM. As such, you can choose to locate your couchdb-lucene server on a different machine to couchdb if you wish, or keep it on the same machine, it's your call.

<h1>Start couchdb-lucene</h1>

To start couchdb-lucene, run;
<pre>
bin/run
</pre>

To stop couchdb-lucene, simply kill the Java process.

<h1>Indexing Strategy</h1>

<h2>Document Indexing</h2>

You must supply a index function in order to enable couchdb-lucene as, by default, nothing will be indexed. To suppress a document from the index, return null. It's more typical to return a single Document object which contains everything you'd like to query and retrieve. You may also return an array of Document objects if you wish.

You may add any number of index views in any number of design documents. All searches will be constrained to documents emitted by the index functions.

Here's an complete example of a design document with couchdb-lucene features:

<pre>
{
    "_id":"_design/a_design_document_with_any_name_you_like",
    "fulltext": {
        "by_subject": {
            "index":"function(doc) { var ret=new Document(); ret.add(doc.subject); return ret }"
        },
        "by_content": {
            "index":"function(doc) { var ret=new Document(); ret.add(doc.content); return ret }"
        }
    }
}
</pre>

Here are some example URL's for the given design document;

<pre>
http://localhost:5984/database/_fti/lucene/by_subject?q=hello
http://localhost:5984/database/_fti/lucene/by_content?q=hello
</pre>

A fulltext object contains multiple index view declarations. An index view consists of;

<dl>
<dt>analyzer</dt><dd>(optional) The analyzer to use</dd>
<dt>defaults</dt><dd>(optional) The default for numerous indexing options can be overridden here. A full list of options follows.</dd>
<dt>index</dt><dd>The indexing function itself, documented below.</dd>
</dl>

<h3>The Defaults Object</h3>

The following indexing options can be defaulted;

<table>
  <tr>
    <th>name</th>
    <th>description</th>
    <th>available options</th>
    <th>default</th>
  </tr>
  <tr>
    <th>field</th>
    <td>the field name to index under</td>
    <td>user-defined</td>
    <td>default</td>
  </tr>
  <tr>
    <th>type</th>
    <td>the type of the field</td>
    <td>date, double, float, integer, long, string</td>
    <td>string</td>
  </tr>
  <tr>
    <th>store</th>
    <td>whether the data is stored. The value will be returned in the search result.</td>
    <td>yes, no</td>
    <td>no</td>
  </tr>
  <tr>
    <th>index</th>
    <td>whether (and how) the data is indexed</td>
    <td>analyzed, analyzed_no_norms, no, not_analyzed, not_analyzed_no_norms</td>
    <td>analyzed</td>
  </tr>
</table>

<h3>The Analyzer Option</h3>

Lucene has numerous ways of converting free-form text into tokens, these classes are called Analyzer's. By default, the StandardAnalyzer is used which lower-cases all text, drops common English words ("the", "and", and so on), among other things. This processing might not always suit you, so you can choose from several others by setting the "analyzer" field to one of the following values;

<ul>
<li>brazilian</li>
<li>chinese</li>
<li>cjk</li>
<li>czech</li>
<li>dutch</li>
<li>english</li>
<li>french</li>
<li>german</li>
<li>keyword</li>
<li>perfield</li>
<li>porter</li>
<li>russian</li>
<li>simple</li>
<li>standard</li>
<li>thai</li>
</ul>

The "perfield" option lets you use a different analyzer for different fields and is configured as follows;

<pre>
?analyzer=perfield:{field_name:"analyzer_name"}
</pre>

Unless overridden, any field name not specified will be handled by the standard analyzer. To change the default, use the special default field name;

<pre>
?analyzer=perfield:{default:"keyword"}
</pre>

You should also specify the analyzer in your ddoc too;

<pre>
"fulltext": {
  "idx": {
    "analyzer": "perfield:{default:\"keyword\"}"
  }
}
</pre>

<h3>The Document class</h3>
<commit_msg>tidy document about analyzer<commit_after><h1>WARNING</h1>

0.5 is not yet released and may contain bugs and regressions. However, it is shaping up nicely so I recommend you try it first. 0.4 remains available if you hit insurmountable problems; but please take time to file a ticket at github if you do.

<h1>Version Compatibility</h1>

<table>
<tr><th>CouchDB</th><th>couchdb-lucene</th></tr>
<tr><td>0.9.1, 0.10</td><td>0.4</td></tr>
<tr><td>0.11 (not yet released)</td><td>0.4-maint (0.4 with patch for trunk compatibility)</td></tr>
<tr><td>0.10+</td><td>0.5 (not yet released)</td></tr>
</table>

<h1>Issue Tracking</h1>

Issue tracking at <a href="http://github.com/rnewson/couchdb-lucene/issues">github</a>.

<h1>Minimum System Requirements</h1>

Java 1.5 (or above) is required; the Sun version is recommended as it's regularly tested against.

<h1>Build couchdb-lucene</h1>

<ol>
<li>Install Maven 2.
<li>checkout repository
<li>type 'mvn'
<li>configure couchdb (see below)
</ol>

You will now have a zip file in the target/ directory. This contains all the couchdb-lucene code, dependencies, startup scripts and configuration files to run couchdb-lucene.

<h1>Configure CouchDB</h1>

<pre>
[couchdb]
os_process_timeout=60000 ; increase the timeout from 5 seconds.

[external]
fti=/path/to/python /path/to/couchdb-lucene/tools/couchdb-external-hook.py

[httpd_db_handlers]
_fti = {couch_httpd_external, handle_external_req, <<"fti">>}
</pre>

<h2>Hook options</h2>

<table>
<tr><th>Option</th><th>Meaning</th><th>Default Value</th></tr>
<tr><td>--remote-host</td><td>The hostname of the couchdb-lucene server</td><td>localhost</td></tr>
<tr><td>--remote-port</td><td>The port of the couchdb-lucene server</td><td>5985</td></tr>
<tr><td>--local-key</td><td>The key for the local couchdb instance as known to the couchdb-lucene server</td><td>local</td></tr>
</table>

<h1>Configure couchdb-lucene</h1>

couchdb-lucene runs in a single, standalone JVM. As such, you can choose to locate your couchdb-lucene server on a different machine to couchdb if you wish, or keep it on the same machine, it's your call.

<h1>Start couchdb-lucene</h1>

To start couchdb-lucene, run;
<pre>
bin/run
</pre>

To stop couchdb-lucene, simply kill the Java process.

<h1>Indexing Strategy</h1>

<h2>Document Indexing</h2>

You must supply a index function in order to enable couchdb-lucene as, by default, nothing will be indexed. To suppress a document from the index, return null. It's more typical to return a single Document object which contains everything you'd like to query and retrieve. You may also return an array of Document objects if you wish.

You may add any number of index views in any number of design documents. All searches will be constrained to documents emitted by the index functions.

Here's an complete example of a design document with couchdb-lucene features:

<pre>
{
    "_id":"_design/a_design_document_with_any_name_you_like",
    "fulltext": {
        "by_subject": {
            "index":"function(doc) { var ret=new Document(); ret.add(doc.subject); return ret }"
        },
        "by_content": {
            "index":"function(doc) { var ret=new Document(); ret.add(doc.content); return ret }"
        }
    }
}
</pre>

Here are some example URL's for the given design document;

<pre>
http://localhost:5984/database/_fti/lucene/by_subject?q=hello
http://localhost:5984/database/_fti/lucene/by_content?q=hello
</pre>

A fulltext object contains multiple index view declarations. An index view consists of;

<dl>
<dt>analyzer</dt><dd>(optional) The analyzer to use</dd>
<dt>defaults</dt><dd>(optional) The default for numerous indexing options can be overridden here. A full list of options follows.</dd>
<dt>index</dt><dd>The indexing function itself, documented below.</dd>
</dl>

<h3>The Defaults Object</h3>

The following indexing options can be defaulted;

<table>
  <tr>
    <th>name</th>
    <th>description</th>
    <th>available options</th>
    <th>default</th>
  </tr>
  <tr>
    <th>field</th>
    <td>the field name to index under</td>
    <td>user-defined</td>
    <td>default</td>
  </tr>
  <tr>
    <th>type</th>
    <td>the type of the field</td>
    <td>date, double, float, integer, long, string</td>
    <td>string</td>
  </tr>
  <tr>
    <th>store</th>
    <td>whether the data is stored. The value will be returned in the search result.</td>
    <td>yes, no</td>
    <td>no</td>
  </tr>
  <tr>
    <th>index</th>
    <td>whether (and how) the data is indexed</td>
    <td>analyzed, analyzed_no_norms, no, not_analyzed, not_analyzed_no_norms</td>
    <td>analyzed</td>
  </tr>
</table>

<h3>The Analyzer Option</h3>

Lucene has numerous ways of converting free-form text into tokens, these classes are called Analyzer's. By default, the StandardAnalyzer is used which lower-cases all text, drops common English words ("the", "and", and so on), among other things. This processing might not always suit you, so you can choose from several others by setting the "analyzer" field to one of the following values;

<ul>
<li>brazilian</li>
<li>chinese</li>
<li>cjk</li>
<li>czech</li>
<li>dutch</li>
<li>english</li>
<li>french</li>
<li>german</li>
<li>keyword</li>
<li>perfield</li>
<li>porter</li>
<li>russian</li>
<li>simple</li>
<li>standard</li>
<li>thai</li>
</ul>

The "perfield" option lets you use a different analyzer for different fields and is configured as follows;

<pre>
"analyzer":"perfield:{field_name:\"analyzer_name\"}"
</pre>

Unless overridden, any field name not specified will be handled by the standard analyzer. To change the default, use the special default field name;

<pre>
"analyzer":"perfield:{default:\"keyword\"}"
</pre>

<h3>The Document class</h3>

{'fraction_non_alphanumeric': 0.14231400673982544, 'fraction_numerical': 0.012615570724963276, 'mean_word_length': 4.365785813630041, 'pattern_counts': {'":': 17, '<': 421, '<?xml version=': 0, '>': 421, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6213382', 'n_tokens_mistral': 4123, 'n_tokens_neox': 3975, 'n_words': 1275}