<commit_before>Clucy
=====

[![License EPL](https://img.shields.io/badge/license-EPL-yellow.svg)](https://www.eclipse.org/legal/epl-v10.html)
[![Build Status](https://travis-ci.org/kostafey/clucy.svg?branch=master)](https://travis-ci.org/kostafey/clucy)

Clucy is a Clojure interface to [Lucene](http://lucene.apache.org/).

Installation
------------

To install Clucy, add the following dependency to your `project.clj`
file:

    [org.clojars.kostafey/clucy "0.5.3-SNAPSHOT"]

Usage
-----

#### Search in documents

To use Clucy, first require it:

    (ns example
      (:require [clucy.core :as clucy]))

Then create an index. You can use `(memory-index)`, which stores the search
index in RAM, or `(disk-index "/path/to/a-folder")`, which stores the index in
a folder on disk.

    (def index (clucy/memory-index))

Next, add Clojure maps to the index:

    (clucy/add index
       {:name "Bob", :job "Builder"}
       {:name "Donald", :job "Computer Scientist"})

You can remove maps just as easily:

    (clucy/delete index
       {:name "Bob", :job "Builder"})

Once maps have been added, the index can be searched:

    user=> (clucy/search index "bob" 10)
    ({:name "Bob", :job "Builder"})

    user=> (clucy/search index "scientist" 10)
    ({:name "Donald", :job "Computer Scientist"})

You can search and remove all in one step. To remove all of the
scientists...

    (clucy/search-and-delete index "job:scientist")

#### Search text positions in single document

```clojure
(ns example
  (:use [clucy.core
         clucy.analyzers
         clucy.positions-searcher]))

(binding [*analyzer* (make-analyzer :en)]
  (let [test-text "This is the house that Jack built.
                   This is the malt
                   That lay in the house that Jack built."
        index (doto (memory-index)
                (add (set-field-params
                      test-text
                      {:positions-offsets true})))
        searcher (make-dict-searcher
                  #{"house"
                    "lay"
                    "<NAME>"})
        result-iter (searcher index)]
    (sort-by second
             (show-text-matches result-iter test-text))))
```

    => (["house" 12] ["<NAME>" 23] ["lay" 95] ["house" 106] ["<NAME>" 117])


Storing Fields<commit_msg>Fix README.<commit_after>Clucy
=====

[![License EPL](https://img.shields.io/badge/license-EPL-yellow.svg)](https://www.eclipse.org/legal/epl-v10.html)
[![Build Status](https://travis-ci.org/kostafey/clucy.svg?branch=master)](https://travis-ci.org/kostafey/clucy)

Clucy is a Clojure interface to [Lucene](http://lucene.apache.org/).

Installation
------------

To install Clucy, add the following dependency to your `project.clj`
file:

    [org.clojars.kostafey/clucy "0.5.3-SNAPSHOT"]

Usage
-----

#### Search in documents

To use Clucy, first require it:

    (ns example
      (:require [clucy.core :as clucy]))

Then create an index. You can use `(memory-index)`, which stores the search
index in RAM, or `(disk-index "/path/to/a-folder")`, which stores the index in
a folder on disk.

    (def index (clucy/memory-index))

Next, add Clojure maps to the index:

    (clucy/add index
       {:name "Bob", :job "Builder"}
       {:name "Donald", :job "Computer Scientist"})

You can remove maps just as easily:

    (clucy/delete index
       {:name "Bob", :job "Builder"})

Once maps have been added, the index can be searched:

    user=> (clucy/search index "bob" 10)
    ({:name "Bob", :job "Builder"})

    user=> (clucy/search index "scientist" 10)
    ({:name "Donald", :job "Computer Scientist"})

You can search and remove all in one step. To remove all of the
scientists...

    (clucy/search-and-delete index "job:scientist")

#### Search text positions in single document

```clojure
(ns example
  (:use [clucy.core
         clucy.analyzers
         clucy.positions-searcher]))

(binding [*analyzer* (make-analyzer :class :en)]
  (let [test-text "This is the house that Jack built.
                   This is the malt
                   That lay in the house that Jack built."
        index (doto (memory-index)
                (add (set-field-params
                      test-text
                      {:positions-offsets true})))
        searcher (make-dict-searcher
                  #{"house"
                    "lay"
                    "<NAME>"})
        result-iter (searcher index)]
    (sort-by second
             (show-text-matches result-iter test-text))))
```

    => (["house" 12] ["<NAME>" 23] ["lay" 95] ["house" 106] ["Jack built" 117])


Storing Fields
{'fraction_non_alphanumeric': 0.15051411069787793, 'fraction_numerical': 0.009188361408882083, 'mean_word_length': 2.525057825751735, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 14, 'https://': 8, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6008820', 'n_tokens_mistral': 1603, 'n_tokens_neox': 1488, 'n_words': 444}