<commit_before># clj-fst

FIXME: description

## Installation

Download from http://example.com/FIXME.

## Usage

FIXME: explanation

    $ java -jar clj-fst-0.1.0-standalone.jar [args]

## Options

FIXME: listing of options this app accepts.

## Examples

...

### Bugs

...

### Any Other Sections
### That You Think
### Might be Useful

## License

Copyright © 2014 FIXME

Distributed under the Eclipse Public License either version 1.0 or (at
your option) any later version.<commit_msg>Update README.md file with the proper documentation<commit_after># clj-fst

This Clojure FST implementation is a wrapper above the Lucene FST package which is part of Lucene core.

Finite state transducers are finite state machines with two tapes: an input and an output tape. The automaton map an input string to an output. The output can be another string or an integer.

The FST produced by this application are implemented as a bytes array which makes FST really effective indexes in terms of speed and memory consumption. In fact, a 10 millions terms index will takes roughtly 256 MB of memory (depending of the string composition of the input strings, and if the outputs are integers or strings).

`clj-fst` is an lightning-fast and memory effective way to figure out if something belong to a really huge set of things, or to get the output of an input. This is really simple but has profound implications.

## Installation

### Using Linengen

You can easily install `clj-fst` using Linengen. The only thing you have to do is to add Add `[clj-fst "0.1.0"]` as a dependency to your `project.clj`.

Then make sure that you downloaded this dependency by running the `lein deps` command.

## Documentation

[The complete `clj-fst` documentation is available here.](http://structureddynamics.github.io/clj-fst/)

## Usage

Here is how you can create, populate and save a FST:

```clojure
;; The first thing to do is to create the Builder
(def builder (create-builder! :type :int))

;; This small `sorted-map` defines the things
;; to add to the FST
(def values (into (sorted-map) {"cat" 1
                                "dog" 2
                                "mice" 3}))

;; Populate the FST using that `sorted-map`
(doseq [[input output] values]
  (add! builder {input output}))

;; Save a FST on the file system
(save! "resources/fst.srz" fst)
```

Additionally you can load a previously saved FST:

```clojure
;; Load a FST from the file system
(load! "resources/fst.srz)
```

You can query a FST to get the output related to an input:

```clojure
;; Query the FST
(get-output "cat" fst)
```

You can iterate over a FST using FST enumerations:

```clojure
;; Create the FST enumeration
(def enum (create-enum! fst))

;; Get the first item in the FST
(next! enum)

;; Get the current FST item pointed by the enumerator
(current! enum)

;; Search for different input terms
(get-ceil-term! "cat" enum)

(get-floor-term! "cat" enum)

(get-exact-term! "cat" enum)
```
{'fraction_non_alphanumeric': 0.09090909090909091, 'fraction_numerical': 0.006759040216289287, 'mean_word_length': 3.6835443037974684, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4242269', 'n_tokens_mistral': 918, 'n_tokens_neox': 875, 'n_words': 421}