<commit_before># word2vec
[![Build Status](https://travis-ci.org/sajari/word2vec.svg?branch=master)](https://travis-ci.org/sajari/word2vec)

word2vec provides functionality for loading binary [word2vec](https://code.google.com/p/word2vec) models and performing cosine similarity of combinations of words.

    $ go get github.com/sajari/word2vec/...

This will fetch the code and build several tools into `$GOPATH/bin`, including `word-calc` which performs basic calculations with lists of query words.  For instance: `vec(king) - vec(man) + vec(woman)` would be equivalent to:

    $ word-calc -model /path/to/model.bin -add king,woman -sub man

Alternatively, you can use the `word2vec` package directly:

```go
// Load the model from an io.Reader (i.e. a file).
model, err := word2vec.FromReader(r)
if err != nil {
	log.Fatalf("error loading model: %v", err)
}

// Create an expression.
expr := word2vec.Expr{}
expr.Add(1, "king")
expr.Add(-1, "man")
expr.Add(1, "woman")

// Find the most similar result by cosine similarity.
matches, err := m.CosN(expr, 1)
if err != nil {
	log.Fatalf("error evaluating cosine similarity: %v", err)
}
```

##  word2vec server

The `word-server` tool (see `cmd/word-server`) creates an HTTP server which wraps a word2vec model which can be queried from Go using a [Client](http://godoc.org/github.com/sajari/word2vec#Client), or using the `word-client` tool (see `cmd/word-client`).

    $ word-server -model /path/to/model.bin -listen localhost:1234

A simple code example using `Client`:

```go
c := word2vec.Client{Addr: "localhost:1234"}

// Create an expression.
expr := word2vec.Expr{}
expr.Add(1, "king")
expr.Add(-1, "man")
expr.Add(1, "woman")

// Find the most similar result by cosine similarity.
matches, err := c.CosN(expr, 1)
if err != nil {
	log.Fatalf("error evaluating cosine similarity: %v", err)
}
```<commit_msg>Improved README.<commit_after># word2vec
[![Build Status](https://travis-ci.org/sajari/word2vec.svg?branch=master)](https://travis-ci.org/sajari/word2vec)
[![GoDoc](https://godoc.org/github.com/sajari/word2vec?status.svg)](https://godoc.org/github.com/sajari/word2vec)

word2vec is a Go package which provides functions for querying word2vec models (see [https://code.google.com/p/word2vec](https://code.google.com/p/word2vec)).  Any binary word2vec model file can be loaded and queried.

## Requirements

- [Go 1.4+](http://golang.org/dl/) (only tested on 1.4+)
- A [word2vec](https://code.google.com/p/word2vec) model (binary format)

## Installation

If you haven't setup Go before, you need to first set a `GOPATH` (see [https://golang.org/doc/code.html#GOPATH](https://golang.org/doc/code.html#GOPATH)).

To fetch and build the code:

    $ go get github.com/sajari/word2vec/...

This will build the command line tools (in particular `word-calc`, `word-server`, `word-client`) into `$GOPATH/bin` (assumed to be in your `PATH` already).

## Usage

### word-calc

The `word-calc` tool is a quick way to perform basic word calculations on a word2vec model.  For instance: `vec(king) - vec(man) + vec(woman)` would be equivalent to:

    $ word-calc -model /path/to/model.bin -add king,woman -sub man

See `word-calc -h` for full more details.  Note that `word-calc` first loads the model every time,  and so can appear to be quite slow. Use `word-server` and `word-client` to get better performance when running multiple queries on the same model.

###  word-server and word-client

The `word-server` tool (see `cmd/word-server`) creates an HTTP server which wraps a word2vec model which can be queried from Go using a [Client](http://godoc.org/github.com/sajari/word2vec#Client), or using the `word-client` tool (see `cmd/word-client`).

    $ word-server -model /path/to/model.bin -listen localhost:1234

A simple code example using `Client`:

```go
c := word2vec.Client{Addr: "localhost:1234"}

// Create an expression.
expr := word2vec.Expr{}
expr.Add(1, "king")
expr.Add(-1, "man")
expr.Add(1, "woman")

// Find the most similar result by cosine similarity.
matches, err := c.CosN(expr, 1)
if err != nil {
	log.Fatalf("error evaluating cosine similarity: %v", err)
}
```

### API Example
Alternatively you can interact with a word2vec model directly in your code:

```go
// Load the model from an io.Reader (i.e. a file).
model, err := word2vec.FromReader(r)
if err != nil {
	log.Fatalf("error loading model: %v", err)
}

// Create an expression.
expr := word2vec.Expr{}
expr.Add(1, "king")
expr.Add(-1, "man")
expr.Add(1, "woman")

// Find the most similar result by cosine similarity.
matches, err := model.CosN(expr, 1)
if err != nil {
	log.Fatalf("error evaluating cosine similarity: %v", err)
}
```
{'fraction_non_alphanumeric': 0.14917717460995938, 'fraction_numerical': 0.015387903398162001, 'mean_word_length': 4.176991150442478, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 12, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4163237', 'n_tokens_mistral': 1632, 'n_tokens_neox': 1566, 'n_words': 510}