<commit_before>[![Build Status](https://travis-ci.org/neurosnap/sentences.svg)](https://travis-ci.org/neurosnap/sentences)
[![GODOC](https://godoc.org/github.com/nathany/looper?status.svg)](https://godoc.org/gopkg.in/neurosnap/sentences.v1)
![MIT](https://img.shields.io/packagist/l/doctrine/orm.svg)
[![Go Report Card](https://goreportcard.com/badge/github.com/neurosnap/sentences)](https://goreportcard.com/report/github.com/neurosnap/sentences)

Sentences - A command line sentence tokenizer
=============================================

This command line utility will convert a blob of text into a list of sentences.

* [Docs](https://godoc.org/gopkg.in/neurosnap/sentences.v1)

Install
-------

```
go get gopkg.in/neurosnap/sentences.v1
go install gopkg.in/neurosnap/sentences.v1/_cmd/sentences
```

### Binaries

#### Linux

* [Linux 386](https:///storage.cloud.google.com/go-sentences/sentences_linux-386.tar.gz)
* [Linux AMD64](https://storage.cloud.google.com/go-sentences/sentences_linux-amd64.tar.gz)

#### Mac

* [Darwin 386](https://storage.cloud.google.com/go-sentences/sentences_darwin-386.tar.gz)
* [Darwin AMD64](https://storage.cloud.google.com/go-sentences/sentences_darwin-amd64.tar.gz)
<commit_msg>add demo to readme<commit_after>[![Build Status](https://travis-ci.org/neurosnap/sentences.svg)](https://travis-ci.org/neurosnap/sentences)
[![GODOC](https://godoc.org/github.com/nathany/looper?status.svg)](https://godoc.org/gopkg.in/neurosnap/sentences.v1)
![MIT](https://img.shields.io/packagist/l/doctrine/orm.svg)
[![Go Report Card](https://goreportcard.com/badge/github.com/neurosnap/sentences)](https://goreportcard.com/report/github.com/neurosnap/sentences)

Sentences - A command line sentence tokenizer
=============================================

This command line utility will convert a blob of text into a list of sentences.

* [Demo](https://sentences-231000.appspot.com/)
* [Docs](https://godoc.org/gopkg.in/neurosnap/sentences.v1)

Install
-------

```
go get gopkg.in/neurosnap/sentences.v1
go install gopkg.in/neurosnap/sentences.v1/_cmd/sentences
```

### Binaries

#### Linux

* [Linux 386](https:///storage.cloud.google.com/go-sentences/sentences_linux-386.tar.gz)
* [Linux AMD64](https://storage.cloud.google.com/go-sentences/sentences_linux-amd64.tar.gz)

#### Mac

* [Darwin 386](https://storage.cloud.google.com/go-sentences/sentences_darwin-386.tar.gz)
* [Darwin AMD64](https://storage.cloud.google.com/go-sentences/sentences_darwin-amd64.tar.gz)

{'fraction_non_alphanumeric': 0.21402805611222445, 'fraction_numerical': 0.021643286573146292, 'mean_word_length': 5.178217821782178, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 25, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2026038', 'n_tokens_mistral': 966, 'n_tokens_neox': 910, 'n_words': 98}