<commit_before># Token
[![Build status (master)](https://travis-ci.org/Machtan/token-rs.svg?branch=master)](https://travis-ci.org/Machtan/token-rs)

This is a small package containing a simple string-tokenizer for the rust programming language. The package also contains a simple sentence-splitting iterator.

(The sentence splitter might be moved, once I find out where I want it).

## Documentation

[machtan.github.io/token-rs/token](http://machtan.github.io/token-rs/token)

# Building
Add the following to your Cargo.toml file

```toml
[dependencies.token]
git = "https://github.com/machtan/token-rs"
```

# Examples

```rust
extern crate token;

let separators = vec![' ', '\n', '\t', '\r'];
let source: &str = "    Hello world \n  How do you do\t-Finely I hope";

let mut tokenizer = tokenizer::Tokenizer::new(source.as_bytes(), separators);
println!("Tokenizing...");
for token in tokenizer {
    println!("- Got token: {}", token.unwrap());
}
println!("Done!");
```

# License
MIT (do what you want with it)<commit_msg>Fixed example in README.md<commit_after># Token
[![Build status (master)](https://travis-ci.org/Machtan/token-rs.svg?branch=master)](https://travis-ci.org/Machtan/token-rs)

This is a small package containing a simple string-tokenizer for the rust programming language. The package also contains a simple sentence-splitting iterator.

(The sentence splitter might be moved, once I find out where I want it).

## Documentation

[machtan.github.io/token-rs/token](http://machtan.github.io/token-rs/token)

# Building
Add the following to your Cargo.toml file

```toml
[dependencies.token]
git = "https://github.com/machtan/token-rs"
```

# Examples

```rust
extern crate token;

let separators = vec![' ', '\n', '\t', '\r'];
let source: &str = "    Hello world \n  How do you do\t-Finely I hope";

let mut tokenizer = token::Tokenizer::new(source.as_bytes(), separators);
println!("Tokenizing...");
loop {
    let token = match tokenizer.next().unwrap() {
        Some(token) => token,
        None => { break; }
    };
    println!("- Got token: {}", token);
}
println!("Done!");
```

# License
MIT (do what you want with it)
{'fraction_non_alphanumeric': 0.16171003717472118, 'fraction_numerical': 0.0, 'mean_word_length': 4.200483091787439, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3446096', 'n_tokens_mistral': 706, 'n_tokens_neox': 700, 'n_words': 209}