<commit_before>Rust-PHF
=========

[![Build Status](https://travis-ci.org/sfackler/rust-phf.png?branch=master)](https://travis-ci.org/sfackler/rust-phf)

Rust-PHF is a library to generate efficient lookup tables at compile time using
[perfect hash functions](http://en.wikipedia.org/wiki/Perfect_hash_function).

It currently uses the
[CHD algorithm](http://cmph.sourceforge.net/papers/esa09.pdf) and can generate
a 100,000 entry map in roughly .4 seconds. By default statistics are not
produced, but if you use the `phf_mac` crate with the `stats` feature enabled
(writing `phf_macros/stats` in the `[dependencies]` section of your
`Cargo.toml` instead of `phf_macros`) and set the environment variable
`PHF_STATS` it will issue a compiler note about how long it took.

Documentation is available at https://sfackler.github.io/rust-phf/doc/phf

Example
=======

```rust
#![feature(plugin)]
#![plugin(phf_macros)]

extern crate phf;

#[derive(Clone)]
pub enum Keyword {
    Loop,
    Continue,
    Break,
    Fn,
    Extern,
}

static KEYWORDS: phf::Map<&'static str, Keyword> = phf_map! {
    "loop" => Keyword::Loop,
    "continue" => Keyword::Continue,
    "break" => Keyword::Break,
    "fn" => Keyword::Fn,
    "extern" => Keyword::Extern,
};

pub fn parse_keyword(keyword: &str) -> Option<Keyword> {
    KEYWORDS.get(keyword).cloned()
}
```
<commit_msg>Talk about phf_codegen in README<commit_after>Rust-PHF
=========

[![Build Status](https://travis-ci.org/sfackler/rust-phf.png?branch=master)](https://travis-ci.org/sfackler/rust-phf)

Rust-PHF is a library to generate efficient lookup tables at compile time using
[perfect hash functions](http://en.wikipedia.org/wiki/Perfect_hash_function).

It currently uses the
[CHD algorithm](http://cmph.sourceforge.net/papers/esa09.pdf) and can generate
a 100,000 entry map in roughly .4 seconds. By default statistics are not
produced, but if you use the `phf_mac` crate with the `stats` feature enabled
(writing `phf_macros/stats` in the `[dependencies]` section of your
`Cargo.toml` instead of `phf_macros`) and set the environment variable
`PHF_STATS` it will issue a compiler note about how long it took.

Documentation is available at https://sfackler.github.io/rust-phf/doc/phf

Usage
=====

PHF data structures can be constucted via either the compiler plugins in the
`phf_macros` crate or code generation supported by the `phf_codegen` crate.
Compiler plugins are not a stable part of Rust at the moment, so `phf_macros`
can only be used with nightlies.

phf_macros
===========

```rust
#![feature(plugin)]
#![plugin(phf_macros)]

extern crate phf;

#[derive(Clone)]
pub enum Keyword {
    Loop,
    Continue,
    Break,
    Fn,
    Extern,
}

static KEYWORDS: phf::Map<&'static str, Keyword> = phf_map! {
    "loop" => Keyword::Loop,
    "continue" => Keyword::Continue,
    "break" => Keyword::Break,
    "fn" => Keyword::Fn,
    "extern" => Keyword::Extern,
};

pub fn parse_keyword(keyword: &str) -> Option<Keyword> {
    KEYWORDS.get(keyword).cloned()
}
```

phf_codegen
===========

build.rs

```rust
extern crate phf_codegen;

use std::fs::File;
use std::io::{BufWriter, Write};
use std::path::Path;

fn main() {
    let path = Path::new(env!("OUT_DIR")).join("codegen.rs");
    let mut file = BufWriter::new(File::create(&path).unwrap());

    write!(&mut file, "static KEYWORDS: phf::Map<&'static str, Keyword> = ").unwrap();
    phf_codegen::Map::new()
        .entry("loop", "Keyword::Loop")
        .entry("continue", "Keyword::Continue")
        .entry("break", "Keyword::Break")
        .entry("fn", "Keyword::Fn")
        .entry("extern", "Keyword::Extern")
        .build(&mut file)
        .unwrap();
    write!(&mut file, ";\n").unwrap();
}
```

lib.rs

```rust
extern crate phf;

#[derive(Clone)]
enum Keyword {
    Loop,
    Continue,
    Break,
    Fn,
    Extern,
}

include!(concat!(env!("OUT_DIR"), "/codegen.rs"));

pub fn parse_keyword(keyword: &str) -> Option<Keyword> {
    KEYWORDS.get(keyword).cloned()
}
```

{'fraction_non_alphanumeric': 0.16108054027013508, 'fraction_numerical': 0.004502251125562781, 'mean_word_length': 4.31781914893617, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 22, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4241715', 'n_tokens_mistral': 1410, 'n_tokens_neox': 1380, 'n_words': 368}