<commit_before>DimmWitted
==========

# How fast is DimmWitted?

  - On Amazon EC2's FREE MACHINE (512M memory, 1 core). We can sample 3.6M varialbes/seconds.
  - On a 2-node Amazon EC2 machine, sampling 7 billion random variables, each of which has 10 features, takes 3 minutes. This means we can run inference for all living human beings on this planet with $15 (100 samples!)
  - On Macbook, DimmWitted runs 10x faster than DeepDive's default sampler.

# Usage

See: the [DimmWitted sampler page in DeepDive's documentation](http://deepdive.stanford.edu/doc/basics/sampler.html).

The binary format for DimmWitted's input is specified in [DeepDive's factor graph schema reference](http://deepdive.stanford.edu/doc/advanced/factor_graph_schema.html).

# Installation

First, install dependencies

    make dep

Then

    make

This will use whatever in you $(CXX) variable to compile. We assume that you have > g++4.7.2 or clang++-4.2. To specify a compiler to use, type in something like

    CXX=/dfs/rulk/0/czhang/software/gcc/bin/g++ make

To test, run

    make test


# Reference

[<NAME> and <NAME>. DimmWitted: A study of main-memory statistical analytics. PVLDB, 2014.](http://www.vldb.org/pvldb/vol7/p1283-zhang.pdf)<commit_msg>add help for compiling on Mac<commit_after>DimmWitted
==========

# How fast is DimmWitted?

  - On Amazon EC2's FREE MACHINE (512M memory, 1 core). We can sample 3.6M varialbes/seconds.
  - On a 2-node Amazon EC2 machine, sampling 7 billion random variables, each of which has 10 features, takes 3 minutes. This means we can run inference for all living human beings on this planet with $15 (100 samples!)
  - On Macbook, DimmWitted runs 10x faster than DeepDive's default sampler.

# Usage

See: the [DimmWitted sampler page in DeepDive's documentation](http://deepdive.stanford.edu/doc/basics/sampler.html).

The binary format for DimmWitted's input is specified in [DeepDive's factor graph schema reference](http://deepdive.stanford.edu/doc/advanced/factor_graph_schema.html).

# Installation

First, install dependencies

    make dep

Then

    make

This will use whatever in you $(CXX) variable to compile. We assume that you have > g++4.7.2 or clang++-4.2. To specify a compiler to use, type in something like

    CXX=/dfs/rulk/0/czhang/software/gcc/bin/g++ make

On MacOS, `CXX=/opt/local/bin/clang++ make`

To test, run

    make test

On MacOS, you can use MacPorts to install clang.

    port select --list clang
    sudo port install clang-3.7
    sudo port select --set clang mp-clang-3.7 

You can then compile using

    CXX=/opt/local/bin/clang++ make

# Reference

[<NAME> and <NAME>. DimmWitted: A study of main-memory statistical analytics. PVLDB, 2014.](http://www.vldb.org/pvldb/vol7/p1283-zhang.pdf)
{'fraction_non_alphanumeric': 0.10057887120115774, 'fraction_numerical': 0.026772793053545588, 'mean_word_length': 3.8086956521739133, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4834604', 'n_tokens_mistral': 969, 'n_tokens_neox': 876, 'n_words': 330}