<commit_before>STOKE
=====

STOKE is a stochastic optimizer for the x86_64 instruction set. STOKE uses
random search to explore the extremely high-dimensional space of all possible
program transformations. Although any one random transformation is unlikely to
produce a code sequence that is both correct and an improvement over the
original, the repeated application of millions of transformations is sufficient
to produce novel and non-obvious code sequences that have been shown to
outperform the code produced by general-purpose and domain-specific compilers,
           and in some cases expert hand-written code.

STOKE has appeared in a number of publications. For a thorough introduction to
the design of STOKE, see:

- **Stochastic Superoptimization** -- ASPLOS 2013 ([link](http://cs.stanford.edu/people/eschkufz/research/asplos291-schkufza.pdf)):
- **Data-Driven Equivalence Checking** -- OOPSLA 2013 ([link](http://cs.stanford.edu/people/eschkufz/research/oopsla011-sharma.pdf)):
- **Stochastic Optimization of Floating-Point Programs with Tunable Precision** -- PLDI 2014 ([link](http://cs.stanford.edu/people/eschkufz/research/pldi52-schkufza.pdf)):

Table of Contents
=====
0. [Prerequisites](#prerequisites)
1. [Choosing a STOKE version](#choosing-a-stoke-version)
2. [Building STOKE](#building-stoke)
3. [Using STOKE](#using-stoke)
4. [Additional Features](#additional-features)
5. [Extending STOKE](#extending-stoke)
 1. [Code Organization](#code-organization)
 2. [Initial Search State](#initial-search-state)
 3. [Search Transformations](#search-transformations)
 4. [Cost Function](#cost-function)
 6. [Live-out Error](#live-out-error)
 7. [Verification Strategy](#verification-strategy)
 8. [Command Line Args](#command-line-args)
6. [Contact](#contact)

Prerequisites
=====

STOKE will run on modern 64-bit x86 processors.  We officially support Haswell
processors with AVX2 extensions.  STOKE should also run on Sandy Bridge
systems (with AVX, but not AVX2), and Nehalem systems without either extension;
however officially these targets are not supported.  

To check what level of hardware support you have, run:

    $ less /proc/cpuinfo
    
and check if the following cpu flags are present:

    $ flags: ... avx avx2 bmi bmi2 popcnt ...

If you don't have 'avx' or 'avx2', you will need to compile for nehalem.  If
you have 'avx', but not avx2, you will compile for 'sandybridge'.  If you have
both, you can use the default make targets.  Build instructions are in the next
section.

STOKE is supported on the latest Ubuntu LTS release; in practice, it will also
run on Ubuntu 13.10+ and on debian testing.  If you're trying to get STOKE to
work on another linux distribution, having the right version of g++ is key.
STOKE is supported on 4.8.2 only (this is the current version in Ubuntu 14.04).
It should also work on 4.9.1, but in the past this has required minor tweaks
(4.9.1 is the current version in Debian testing).  g++ 4.7.x and older
definitely will not work.

Most of STOKE's software dependencies are available through apt. These can be
satisfied by typing:

    $ sudo apt-get install git subversion flex bison ccache doxygen g++ g++-multilib ghc libghc-regex-tdfa-dev libghc-regex-compat-dev libghc-split-dev cmake libghc-regex-compat-dev libjsoncpp-dev libcln-dev

The rest of the dependencies will be fetched automatically as part of the build
process.

Choosing a STOKE version
=====

The entire STOKE code base is available on GitHub under the Apache Software
License version 2.0 at [github.com/StanfordPL/stoke-release](https://github.com/StanfordPL/stoke-release/).

We provide both stable releases as well as our latest development code (experimental).  There are
different trade-offs in deciding which one to use:

- **Development (Experimental) branch**:  This is our development branch, that
we use day-to-day.  It contains the latest bug fixes and features, but also the
latest bugs.  Sometimes it noticeably breaks, but usually it's okay.  The only
guarantee is that this branch will always pass our regression tests.  We
sometimes make non-backwards compatible changes, such as changing the syntax of
command-line arguments.  We can't promise support for this, but feel free to
ask.
- **Release**:  We typically release the version that corresponds to the papers
we write.  Most likely this is a version of STOKE that has been used to run the
experiments for a paper, and at least worked for that task.  However, we rarely
update releases, and so they might contain bugs, some of which may have already
been fixed by us.  We also don't tend to release very often, so a release might
be considerably out-of-date.

We leave it to the user to decide which version works best for them.  If you find a bug, please try the development branch first to see if has already been fixed.

To get the development branch, type:

    $ git clone -b development https://github.com/StanfordPL/stoke-release

The `master` branch always points to the latest release (plus potentially some back-ported bugfixes).  To get it, type:

    $ git clone -b master https://github.com/StanfordPL/stoke-release

or alternatively download it under the releases on GitHub.

<commit_msg>minor readme change<commit_after>STOKE
=====

STOKE is a stochastic optimizer for the x86_64 instruction set. STOKE uses
random search to explore the extremely high-dimensional space of all possible
program transformations. Although any one random transformation is unlikely to
produce a code sequence that is both correct and an improvement over the
original, the repeated application of millions of transformations is sufficient
to produce novel and non-obvious code sequences that have been shown to
outperform the code produced by general-purpose and domain-specific compilers,
and in some cases expert hand-written code.

STOKE has appeared in a number of publications. For a thorough introduction to
the design of STOKE, see:

- **Stochastic Superoptimization** -- ASPLOS 2013 ([link](http://cs.stanford.edu/people/eschkufz/research/asplos291-schkufza.pdf)):
- **Data-Driven Equivalence Checking** -- OOPSLA 2013 ([link](http://cs.stanford.edu/people/eschkufz/research/oopsla011-sharma.pdf)):
- **Stochastic Optimization of Floating-Point Programs with Tunable Precision** -- PLDI 2014 ([link](http://cs.stanford.edu/people/eschkufz/research/pldi52-schkufza.pdf)):

Table of Contents
=====
0. [Prerequisites](#prerequisites)
1. [Choosing a STOKE version](#choosing-a-stoke-version)
2. [Building STOKE](#building-stoke)
3. [Using STOKE](#using-stoke)
4. [Additional Features](#additional-features)
5. [Extending STOKE](#extending-stoke)
 1. [Code Organization](#code-organization)
 2. [Initial Search State](#initial-search-state)
 3. [Search Transformations](#search-transformations)
 4. [Cost Function](#cost-function)
 6. [Live-out Error](#live-out-error)
 7. [Verification Strategy](#verification-strategy)
 8. [Command Line Args](#command-line-args)
6. [Contact](#contact)

Prerequisites
=====

STOKE will run on modern 64-bit x86 processors.  We officially support Haswell
processors with AVX2 extensions.  STOKE should also run on Sandy Bridge
systems (with AVX, but not AVX2), and Nehalem systems without either extension;
however officially these targets are not supported.  

To check what level of hardware support you have, run:

    $ less /proc/cpuinfo
    
and check if the following cpu flags are present:

    flags: ... avx avx2 bmi bmi2 popcnt ...

If you don't have 'avx' or 'avx2', you will need to compile for nehalem.  If
you have 'avx', but not avx2, you will compile for 'sandybridge'.  If you have
both, you can use the default make targets.  Build instructions are in the next
section.

STOKE is supported on the latest Ubuntu LTS release; in practice, it will also
run on Ubuntu 13.10+ and on debian testing.  If you're trying to get STOKE to
work on another linux distribution, having the right version of g++ is key.
STOKE is supported on 4.8.2 only (this is the current version in Ubuntu 14.04).
It should also work on 4.9.1, but in the past this has required minor tweaks
(4.9.1 is the current version in Debian testing).  g++ 4.7.x and older
definitely will not work.

Most of STOKE's software dependencies are available through apt. These can be
satisfied by typing:

    $ sudo apt-get install git subversion flex bison ccache doxygen g++ g++-multilib ghc libghc-regex-tdfa-dev libghc-regex-compat-dev libghc-split-dev cmake libghc-regex-compat-dev libjsoncpp-dev libcln-dev

The rest of the dependencies will be fetched automatically as part of the build
process.

Choosing a STOKE version
=====

The entire STOKE code base is available on GitHub under the Apache Software
License version 2.0 at [github.com/StanfordPL/stoke-release](https://github.com/StanfordPL/stoke-release/).

We provide both stable releases as well as our latest development code.  There are different trade-offs in deciding which one to use:

- **Development branch**:  This is our development branch, that
we use day-to-day.  It contains the latest bug fixes and features, but also the
latest bugs.  Sometimes it noticeably breaks, but usually it's okay.  The only
guarantee is that this branch will always pass our regression tests.  We
sometimes make non-backwards compatible changes, such as changing the syntax of
command-line arguments.  We can't promise support for this, but feel free to
ask.
- **Release**:  We typically release the version that corresponds to the papers
we write.  Most likely this is a version of STOKE that has been used to run the
experiments for a paper, and at least worked for that task.  However, we rarely
update releases, and so they might contain bugs, some of which may have already
been fixed by us.  We also don't tend to release very often, so a release might
be considerably out-of-date.

We leave it to the user to decide which version works best for them.  If you find a bug, please try the development branch first to see if has already been fixed.

To get the development branch, type:

    $ git clone -b development https://github.com/StanfordPL/stoke-release

The `master` branch always points to the latest release (plus potentially some back-ported bugfixes).  To get it, type:

    $ git clone -b master https://github.com/StanfordPL/stoke-release

or alternatively download it under the releases on GitHub.


{'fraction_non_alphanumeric': 0.07767831360092406, 'fraction_numerical': 0.013283280392723073, 'mean_word_length': 4.336414997431946, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '581397', 'n_tokens_mistral': 3120, 'n_tokens_neox': 2872, 'n_words': 1376}