<commit_before># Sheer

Sheer is a tool for [loading arbitrary content](#indexing) into [Elasticsearch](http://www.elasticsearch.org) and [serving that content](#serving) on the web using [Jinja2 templates](http://jinja.pocoo.org/). 

If you're not familiar with Elasticsearch, it is highly recommended that you read the Elasticsearch Definitive Guide's [Finding Your Feet](http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/_finding_your_feet.html).

* [Requirements](#requirements)
* [Installation](#installation)
* [Quick Start](#quick-start)
* [Indexing](#indexing)
    * [Reindexing](#reindexing)
    * [Partial Indexing](#partial-indexing)
	* [Sheer Index Settings](#sheer-index-settings)
	* [Content Processors](#content-processors)
	* [Mappings](#mappings)
* [Serving](#serving)
	* [Templates](#templates)
		* [Context](#context)
			* [`selected_filters_for_field`](#selected_filters_for_fieldfieldname)
			* [`is_filter_selected`](#is_filter_selectedfieldname-value)
			* [`queries`](#queries)
			* [`more_like_this`](#more_like_thishit)
			* [`get_document`](#get_documentdoctype-docid)
	* [Elasticsearch Lookup URLs](#elasticsearch-lookup-urls)
	* [Blueprints](#blueprints)
	* [Query API](#query-api)
		* [`QueryFinder`](#queryfinder)
		* [`Query`](#query)
		* [`QueryResult`](#queryresult)
		* [`QueryHit`](#queryhit)
- [Licensing](#licensing)


## Requirements

Sheer is a Python application that requires:

- [Elasticsearch](http://www.elasticsearch.org/)
- [Flask](http://flask.pocoo.org/)

Recommended for installing and running Sheer:

- [virtualenv](https://virtualenv.pypa.io/en/latest/)
- [virtualenvwrapper](https://virtualenvwrapper.readthedocs.org)

Running tests requires:

- [nose](https://nose.readthedocs.org/en/latest/)
- [mock](http://www.voidspace.org.uk/python/mock/)

## Installation

To install Sheer you will first need to install 
[Elasticsearch](http://www.elasticsearch.org/). This can be acomplished 
a number of ways, many of which are detailed in 
[in the Elasticsearch documentation](http://www.elastic.co/guide/en/elasticsearch/reference/current/setup.html). 
On Mac OS X it can be installed using [homebrew](http://brew.sh/):

```
brew install elasticsearch
```

There are also [Elasticsearch apt and Yum repositories](http://www.elastic.co/guide/en/elasticsearch/reference/current/setup-repositories.html).

To install Sheer itself, it is recommended to create a 
[`virtualenv`](https://virtualenv.pypa.io/en/latest/) using 
[`virtualenvwrapper`](https://virtualenvwrapper.readthedocs.org).

```
mkvirtualenv sheer
workon sheer
```

Then you can clone the Sheer repository and install the Python 
requirements using `pip`:

```shell
git clone https://github.com/cfpb/sheer
pip install -r sheer/requirements.txt
```

You can then install Sheer with `pip`. `pip -e` installs Sheer in
"editable" mode, which means it runs from the path where you've cloned
it, and any changes you `git pull` from upstream won't have to be
installed again.

```shell
pip install -e sheer
```

## Quick Start

This quick start assumes you have an existing Sheer site you want to
load content for and serve.

```shell
cd path/to/my/sheer/site
```

Index the site's content in Elasticsearch:

```shell
sheer index
```

Serve the site at [http://localhost:7000](http://localhost:7000):<commit_msg>Add testing section to README<commit_after># Sheer

Sheer is a tool for [loading arbitrary content](#indexing) into [Elasticsearch](http://www.elasticsearch.org) and [serving that content](#serving) on the web using [Jinja2 templates](http://jinja.pocoo.org/). 

If you're not familiar with Elasticsearch, it is highly recommended that you read the Elasticsearch Definitive Guide's [Finding Your Feet](http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/_finding_your_feet.html).

* [Requirements](#requirements)
* [Installation](#installation)
* [Testing](#testing)
* [Quick Start](#quick-start)
* [General Usage](#general-usage)
* [Indexing](#indexing)
    * [Reindexing](#reindexing)
    * [Partial Indexing](#partial-indexing)
	* [Sheer Index Settings](#sheer-index-settings)
	* [Content Processors](#content-processors)
	* [Mappings](#mappings)
* [Serving](#serving)
	* [Templates](#templates)
		* [Context](#context)
			* [`selected_filters_for_field`](#selected_filters_for_fieldfieldname)
			* [`is_filter_selected`](#is_filter_selectedfieldname-value)
			* [`queries`](#queries)
			* [`more_like_this`](#more_like_thishit)
			* [`get_document`](#get_documentdoctype-docid)
	* [Elasticsearch Lookup URLs](#elasticsearch-lookup-urls)
	* [Blueprints](#blueprints)
	* [Query API](#query-api)
		* [`QueryFinder`](#queryfinder)
		* [`Query`](#query)
		* [`QueryResult`](#queryresult)
		* [`QueryHit`](#queryhit)
- [Licensing](#licensing)


## Requirements

Sheer is a Python application that requires:

- [Elasticsearch](http://www.elasticsearch.org/)
- [Flask](http://flask.pocoo.org/)

Recommended for installing and running Sheer:

- [virtualenv](https://virtualenv.pypa.io/en/latest/)
- [virtualenvwrapper](https://virtualenvwrapper.readthedocs.org)

Running tests requires:

- [nose](https://nose.readthedocs.org/en/latest/)
- [mock](http://www.voidspace.org.uk/python/mock/)

## Installation

To install Sheer you will first need to install 
[Elasticsearch](http://www.elasticsearch.org/). This can be acomplished 
a number of ways, many of which are detailed in 
[in the Elasticsearch documentation](http://www.elastic.co/guide/en/elasticsearch/reference/current/setup.html). 
On Mac OS X it can be installed using [homebrew](http://brew.sh/):

```
brew install elasticsearch
```

There are also [Elasticsearch apt and Yum repositories](http://www.elastic.co/guide/en/elasticsearch/reference/current/setup-repositories.html).

To install Sheer itself, it is recommended to create a 
[`virtualenv`](https://virtualenv.pypa.io/en/latest/) using 
[`virtualenvwrapper`](https://virtualenvwrapper.readthedocs.org).

```
mkvirtualenv sheer
workon sheer
```

Then you can clone the Sheer repository and install the Python 
requirements using `pip`:

```shell
git clone https://github.com/cfpb/sheer
pip install -r sheer/requirements.txt
```

You can then install Sheer with `pip`. `pip -e` installs Sheer in
"editable" mode, which means it runs from the path where you've cloned
it, and any changes you `git pull` from upstream won't have to be
installed again.

```shell
pip install -e sheer
```

## Testing

To run the Sheer tests, you'll need the Python packages 
[nose](https://nose.readthedocs.org/en/latest/) and 
[mock](http://www.voidspace.org.uk/python/mock/) installed. Both can be
installed via `pip`:

```shell
pip install nose mock
```

Both are also installed by the Sheer [`requirements.txt` file](#installation).

To run the tests, simply run:

```shell
nosetest sheer
```

## Quick Start

This quick start assumes you have an existing Sheer site you want to
load content for and serve.

```shell
cd path/to/my/sheer/site
```

Index the site's content in Elasticsearch:

```shell
sheer index
```

Serve the site at [http://localhost:7000](http://localhost:7000):
{'fraction_non_alphanumeric': 0.144681448217794, 'fraction_numerical': 0.0025259612685938817, 'mean_word_length': 5.26274165202109, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 13, 'lorem ipsum': 0, 'www.': 15, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2667853', 'n_tokens_mistral': 2425, 'n_tokens_neox': 2236, 'n_words': 586}