<commit_before># Deconst Jekyll Preparer

.md :point_right: .json :point_right: :wrench: :point_right: content service

[![Docker Repository on Quay.io](https://quay.io/repository/deconst/preparer-jekyll/status "Docker Repository on Quay.io")](https://quay.io/repository/deconst/preparer-jekyll)

*preparermd* builds each page of a [Jekyll site](http://jekyllrb.com/) into custom JSON metadata envelopes and broadcasts them to a [content service](https://github.com/deconst/content-service) that performs storage and indexing for presentation and search.

It's intended to be used within a CI system to present content to the rest of the build pipeline.

## Running Locally

To run the Jekyll preparer locally, you'll need to install:

 * [Docker](https://docs.docker.com/installation/#installation) for your platform. Choose the boot2docker option for Mac OS X or Windows.

Once you have Docker set up, export any desired configuration variables, run `deconst-preparer-jekyll.sh` with the path of a control repository clone to prepare the Jekyll content found there.

```bash
export CONTENT_STORE_URL=http://my-content-store.com:9000/
export CONTENT_ID_BASE=https://github.com/myorg/myrepo

./deconst-preparer-jekyll.sh /path/to/control-repo
```

### Configuration

The following values must be present in the build environment to submit assets:

 * `CONTENT_STORE_URL` must be the base URL of the publicly available content store service. The prepare script defaults this to one consistent with our docker-compose setups.
 * `CONTENT_ID_BASE` must be set to a prefix that's unique among the content repositories associated with the target deconst instance. Our convention is to use the base URL of the GitHub repository.
 * `TRAVIS_PULL_REQUEST` must be set to `"false"`. Travis automatically sets this value for your build environment on the primary branch of your repository.<commit_msg>Document the API key env var in the README.<commit_after># Deconst Jekyll Preparer

.md :point_right: .json :point_right: :wrench: :point_right: content service

[![Docker Repository on Quay.io](https://quay.io/repository/deconst/preparer-jekyll/status "Docker Repository on Quay.io")](https://quay.io/repository/deconst/preparer-jekyll)

*preparermd* builds each page of a [Jekyll site](http://jekyllrb.com/) into custom JSON metadata envelopes and broadcasts them to a [content service](https://github.com/deconst/content-service) that performs storage and indexing for presentation and search.

It's intended to be used within a CI system to present content to the rest of the build pipeline.

## Running Locally

To run the Jekyll preparer locally, you'll need to install:

 * [Docker](https://docs.docker.com/installation/#installation) for your platform. Choose the boot2docker option for Mac OS X or Windows.

Once you have Docker set up, export any desired configuration variables, run `deconst-preparer-jekyll.sh` with the path of a control repository clone to prepare the Jekyll content found there.

```bash
export CONTENT_STORE_URL=http://my-content-store.com:9000/
export CONTENT_STORE_APIKEY="<KEY>..."
export CONTENT_ID_BASE=https://github.com/myorg/myrepo

./deconst-preparer-jekyll.sh /path/to/control-repo
```

### Configuration

The following values must be present in the build environment to submit assets:

 * `CONTENT_STORE_URL` must be the base URL of the publicly available content store service. The prepare script defaults this to one consistent with our docker-compose setups.
 * `CONTENT_STORE_APIKEY` must be a valid API key issued by the content service. See [the content service documentation](https://github.com/deconst/content-service#post-keysnamedname) for instructions on generating an API key.
 * `CONTENT_ID_BASE` must be set to a prefix that's unique among the content repositories associated with the target deconst instance. Our convention is to use the base URL of the GitHub repository.
 * `TRAVIS_PULL_REQUEST` must be set to `"false"`. Travis automatically sets this value for your build environment on the primary branch of your repository.
{'fraction_non_alphanumeric': 0.08165767533104464, 'fraction_numerical': 0.0024521824423737125, 'mean_word_length': 5.01622418879056, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 4, 'https://': 11, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5053341', 'n_tokens_mistral': 1169, 'n_tokens_neox': 1105, 'n_words': 475}