<commit_before>kleineAnfragen.
===============

Liberating *kleine Anfragen* from *Parlamentsdokumentationssysteme*.

Development
-----------

For a simple and quick development environment, [fig](http://fig.sh) is used. Install [docker](https://docs.docker.com/installation/) and [fig](http://www.fig.sh/install.html), then run:

    fig up

Fig downloads the required services (postgres, elasticsearch, redis, ...) as docker containers and links them with the app.
If you want to look into postgres or elasticsearch, uncomment the `ports` section in `fig.yml`.

You may be required to execute database migrations. Try this:

    fig run web rake db:migrate
    fig run web rake db:seed

To get a rails console, run:

    fig run web rails c

Dependencies
------------

* ruby 2.1.5
* postgres
* elasticsearch (for search)
* redis (for resque)
* nodejs (for asset compiling)
* [tika](http://tika.apache.org) (for extracting text from pdfs)
* [Nomenklatura](https://github.com/pudo/nomenklatura) (for normalization of people names, ministries and parties)
* Poppler / pdftoppm (for thumbnailing)
* [image_optim binaries](https://github.com/toy/image_optim#binaries-installation) (for compressing thumbnails)

Configuration
-------------

* `config/application.rb`

	Please change the `config.x.user_agent` to your own email address.<commit_msg>Add nomenklatura setup instructions to readme<commit_after>kleineAnfragen.
===============

Liberating *kleine Anfragen* from *Parlamentsdokumentationssysteme*.

Development
-----------

For a simple and quick development environment, [fig](http://fig.sh) is used. Install [docker](https://docs.docker.com/installation/) and [fig](http://www.fig.sh/install.html), then run:

    fig up

Fig downloads the required services (postgres, elasticsearch, redis, ...) as docker containers and links them with the app.
If you want to look into postgres or elasticsearch, uncomment the `ports` section in `fig.yml`.

You may be required to execute database migrations. Try this:

    fig run web rake db:migrate
    fig run web rake db:seed

To get a rails console, run:

    fig run web rails c

### Normalizing Names with Nomenklatura

For normalizing names of people, parties and ministries, we use [Nomenklatura](https://github.com/pudo/nomenklatura).

If you want to use nomenklatura while developing, you need to edit fig.yml:
* Uncomment the nomenklatura link
* the `NOMENKLATURA_` environment variables
* the whole nomenklatura image
* set `GITHUB_CLIENT_ID` and `GITHUB_CLIENT_SECRET` to those of a [new Github OAuth application](https://github.com/settings/applications/new).

After your next `fig up` login to your nomenklatura instance (reachable at http://localhost:8080) and get the API key from the _profile_ link. Insert it into fig.yml.

kleineAnfragen needs multiple Datasets with the following identifiers that must be created in Nomenklatura:

* `ka-parties`
* `ka-people-XX` (replace XX with a two letter state)
* `ka-ministries-XX` (replace XX with a two letter state)

Dependencies
------------

* ruby 2.1.5
* postgres
* elasticsearch (for search)
* redis (for resque)
* nodejs (for asset compiling)
* [tika](http://tika.apache.org) (for extracting text from pdfs)
* [Nomenklatura](https://github.com/pudo/nomenklatura) (for normalization of people names, ministries and parties)
* Poppler / pdftoppm (for thumbnailing)
* [image_optim binaries](https://github.com/toy/image_optim#binaries-installation) (for compressing thumbnails)

Configuration
-------------

* `config/application.rb`

	Please change the `config.x.user_agent` to your own email address.
{'fraction_non_alphanumeric': 0.12143845089903181, 'fraction_numerical': 0.0027662517289073307, 'mean_word_length': 4.263464337700146, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 8, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3239121', 'n_tokens_mistral': 1126, 'n_tokens_neox': 1078, 'n_words': 387}