<commit_before>Mapa76
======

Main repository for [Mapa76 project](http://mapa76.info/) (in spanish).


Dependencies
------------

  * Ruby 1.9+
  * Bundler (`bundler` gem)
  * MongoDB server
  * FreeLing 3.0


Install
-------

First, run `bundle install` to install all gem dependencies.

    $ bundle install

Create your MongoDB configuration file based on the sample file, and modify the
connection options to suit your needs:

    $ cp config/mongoid.yml.sample config/mongoid.yml


### FreeLing ###

The NER module currently uses FreeLing, an open source suite of language
analyzers written in C++.

#### Download ####

This has been tested on *FreeLing 3.0a1* only. Because this is an alpha
release, there are no binary packages available. You can download the source
[here](http://devel.cpl.upc.edu/freeling/downloads/16) (114Mb~) and compile it.

#### Compile and Install ####

For compiling the source, you need the `libboost` and `libicu` libraries. On
Debian / Ubuntu machines, you can run:

    # apt-get install libboost-dev libboost-filesystem-dev libboost-program-options-dev libboost-regex-dev libicu-dev

Then, just execute `./configure`, `make` and `make install` as usual.


Usage
-----

Start the Padrino server and visit [http://localhost:3000/](http://localhost:3000/)

    $ bundle exec padrino start<commit_msg>update README with instructions on how to start workers with Resque<commit_after>Mapa76
======

Main repository for [Mapa76 project](http://mapa76.info/) (in spanish).


Dependencies
------------

  * Ruby 1.9+
  * Bundler (`bundler` gem)
  * MongoDB server
  * FreeLing 3.0


Install
-------

First, run `bundle install` to install all gem dependencies.

    $ bundle install

Create your MongoDB configuration file based on the sample file, and modify the
connection options to suit your needs:

    $ cp config/mongoid.yml.sample config/mongoid.yml


### FreeLing ###

The NER module currently uses FreeLing, an open source suite of language
analyzers written in C++.

#### Download ####

This has been tested on *FreeLing 3.0a1* only. Because this is an alpha
release, there are no binary packages available. You can download the source
[here](http://devel.cpl.upc.edu/freeling/downloads/16) (114Mb~) and compile it.

#### Compile and Install ####

For compiling the source, you need the `libboost` and `libicu` libraries. On
Debian / Ubuntu machines, you can run:

    # apt-get install libboost-dev libboost-filesystem-dev libboost-program-options-dev libboost-regex-dev libicu-dev

Then, just execute `./configure`, `make` and `make install` as usual.


Usage
-----

Start the Padrino server and visit [http://localhost:3000/](http://localhost:3000/)

    $ bundle exec padrino start

To start workers for document processing, you need to run at least one Resque worker:

    $ QUEUE=* VERBOSE=1 bundle exec rake resque:work

you can run multiple workers with the `resque:workers` task:

    $ COUNT=2 QUEUE=* bundle exec rake resque:workers
{'fraction_non_alphanumeric': 0.11528150134048257, 'fraction_numerical': 0.018096514745308313, 'mean_word_length': 3.7156398104265405, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1805380', 'n_tokens_mistral': 955, 'n_tokens_neox': 875, 'n_words': 345}