<commit_before># GwentAPI
The code for GwentAPI

GwentAPI is a RESTful API dedicated to provide information regarding GWENT®: The Witcher Card Game. You may use its interface to consume information on cards, factions, etc.

## Requirement & dependencies

* Bash (use MINGW/git bash or cygwin for windows)
* [Go 1.8](https://golang.org/dl/)
* MongoDB 3.4
* Python 3.4.2
    * pymongo 3.4.0
* Git (optional)
* [jq-1.5](https://stedolan.github.io/jq/)

## Installation

Assuming you installed and configure all the required softwares listed above and that you have a working Go setup:

### Build the software
1. go get github.com/GwentAPI/gwentapi
2. From the project directory, initialize the submodules:
    * ``git submodule init``
    * ``git submodule update``
3. GwentAPI uses the expimental [dep](https://github.com/golang/dep) tool to manage the Go dependencies, however they are included in the ``vendor`` directory to help build the software on CI platforms.
4. Change your monboDB credentials in the ``init()`` of the ``/dataLayer/dal/db.go`` file if needed. By default it connects to localhost on no particular users (no SSL, access control not enabled).
5. Compile:
    * ``go build``
6. Optionally set the version of the binary (displayed with the -v flag) by changing the value of ``version`` in ``main.go`` before compiling or set it at compile time with the following command:
    * ``go build -ldflags="-X main.version=<versionString>" ``
7. Optionally modify the ``baseURL`` in the ``config.toml`` file to change where resources are pointing to.

**The program will run on port 8080 by default.**
    
### Feed the database

The ``data`` submodule contains everything needed to feed the database.

Assuming that you have a running mongoDB instance :

1. Run the ``extract.sh`` bash script, giving it the latest.jsonl file as a parameter:
    * ``./extract.sh input/latest.jsonl`` which will use jq to extract and create collection related .jsonl files on the same directory as the script.
2. Assuming that ``db_setup.py`` is in the same location as the newly generated files, populate the database by running db_setup.py (the command differs depending on your setup):
    * ``python3 db_setup.py``
    
``db_setup.py`` will create the database if not present along with all the collections. If a collection is already present, the script will refuse to run unless you force it:
* ``python3 db_setup.py -f`` forcing the script will drop all the collections before re-populating the database.

If you enabled authentication on mongoDB or you are using a different port/host other than the default port on localhost, the script support overriding the values. Learn more with:
``python3 db_setup.py --help``. SSL is not supported. Remember that the go program **doesn't** support those different configurations at the moment.


## How to contribute?

TODO: steps on installing goa.

## Where are the artworks and why does latest.jsonl refers to gwentify?

latest.jsonl is generated by [Arachas](https://github.com/GwentAPI/arachas) a web crawler that extract data from [gwentify.com](gwentify.com).

GwentAPI doesn't hot link to gwentify and so, the database population script will replace the url. The artworks isn't included because git doesn't like binary files and neither do I like including binary files in git.<commit_msg>fix readme link<commit_after># GwentAPI
The code for GwentAPI

GwentAPI is a RESTful API dedicated to provide information regarding GWENT®: The Witcher Card Game. You may use its interface to consume information on cards, factions, etc.

## Requirement & dependencies

* Bash (use MINGW/git bash or cygwin for windows)
* [Go 1.8](https://golang.org/dl/)
* MongoDB 3.4
* Python 3.4.2
    * pymongo 3.4.0
* Git (optional)
* [jq-1.5](https://stedolan.github.io/jq/)

## Installation

Assuming you installed and configure all the required softwares listed above and that you have a working Go setup:

### Build the software
1. go get github.com/GwentAPI/gwentapi
2. From the project directory, initialize the submodules:
    * ``git submodule init``
    * ``git submodule update``
3. GwentAPI uses the expimental [dep](https://github.com/golang/dep) tool to manage the Go dependencies, however they are included in the ``vendor`` directory to help build the software on CI platforms.
4. Change your monboDB credentials in the ``init()`` of the ``/dataLayer/dal/db.go`` file if needed. By default it connects to localhost on no particular users (no SSL, access control not enabled).
5. Compile:
    * ``go build``
6. Optionally set the version of the binary (displayed with the -v flag) by changing the value of ``version`` in ``main.go`` before compiling or set it at compile time with the following command:
    * ``go build -ldflags="-X main.version=<versionString>" ``
7. Optionally modify the ``baseURL`` in the ``config.toml`` file to change where resources are pointing to.

**The program will run on port 8080 by default.**
    
### Feed the database

The ``data`` submodule contains everything needed to feed the database.

Assuming that you have a running mongoDB instance :

1. Run the ``extract.sh`` bash script, giving it the latest.jsonl file as a parameter:
    * ``./extract.sh input/latest.jsonl`` which will use jq to extract and create collection related .jsonl files on the same directory as the script.
2. Assuming that ``db_setup.py`` is in the same location as the newly generated files, populate the database by running db_setup.py (the command differs depending on your setup):
    * ``python3 db_setup.py``
    
``db_setup.py`` will create the database if not present along with all the collections. If a collection is already present, the script will refuse to run unless you force it:
* ``python3 db_setup.py -f`` forcing the script will drop all the collections before re-populating the database.

If you enabled authentication on mongoDB or you are using a different port/host other than the default port on localhost, the script support overriding the values. Learn more with:
``python3 db_setup.py --help``. SSL is not supported. Remember that the go program **doesn't** support those different configurations at the moment.


## How to contribute?

TODO: steps on installing goa.

## Where are the artworks and why does latest.jsonl refers to gwentify?

latest.jsonl is generated by [Arachas](https://github.com/GwentAPI/arachas) a web crawler that extract data from [https://gwentify.com](gwentify.com).

GwentAPI doesn't hot link to gwentify and so, the database population script will replace the url. The artworks isn't included because git doesn't like binary files and neither do I like including binary files in git.
{'fraction_non_alphanumeric': 0.08442916915720262, 'fraction_numerical': 0.008368200836820083, 'mean_word_length': 4.241190289741581, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 9, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1607990', 'n_tokens_mistral': 1895, 'n_tokens_neox': 1786, 'n_words': 931}