<commit_before>![CI status](https://codeship.com/projects/401c8740-652d-0134-fcd9-3aa7f9d29c3d/status?branch=dev)
# Beit-Hatfusot Backend

Welcome to the API server of the museum of the jewish people.  The code is in
Python using Flask and MongoDB.  The API docs are at
http://api.dbs.bh.org.il/v1/docs

The server is built on top of 6 datasets the museum has collected in the last four
decades.  We have >72,000 Pictures and >1,100 movies. We have text articles
about >6,000 communities >8,000 luminaries and >17,000 family names.

The last dataset is of family trees.
People have contributed to the museum ~11,000 trees with ~5,000,000 individuals.

## Installation

This server uses MongoDB so you'll have to have one up and running.
Mongo installation is quite simple, just read the
[manual](https://docs.mongodb.com/manual/installation/).

While a search engine is not a must, it is recommended to install
[Elasticsearch](https://www.elastic.co/downloads/elasticsearch).
Once you have MongoDB installed you can go on to install the python code:

### Linux

	$ sudo apt-get install libffi-dev libjpeg62 libjpeg62-dev zlib1g-dev libssl-dev
    $ git clone [your_fork@github...]
    $ cd dbs-back
    $ virtualenv env
    $ . env/bin/activate
    $ pip install -r requirements.txt

## Getting a DB

To play with the server you'll need some data in your db. Please download the
tar ball from [here](https://storage.googleapis.com/bhs-resources/bhdata.tgz) and run

    $ tzr xf bhdata.tgz
    $ mongorestore -d bhdata dump/bhdata

To load the search engine with the dump you restores run:

    $ python scripts/dump_mongo_to_es.py --db bhdata

## Testing

    $ py.test tests

## Running a local server

A local server uses port 5000 and is activated by:

    $ python scripts/runserver.py

## Using the shell

If you want to play around with the flask shell you can start it using a special
script and gain access to the `app` object.

 $ python scripts/flask_shell.py

## Getting a the frontend

The frontend that uses this API server lives
[here](https://github.com/Beit-Hatfutsot/dbs-front). To run the fronend so it
will use your local server you just need to run, in separate window:

    $ API_SERVER=local gulp serve

(the API_SERVER part tells the frontend to use a server at localhost:5000)

## Migration

Part of the code in this repository deals with migrating the databases from the
legacy MSSQL server.  You don't haver to do it, as you get all the data you need
from the dump above, but if you want to contribute to the migration you'll need 
access to MoJP local server on MoJP local network.
The migration process is executed done using a celery based task
queue and redis as a storage backend.  To install the required packages run:

    $ pip install -r requirements.migrate.txt

To activate the migration worker run:

    $ celery -A migration.tasks worker --loglevel info

Once the worker is listening, run in a separate window:

    $ python scripts/migrate.py --lasthours 200

## Contributing

Contributions from both Jews and Gentiles are welcomed! We even have a
`beginner` label to help you start with (hopefully) simple issues.
Once you have an issue, just follow these simple steps:
<commit_msg>updating migration section<commit_after>![CI status](https://codeship.com/projects/401c8740-652d-0134-fcd9-3aa7f9d29c3d/status?branch=dev)
# Beit-Hatfusot Backend

Welcome to the API server of the museum of the jewish people.  The code is in
Python using Flask and MongoDB.  The API docs are at
http://api.dbs.bh.org.il/v1/docs

The server is built on top of 6 datasets the museum has collected in the last four
decades.  We have >72,000 Pictures and >1,100 movies. We have text articles
about >6,000 communities >8,000 luminaries and >17,000 family names.

The last dataset is of family trees.
People have contributed to the museum ~11,000 trees with ~5,000,000 individuals.

## Installation

This server uses MongoDB so you'll have to have one up and running.
Mongo installation is quite simple, just read the
[manual](https://docs.mongodb.com/manual/installation/).

While a search engine is not a must, it is recommended to install
[Elasticsearch](https://www.elastic.co/downloads/elasticsearch).
Once you have MongoDB installed you can go on to install the python code:

### Linux

	$ sudo apt-get install libffi-dev libjpeg62 libjpeg62-dev zlib1g-dev libssl-dev
    $ git clone [your_fork@github...]
    $ cd dbs-back
    $ virtualenv env
    $ . env/bin/activate
    $ pip install -r requirements.txt

## Getting a DB

To play with the server you'll need some data in your db. Please download the
tar ball from [here](https://storage.googleapis.com/bhs-resources/bhdata.tgz) and run

    $ tzr xf bhdata.tgz
    $ mongorestore -d bhdata dump/bhdata

To load the search engine with the dump you restores run:

    $ python scripts/dump_mongo_to_es.py --db bhdata

## Testing

    $ py.test tests

## Running a local server

A local server uses port 5000 and is activated by:

    $ python scripts/runserver.py

## Using the shell

If you want to play around with the flask shell you can start it using a special
script and gain access to the `app` object.

 $ python scripts/flask_shell.py

## Getting a the frontend

The frontend that uses this API server lives
[here](https://github.com/Beit-Hatfutsot/dbs-front). To run the fronend so it
will use your local server you just need to run, in separate window:

    $ API_SERVER=local gulp serve

(the API_SERVER part tells the frontend to use a server at localhost:5000)

## Migration

Part of the code in this repository deals with migrating the databases from the
legacy MSSQL server.  You don't haver to do it, as you get all the data you need
from the dump above, but if you want to contribute to the migration you'll need 
access to MoJP local server on MoJP local network.
The migration process is executed done using a celery based task
queue and redis as a storage backend.  To install the required packages run:

    $ pip install -r requirements.migrate.txt

You'll also need to create some local system folders:

    # mkdir /run/bhs
    # mkdir /etc/bhs

And get the super secret `migrate_config.yaml` file into `/etc/bhs`.
To activate the migration worker run:

    $ celery -A migration.tasks worker --loglevel info

Once the worker is listening, run in a separate window:

    $ python scripts/migrate.py --lasthours 200

## Contributing

Contributions from both Jews and Gentiles are welcomed! We even have a
`beginner` label to help you start with (hopefully) simple issues.
Once you have an issue, just follow these simple steps:

{'fraction_non_alphanumeric': 0.06789378394689197, 'fraction_numerical': 0.022027761013880508, 'mean_word_length': 3.7862815884476535, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 13, 'https://': 10, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1131869', 'n_tokens_mistral': 2091, 'n_tokens_neox': 1922, 'n_words': 912}