<commit_before>pheme.warehouse
===============

**Public Health EHR Message Engine (PHEME), Data Warehouse Module**

Module responsible for creation and maintenance of the `Data Warehouse`
used by the PHEME system.  Several `Mirth Connect`_ channels process
HL/7 batch files writing portions of their content to a warehouse
schema for consumption by other PHEME modules.

Requirements
------------

* Install `Mirth Connect`_ and provide the ``admin`` user with a
  secure password and set this password in
  ``$MIRTH_HOME/conf/mirth-cli-config.properties`` for
  ``pheme.warehouse`` command line tools proper function.
* `PostgreSQL`_
* `Python`_ 2.7.*

Install
-------

Beyond the requirements listed above, ``pheme.warehouse`` is
dependent on the ``pheme.util`` module.  Although future builds may
automatically pick it up, for now, clone and build it in the same
virtual environment (or native environment) being used for
``pheme.warehouse``::

    git clone https://github.com/pbugni/pheme.util.git
    cd pheme.util
    ./setup.py develop
    cd ..

Then clone and build this module::

    git clone https://github.com/pbugni/pheme.warehouse.git
    cd pheme.warehouse
    ./setup.py develop

A pheme config file (see ``pheme.util.config``) must specify where
`Mirth Connect` was installed::

    [mirth]
    mirth_home=/opt/mirth/current

Transformation of the mirth channels uses values in the pheme config
file.  (NB for the tests to pass, ``create_table_user`` must be the
same as ``database_user``, however for security reasons, this should
not be the case on production systems) Values needed include::

    [mirth]
    create_table_user=system
    create_table_password=password
    database=warehouse
    database_user=user
    database_password=password
    input_dir=/opt/pheme/filedrops
    error_dir=/opt/pheme/error
    output_dir=/opt/pheme/processed

To transform the channels, provide the checked out location of the
channels (i.e. ``pheme.warehouse/channels``) and a temporary directory for
output::

    transform_channels channels /tmp/transformed

Deploy the transformed channels.  See `Mirth Connect` requirement
above and make sure it's running::

    deploy_channels /tmp/transformed

Running
-------

The executable programs provided by ``pheme.warehouse`` are listed
under [console_scripts] within the project's setup.py file.  All take
the standard help options [-h, --help].  Invoke with help for more
information::

    mirth_channel_transform --help<commit_msg>Added missing block on database configuration and table creation.<commit_after>pheme.warehouse
===============

**Public Health EHR Message Engine (PHEME), Data Warehouse Module**

Module responsible for creation and maintenance of the `Data Warehouse`
used by the PHEME system.  Several `Mirth Connect`_ channels process
HL/7 batch files writing portions of their content to a warehouse
schema for consumption by other PHEME modules.

Requirements
------------

* Install `Mirth Connect`_ and provide the ``admin`` user with a
  secure password and set this password in
  ``$MIRTH_HOME/conf/mirth-cli-config.properties`` for
  ``pheme.warehouse`` command line tools proper function.
* `PostgreSQL`_
* `Python`_ 2.7.*

Install
-------

Beyond the requirements listed above, ``pheme.warehouse`` is
dependent on the ``pheme.util`` module.  Although future builds may
automatically pick it up, for now, clone and build it in the same
virtual environment (or native environment) being used for
``pheme.warehouse``::

    git clone https://github.com/pbugni/pheme.util.git
    cd pheme.util
    ./setup.py develop
    cd ..

Then clone and build this module::

    git clone https://github.com/pbugni/pheme.warehouse.git
    cd pheme.warehouse
    ./setup.py develop

A pheme config file (see ``pheme.util.config``) must specify where
`Mirth Connect` was installed::

    [mirth]
    mirth_home=/opt/mirth/current

Transformation of the mirth channels uses values in the pheme config
file.  (NB for the tests to pass, ``create_table_user`` must be the
same as ``database_user``, however for security reasons, this should
not be the case on production systems) Values needed include::

    [mirth]
    create_table_user=system
    create_table_password=password
    database=warehouse
    database_user=user
    database_password=password
    input_dir=/opt/pheme/filedrops
    error_dir=/opt/pheme/error
    output_dir=/opt/pheme/processed

For database creation to work, postgres must be configured to handle
the user named above.  Substitute in the database, user and password
values below to match the settings used in the pheme config file::

    $ sudo -u postgres createdb warehouse
    $ sudo -u postgres createuser system
    Shall the new role be a superuser? (y/n) n
    Shall the new role be allowed to create databases? (y/n) n
    Shall the new role be allowed to create more new roles? (y/n) n
    $ sudo -u postgres psql
    postgres=# alter user system password 'password';
    ALTER ROLE
    postgres=# alter database warehouse owner to system;
    ALTER DATABASE

The `setup.py develop` call above in pheme.warehouse also created an
executable for database schema creation.  NB, this will destroy any
data in the named database (which is why it prompts for `destroy`)::

    create_warehouse_tables

To transform the channels, provide the checked out location of the
channels (i.e. ``pheme.warehouse/channels``) and a temporary directory for
output::

    transform_channels channels /tmp/transformed

Deploy the transformed channels.  See `Mirth Connect` requirement
above and make sure it's running::

    deploy_channels /tmp/transformed

Running
-------

The executable programs provided by ``pheme.warehouse`` are listed
under [console_scripts] within the project's setup.py file.  All take
the standard help options [-h, --help].  Invoke with help for more
information::

    mirth_channel_transform --help
{'fraction_non_alphanumeric': 0.08816078365141024, 'fraction_numerical': 0.0010133423408208073, 'mean_word_length': 3.951505016722408, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3244097', 'n_tokens_mistral': 1730, 'n_tokens_neox': 1618, 'n_words': 684}