<commit_before>
===========
Development
===========


With docker
===========


Build image


.. code-block:: bash

   docker build -t pg_bawler .


Run tests


.. code-block:: bash

   docker run \
      -u root \
      -v $(pwd):/opt/pg_bawler \
      -v /var/run/docker.sock:/var/run/docker.sock \
      -t pg_bawler \
      /bin/sh runtests.sh



With local virtualenv
=====================


.. note:: Make sure that you are using python 3.5+

Create virtualenv for example with `venv <https://docs.python.org/3/library/venv.html>`_.

.. code-block:: bash

   python3 -m venv /path/to/new/virtual/environment


Activate created environment

.. code-block:: bash

   source /path/to/new/virtua/environment


With activated virtual environment, install dependencies

.. code-block:: bash

   pip install \
       -r requirements.txt \
       -r test-requirements.txt \
       -r dev-requirements.txt


To verify the installation run the tests

.. code-block:: bash<commit_msg>Fix virtualevn activation step<commit_after>
===========
Development
===========


With docker
===========


Build image


.. code-block:: bash

   docker build -t pg_bawler .


Run tests


.. code-block:: bash

   docker run \
      -u root \
      -v $(pwd):/opt/pg_bawler \
      -v /var/run/docker.sock:/var/run/docker.sock \
      -t pg_bawler \
      /bin/sh runtests.sh



With local virtualenv
=====================


.. note:: Make sure that you are using python 3.5+

Create virtualenv for example with `venv <https://docs.python.org/3/library/venv.html>`_.

.. code-block:: bash

   python3 -m venv /path/to/new/virtual/environment


Activate created environment

.. code-block:: bash

   source /path/to/new/virtua/environment/bin/activate


With activated virtual environment, install dependencies

.. code-block:: bash

   pip install \
       -r requirements.txt \
       -r test-requirements.txt \
       -r dev-requirements.txt


To verify the installation run the tests

.. code-block:: bash
{'fraction_non_alphanumeric': 0.16096579476861167, 'fraction_numerical': 0.004024144869215292, 'mean_word_length': 2.5517857142857143, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3981043', 'n_tokens_mistral': 688, 'n_tokens_neox': 611, 'n_words': 172}