<commit_before># pyontutils
python utilities for working with ontologies

## Requirements
This repo requires Python3.6 or later.
See Pipfile for additional requirements.
ontload requires Java8 and >=maven3.3 in order to build SciGraph.
[parcellation.py](pyontutils/parcellation.py) requires [FSL](https://fsl.fmrib.ox.ac.uk/fsl/fslwiki/)
to be installed or you need to obtain the [atlases](https://fsl.fmrib.ox.ac.uk/fsl/fslwiki/Atlases) in
some other way. In order to build all the packages required by this repo you will need
gcc (and toolchain) installed and will need to have the development packages for
`postgresql`, `libxml`, and `protobuf`  installed on your system.

## Installation
The easiest way to install pyontutils is to use pipenv. It makes it easy to manage
specific version of packages needed by pyontutils. For example in order to get good
(deterministic) ttl serialization from these tools you need to use my modified version
of rdflib (see https://github.com/RDFLib/rdflib/pull/649 and https://github.com/RDFLib/rdflib/pull/793).
[Pipenv](https://pipenv.readthedocs.io/en/latest/#install-pipenv-today) makes it easier
to accomplish this.

1. In your preferred folder `git clone https://github.com/tgbugs/pyontutils.git`
2. `cd pyontutils`
3. `pipenv install --skip-lock`. Note that `mysql-connector` (aka `mysql-connector-python`) often cannot find the
files it needs to build.  When installing pass them in as environment variables (you may need to adjust
exact paths for your system). `MYSQLXPB_PROTOBUF_INCLUDE_DIR=/usr/include/google/protobuf MYSQLXPB_PROTOBUF_LIB_DIR=/usr/lib64 MYSQLXPB_PROTOC=/usr/bin/protoc pipenv install --skip-lock`
4. `pipenv shell` to enter the virtual environment where everything should work.

Alternately, if manage your packages via another system you can create a development setup
by adding this folder to your `PYTHONPATH` environment variable using
`export PYTHONPATH=PYTHONPATH:"$(pwd)"` from the location of this readme.
If you use a development setup you will need to create symlinks described below.<commit_msg>readme further clarification for mysql connector issues<commit_after># pyontutils
python utilities for working with ontologies

## Requirements
This repo requires Python3.6 or later.
See Pipfile for additional requirements.
ontload requires Java8 and >=maven3.3 in order to build SciGraph.
[parcellation.py](pyontutils/parcellation.py) requires [FSL](https://fsl.fmrib.ox.ac.uk/fsl/fslwiki/)
to be installed or you need to obtain the [atlases](https://fsl.fmrib.ox.ac.uk/fsl/fslwiki/Atlases) in
some other way. In order to build all the packages required by this repo you will need
gcc (and toolchain) installed and will need to have the development packages for
`postgresql`, `libxml`, and `protobuf`  installed on your system.

## Installation
The easiest way to install pyontutils is to use pipenv. It makes it easy to manage
specific version of packages needed by pyontutils. For example in order to get good
(deterministic) ttl serialization from these tools you need to use my modified version
of rdflib (see https://github.com/RDFLib/rdflib/pull/649 and https://github.com/RDFLib/rdflib/pull/793).
[Pipenv](https://pipenv.readthedocs.io/en/latest/#install-pipenv-today) makes it easier
to accomplish this.

1. In your preferred folder `git clone https://github.com/tgbugs/pyontutils.git`
2. `cd pyontutils`
3. `pipenv install --skip-lock`. Note that `mysql-connector` (aka `mysql-connector-python`) often cannot find the
files it needs to build.  When installing pass them in as environment variables (you may need to adjust
exact paths for your system). `MYSQLXPB_PROTOBUF_INCLUDE_DIR=/usr/include/google/protobuf MYSQLXPB_PROTOBUF_LIB_DIR=/usr/lib64 MYSQLXPB_PROTOC=/usr/bin/protoc pipenv install --skip-lock`. There are some systems on which even this is not sufficient. If you encounter this situation modify to mysql-connector entry in Pipfile from `"*"` => `"==2.1.6"`. And then run the command without environment variables.
4. `pipenv shell` to enter the virtual environment where everything should work.

Alternately, if manage your packages via another system you can create a development setup
by adding this folder to your `PYTHONPATH` environment variable using
`export PYTHONPATH=PYTHONPATH:"$(pwd)"` from the location of this readme.
If you use a development setup you will need to create symlinks described below.
{'fraction_non_alphanumeric': 0.08117913832199547, 'fraction_numerical': 0.008390022675736962, 'mean_word_length': 4.960810810810811, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 6, 'https://': 12, 'lorem ipsum': 0, 'www.': 0, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3151496', 'n_tokens_mistral': 1311, 'n_tokens_neox': 1260, 'n_words': 539}