<commit_before>surechembl-data-client
======================

A collection of scripts for retrieving SureChEMBL data and building a relational database of patent chemistry. 

Please clone this repository to your local environment and follow the instructions below. 

Released under the MIT license. Access to the SureChEMBL data must be obtained from EBI-EMBL.

# Installation

## Core dependencies

Please make sure that the following dependencies are available:

* Python 2.7
* pip
* RDBMS of your choice
* Python DB API compatible client (see below)

Note that the client code has been tested against Oracle, but is written in a DB-agnostic way to simplify usage in
other scenarios.

## Python libraries

The complete set of required libraries can be found here:

    src/requirements.txt

This file was generated using pip's 'freeze' command. Use this file with your version of pip to ensure all required libraries are available.

## Database Client - Oracle Example

Install the oracle instant client and cx_oracle, as per [these](http://www.cs.utexas.edu/~mitra/csSpring2012/cs327/cx_mac.html) instructions.

_Hints:_

You can use pip to install cx_oracle.

Depending on your python configuration, you may need to install via sudo, e.g.

    sudo -E pip install cx_oracle

The following environment variables are required to use cx_oracle from python:

    ORACLE_HOME=/your/instant_client/install
    DYLD_LIBRARY_PATH=/your/instant_client/install

## Schema installation

The database schema can be found in:

    schema/sc_data.sql

It is designed to be RDBMS agnostic, but has only been tested with Oracle XE and MySQL. 

**NOTE: The schema requires minor alterations depending on your RDBMS - see the inline instructions in sc_data.sql** 

As an alternative to sc_data.sql, a SQL Alchemy schema can be used to generate the database schema. See the following test
for an example of how to do this:

    data_load_test.py - DataLoaderTests.setUp()

Note that an Oracle-specific helper script is provided to simplify user and tablespace creation prior to 
installation of the schema (with small data allocations that are only suitable for small-scale testing).

    schema/create_user.osql


# Running the tests

The unit tests cover key functionality for data retrieval and loading, and can be run as follows.

Ensure the surechembl-data-client folder is on your PYTHONPATH, e.g.:

    export PYTHONPATH=~/workspaces/surechembl/surechembl-data-client

Move to the tests folder, and run the tests:

    cd ~/workspaces/surechembl/surechembl-data-client/src/tests
    ./ftp_test.py
    ./data_load_test.py


# How to use the SureChEMBL Data Client<commit_msg>Add link to schema docs<commit_after>surechembl-data-client
======================

A collection of scripts for retrieving SureChEMBL data and building a relational database of patent chemistry. 

Please clone this repository to your local environment and follow the instructions below. 

Released under the MIT license. Access to the SureChEMBL data must be obtained from EBI-EMBL.

# Installation

## Core dependencies

Please make sure that the following dependencies are available:

* Python 2.7
* pip
* RDBMS of your choice
* Python DB API compatible client (see below)

Note that the client code has been tested against Oracle, but is written in a DB-agnostic way to simplify usage in
other scenarios.

## Python libraries

The complete set of required libraries can be found here:

    src/requirements.txt

This file was generated using pip's 'freeze' command. Use this file with your version of pip to ensure all required libraries are available.

## Database Client - Oracle Example

Install the oracle instant client and cx_oracle, as per [these](http://www.cs.utexas.edu/~mitra/csSpring2012/cs327/cx_mac.html) instructions.

_Hints:_

You can use pip to install cx_oracle.

Depending on your python configuration, you may need to install via sudo, e.g.

    sudo -E pip install cx_oracle

The following environment variables are required to use cx_oracle from python:

    ORACLE_HOME=/your/instant_client/install
    DYLD_LIBRARY_PATH=/your/instant_client/install

## Schema installation

The database schema can be found in:

    schema/sc_data.sql

It is designed to be RDBMS agnostic, but has only been tested with Oracle XE and MySQL.
 
A detailed description of the schema can be found in Google Docs, [here](https://docs.google.com/document/d/1INrMl63bp0Ut7hi_BvCXmW39SS62QeYL99lKB3PdwE4/edit#heading=h.6senzsu0y7u).

**NOTE: The schema requires minor alterations depending on your RDBMS - see the inline instructions in sc_data.sql** 

As an alternative to sc_data.sql, a SQL Alchemy schema can be used to generate the database schema. See the following test
for an example of how to do this:

    data_load_test.py - DataLoaderTests.setUp()

Note that an Oracle-specific helper script is provided to simplify user and tablespace creation prior to 
installation of the schema (with small data allocations that are only suitable for small-scale testing).

    schema/create_user.osql


# Running the tests

The unit tests cover key functionality for data retrieval and loading, and can be run as follows.

Ensure the surechembl-data-client folder is on your PYTHONPATH, e.g.:

    export PYTHONPATH=~/workspaces/surechembl/surechembl-data-client

Move to the tests folder, and run the tests:

    cd ~/workspaces/surechembl/surechembl-data-client/src/tests
    ./ftp_test.py
    ./data_load_test.py


# How to use the SureChEMBL Data Client
{'fraction_non_alphanumeric': 0.05972850678733032, 'fraction_numerical': 0.006153846153846154, 'mean_word_length': 4.208294062205467, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 1, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2026589', 'n_tokens_mistral': 1611, 'n_tokens_neox': 1496, 'n_words': 698}