<commit_before>mesh-tree
========

Utility functions for traversing the Medical Subject Heading (MeSH) ontology tree

## Setup

#### 1. Fetch data

Raw data are available
[here](https://www.nlm.nih.gov/mesh/filelist.html) as ascii and XML formats. Registration is required: see the
[Memorandum of Understanding](https://www.nlm.nih.gov/mesh/2014/download/termscon.html).

Run `./bin/fetch_mesh.sh`. The script will prompt for the email used in registration. All relevant MeSH files are downloaded into `$HOME/data/mesh/`.

#### 2. Transform to RDF

See [HHS/meshrdf](https://github.com/HHS/meshrdf) for instructions on transforming MeSH from XML to RDF N-triples using Saxon (java). Saxon 9.6 home edition was used, which works well. Additional details are available at the meshrdf [website](http://hhs.github.io/meshrdf/). Note: as of March 2015, there is an error in the `qual2015.dtd` file where the line `<!ENTITY  % DescriptorRecordSet SYSTEM "desc2014.dtd">` needs to be changed to `desc2015.dtd` given that all the most up-to-date files are used. Export the path of Saxon `$SAXON_JAR` and run `./bin/mesh_xml2rdf.sh`. The RDF N-triples file will be produced at `$HOME/data/mesh/RDF`.

#### 3. Install dependencies

`npm install`

#### 4. Initialize DB

We utilize LevelGraph (which is built on top of LevelDB) with the LevelGraph-N3 extension for storing and accessing the MeSH ontology as RDF N-triples. Run `npm run initdb` to stream the RDF data into the datastore.

## Tests

`npm test` runs Mocha tests.

## Usage

This package can be run as a service (TODO) or be imported as a module.

#### Running as service

Can be run as service with command `npm run server`, which listens for requests at `localhost` on port `7770` by default. The API is simply is a serialized JSON object with the following fields:

`job_id`: job unique identifier
`taskname`: function name as a string (see API below)
`payload`: argument(s), for example `desc_ui`

An simple example of a client function in python:

```
def mesh_tree_rpc(taskname, *args):
    HOST = '127.0.0.1'
    PORT = 7770
    req_socket = zmq.Context().socket(zmq.REQ)
    req_socket.connect('tcp://{}:{}'.format(HOST, PORT))
    req_obj = {
        'job_id': str(uuid.uuid4()),
        'taskname': taskname,
        'payload': args
    }
    req_socket.send(json.dumps(req_obj).encode('utf8'))
    result = json.loads(req_socket.recv().decode('utf8'))
    return result
```

#### API

##### getTreeNumbersByDescUI (desc_ui)

Returns array of tree numbers by descriptor record unique identifier.

Example: 'D000001' returns ['D03.438.221.173']
<commit_msg>update README; rpc server tests passing; closes #7<commit_after>mesh-tree
========

Utility functions for traversing the Medical Subject Heading (MeSH) ontology tree

## Setup

#### 1. Fetch data

Raw data are available
[here](https://www.nlm.nih.gov/mesh/filelist.html) as ascii and XML formats. Registration is required: see the
[Memorandum of Understanding](https://www.nlm.nih.gov/mesh/2014/download/termscon.html).

Run `./bin/fetch_mesh.sh`. The script will prompt for the email used in registration. All relevant MeSH files are downloaded into `$HOME/data/mesh/`.

#### 2. Transform to RDF

See [HHS/meshrdf](https://github.com/HHS/meshrdf) for instructions on transforming MeSH from XML to RDF N-triples using Saxon (java). Saxon 9.6 home edition was used, which works well. Additional details are available at the meshrdf [website](http://hhs.github.io/meshrdf/). Note: as of March 2015, there is an error in the `qual2015.dtd` file where the line `<!ENTITY  % DescriptorRecordSet SYSTEM "desc2014.dtd">` needs to be changed to `desc2015.dtd` given that all the most up-to-date files are used. Export the path of Saxon `$SAXON_JAR` and run `./bin/mesh_xml2rdf.sh`. The RDF N-triples file will be produced at `$HOME/data/mesh/RDF`.

#### 3. Install dependencies

`npm install`

#### 4. Initialize DB

We utilize LevelGraph (which is built on top of LevelDB) with the LevelGraph-N3 extension for storing and accessing the MeSH ontology as RDF N-triples. Run `npm run initdb` to stream the RDF data into the datastore.

## Tests

`npm test` runs Mocha tests.

## Usage

This package can be run as a service or be imported as a module.

#### Running as service

Can be run as service with command `npm run server`, which listens for requests at `localhost` on port `7770` by default. The API is simply is a serialized JSON object with the following fields:

`job_id`: job unique identifier

`taskname`: function name as a string (see API below)

`payload`: argument(s), for example `desc_ui`

An simple example of a client function in python:

```
def mesh_tree_rpc(taskname, *args):
    HOST = '127.0.0.1'
    PORT = 7770
    req_socket = zmq.Context().socket(zmq.REQ)
    req_socket.connect('tcp://{}:{}'.format(HOST, PORT))
    req_obj = {
        'job_id': str(uuid.uuid4()),
        'taskname': taskname,
        'payload': args
    }
    req_socket.send(json.dumps(req_obj).encode('utf8'))
    result = json.loads(req_socket.recv().decode('utf8'))
    return result
```

#### API

##### getTreeNumbersByDescUI (desc_ui)

Returns array of tree numbers by descriptor record unique identifier.

Example: 'D000001' returns ['D03.438.221.173']

{'fraction_non_alphanumeric': 0.11674259681093394, 'fraction_numerical': 0.02372817008352316, 'mean_word_length': 4.071222329162657, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 6, 'lorem ipsum': 0, 'www.': 4, 'xml': 2}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6354860', 'n_tokens_mistral': 1762, 'n_tokens_neox': 1612, 'n_words': 632}