<commit_before># The Genomics DeepDive (GDD) Project

We have a wiki for this project at
<http://dev.stanford.edu/confluence/display/collaborators/DeepDive+Genomics>

### Running GDD: Basics

*NOTE [Mac OSX]: If running on Mac OSX, some of the scripts used below (those using the linux `readlink` command) may fail; recommended fix for now is to hardcode paths locally...*

1. Copy template file `env.sh` to `env_local.sh` and modify this file (it's ignored by git, and prefered by the run script).  Make sure to set:
	* database variables: user, host, port, db-name, db-type (postgres vs. greenplum)
	* _memory and parallelism options_
	* _relevant library paths_
	* _[GREENPLUM ONLY] port for gpfdist_

2. Create the database to be used if necessary

3. [IF NO INPUT DATA LOADED] **If no existing input data (e.g. `sentences` and/or `sentences_input` tables) is loaded:** Create input schema by running `./util/create_input_schema.sh` (**NOTE: this will drop any input data already loaded!**).  Then load data: if from tsv file you can use (usually loading `sentences_input` -> `TABLE_NAME=sentences_input`):

		./util/copy_table_from_file.sh [DB_NAME] [TABLE_NAME] [TSV_FILE_PATH]

  NOTE: some ready-to-use data files are available on `raiders2` at `/lfs/raiders2/0/robinjia/data/genomics_sentences_input_data/`.
`genomics_10k_sentences.tsv` has 10k sentences (good for quick testing and debugging) and `genomics_50k_docs.tsv` has 50k documents 
(gene extraction takes about 2 hours on `raiders2`).
There is also `plos_all_docs.tsv`, a separate, cleaner dataset based on HTML dumps from PLoS.
If you want to run certain mindtagger tasks (namely pheno-recall),
you will also need to copy 
`/lfs/raiders2/0/robinjia/data/genomics_sentences_input_data/hpo_to_plos_doc_id.tsv`
to the `hpo_to_doc_via_mesh` table.

4. To refresh / create the schema, run `./util/create_schema.sh`- *note that this will drop any output data from previous runs*.

5. Make sure that the custom user functions have been loaded into Postgres for this user; to do so run `./util/add_user_functions.sh`.

6. [GREENPLUM ONLY] Make sure that GreenPlum's parallel file distribution server, `gpfdist`, is running with the correct settings (e.g. run `ps aux | grep gpfdist`; make sure that an intance is running with the correct $GPPATH and $GPPORT).  If not, then start a new one running on a free port:

		gpfdist -d ${GPPATH} -p ${GPPORT} -m 268435456 &

7. Fetch and process ontology files: `cd onto; ./make_all.sh`

8. Run the appropriate pipeline: `./run.sh [PIPELINE_NAME]`
<commit_msg>Adds some notes on how to run Dashboard<commit_after># The Genomics DeepDive (GDD) Project

We have a wiki for this project at
<http://dev.stanford.edu/confluence/display/collaborators/DeepDive+Genomics>

### Running GDD: Basics

*NOTE [Mac OSX]: If running on Mac OSX, some of the scripts used below (those using the linux `readlink` command) may fail; recommended fix for now is to hardcode paths locally...*

1. Copy template file `env.sh` to `env_local.sh` and modify this file (it's ignored by git, and prefered by the run script).  Make sure to set:
	* database variables: user, host, port, db-name, db-type (postgres vs. greenplum)
	* _memory and parallelism options_
	* _relevant library paths_
	* _[GREENPLUM ONLY] port for gpfdist_

2. Create the database to be used if necessary

3. [IF NO INPUT DATA LOADED] **If no existing input data (e.g. `sentences` and/or `sentences_input` tables) is loaded:** Create input schema by running `./util/create_input_schema.sh` (**NOTE: this will drop any input data already loaded!**).  Then load data: if from tsv file you can use (usually loading `sentences_input` -> `TABLE_NAME=sentences_input`):

		./util/copy_table_from_file.sh [DB_NAME] [TABLE_NAME] [TSV_FILE_PATH]

  NOTE: some ready-to-use data files are available on `raiders2` at `/lfs/raiders2/0/robinjia/data/genomics_sentences_input_data/`.
`genomics_10k_sentences.tsv` has 10k sentences (good for quick testing and debugging) and `genomics_50k_docs.tsv` has 50k documents 
(gene extraction takes about 2 hours on `raiders2`).
There is also `plos_all_docs.tsv`, a separate, cleaner dataset based on HTML dumps from PLoS.
If you want to run certain mindtagger tasks (namely pheno-recall),
you will also need to copy 
`/lfs/raiders2/0/robinjia/data/genomics_sentences_input_data/hpo_to_plos_doc_id.tsv`
to the `hpo_to_doc_via_mesh` table.

4. To refresh / create the schema, run `./util/create_schema.sh`- *note that this will drop any output data from previous runs*.

5. Make sure that the custom user functions have been loaded into Postgres for this user; to do so run `./util/add_user_functions.sh`.

6. [GREENPLUM ONLY] Make sure that GreenPlum's parallel file distribution server, `gpfdist`, is running with the correct settings (e.g. run `ps aux | grep gpfdist`; make sure that an intance is running with the correct $GPPATH and $GPPORT).  If not, then start a new one running on a free port:

		gpfdist -d ${GPPATH} -p ${GPPORT} -m 268435456 &

7. Fetch and process ontology files: `cd onto; ./make_all.sh`

8. Run the appropriate pipeline: `./run.sh [PIPELINE_NAME]`

### Running Dashboard for Reports

Make sure you have run `util/update-mindbender.sh` at least once.
It will download the `util/mindbender` command, which includes Mindtagger as well as Dashboard.

To produce a set of reports using Dashboard after a GDD run, use the following steps:
```bash
(
. env_local.sh
export GDD_PIPELINE=
util/mindbender snapshot
)
```
This will produce a set of reports under a directory pointed by `snapshot/LATEST` as configured in `snapshot-default.conf`.

To view the produced reports, you can use the Dashboard GUI by starting it with the following command and opening the URL it prints:
```bash
(
. env_local.sh
export GDD_PIPELINE=
PORT=12345 util/mindbender dashboard
)
```
You may need to change `PORT=12345` value if someone else is already using it.
When Dashboard URL is loaded in your web browser, you can navigate to the first snapshot in the top "View Snapshots" dropdown.

{'fraction_non_alphanumeric': 0.08884127245755728, 'fraction_numerical': 0.012197132025712872, 'mean_word_length': 4.541552511415525, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '30982', 'n_tokens_mistral': 2029, 'n_tokens_neox': 1921, 'n_words': 813}