<commit_before>### Parser

A simple XML / HTML parser for pulling out document sections, with some markdown-esque preservation of xml/html tags.  This is run as the first component of the end-to-end preprocessing:

1. **XML parsing [this lib]**
2. NLP preprocessing [bazaar / coreNLP]
3. Loading into db [psql]

To install and compile, run:

    ./get_deps.sh
    ant

To run end-to-end, make sure the `BAZAAR_DIR`, `PARALLELISM` and db variables are set correctly in `dd-genomics/env_local.sh`, then run, specifying whether to add to an existing table or create one:
  
    ./run.sh [In: a directory of / single xml file] [out: db table] ["create"|"add"]

#### To-do:
* Parse document metadata
* Parallelize XML parsing?<commit_msg>Some notes on parser testing<commit_after>### Parser

A simple XML / HTML parser for pulling out document sections, with some markdown-esque preservation of xml/html tags.  This is run as the first component of the end-to-end preprocessing:

1. **XML parsing [this lib]**
2. NLP preprocessing [bazaar / coreNLP]
3. Loading into db [psql]

To install and compile, run:

    ./get_deps.sh
    ant

To run end-to-end, make sure the `BAZAAR_DIR`, `PARALLELISM` and db variables are set correctly in `dd-genomics/env_local.sh`, then run, specifying whether to add to an existing table or create one:
  
    ./run.sh [In: a directory of / single xml file] [out: db table] ["create"|"add"]

#### To-do:
* Parse document metadata
* Parallelize XML parsing?

### Notes (bazaar):
* Testing pmc.json on raiders (80 cores): 80k lines in 20m ~= 250K/hr
* Testing pmc.json on an ec2-m3.large (2 cores): 10K lines in 15m ~= 40K/hr
* *Testing pmc.json on an ec2-c3.4xlarge (16 cores): 80k lines in 15m ~= 300K/hr*
  * Memory errors... seems like the ratio of memory:cores too low?
* Testing pmc.json on **two** ec2-m3.2xlarges (2 x 8 cores): 80k lines in 26m ~= 200K/hr
{'fraction_non_alphanumeric': 0.10922587486744433, 'fraction_numerical': 0.02545068928950159, 'mean_word_length': 3.765151515151515, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 4}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6048200', 'n_tokens_mistral': 674, 'n_tokens_neox': 617, 'n_words': 264}