<commit_before>Struct-o-Miner
==============

Python package for extracting structured data from XML/HTML documents

**Work in progress**. Examples can be found in the [examples](https://github.com/aGHz/structominer/tree/master/examples) directory.<commit_msg>Added instructions to run examples<commit_after>Struct-o-Miner
==============

Python package for extracting structured data from XML/HTML documents

**Work in (frantic) progress**.

Examples can be found in the [examples](https://github.com/aGHz/structominer/tree/master/examples) directory. To run them while waiting for proper python packaging:

    # install libxml2 using your native package manager
    git clone https://github.com/aGHz/structominer.git structominer
    cd structominer
    virtualenv --distribute --no-site-packages .
    . bin/activate
    pip install lxml requests
    PYTHONPATH="../" python examples/hn.py
{'fraction_non_alphanumeric': 0.1187010078387458, 'fraction_numerical': 0.0011198208286674132, 'mean_word_length': 4.451219512195122, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4766043', 'n_tokens_mistral': 248, 'n_tokens_neox': 231, 'n_words': 79}