<commit_before>parserator
==========
A toolkit for making domain-specific probabilistic parsers

[![Build Status](https://travis-ci.org/datamade/parserator.svg?branch=master)](https://travis-ci.org/datamade/parserator)

Want a domain-specific parser that learns to parse strings probabilistically? Here's a tool that will help you make one! All you need is some training data to teach the parser about its domain.

## What a probabilistic parser can do
Given a string, a parser will break it out into labeled components. The parser uses [conditional random fields](http://en.wikipedia.org/wiki/Conditional_random_field) to label components based on (1) features of the component string and (2) the order of labels.

A neat thing about a probabilistic approach (as opposed to a rule-based approach) is that the parser can continually learn from new training data, and continually improve its performance.

## parserator examples

* [usaddress](https://github.com/datamade/usaddress) - Our first probabilistic parser and the basis for the parserator toolkit, it parses any address in the United States. [Read our blog post on how it works](http://datamade.us/blog/parsing-addresses-with-usaddress/).
* [name-parser](https://github.com/datamade/name-parser) - Parser for romanized person names. 

## How to make a parser using parserator
1. **Initialize a new parser**

    ```
    pip install parserator  
    parserator init [YOUR PARSER NAME]  
    ```  
    This will initialize a new parser in your current directory. For example, running ```parserator init foo``` will generate the following directories and files:
    * foo/
      - \_\_init\_\_.py
    * foo_data/
      - labeled_xml/
      - unlabeled/
    * setup.py
    * tests/
      - test_tokenizing.py
2. **Configure the parser to your domain**<commit_msg>more on what parserator is good for<commit_after>parserator
==========
A toolkit for making domain-specific probabilistic parsers

[![Build Status](https://travis-ci.org/datamade/parserator.svg?branch=master)](https://travis-ci.org/datamade/parserator)

Want a domain-specific parser that learns to parse strings probabilistically? Here's a tool that will help you make one! All you need is some training data to teach the parser about its domain.

## What a probabilistic parser does
Given a string, a parser will break it out into labeled components. The parser uses [conditional random fields](http://en.wikipedia.org/wiki/Conditional_random_field) to label components based on (1) features of the component string and (2) the order of labels.

## parserator examples

* [usaddress](https://github.com/datamade/usaddress) - Our first probabilistic parser and the basis for the parserator toolkit, it parses any address in the United States. [Read our blog post on how it works](http://datamade.us/blog/parsing-addresses-with-usaddress/).
* [name-parser](https://github.com/datamade/name-parser) - Parser for romanized person names. 

## When is a probabalistic parser useful?
A probabilistic parser is particularly useful for sets of strings that have common structure/patterns, but can deviate from those patterns in ways that are difficult to anticipate with hard coded rules.

For example, in most cases, US addresses start with street number. But there are exceptions: sometimes valid addresses deviate from this pattern (e.g. addresses starting with building name, PO box) and furthermore, addresses in real datasets often include typos & errors. Because there are infinitely many patterns and possible typos to account for, a probabilistic parser is well-suited to parse US addresses.

A neat thing about a probabilistic approach (as opposed to a rule-based approach) is that the parser can continually learn from new training data, and continually improve its performance.

Some other examples of domains where a probabilistic parser can be useful:
- addresses in another country
- product names/descriptions (e.g. parsing 'Twizzlers Twists, Strawberry, 16-Ounce Bags (Pack of 6)' into brand, item, flavor, weight, etc)
- citations

## How to make a parser using parserator
1. **Initialize a new parser**

    ```
    pip install parserator  
    parserator init [YOUR PARSER NAME]  
    ```  
    This will initialize a new parser in your current directory. For example, running ```parserator init foo``` will generate the following directories and files:
    * foo/
      - \_\_init\_\_.py
    * foo_data/
      - labeled_xml/
      - unlabeled/
    * setup.py
    * tests/
      - test_tokenizing.py
2. **Configure the parser to your domain**
{'fraction_non_alphanumeric': 0.08112836212551935, 'fraction_numerical': 0.0024054231357970696, 'mean_word_length': 4.070953436807096, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6970145', 'n_tokens_mistral': 1232, 'n_tokens_neox': 1180, 'n_words': 551}