<commit_before>===============================
Match
===============================


.. image:: https://img.shields.io/pypi/v/match.svg
        :target: https://pypi.python.org/pypi/match

.. image:: https://img.shields.io/travis/kvh/match.svg
        :target: https://travis-ci.org/kvh/match

.. image:: https://readthedocs.org/projects/match/badge/?version=latest
        :target: https://match.readthedocs.io/en/latest/?badge=latest
        :alt: Documentation Status

.. image:: https://pyup.io/repos/github/kvh/match/shield.svg
     :target: https://pyup.io/repos/github/kvh/match/
     :alt: Updates


Probabilistic Entity Matching


* Free software: MIT license
* Documentation: https://match.readthedocs.io.


Installation
--------

* TODO

Usage
--------

Basic entity detection and matching for built-in types.

.. code:: python

    >>> import match

    # Auto detect entity type
    >>> match.detect_type('608-555-5555')
    (1, PhoneNumberType)
    >>> match.detect_type('<EMAIL>')
    (1, EmailType)
    >>> match.detect_type('<NAME>')
    (.95, FullNameType)
    >>> match.detect_type('Hi, how are you?')
    (1, StringType)

    # Score similarities intelligently based on detected type
    >>> match.score_similarity('<NAME>', '<NAME>')
    (.92, FullNameType)
    >>> match.score_similarity('123 easy st, NY, NY', '123 Easy Street, New York City')
    (.98, AddressType)
    >>> match.score_similarity('Hi, how are you Joe?', 'hi how are you doing joe?')
    (.81, StringType)
    >>> match.score_similarity_as_type('608-555-5555', '608-555-5554', 'phonenumber')
    .0
    >>> match.score_similarity_as_type('608-555-5555', '608-555-5554', 'string')
    .9

    # Parse entity based on detected type
    >>> match.parse('608-555-5555')
    ('+1 608 555 5555', PhoneNumberType)
    >>> match.parse(' march 3rd, 1997', to_object=True)
    (datetime.datetime(1997, 3, 3), DateTimeType)
    >>> match.parse_as_type(' march 3rd, 1997', 'email')
    None


Probabilistic matching, based on frequencies in a given corpus.

.. code:: python

    >>> from match import similarities
    >>> import random

    # Build similarity model from weighted random corpus of a's, b's, c's, and d's
    >>> corpus = random.sample('a'*10000 + ' '*10000 + 'b'*1000 + 'c'*100 + 'd'*10, k=21110)
    >>> psim = similarities.ProbabilisticNgramSimilarity(corpus, grams=2)
    >>> psim.similarity('ab ba c', 'ab ba d') # Lower similarity since 'a' is common
    .6
    >>> psim.similarity('db bd c', 'db bd a') # Higher similarity since 'd' is rare
    .8


Custom types

.. code:: python

    >>> from match.similarity import ProbabilisticDiceCoefficient

    # Build similarity model from custom corpus
    >>> corpus = ''.join(['cheddar', 'brie', 'guyere', 'mozzarella', 'parmesian', 'jack', 'colby'])
    >>> cheese_sim = ProbabilisticDiceCoefficient(corpus)
    >>> match.add_type('cheese', StringType(similarity_measure=cheese_sim))
    >>> match.detect_type('colby jack')
    (.8, 'cheese')


Credits
---------

This package was created with Cookiecutter_ and the `audreyr/cookiecutter-pypackage`_ project template.

.. _Cookiecutter: https://github.com/audreyr/cookiecutter
.. _`audreyr/cookiecutter-pypackage`: https://github.com/audreyr/cookiecutter-pypackage
<commit_msg>More examples<commit_after>===============================
Match
===============================


.. image:: https://img.shields.io/pypi/v/match.svg
        :target: https://pypi.python.org/pypi/match

.. image:: https://img.shields.io/travis/kvh/match.svg
        :target: https://travis-ci.org/kvh/match

.. image:: https://readthedocs.org/projects/match/badge/?version=latest
        :target: https://match.readthedocs.io/en/latest/?badge=latest
        :alt: Documentation Status

.. image:: https://pyup.io/repos/github/kvh/match/shield.svg
     :target: https://pyup.io/repos/github/kvh/match/
     :alt: Updates


Probabilistic Entity Matching


* Free software: MIT license
* Documentation: https://match.readthedocs.io.


Installation
--------

* TODO

Usage
--------

Basic entity detection and matching for built-in types.

.. code:: python

    >>> import match

    # Auto detect entity type
    >>> match.detect_type('608-555-5555')
    (1, PhoneNumberType)
    >>> match.detect_type('<EMAIL>')
    (1, EmailType)
    >>> match.detect_type('<NAME>')
    (.95, FullNameType)
    >>> match.detect_type('Hi, how are you?')
    (1, StringType)
    >>> match.score_types('Score this! @squaredloss')
    [(0, EmailType), (.05, FullNameType), (0, PhoneNumberType), (1, StringType), (0, DateTimeType), ...

    # Score similarities intelligently based on detected type
    >>> match.score_similarity('<NAME>', '<NAME>')
    (.92, FullNameType)
    >>> match.score_similarity('123 easy st, NY, NY', '123 Easy Street, New York City')
    (.98, AddressType)
    >>> match.score_similarity('223 easy st, NY, NY', '123 easy st, NY, NY')
    (.6, AddressType)
    >>> match.score_similarity('Hi, how are you Joe?', 'hi how are you doing joe?')
    (.81, StringType)
    >>> match.score_similarity_as_type('608-555-5555', '608-555-5554', 'phonenumber')
    .0
    >>> match.score_similarity_as_type('608-555-5555', '608-555-5554', 'string')
    .9

    # Parse entity (to normalized string or object) based on detected type
    >>> match.parse('(608) 555-5555')
    ('+1 608 555 5555', PhoneNumberType)
    >>> match.parse('6085555555')
    ('+1 608 555 5555', PhoneNumberType)
    >>> match.parse(' march 3rd, 1997', to_object=True)
    (datetime.datetime(1997, 3, 3), DateTimeType)
    >>> match.parse_as_type(' march 3rd, 1997', 'email')
    None


Probabilistic matching, based on frequencies in a given corpus.

.. code:: python

    >>> from match import similarities
    >>> import random

    # Build similarity model from weighted random corpus of a's, b's, c's, and d's
    >>> corpus = random.sample('a'*10000 + ' '*10000 + 'b'*1000 + 'c'*100 + 'd'*10, k=21110)
    >>> psim = similarities.ProbabilisticNgramSimilarity(corpus, grams=2)
    >>> psim.similarity('ab ba c', 'ab ba d')
    .6  # Lower similarity since 'a' is common
    >>> psim.similarity('db bd c', 'db bd a')
    .8  # Higher similarity since 'd' is rare


Custom types

.. code:: python

    >>> from match.similarity import ProbabilisticDiceCoefficient

    # Build similarity model from custom corpus
    >>> corpus = ''.join(['cheddar', 'brie', 'guyere', 'mozzarella', 'parmesian', 'jack', 'colby'])
    >>> cheese_sim = ProbabilisticDiceCoefficient(corpus)
    >>> match.add_type('cheese', StringType(similarity_measure=cheese_sim))
    >>> match.detect_type('colby jack')
    (.8, 'cheese')


Credits
---------

This package was created with Cookiecutter_ and the `audreyr/cookiecutter-pypackage`_ project template.

.. _Cookiecutter: https://github.com/audreyr/cookiecutter
.. _`audreyr/cookiecutter-pypackage`: https://github.com/audreyr/cookiecutter-pypackage

{'fraction_non_alphanumeric': 0.18127979199768887, 'fraction_numerical': 0.04362270691896577, 'mean_word_length': 3.5373525557011796, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 164, 'https://': 22, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2421329', 'n_tokens_mistral': 2599, 'n_tokens_neox': 2251, 'n_words': 538}