<commit_before># libpostal: international street address NLP

[![Build Status](https://travis-ci.org/openvenues/libpostal.svg?branch=master)](https://travis-ci.org/openvenues/libpostal)

libpostal is a fast statistical parser/normalizer for street addresses around the world.

:jp: :us: :gb: :ru: :fr: :kr: :it: :es: :cn: :de:

Addresses and the geographic coordinates they represent are essential for any
location-based application (map search, transportation, on-demand/delivery
services, check-ins, reviews). Yet even the simplest addresses are packed with
local conventions, abbreviations and context, making them difficult to
index/query effectively with traditional full-text search engines, which are
designed for document indexing. This library helps convert the free-form
addresses that humans use into clean normalized forms suitable for machine
comparison and full-text indexing.

While libpostal is not itself a full geocoder, it can be used as a preprocessing step to make any geocoding application smarter, simpler, and more consistent internationally.

The core library is written in pure C. Language bindings for [Python](https://github.com/openvenues/pypostal), [Ruby](https://github.com/openvenues/ruby_postal), [Go](https://github.com/openvenues/gopostal), [Java](https://github.com/openvenues/jpostal), [PHP](https://github.com/openvenues/php-postal), and [NodeJS](https://github.com/openvenues/node-postal) are officially supported and it's easy to write bindings in other languages.

Examples of normalization
-------------------------

The expand_address API converts messy real-world addresses into normalized
equivalents suitable for search indexing, hashing, etc. 

Here's an interactive example using the Python binding:

![expand](https://cloud.githubusercontent.com/assets/238455/13210468/b1676b72-d8fd-11e5-8557-ded418f6ffda.gif)

libpostal contains an OSM-trained language classifier to detect which language(s) are used in a given
address so it can apply the appropriate normalizations. The only input needed is the raw address string. 
Here's a short list of some less straightforward normalizations in various languages.

| Input                               | Output (may be multiple in libpostal)   |
| ----------------------------------- |-----------------------------------------|
| One-hundred twenty E 96th St        | 120 east 96th street                    |
| C/ Ocho, P.I. 4                     | calle 8 polígono industrial 4           |
| V XX Settembre, 20                  | via 20 settembre 20                     |
| Quatre vignt douze R. de l'Église   | 92 rue de l' église                     |
| ул Каретный Ряд, д 4, строение 7    | улица каретныи ряд дом 4 строение 7     |
| ул Каретный Ряд, д 4, строение 7    | ulitsa karetnyy ryad dom 4 stroyeniye 7 |
| Marktstrasse 14                     | markt straße 14                         |

libpostal currently supports these types of normalizations in *60+ languages*,
and you can [add more](https://github.com/openvenues/libpostal/tree/master/resources/dictionaries) <commit_msg>[docs] Adding blog post link and license to the README<commit_after># libpostal: international street address NLP

[![Build Status](https://travis-ci.org/openvenues/libpostal.svg?branch=master)](https://travis-ci.org/openvenues/libpostal) [![License](https://img.shields.io/github/license/openvenues/libpostal.svg)]

:jp: :us: :gb: :ru: :fr: :kr: :it: :es: :cn: :de:

libpostal is a C library for parsing/normalizing street addresses around the world. This [introductory blog post](https://medium.com/@albarrentine/statistical-nlp-on-openstreetmap-b9d573e6cc86) is a good overview of the research and thought process behind libpostal.

Addresses and the geographic coordinates they represent are essential for any location-based application (map search, transportation, on-demand/delivery services, check-ins, reviews). Yet even the simplest addresses are packed with local conventions, abbreviations and context, making them difficult to index/query effectively with traditional full-text search engines, which are designed for document indexing. This library helps convert the free-form addresses that humans use into clean normalized forms suitable for machine comparison and full-text indexing.

While libpostal is not itself a full geocoder, it can be used as a preprocessing step to make any geocoding application smarter, simpler, and more consistent internationally.

The core library is written in pure C. Language bindings for [Python](https://github.com/openvenues/pypostal), [Ruby](https://github.com/openvenues/ruby_postal), [Go](https://github.com/openvenues/gopostal), [Java](https://github.com/openvenues/jpostal), [PHP](https://github.com/openvenues/php-postal), and [NodeJS](https://github.com/openvenues/node-postal) are officially supported and it's easy to write bindings in other languages.

Examples of normalization
-------------------------

The expand_address API converts messy real-world addresses into normalized
equivalents suitable for search indexing, hashing, etc. 

Here's an interactive example using the Python binding:

![expand](https://cloud.githubusercontent.com/assets/238455/13210468/b1676b72-d8fd-11e5-8557-ded418f6ffda.gif)

libpostal contains an OSM-trained language classifier to detect which language(s) are used in a given
address so it can apply the appropriate normalizations. The only input needed is the raw address string. 
Here's a short list of some less straightforward normalizations in various languages.

| Input                               | Output (may be multiple in libpostal)   |
| ----------------------------------- |-----------------------------------------|
| One-hundred twenty E 96th St        | 120 east 96th street                    |
| C/ Ocho, P.I. 4                     | calle 8 polígono industrial 4           |
| V XX Settembre, 20                  | via 20 settembre 20                     |
| Quatre vignt douze R. de l'Église   | 92 rue de l' église                     |
| ул Каретный Ряд, д 4, строение 7    | улица каретныи ряд дом 4 строение 7     |
| ул Каретный Ряд, д 4, строение 7    | ulitsa karetnyy ryad dom 4 stroyeniye 7 |
| Marktstrasse 14                     | markt straße 14                         |

libpostal currently supports these types of normalizations in *60+ languages*,
and you can [add more](https://github.com/openvenues/libpostal/tree/master/resources/dictionaries) 
{'fraction_non_alphanumeric': 0.11057841014537581, 'fraction_numerical': 0.020878441076399627, 'mean_word_length': 3.1243622448979593, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 22, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '581639', 'n_tokens_mistral': 1908, 'n_tokens_neox': 1743, 'n_words': 670}