<commit_before>#!/usr/bin/python
# -*- coding: utf-8; -*-
"""Field mapping.

When you have a big pile of dicts in slightly different formats, and
you want to transform them to dicts in a consistent format, this
module lets you write the field mappings in a small amount of code and
execute them reasonably quickly.

Quick start:

    >>> m = FieldMapper({'age': ['Age', 'age'], 'lastname': 'sname'})
    >>> [m.map(x) for x in [dict(Age=30, sname='Smith'),
    ...                     dict(age=49, sname='Wilson'),
    ...                     dict(x=3, y=4, lastname='Jones')]]
    ... #doctest: +NORMALIZE_WHITESPACE
    [{'lastname': 'Smith', 'age': 30,
      'original_data': {'Age': 30, 'sname': 'Smith'}},
     {'lastname': 'Wilson', 'age': 49,
      'original_data': {'age': 49, 'sname': 'Wilson'}},
     {'original_data': {'y': 4, 'lastname': 'Jones', 'x': 3}}]

You can include functions as mappings; it looks for their argument
names in the input data.

    >>> m = FieldMapper({'age': ['age', lambda birthyear: 2008 - birthyear]})
    >>> m.map({'age': 27})
    {'age': 27, 'original_data': {'age': 27}}
    >>> m.map({'birthyear': 1970})
    {'age': 38, 'original_data': {'birthyear': 1970}}
    >>> m = FieldMapper({'date': lambda year, month, day:
    ...                          '%s-%s-%s' % (year, month, day)})
    >>> m.map(dict(year=2008, month=11, day=19))['date']
    '2008-11-19'

You can construct (slightly) more complicated pipelines with `Reformat`:

    >>> def invert_name(n):
    ...     last, first = n.split(',')
    ...     return '%s %s' % (first.strip(), last.strip())
    >>> m = FieldMapper({'name': Reformat(format=invert_name,
    ...                                   source=['name', 'fullname'])})
    >>> m.map({'name': '<NAME>'})['name']
    '<NAME>'
    >>> m.map({'fullname': 'Smith,John'})['name']
    '<NAME>'

Finally, you can use `CatchAllField` (q.v.) to handle cases where these
tools don’t cut it.

BUGS:

-  Any particular input field can be used reliably by at most one
   output field.  If you try to use the same input field for more than
   one thing, you may get an `AssertionError`.  (You can often work<commit_msg>made parse/field_mapper.py comments more markdowny<commit_after>#!/usr/bin/python
# -*- coding: utf-8; -*-
"""Field mapping
================

When you have a big pile of dicts in slightly different formats, and
you want to transform them to dicts in a consistent format, this
module lets you write the field mappings in a small amount of code and
execute them reasonably quickly.

Quick start
-----------

    >>> m = FieldMapper({'age': ['Age', 'age'], 'lastname': 'sname'})
    >>> [m.map(x) for x in [dict(Age=30, sname='Smith'),
    ...                     dict(age=49, sname='Wilson'),
    ...                     dict(x=3, y=4, lastname='Jones')]]
    ... #doctest: +NORMALIZE_WHITESPACE
    [{'lastname': 'Smith', 'age': 30,
      'original_data': {'Age': 30, 'sname': 'Smith'}},
     {'lastname': 'Wilson', 'age': 49,
      'original_data': {'age': 49, 'sname': 'Wilson'}},
     {'original_data': {'y': 4, 'lastname': 'Jones', 'x': 3}}]

You can include functions as mappings; it looks for their argument
names in the input data.

    >>> m = FieldMapper({'age': ['age', lambda birthyear: 2008 - birthyear]})
    >>> m.map({'age': 27})
    {'age': 27, 'original_data': {'age': 27}}
    >>> m.map({'birthyear': 1970})
    {'age': 38, 'original_data': {'birthyear': 1970}}
    >>> m = FieldMapper({'date': lambda year, month, day:
    ...                          '%s-%s-%s' % (year, month, day)})
    >>> m.map(dict(year=2008, month=11, day=19))['date']
    '2008-11-19'

You can construct (slightly) more complicated pipelines with `Reformat`:

    >>> def invert_name(n):
    ...     last, first = n.split(',')
    ...     return '%s %s' % (first.strip(), last.strip())
    >>> m = FieldMapper({'name': Reformat(format=invert_name,
    ...                                   source=['name', 'fullname'])})
    >>> m.map({'name': '<NAME>'})['name']
    '<NAME>'
    >>> m.map({'fullname': 'Smith,John'})['name']
    '<NAME>'

Finally, you can use `CatchAllField` (q.v.) to handle cases where these
tools don’t cut it.

BUGS
----

-  Any particular input field can be used reliably by at most one
   output field.  If you try to use the same input field for more than
   one thing, you may get an `AssertionError`.  (You can often work
{'fraction_non_alphanumeric': 0.19710013593112824, 'fraction_numerical': 0.02401449932034436, 'mean_word_length': 2.9070796460176993, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 75, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4123770', 'n_tokens_mistral': 1559, 'n_tokens_neox': 1412, 'n_words': 459}