<commit_before>Laboratory! |Build Status|
==========================

A Python library for carefully refactoring critical paths (and a port of
`GitHub's Scientist`_).

.. _GitHub's Scientist: https://github.com/github/scientist


Why?
----

See GitHub's blog post |--| http://githubengineering.com/scientist/


But how?
--------

Imagine you've implemented a complex caching strategy for some objects in your
database and a stale cache is simply not acceptable.  How could you test this
and ensure parity with your previous implementation, under load, with
production data?  Run it in production!

Laboratory will:

-  Run both the new and the old code
-  Compare their results
-  Record timing information about all code
-  Swallow and record exceptions in the new code
-  Publish all of this information

Of course, you're still unsure your candidate code works correctly, so
laboratory will always return the result from the control block.

.. code:: python

    import laboratory

    experiment = laboratory.Experiment()
    with experiment.control() as c:
        c.record(get_objects_from_database())

    with experiment.candidate() as c:
        c.record(get_objects_from_cache())

    objects = experiment.run()


Publishing results
------------------

This data is useless unless we can do something with it. Laboratory makes no
assumptions about how to do this |--| it's entirely for you to implement to suit
your needs.  For example, timing data can be sent to graphite, and mismatches
can be placed in a capped collection in redis for debugging later.

The publish method is passed a ``Result`` instance, with control and candidate
data is available in ``Result.control`` and ``Result.observations``
respectively.

.. code:: python

    class MyExperiment(laboratory.Experiment):
        def publish(self, result):
            statsd.timing('MyExperiment.control', result.control.duration)
            for o in result.observations:
                statsd.timing('MyExperiment.%s' % o.name, o.duration)


Controlling comparison
----------------------
<commit_msg>Show decorator usage in readme<commit_after>Laboratory! |Build Status|
==========================

A Python library for carefully refactoring critical paths (and a port of
`GitHub's Scientist`_).

.. _GitHub's Scientist: https://github.com/github/scientist


Why?
----

See GitHub's blog post |--| http://githubengineering.com/scientist/


But how?
--------

Imagine you've implemented a complex caching strategy for some objects in your
database and a stale cache is simply not acceptable.  How could you test this
and ensure parity with your previous implementation, under load, with
production data?  Run it in production!

Laboratory will:

-  Run both the new and the old code
-  Compare their results
-  Record timing information about all code
-  Swallow and record exceptions in the new code
-  Publish all of this information

Of course, you're still unsure your candidate code works correctly, so
laboratory will always return the result from the control block.

.. code:: python

    import laboratory

    experiment = laboratory.Experiment()
    with experiment.control() as c:
        c.record(get_objects_from_database())

    with experiment.candidate() as c:
        c.record(get_objects_from_cache())

    objects = experiment.run()

Note that the ``Experiment`` class can also be used as a decorator.

.. code:: python

    @Experiment(candidate=get_objects_from_cache)
    def get_objects_from_database():
        return True


Publishing results
------------------

This data is useless unless we can do something with it. Laboratory makes no
assumptions about how to do this |--| it's entirely for you to implement to suit
your needs.  For example, timing data can be sent to graphite, and mismatches
can be placed in a capped collection in redis for debugging later.

The publish method is passed a ``Result`` instance, with control and candidate
data is available in ``Result.control`` and ``Result.observations``
respectively.

.. code:: python

    class MyExperiment(laboratory.Experiment):
        def publish(self, result):
            statsd.timing('MyExperiment.control', result.control.duration)
            for o in result.observations:
                statsd.timing('MyExperiment.%s' % o.name, o.duration)


Controlling comparison
----------------------

{'fraction_non_alphanumeric': 0.10403299725022915, 'fraction_numerical': 0.0, 'mean_word_length': 3.386934673366834, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2293884', 'n_tokens_mistral': 1179, 'n_tokens_neox': 1101, 'n_words': 491}