<commit_before>.. rdc-etl documentation master file, created by
   sphinx-quickstart on Wed Nov  6 11:08:57 2013.
   You can adapt this file completely to your liking, but it should at least
   contain the root `toctree` directive.

Welcome to rdc-etl's documentation!
===================================

Contents:

.. toctree::
   :maxdepth: 2



Indices and tables<commit_msg>doc: basic overview<commit_after>rdc.etl
=======

Contents:

.. toctree::
   :maxdepth: 2


Overview
=======

Extract Transform Load (ETL) toolkit for python.

Toolkit for doing data integration related work. Unlike java based tools like
talend or pentaho data-integration, this is a DIY framework, and if you're
looking for a WYSIWIG ETL engine, you should probably go back to the previously
cited ones.


Create a Harness
::::::::::::::::

* Connect transformations
* Runnable
* Manage threading

>>> from rdc.etl.harness.threaded import ThreadedHarness as Harness
>>> harness = Harness()
>>> # ... add your transforms here (see below)
>>> harness() # run!


Create transformations
::::::::::::::::::::::

* Various types, some transformations takes input and yield output, while some
  are input-only ("load") or output-only ("extract").

>>> from rdc.etl.transform.extract import Extract
>>> extract = Extract(stream_data=({'foo': 'bar'}, {'foo': 'baz'}))
>>> from rdc.etl.transform.simple import SimpleTransform
>>> transform = SimpleTransform()
>>> transform.add('foo').filter('upper')
>>> from rdc.etl.transform.util import Log
>>> load = Log()


Tie transformations together
::::::::::::::::::::::::::::

The default harness has a `chain_add` shortcut that allows to easily create and
connect together a linear chain of transformations (by linear, we mean that
every transform default output channel is connected to the next transformation
default input channel). This is a very common case, and should be enough to
play around a bit.

>>> harness.chain_add(extract, transform, load)

This code, without using the shortcut, would need to be written as follow.

>>> _extract = harness.add(extract)
>>> _transform = harness.add(transform)
>>> _transform.set_input_from(_extract)
>>> _load = harness.add(load)
>>> _load.set_input_from(_transform)

The difference between original transformation instances and their underscore
prefixed counterpart (for example, difference between extract and _extract) is
that the underscor one is an instance `ThreadedTransform`, which is a decorator
adding threading ability _and_ io queues. You won't be able to
.`set_input_from`() on or from a non decorated transformation.

This API is a work in progress, and IO channels are being reworked. Expect
changes on this part.


Run the job
:::::::::::

Once your job has been configured in your harness instance, you can run it.

>>> harness()


Indices and tables
{'fraction_non_alphanumeric': 0.12751677852348994, 'fraction_numerical': 0.0045920169551395265, 'mean_word_length': 4.937106918238993, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 57, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3458598', 'n_tokens_mistral': 796, 'n_tokens_neox': 744, 'n_words': 333}