<commit_before>Design
======
The Yahoo pipelines are translated into pipelines of Python generators which 
should give a close match to the original data flow. Each call to the final
generator will ripple through the pipeline issuing `.next()` calls until the 
source is exhausted.

The modules are topologically sorted to give their creation order. 
The main output and inputs are connected via the yielded values and the 
first parameter. Other inputs are passed as named parameters referencing the 
input module.

The JSON representation of the configuration parameters maps closely onto 
Python dictionaries and so is left as-is and passed and parsed as-and-when 
needed.

Each Yahoo module is coded as a separate Python module. This might help in
future if the generators are made to run on separate processors/machines and 
we could use queues to plumb them together.


Setting up the environment
==========================
If using a Python version before 2.6 then simplejson is needed:
  
  * http://pypi.python.org/pypi/simplejson

Put the source code in a directory, say, `pipeline/pipe2py`.

Make the package available to Python, e.g.

  export PYTHONPATH=pipeline


Unit tests
==========
Run in the test directory::

  python testbasics.py

In test-mode, modules needing user input use the default values rather than 
prompting the user. This is done by setting `context.test==True`.


Usage
=====
There are two ways to translate a Yahoo pipe into Python. One outputs a Python 
script which wraps the pipeline in a function which can then be imported and 
run from another Python program (i.e. compiled). The other interprets the 
pipeline on-the-fly and executes it within the current process 
(i.e. interpreted).

1. Compiling a pipeline to a Python script
------------------------------------------
Both of the following will create a python file named after the input argument 
with a .py extension (using the `compile.parse_and_write_pipe` function). This 
file can then be run directly or imported into other pipelines.

The first pulls the pipeline definition directly from Yahoo. The second loads 
the pipeline definition from a file:

  * python compile.py -p pipelineid
  
  or
  
  * python compile.py pipelinefile
  
Subpiplines are expected to be contained in python files named pipe_PIPEID.py,
where `PIPEID` is the Yahoo ID for the pipeline, e.g.

  pipe_2de0e4517ed76082dcddf66f7b218057.py

So if you do use the second option you should store your pipeline definitions 
in files named the same way, e.g.

  pipe_2de0e4517ed76082dcddf66f7b218057.json

then compile.py will output files with the expected naming convention.
  
2. Interpreting a pipeline and executing in-process
---------------------------------------------------
Example::

    from pipe2py.compile import parse_and_build_pipe
    from pipe2py import Context

    pipe_def = """json representation of the pipe"""

    p = parse_and_build_pipe(Context(), pipe_def)

    for i in p:
        print i


Inputs
======
Some pipelines need to prompt the user for input values. When running a
compiled pipe, it defaults to prompting the user via the console, but in other
situations this may not be appropriate, e.g. when integrating with a website. 
In such cases, the input values can instead be read from the pipe's context (a 
set of values passed into every pipe). The context.inputs dictionary can be 
pre-populated with user input before the pipe is executed. 

To determine which prompts are needed, the pipeline can be called initially 
with `context.describe_input==True`, and this will return a list of tuples 
defining the inputs needed (it will not execute the pipe), e.g.::

    context = Context(describe_input=True)
    p = pipe_ac45e9eb9b0174a4e53f23c4c9903c3f(context, None)
    need_inputs = p
    print need_inputs

    >>> [(u'0', u'username', u'Twitter username', u'text', u''), 
    ...  (u'1', u'statustitle', 
    ...   u'Status title [string] or [logo] means twitter icon', u'text', 
    ...   u'logo')]

Each tuple is of the form::

  (position,
   name,
   prompt,
   type,
   default)

The list of tuples is sorted by position, i.e. the order in which they should 
be presented to the user. The name should be used as a key in the 
`context.inputs` dictionary. The prompt is the prompt for the user. Type is 
the data type, e.g. text, number. And default is the default value (used if no 
value is given), e.g. to run the above pipe with pre-defined inputs, and so no
console prompting::

    context = Context(inputs={'username':'greg', 'statustitle':'logo'})
    p = pipe_ac45e9eb9b0174a4e53f23c4c9903c3f(context, None)
    for i in p:
        print i
<commit_msg>Tweak readme about console prompts<commit_after>Design
======
The Yahoo pipelines are translated into pipelines of Python generators which 
should give a close match to the original data flow. Each call to the final
generator will ripple through the pipeline issuing `.next()` calls until the 
source is exhausted.

The modules are topologically sorted to give their creation order. 
The main output and inputs are connected via the yielded values and the 
first parameter. Other inputs are passed as named parameters referencing the 
input module.

The JSON representation of the configuration parameters maps closely onto 
Python dictionaries and so is left as-is and passed and parsed as-and-when 
needed.

Each Yahoo module is coded as a separate Python module. This might help in
future if the generators are made to run on separate processors/machines and 
we could use queues to plumb them together.


Setting up the environment
==========================
If using a Python version before 2.6 then simplejson is needed:
  
  * http://pypi.python.org/pypi/simplejson

Put the source code in a directory, say, `pipeline/pipe2py`.

Make the package available to Python, e.g.

  export PYTHONPATH=pipeline


Unit tests
==========
Run in the test directory::

  python testbasics.py

In test-mode, modules needing user input use the default values rather than 
prompting the user. This is done by setting `context.test==True`.


Usage
=====
There are two ways to translate a Yahoo pipe into Python. One outputs a Python 
script which wraps the pipeline in a function which can then be imported and 
run from another Python program (i.e. compiled). The other interprets the 
pipeline on-the-fly and executes it within the current process 
(i.e. interpreted).

1. Compiling a pipeline to a Python script
------------------------------------------
Both of the following will create a python file named after the input argument 
with a .py extension (using the `compile.parse_and_write_pipe` function). This 
file can then be run directly or imported into other pipelines.

The first pulls the pipeline definition directly from Yahoo. The second loads 
the pipeline definition from a file:

  * python compile.py -p pipelineid
  
  or
  
  * python compile.py pipelinefile
  
Subpiplines are expected to be contained in python files named pipe_PIPEID.py,
where `PIPEID` is the Yahoo ID for the pipeline, e.g.

  pipe_2de0e4517ed76082dcddf66f7b218057.py

So if you do use the second option you should store your pipeline definitions 
in files named the same way, e.g.

  pipe_2de0e4517ed76082dcddf66f7b218057.json

then compile.py will output files with the expected naming convention.
  
2. Interpreting a pipeline and executing in-process
---------------------------------------------------
Example::

    from pipe2py.compile import parse_and_build_pipe
    from pipe2py import Context

    pipe_def = """json representation of the pipe"""

    p = parse_and_build_pipe(Context(), pipe_def)

    for i in p:
        print i


Inputs
======
Some pipelines need to prompt the user for input values. When running a
compiled pipe, it defaults to prompting the user via the console, but in other
situations this may not be appropriate, e.g. when integrating with a website. 
In such cases, the input values can instead be read from the pipe's context (a 
set of values passed into every pipe). The context.inputs dictionary can be 
pre-populated with user input before the pipe is executed. 

To determine which prompts are needed, the pipeline can be called initially 
with `context.describe_input==True`, and this will return a list of tuples 
defining the inputs needed (it will not execute the pipe), e.g.::

    context = Context(describe_input=True)
    p = pipe_ac45e9eb9b0174a4e53f23c4c9903c3f(context, None)
    need_inputs = p
    print need_inputs

    >>> [(u'0', u'username', u'Twitter username', u'text', u''), 
    ...  (u'1', u'statustitle', 
    ...   u'Status title [string] or [logo] means twitter icon', u'text', 
    ...   u'logo')]

Each tuple is of the form::

  (position,
   name,
   prompt,
   type,
   default)

The list of tuples is sorted by position, i.e. the order in which they should 
be presented to the user. The name should be used as a key in the 
`context.inputs` dictionary. The prompt is the prompt for the user. Type is 
the data type, e.g. text, number. And default is the default value (used if no 
value is given), e.g. to run the above pipe with pre-defined inputs, and no
console prompting::

    context = Context(inputs={'username':'greg', 'statustitle':'logo'}, 
                      console=False)
    p = pipe_ac45e9eb9b0174a4e53f23c4c9903c3f(context, None)
    for i in p:
        print i

{'fraction_non_alphanumeric': 0.08558080271819919, 'fraction_numerical': 0.018475260140157147, 'mean_word_length': 3.4895138226882745, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6142624', 'n_tokens_mistral': 2744, 'n_tokens_neox': 2536, 'n_words': 1276}