<commit_before>

``py.execnet`` allows to:

* instantiate local or remote Python Processes
* send code for execution in one or many processes 
* asynchronously send and receive data between processes through channels 
* completely avoid manual installation steps on remote places

.. contents::

Gateways: immediately spawn local or remote process
----------------------------------------------------

In order to send code to a remote place or a subprocess
you need to instantiate a so-called Gateway object.  
There are currently three Gateway classes:

* :api:`py.execnet.PopenGateway` to open a subprocess 
  on the local machine.  Useful for making use 
  of multiple processors to to contain code execution
  in a separated environment. 

* :api:`py.execnet.SshGateway` to connect to 
  a remote ssh server and distribute execution to it. 

* :api:`py.execnet.SocketGateway` a way to connect to 
  a remote Socket based server. *Note* that this method
  requires a manually started 
  :source:py/execnet/script/socketserver.py
  script.  You can run this "server script" without 
  having the py lib installed on the remote system
  and you can setup it up as permanent service. 


remote_exec: execute source code remotely 
--------------------------------------------

All gateways offer remote code execution via this high level function::

    def remote_exec(source): 
        """return channel object for communicating with the asynchronously 
        executing 'source' code which will have a corresponding 'channel' 
        object in its executing namespace."""

With `remote_exec` you send source code to the other
side and get both a local and a remote Channel_ object,
which you can use to have the local and remote site
communicate data in a structured way.   Here is 
an example for reading the PID::

  >>> import py 
  >>> gw = py.execnet.PopenGateway()
  >>> channel = gw.remote_exec("""
  ...     import os
  ...     channel.send(os.getpid())
  ... """)
  >>> remote_pid = channel.receive()
  >>> remote_pid != py.std.os.getpid()
  True

.. _`Channel`: 
.. _`channel-api`: 
.. _`exchange data`: 

Channels: bidirectionally exchange data between hosts 
-------------------------------------------------------------

A channel object allows to send and receive data between 
two asynchronously running programs.  When calling
`remote_exec` you will get a channel object back and 
the code fragement running on the other side will 
see a channel object in its global namespace. 

Here is the interface of channel objects::

    #
    # API for sending and receiving anonymous values
    #
    channel.send(item): 
        sends the given item to the other side of the channel, 
        possibly blocking if the sender queue is full. 
        Note that items need to be marshallable (all basic 
        python types are).

    channel.receive():
        receives an item that was sent from the other side, 
        possibly blocking if there is none. 
        Note that exceptions from the other side will be 
        reraised as gateway.RemoteError exceptions containing 
        a textual representation of the remote traceback. 

    channel.waitclose(timeout=None): 
        wait until this channel is closed.  Note that a closed
        channel may still hold items that will be received or 
        send. Note that exceptions from the other side will be 
        reraised as gateway.RemoteError exceptions containing 
        a textual representation of the remote traceback. 

    channel.close(): 
        close this channel on both the local and the remote side. 
        A remote side blocking on receive() on this channel 
        will get woken up and see an EOFError exception. 


.. _xspec:


XSpec: string specification for gateway type and configuration
-------------------------------------------------------------------

``py.execnet`` supports a simple extensible format for 
specifying and configuring Gateways for remote execution.  
You can use a string speficiation to make a new gateway, 
for example a new SshGateway::

    gateway = py.execnet.makegateway("ssh=myhost")

Let's look at some examples for valid specifications. 
Specification for an ssh connection to `wyvern`, running on python2.4 in the (newly created) 'mycache'  subdirectory::

    ssh=wyvern//python=python2.4//chdir=mycache

Specification of a python2.5 subprocess; with a low CPU priority ("nice" level). Current dir will be the current dir of the instantiator (that's true for all 'popen' specifications unless they specify 'chdir')::

    popen//python=2.5//nice=20

Specification of a Python Socket server process that listens on 192.168.1.4:8888; current dir will be the 'pyexecnet-cache' sub directory which is used a default for all remote processes::

    socket=192.168.1.4:8888

More generally, a specification string has this general format::

    key1=value1//key2=value2//key3=value3

If you omit a value, a boolean true value is assumed.  Currently
the following key/values are supported: 

* ``popen`` for a PopenGateway
* ``ssh=host`` for a SshGateway
* ``socket=address:port`` for a SocketGateway 
* ``python=executable`` for specifying Python executables
* ``chdir=path`` change remote working dir to given relative or absolute path
* ``nice=value`` decrease remote nice level if platforms supports it 
  

Examples fo py.execnet usage 
-------------------------------------

compare cwd() of Popen Gateways
++++++++++++++++++++++++++++++++++++++++

A PopenGateway has the same working directory as the instantiatior::

    >>> import py, os
    >>> gw = py.execnet.PopenGateway()
    >>> ch = gw.remote_exec("import os; channel.send(os.getcwd())")
    >>> res = ch.receive()
    >>> assert res == os.getcwd()
    >>> gw.exit()

synchronously receive results from two sub processes <commit_msg>[svn r63216] 72: the code fragement -> fragment 114: You can use a string speficiation -> specification 147: Examples fo -> of<commit_after>

``py.execnet`` allows to:

* instantiate local or remote Python Processes
* send code for execution in one or many processes 
* asynchronously send and receive data between processes through channels 
* completely avoid manual installation steps on remote places

.. contents::

Gateways: immediately spawn local or remote process
----------------------------------------------------

In order to send code to a remote place or a subprocess
you need to instantiate a so-called Gateway object.  
There are currently three Gateway classes:

* :api:`py.execnet.PopenGateway` to open a subprocess 
  on the local machine.  Useful for making use 
  of multiple processors to to contain code execution
  in a separated environment. 

* :api:`py.execnet.SshGateway` to connect to 
  a remote ssh server and distribute execution to it. 

* :api:`py.execnet.SocketGateway` a way to connect to 
  a remote Socket based server. *Note* that this method
  requires a manually started 
  :source:py/execnet/script/socketserver.py
  script.  You can run this "server script" without 
  having the py lib installed on the remote system
  and you can setup it up as permanent service. 


remote_exec: execute source code remotely 
--------------------------------------------

All gateways offer remote code execution via this high level function::

    def remote_exec(source): 
        """return channel object for communicating with the asynchronously 
        executing 'source' code which will have a corresponding 'channel' 
        object in its executing namespace."""

With `remote_exec` you send source code to the other
side and get both a local and a remote Channel_ object,
which you can use to have the local and remote site
communicate data in a structured way.   Here is 
an example for reading the PID::

  >>> import py 
  >>> gw = py.execnet.PopenGateway()
  >>> channel = gw.remote_exec("""
  ...     import os
  ...     channel.send(os.getpid())
  ... """)
  >>> remote_pid = channel.receive()
  >>> remote_pid != py.std.os.getpid()
  True

.. _`Channel`: 
.. _`channel-api`: 
.. _`exchange data`: 

Channels: bidirectionally exchange data between hosts 
-------------------------------------------------------------

A channel object allows to send and receive data between 
two asynchronously running programs.  When calling
`remote_exec` you will get a channel object back and 
the code fragment running on the other side will 
see a channel object in its global namespace. 

Here is the interface of channel objects::

    #
    # API for sending and receiving anonymous values
    #
    channel.send(item): 
        sends the given item to the other side of the channel, 
        possibly blocking if the sender queue is full. 
        Note that items need to be marshallable (all basic 
        python types are).

    channel.receive():
        receives an item that was sent from the other side, 
        possibly blocking if there is none. 
        Note that exceptions from the other side will be 
        reraised as gateway.RemoteError exceptions containing 
        a textual representation of the remote traceback. 

    channel.waitclose(timeout=None): 
        wait until this channel is closed.  Note that a closed
        channel may still hold items that will be received or 
        send. Note that exceptions from the other side will be 
        reraised as gateway.RemoteError exceptions containing 
        a textual representation of the remote traceback. 

    channel.close(): 
        close this channel on both the local and the remote side. 
        A remote side blocking on receive() on this channel 
        will get woken up and see an EOFError exception. 


.. _xspec:


XSpec: string specification for gateway type and configuration
-------------------------------------------------------------------

``py.execnet`` supports a simple extensible format for 
specifying and configuring Gateways for remote execution.  
You can use a string spefication to make a new gateway, 
for example a new SshGateway::

    gateway = py.execnet.makegateway("ssh=myhost")

Let's look at some examples for valid specifications. 
Specification for an ssh connection to `wyvern`, running on python2.4 in the (newly created) 'mycache'  subdirectory::

    ssh=wyvern//python=python2.4//chdir=mycache

Specification of a python2.5 subprocess; with a low CPU priority ("nice" level). Current dir will be the current dir of the instantiator (that's true for all 'popen' specifications unless they specify 'chdir')::

    popen//python=2.5//nice=20

Specification of a Python Socket server process that listens on 192.168.1.4:8888; current dir will be the 'pyexecnet-cache' sub directory which is used a default for all remote processes::

    socket=192.168.1.4:8888

More generally, a specification string has this general format::

    key1=value1//key2=value2//key3=value3

If you omit a value, a boolean true value is assumed.  Currently
the following key/values are supported: 

* ``popen`` for a PopenGateway
* ``ssh=host`` for a SshGateway
* ``socket=address:port`` for a SocketGateway 
* ``python=executable`` for specifying Python executables
* ``chdir=path`` change remote working dir to given relative or absolute path
* ``nice=value`` decrease remote nice level if platforms supports it 
  

Examples of py.execnet usage 
-------------------------------------

compare cwd() of Popen Gateways
++++++++++++++++++++++++++++++++++++++++

A PopenGateway has the same working directory as the instantiatior::

    >>> import py, os
    >>> gw = py.execnet.PopenGateway()
    >>> ch = gw.remote_exec("import os; channel.send(os.getcwd())")
    >>> res = ch.receive()
    >>> assert res == os.getcwd()
    >>> gw.exit()

synchronously receive results from two sub processes 
{'fraction_non_alphanumeric': 0.11668224696184244, 'fraction_numerical': 0.007903458825529022, 'mean_word_length': 2.8799868117375538, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 72, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 4, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3664953', 'n_tokens_mistral': 3159, 'n_tokens_neox': 2947, 'n_words': 1408}