<jupyter_start><jupyter_text>Overview and getting started IntroductionLet's start with an overview of IPython's architecture for paralleland distributed computing. This architecture abstracts out parallelismin a very general way, which enables IPython to support many differentstyles of parallelism including:-   Single program, multiple data (SPMD) parallelism-   Multiple program, multiple data (MPMD) parallelism-   Message passing using MPI or ØMQ-   Task farming-   Data parallel-   Coordination of distributed processes-   Combinations of these approaches-   Custom user defined approachesMost importantly, IPython enables all types of parallel applications tobe developed, executed, debugged and monitored *interactively*. Hence,the `I` in `IPython`. Some example use cases for`IPython.parallel`:-   Quickly parallelize algorithms that are embarrassingly parallel    using a number of simple approaches. Many simple things can be    parallelized interactively in one or two lines of code.-   Steer traditional MPI applications on a supercomputer from an    IPython session on your laptop.-   Analyze and visualize large datasets (that could be remote and/or    distributed) interactively using IPython and tools like    matplotlib/TVTK.-   Develop, test and debug new parallel algorithms (that may use MPI or PyZMQ)    interactively.-   Tie together multiple MPI jobs running on different systems into one    giant distributed and parallel system.-   Start a parallel job on your cluster and then have a remote    collaborator connect to it and pull back data into their local    IPython session for plotting and analysis.-   Run a set of tasks on a set of CPUs using dynamic load balancing.  Architecture overviewThe IPython architecture consists of four components:-   The IPython engine-   The IPython hub-   The IPython schedulers-   The cluster clientThese components live in the `IPython.parallel` package and areinstalled with IPython.  IPython engineThe IPython engine is a Python instance that accepts Python commands overa network connection.  When multiple engines are started, paralleland distributed computing becomes possible. An important property of anIPython engine is that it blocks while user code is being executed. Readon for how the IPython controller solves this problem to expose a cleanasynchronous API to the user.  IPython controllerThe IPython controller processes provide an interface for working with aset of engines. At a general level, the controller is a collection ofprocesses to which IPython engines and clients can connect. Thecontroller is composed of a `Hub` and a collection of`Schedulers`, which may be in processes or threads.The controller provides a single point of contact for users whowish to utilize the engines in the cluster. There is a variety ofdifferent ways of working with a controller, but all of thesemodels are implemented via the `View.apply` method, afterconstructing `View` objects to represent different collections engines.The two primary models for interacting with engines are:-   A **Direct** interface, where engines are addressed explicitly.-   A **LoadBalanced** interface, where the Scheduler is trusted with    assigning work to appropriate engines.Advanced users can readily extend the View models to enable other stylesof parallelism.  The HubThe center of an IPython cluster is the Hub.  The Hub can be viewed as an über-logger, which keeps track of engine connections, schedulers, clients, as well as persist alltask requests and results in a database for later use.  SchedulersAll actions that can be performed on the engine go through a Scheduler.While the engines themselves block when user code is run, the schedulershide that from the user to provide a fully asynchronous interface to aset of engines.  Each Scheduler is a small GIL-less function in C providedby pyzmq (the Python load-balanced scheduler being an exception).   ØMQ and PyZMQAll of this is implemented with the lovely ØMQ messaging library,and pyzmq, the lightweight Python bindings, which allows very fastzero-copy communication of objects like numpy arrays.  IPython client and viewsThere is one primary object, the `Client`, forconnecting to a cluster. For each execution model, there is acorresponding `View`. These views allow users tointeract with a set of engines through the interface. Here are the twodefault views:-   The `DirectView` class for explicit addressing.-   The `LoadBalancedView` class for destination-agnostic    scheduling.  Getting Started Starting the IPython controller and enginesTo follow along with this tutorial, you will need to start the IPythoncontroller and four IPython engines. The simplest way of doing this iswith the [clusters tab](/clusters),or you can use the `ipcluster` command in a terminal:    $ ipcluster start -n 4There isn't time to go into it here, but ipcluster can be used to start enginesand the controller with various batch systems including:* SGE* PBS* LSF* MPI* SSH* WinHPCMore information on starting and configuring the IPython cluster in [the IPython.parallel docs](http://ipython.org/ipython-doc/stable/parallel/parallel_process.html).Once you have started the IPython controller and one or more engines,you are ready to use the engines to do something useful. To make sure everything is working correctly, let's do a very simple demo:<jupyter_code>from IPython import parallel
rc = parallel.Client()
rc.block = True
rc.ids
def mul(a,b):
    return a*b
def summary():
    """summarize some info about this process"""
    import os
    import socket
    import sys
    return {
        'cwd': os.getcwd(),
        'Python': sys.version,
        'hostname': socket.gethostname(),
        'pid': os.getpid(),
    }
mul(5,6)
summary()<jupyter_output><empty_output><jupyter_text>What does it look like to call this function remotely?Just turn `f(*args, **kwargs)` into `view.apply(f, *args, **kwargs)`!<jupyter_code>rc[0].apply(mul, 5, 6)
rc[0].apply(summary)<jupyter_output><empty_output><jupyter_text>And the same thing in parallel?<jupyter_code>rc[:].apply(mul, 5, 6)
rc[:].apply(summary)<jupyter_output><empty_output><jupyter_text>Python has a builtin map for calling a function with a variety of arguments<jupyter_code>map(mul, range(1,10), range(2,11))<jupyter_output><empty_output><jupyter_text>So how do we do this in parallel?<jupyter_code>view = rc.load_balanced_view()
view.map(mul, range(1,10), range(2,11))<jupyter_output><empty_output><jupyter_text>And a preview of parallel magics:<jupyter_code>%%px
import os, socket
print os.getpid()
print socket.gethostname()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.05668934240362812, 'fraction_numerical': 0.0031746031746031746, 'mean_word_length': 4.971119133574007, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '131705', 'license': 'MIT', 'n_tokens_mistral': 1708, 'n_tokens_neox': 1657, 'n_words': 849, 'path': 'IPython-parallel-tutorial/Overview.ipynb', 'repo_name': 'sunny2309/scipy_conf_notebooks', 'size': '6615'}