<jupyter_start><jupyter_text><img src="https://www.colorado.edu/rc/sites/default/files/page/logo.png"     alt="Logo for Research Computing @ University of Colorado Boulder"     width="400" />      Running functions on IPython engines  Start cluster / Load the module<jupyter_code>!ipcluster start -n 4 --daemonize
import ipyparallel<jupyter_output><empty_output><jupyter_text>Create a client instance, used to connect the controller to the remote engines<jupyter_code>profile = 'example-shas'
rc = ipyparallel.Client(profile=profile)
nengines = len(rc)
nengines
rc.ids<jupyter_output><empty_output><jupyter_text>Create a `DirectView`A `DirectView` is created by list access to a client instance:<jupyter_code>dvall  = rc[:]
print(dvall)
type(dvall)<jupyter_output><empty_output><jupyter_text>Running Python functions on the engines BlockingIn blocking mode the `apply` function waits until all function calls are completed.<jupyter_code>import os, socket
dvall.block = True
hostnames = dvall.apply(socket.gethostname)
print(hostnames)<jupyter_output>['schleppy-pro.local', 'schleppy-pro.local', 'schleppy-pro.local', 'schleppy-pro.local']<jupyter_text>`apply_sync` is blocking<jupyter_code>dvall.block = False
pids = dvall.apply_sync(os.getpid)
print(pids)
for i in range(nengines):
    istr = '{:02d}'.format(i)  # returns a 2-digit string whose value is i
    pstr = str(pids[i])
    hstr = str(hostnames[i])
    msg = 'Engine '+istr+':   pid = '+pstr+';  hostname ='+hstr
    print(msg)<jupyter_output>Engine 00:   pid = 32167;  hostname =schleppy-pro.local
Engine 01:   pid = 32168;  hostname =schleppy-pro.local
Engine 02:   pid = 32169;  hostname =schleppy-pro.local
Engine 03:   pid = 32170;  hostname =schleppy-pro.local<jupyter_text>Non blocking<jupyter_code>def mypid():
    import time, os
    time.sleep(10)
    pid = os.getpid()
    return pid<jupyter_output><empty_output><jupyter_text>Note the import to make sure the module is imported on the engines.<jupyter_code>dvall.block = False
res = dvall.apply(mypid)
res<jupyter_output><empty_output><jupyter_text>Checking if result is ready<jupyter_code>res.ready()<jupyter_output><empty_output><jupyter_text>Blocking to get the result<jupyter_code>pids = res.get()
pids
!ipcluster stop<jupyter_output>2019-05-20 21:16:54.776 [IPClusterStop] Stopping cluster [pid=32157] with [signal=<Signals.SIGINT: 2>]
{'fraction_non_alphanumeric': 0.11802030456852793, 'fraction_numerical': 0.025380710659898477, 'mean_word_length': 6.017804154302671, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 36, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '255896', 'license': 'MIT', 'n_tokens_mistral': 909, 'n_tokens_neox': 845, 'n_words': 190, 'path': '04-function-engines.ipynb', 'repo_name': 'ResearchComputing/RMACC-2019-ipyparallel', 'size': '2364'}