<jupyter_start><jupyter_text>IPython's Data Publication API IPython has an API that allows IPython Engines to publish data back to the Client.  This Notebook shows how this API works.  Setup We begin by enabling matplotlib plotting and creating a `Client` object to work with an IPython cluster.<jupyter_code>%matplotlib inline
import ipyparallel as ipp
c = ipp.Client()
dv = c[:]
dv.block = False
dv<jupyter_output><empty_output><jupyter_text>Simple publication Here is a simple Python function we are going to run on the Engines.  This function uses `publish_data` to publish a simple Python dictionary when it is run.<jupyter_code>def publish_it():
    from ipyparallel.datapub import publish_data
    publish_data(dict(a='hi2'))<jupyter_output><empty_output><jupyter_text>We run the function on the Engines using `apply_async` and save the returned `AsyncResult` object:<jupyter_code>ar = dv.apply_async(publish_it)<jupyter_output><empty_output><jupyter_text>The published data from each engine is then available under the `.data` attribute of the `AsyncResult` object.<jupyter_code>ar.data<jupyter_output><empty_output><jupyter_text>Each time `publish_data` is called, the `.data` attribute is updated with the most recently published data.  Simulation loop In many cases, the Engines will be running a simulation loop and we will want to publish data at each time step of the simulation.  To show how this works, we create a mock simulation function that iterates over a loop and publishes a NumPy array and loop variable at each time step. By inserting a call to `time.sleep(1)`, we ensure that new data will be published every second.<jupyter_code>def simulation_loop():
    from ipyparallel.datapub import publish_data
    import time
    import numpy as np
    for i in range(10):
        publish_data(dict(a=np.random.rand(20), i=i))
        time.sleep(2)<jupyter_output><empty_output><jupyter_text>Again, we run the `simulation_loop` function in parallel using `apply_async` and save the returned `AsyncResult` object.<jupyter_code>ar = dv.apply_async(simulation_loop)<jupyter_output><empty_output><jupyter_text>New data will be published by the Engines every second.  Anytime we access `ar.data`, we will get the most recently published data.<jupyter_code>import matplotlib.pyplot as plt
data = ar.data
for i, d in enumerate(data):
    plt.plot(d['a'], label='engine: '+str(i))
plt.title('Data published at time step: ' + str(data[0]['i']))
plt.legend()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08133012820512821, 'fraction_numerical': 0.003205128205128205, 'mean_word_length': 5.694369973190349, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '255258', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 763, 'n_tokens_neox': 749, 'n_words': 283, 'path': 'examples/Data Publication API.ipynb', 'repo_name': 'vertingo/Anaconda_Videos_Tutos', 'size': '2496'}