<jupyter_start><jupyter_text>Reusing a pool of workers Some algorithms require to make several consecutive calls to a parallel function interleaved with processing of the intermediate results. Calling Parallel several times in a loop is sub-optimal because it will create and destroy a pool of workers (threads or processes) several times which can cause a significant overhead.For this case it is more efficient to use the context manager API of the Parallel class to re-use the same pool of workers for several calls to the Parallel object:<jupyter_code>with Parallel(n_jobs=2) as parallel:
    accumulator = 0.
    n_iter = 0
    while accumulator < 1000:
        results = parallel(delayed(sqrt)(accumulator + i ** 2)for i in range(5))
        accumulator += sum(results)  # synchronization barrier
        n_iter += 1
(accumulator, n_iter)<jupyter_output><empty_output><jupyter_text>Working with numerical data in shared memory (memmaping)  Automated array to memmap conversion<jupyter_code>import numpy as np
from joblib import Parallel, delayed
from joblib.pool import has_shareable_memory

Parallel(n_jobs=2, max_nbytes=1e6)(
    delayed(has_shareable_memory)(np.ones(int(i)))
    for i in [1e2, 1e4, 1e6])
import joblib
joblib.__version__<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06342991386061081, 'fraction_numerical': 0.014878621769772905, 'mean_word_length': 5.114832535885167, 'pattern_counts': {'":': 0, '<': 10, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '302788', 'license': 'Apache-2.0', 'n_tokens_mistral': 382, 'n_tokens_neox': 364, 'n_words': 145, 'path': 'concurrency/joblib/Embarrassingly_parallel_for_loops.ipynb', 'repo_name': 'scotthuang1989/Python-3-Module-of-the-Week', 'size': '1277'}