<jupyter_start><jupyter_text>Multiprocessing Module[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/borchero/pyblaze/blob/master/docs/examples/multiprocessing.ipynb)[![Download Jupyter Notebbok](https://img.shields.io/badge/Github-Download-brightgreen)](https://github.com/borchero/pyblaze/blob/master/docs/examples/multiprocessing.ipynb)When working with independent data, spreading computations across multiple cores often provides an easy way to (linearly) increase a program’s throughput.One of the most common forms of parallelism is to split loops across multiple threads. Unfortunately, Python does not support such parallelism easily, especially if computations are CPU-bound. Using simple threads is often not an option due to the GIL and working with processes is often tedious. This is particularly true when working with PyTorch tensors as they have to be passed between processes over queues.The `multiprocessing` module of PyBlaze aims to make it possible to speed up computations easily, providing a high-level interface.  VectorizationPyBlaze refers to vectorization as the process of parallelizing for-loops of the following form:```pythonresult = []for item in iterable:    result.append(map(item))```PyBlaze’s class providing this functionality is the `Vectorizer` class in the `multiprocessing` module. In the background, the vectorizer handles everything such as creating processes, ensuring their shutdown, passing items and results between processes. Due to the class’s simplicity, it can often be used as a drop-in replacement for existing for-loops which not only reduces runtime but enhances readability.  Example ProgramConsider, for example, an array `text` of strings which you want to tokenize according to a complex function `tokenize`. The function takes as input a single string and returns its tokenization.<jupyter_code>import numpy as np
import time

texts = [
    ''.join(np.random.choice(['a', 'b', 'c', 'd', 'e', ' '], size=(20,)))
    for _ in range(100)
]

def tokenize(text):
    time.sleep(0.01)
    return text.split()<jupyter_output><empty_output><jupyter_text>The sequential implementation is very easy, however, not particularly efficient:<jupyter_code>def sequential(texts):
    return [tokenize(t) for t in texts]<jupyter_output><empty_output><jupyter_text>The `Vectorizer` can easily be used to compute the tokenizations of all texts in parallel:<jupyter_code>import pyblaze.multiprocessing as xmp

def parallel(texts):
    tokenizer = xmp.Vectorizer(tokenize, num_workers=4)
    return tokenizer.process(texts)<jupyter_output><empty_output><jupyter_text>We can now compare the runtime of the sequential and vectorized implementation:<jupyter_code>%timeit sequential(texts)
%timeit parallel(texts)<jupyter_output>280 ms ± 1.17 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
{'fraction_non_alphanumeric': 0.08347706105553639, 'fraction_numerical': 0.005864091065884788, 'mean_word_length': 5.872037914691943, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '101875', 'license': 'MIT', 'n_tokens_mistral': 835, 'n_tokens_neox': 796, 'n_words': 307, 'path': 'docs/examples/multiprocessing.ipynb', 'repo_name': 'Greenroom-Robotics/pyblaze', 'size': '2899'}