<jupyter_start><jupyter_text>Multiprocessing for map dataIn this notebook we will demonstrate how to apply multiprocessing for images with map data.First we will demonstrate the transformation we want to apply, and then we will run it on multiple map sections at the same time with multiprocessing.<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline

import multiprocessing
import pickle
import PIL
from PIL import Image
from pathlib import Path
data_folder = Path("../data/")<jupyter_output><empty_output><jupyter_text>What we want to do is load a TIFF image with the data for a map layer, process it and then save the results.  Example process  Loading the image<jupyter_code>Image.MAX_IMAGE_PIXELS = 2000000000  # Increase PIL image load limit
filename="Hansen_GFC-2018-v1.6_treecover2000_50N_010W.tif"
granule = Image.open(data_folder/"hansen"/filename)
granule.size<jupyter_output><empty_output><jupyter_text>Processing the image<jupyter_code>granule_resized = granule.resize((1000, 1000), PIL.Image.LINEAR)
granule_array = np.array(granule_resized)<jupyter_output><empty_output><jupyter_text>Saving the image to file<jupyter_code>processed_filename = filename[:-3]+"pickle"
with open(data_folder/"processed/hansen"/processed_filename, "wb") as file:
    pickle.dump(granule_array, file)<jupyter_output><empty_output><jupyter_text>We can check that it was stored properly by loading it again.<jupyter_code>with open(data_folder/"processed/hansen"/processed_filename, "rb") as file:
    array_check = pickle.load(file)
print(np.all(array_check == granule_array))
del array_check<jupyter_output>True<jupyter_text>As a functionWe can turn the previous process in to a function for easier use later on.<jupyter_code>def process_image(image):
    # Just a simple example of a process, but non-trivial in terms of time
    return np.array(image.resize((1000, 1000), PIL.Image.LINEAR))

def image_pipeline(origin, destination=None):
    origin=Path(origin)
    if destination is None:
        filename = origin.stem + ".pickle"
        destination = origin.parent.parent/"processed"/origin.parts[-2]/filename
    loaded = Image.open(origin)
    processed = process_image(loaded)
    with destination.open("wb") as file:
        pickle.dump(processed, file)
    return destination  # Just to return something<jupyter_output><empty_output><jupyter_text>And we can check that it worked.<jupyter_code>%%time

origin = data_folder/"hansen"/"Hansen_GFC-2018-v1.6_treecover2000_50N_000E.tif"
image_pipeline(origin)

destination = data_folder/"processed/hansen/Hansen_GFC-2018-v1.6_treecover2000_50N_000E.pickle"
Path(destination).is_file()  # This checks that the file now exists<jupyter_output>CPU times: user 9.84 s, sys: 955 ms, total: 10.8 s
Wall time: 11.4 s<jupyter_text>MultiprocessingNow we use our huble pipeline with `multiprocessing`.First it would be useful to know how many CPU we have in this machine. This is easy to do with `multiprocessing.cpu_count()`<jupyter_code>print(f"There are {multiprocessing.cpu_count()} CPU available.", )<jupyter_output>There are 2 CPU available.<jupyter_text>There are various ways to start a multiprocess, but we are going to use the `Pool`. We need to gather of all the inputs we want to our pipeline in a list.<jupyter_code>input_list = list(origin.parent.glob("*.tif"))<jupyter_output><empty_output><jupyter_text>Then, we define a `Pool` manager with a set number of maximum concurrent processes.<jupyter_code>max_processes = multiprocessing.cpu_count() // 2  # Let's use at most half of our processors here
pool = multiprocessing.Pool(max_processes)<jupyter_output><empty_output><jupyter_text>Finally, we execute it over all out inputs with the `.map` method.<jupyter_code>%%time
results = pool.map(image_pipeline, input_list)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09064935064935065, 'fraction_numerical': 0.0225974025974026, 'mean_word_length': 6.118299445471349, 'pattern_counts': {'":': 0, '<': 42, '<?xml version=': 0, '>': 42, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '245477', 'license': 'MIT', 'n_tokens_mistral': 1329, 'n_tokens_neox': 1227, 'n_words': 355, 'path': 'notebooks/image_multiprocess_test.ipynb', 'repo_name': 'miguel-bm/pandora', 'size': '3850'}