<jupyter_start><jupyter_text>Benchmark of EXtra-foamAuthor: <NAME>, March 03, 2020<jupyter_code>from extra_foam.algorithms import mask_image_data, nanmean_image_data, nansum
import random
import numpy as np
import multiprocessing as mp

mp.cpu_count()<jupyter_output><empty_output><jupyter_text>Assume we want to process a train of detector images.<jupyter_code>images = np.random.randn(128, 1200, 1200).astype(np.float32)
images[:, fd00:a516:7c1b:17cd:6d81:2137:bd2a:2c5b, ::2] = np.nan<jupyter_output><empty_output><jupyter_text>Benchmark nanmean<jupyter_code>%timeit np.nanmean(images, axis=0)
%timeit nanmean_image_data(images)<jupyter_output>13.5 ms ± 759 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)<jupyter_text>A common use case is to apply `nanmean` to a list of selected images.<jupyter_code># select 120 images out of 128 ones
selected = random.sample(range(images.shape[0]), 120)
# numpy is slower than operating on all the images since it copies the data when 'selected' is a list.
%timeit np.nanmean(images[selected], axis=0)
%timeit nanmean_image_data(images, kept=selected)<jupyter_output>12.1 ms ± 117 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)<jupyter_text>Benchmark masking threshold mask<jupyter_code>imgs = images.copy()
%timeit imgs[(imgs > 1) | (imgs < -1)] = np.nan
imgs = images.copy()
%timeit mask_image_data(imgs, threshold_mask=(-1, 1))<jupyter_output>19.6 ms ± 533 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)<jupyter_text>image mask<jupyter_code>image_mask = np.ones((1200, 1200), dtype=bool)
image_mask[::3, ::3] = np.nan
imgs = images.copy()
%timeit imgs[:, image_mask] = np.nan
imgs = images.copy()
%timeit mask_image_data(imgs, image_mask=image_mask)<jupyter_output>27.5 ms ± 71.5 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)<jupyter_text>threshold mask and image mask<jupyter_code>imgs = images.copy()
%timeit imgs[(image_mask) | (imgs > 1) | (imgs < -1)] = np.nan
imgs = images.copy()
%timeit mask_image_data(imgs, image_mask=image_mask, threshold_mask=(-1, 1))<jupyter_output>27.5 ms ± 37.6 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)<jupyter_text>Benchmark nan-statistics Non-paralleled funtion can be also much faster than `numpy`.<jupyter_code>images = np.random.randn(1200, 1200).astype(np.float32)
images[::2, ::2] = np.nan
%timeit np.nansum(images)
%timeit nansum(images)<jupyter_output>1.48 ms ± 60.1 ns per loop (mean ± std. dev. of 7 runs, 1000 loops each)
{'fraction_non_alphanumeric': 0.12956945572705117, 'fraction_numerical': 0.05645816409423233, 'mean_word_length': 6.139130434782609, 'pattern_counts': {'":': 0, '<': 30, '<?xml version=': 0, '>': 30, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '458519', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1047, 'n_tokens_neox': 938, 'n_words': 260, 'path': 'benchmarks/benchmark_demo.ipynb', 'repo_name': 'zhujun98/EXtra-foam', 'size': '2427'}