<jupyter_start><jupyter_text>Test joblib<jupyter_code>from joblib import Parallel, delayed

import multiprocessing


from math import sqrt
import time as t
# check no of cores on your machine
# https://stackoverflow.com/questions/1006289/how-to-find-out-the-number-of-cpus-using-python

cores = multiprocessing.cpu_count()<jupyter_output><empty_output><jupyter_text>Simulate a time consuming ML job<jupyter_code>def ml_func(x):
    t.sleep(1)<jupyter_output><empty_output><jupyter_text>Parralelize with one job<jupyter_code>time res = Parallel(n_jobs=1)(delayed(ml_func)(i) for i in range(10))<jupyter_output>CPU times: user 4.89 ms, sys: 2.15 ms, total: 7.04 ms
Wall time: 10 s<jupyter_text>Parralelize with two job<jupyter_code>time res = Parallel(n_jobs=2)(delayed(f)(i) for i in range(10))<jupyter_output>CPU times: user 30.6 ms, sys: 16.8 ms, total: 47.4 ms
Wall time: 5.84 s<jupyter_text>Parralelize with 4 jobswe have 4 cores on this machine<jupyter_code>time res = Parallel(n_jobs=cores)(delayed(ml_func)(i**2) for i in range(10))<jupyter_output>CPU times: user 17.7 ms, sys: 3.81 ms, total: 21.5 ms
Wall time: 3.02 s<jupyter_text>No parallelization<jupyter_code>time res = [ml_func(i) for i in range(10)]<jupyter_output>CPU times: user 1.37 ms, sys: 1.85 ms, total: 3.22 ms
Wall time: 10 s
{'fraction_non_alphanumeric': 0.11864406779661017, 'fraction_numerical': 0.05161787365177196, 'mean_word_length': 6.422857142857143, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '89394', 'license': 'Apache-2.0', 'n_tokens_mistral': 567, 'n_tokens_neox': 511, 'n_words': 129, 'path': 'notebooks/basic_demo.ipynb', 'repo_name': 'kjahan/parallelizer', 'size': '1298'}