<jupyter_start><jupyter_text>Different download and save approaches in Python> A comparison of synchronous, multiprocess, and async approaches to downloading and saving files- toc: true - badges: true- comments: false- categories: [jupyter] In this example I wanted to demonstrate the big differences in times between downloading files synchronously, in parallel processes, and asynchronously using async and await packages. The results shown here were run on my laptop using a Intel® Core™ i9-9980HK CPU @ 2.40GHz × 16 processor. I download 15 CSV files totalling about 26.7 megabytes at the time of writing.I expect the results to change significantly based on the number and size of the files but I think this nicely demonstrates for a relatively small number of largish files async performs better (in general - depending on the number of cores available) compared to parallel downloading with multiple processes and asynchronous downloading is significantly faster than synchronous sequential downloading files.  Requirements (Conda)<jupyter_code>%conda install -c conda-forge -y aiofiles=0.5 aiohttp=3.6 joblib=0.16 requests=2.24<jupyter_output><empty_output><jupyter_text>Requirements (Pip)<jupyter_code>%pip install aiofiles==0.5 aiohttp==3.6 joblib==0.16 requests==2.24<jupyter_output><empty_output><jupyter_text>Prepare a list of CSV files to download<jupyter_code>file_url_list = [f"http://data.wa.aemo.com.au/datafiles/load-summary/load-summary-{year}.csv" for year in range(2006, 2021)]<jupyter_output><empty_output><jupyter_text>Download 15 csv files synchronously<jupyter_code>import requests
def download_file(file_url: str):
    response = requests.get(file_url)
    if response.status_code == 200:
        file_name = file_url.split("/")[-1]
        with open(file_name, mode="wb") as file:
            file.write(response.content)
%%time
for file_url in file_url_list:
    download_file(file_url)<jupyter_output>CPU times: user 791 ms, sys: 481 ms, total: 1.27 s
Wall time: 1min 17s<jupyter_text>Download 15 csv files in parallel processes using all available cores<jupyter_code>from joblib import Parallel, delayed
%%time
Parallel(n_jobs=-1)(delayed(download_file)(file_url) for file_url in file_url_list)
pass # pass just so Jupyter doesn't show the list of null returns from Parallel<jupyter_output>CPU times: user 59.6 ms, sys: 91.8 ms, total: 151 ms
Wall time: 10.9 s<jupyter_text>Download 15 csv files asynchronously using async and await libraries<jupyter_code>import asyncio
import aiofiles
import aiohttp

from typing import List
async def download_file(session: aiohttp.ClientSession, file_url: str):
    async with session.get(file_url) as resp:
        file_name = file_url.split("/")[-1]
        if resp.status == 200:
            async with aiofiles.open(file_name, mode="wb") as f:
                await f.write(await resp.read())


async def download_file_list(file_url_list: List[str]):
    async with aiohttp.ClientSession() as session:
        # If downloading a very large number of files you'd probably need to limit
        # the number of concurrent requests that could be sent at once
        tasks = [download_file(session, file_url) for file_url in file_url_list]
        await asyncio.gather(*tasks)
import time
t0 = time.time()
await download_file_list(file_url_list)
t1 = time.time()
print(f"Wall time: {t1 - t0:0.2}s")
# Using time functions since %%time magic errors when used in a cell with await<jupyter_output>Wall time: 8.0s
{'fraction_non_alphanumeric': 0.08108108108108109, 'fraction_numerical': 0.025876940770557792, 'mean_word_length': 4.936860068259386, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 23, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 19, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '566447', 'license': 'Apache-2.0', 'n_tokens_mistral': 1120, 'n_tokens_neox': 1034, 'n_words': 377, 'path': '_notebooks/2020-09-28-compare-download-code.ipynb', 'repo_name': 'owenlamont/blog', 'size': '3478'}