<jupyter_start><jupyter_text>Function to benchmarkWe'll look at a simple function to start off with, that just returns the mean of the data<jupyter_code>def func(da):    
    return da.mean()<jupyter_output><empty_output><jupyter_text>Running OnceJust running it once gives a basic idea of performance. I've used the chunking in the NetCDF file as a starting point.<jupyter_code>climtas.profile.profile_once('/g/data/ub4/era5/netcdf/surface/t2m/2019/t2m_era5_global_20190101_*.nc',
                             variable='t2m',
                             function=func,
                             chunks={'time':93, 'latitude': 91, 'longitude': 180})<jupyter_output><empty_output><jupyter_text>BenchmarkingNow let's benchmark with a few different chunk shapes. The function gets run three times with each of the chunk options and the minimum time taken is returned<jupyter_code>climtas.profile.benchmark('/g/data/ub4/era5/netcdf/surface/t2m/2019/t2m_era5_global_20190101_*.nc',
                          variable='t2m',
                          function=func,
                          chunks={'time':[93, 93, 93], 'latitude': [91, 91*2, 91], 'longitude': [180, 180*2, 180//2]})<jupyter_output><empty_output><jupyter_text>DistributedLet's switch to Dask's distributed mode, and process a whole year of data on 4 cpus<jupyter_code>import climtas.nci
client = climtas.nci.GadiClient()
client
climtas.profile.benchmark('/g/data/ub4/era5/netcdf/surface/t2m/2019/t2m_era5_global_*.nc',
                          variable='t2m',
                          function=func,
                          chunks={'time':[93, 93, 93], 'latitude': [91, 91*2, 91], 'longitude': [180, 180*2, 180//2]})<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11771561771561771, 'fraction_numerical': 0.05944055944055944, 'mean_word_length': 3.0495283018867925, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '419012', 'license': 'Apache-2.0', 'n_tokens_mistral': 628, 'n_tokens_neox': 562, 'n_words': 125, 'path': 'notebooks/Benchmarking.ipynb', 'repo_name': 'ScottWales/climtas', 'size': '1716'}