<jupyter_start><jupyter_text>Percentile==========<jupyter_code>import xarray
import climtas
import dask.array
import pandas
import numpy<jupyter_output><empty_output><jupyter_text>We have hourly input data for a year that we want calculate the 90th percentile along the time axis for each grid point<jupyter_code>time = pandas.date_range('20010101', '20020101', freq='H', closed='left')

data = dask.array.random.random((len(time),50,100), chunks=(24*60,25,25))
lat = numpy.linspace(-90, 90, data.shape[1])
lon = numpy.linspace(-180, 180, data.shape[2], endpoint=False)

da = xarray.DataArray(data, coords=[('time', time), ('lat', lat), ('lon', lon)], name='temperature')
da<jupyter_output><empty_output><jupyter_text>The Xarray way is to use [xarray.DataArray.quantile](http://xarray.pydata.org/en/stable/generated/xarray.DataArray.quantile.html), however for a dataset chunked along the time axis this will give an error message:<jupyter_code>try:
    da.quantile(0.9, 'time')
except Exception as e:
    print('Error:', e)<jupyter_output>Error: dimension time on 0th function argument to apply_ufunc with dask='parallelized' consists of multiple chunks, but is also a core dimension. To fix, either rechunk into a single dask array chunk along this dimension, i.e., ``.chunk(time: -1)``, or pass ``allow_rechunk=True`` in ``dask_gufunc_kwargs`` but beware that this may significantly increase memory usage.<jupyter_text>To get this to work we must rechunk the data so it is not chunked on the time axis, a very expensive operation for large datasets where variables are split into multiple files for different years.<jupyter_code>da.chunk({'time': None}).quantile(0.9, 'time')<jupyter_output><empty_output><jupyter_text>Dask has an approximate percentile operation that works on data chunked along time, however that will only run on one dimensional data<jupyter_code>dask.array.percentile(da.data[:, 30, 60], 90)<jupyter_output><empty_output><jupyter_text>[climtas.blocked.approx_percentile](api/blocked.rstclimtas.blocked.approx_percentile) extends the Dask parallel approximate percentile calculation to multi-dimensional datasets, so large datasets don't need to be rechunked.Note that this is an approximation to the true percentile, check it behaves appropriately for your dataset by comparing `approx_percentile` and `numpy.percentile` on a subset of the data.<jupyter_code>climtas.approx_percentile(da, 90, 'time')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10961695191524043, 'fraction_numerical': 0.023227383863080684, 'mean_word_length': 6.893890675241158, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '85672', 'license': 'Apache-2.0', 'n_tokens_mistral': 803, 'n_tokens_neox': 751, 'n_words': 245, 'path': 'doc/percentile.ipynb', 'repo_name': 'ScottWales/climtas', 'size': '2454'}