<jupyter_start><jupyter_text>Download model data using climet-lab----This notebook shows how to use the climet-lab data store to download the data for ECMWF, ECCC, and NCEP.More information about climet-lab, what models and variables are available can be found here:https://github.com/ecmwf-lab/climetlab-s2s-ai-challenge<jupyter_code>import climetlab as cml
import xarray as xr
xr.set_options(keep_attrs=True)

from dask.distributed import Client
import dask.config
dask.config.set({"array.slicing.split_large_chunks": False})
client = Client("tcp://10.12.206.54:34204")<jupyter_output><empty_output><jupyter_text>This command shows the climet-lab settings.Depending on how much data you are downloading, you will want to change your cache-directory, maximum-cache-size, and number-of-dowload-threads. There are a few examples below this cell.<jupyter_code>cml.settings
# good to set that cache does not delete files
#cml.settings.set("maximum-cache-size", "1800GB")
# increase parallel downloads: default 5
#cml.settings.set("number-of-download-threads", 53)<jupyter_output><empty_output><jupyter_text>What are you downloading?Here you choose your variable, model, and pressure level (if applicable). This only downloads one variable at a time, so you will need to rerun it if you want multiple variables.<jupyter_code>var = 'v'
model = 'eccc'
plev = 850
ds = cml.load_dataset('s2s-ai-challenge-training-input', origin=model, parameter=var, format='netcdf').to_xarray()<jupyter_output>By downloading data from this dataset, you agree to the terms and conditions defined at https://apps.ecmwf.int/datasets/data/s2s/licence/. If you do not agree with such terms, do not download the data.<jupyter_text>- Renaming variables to work in climpred and dropping coordinates we don't need.<jupyter_code>ds = ds.rename({"realization": "member","forecast_time": "init","lead_time": "lead","latitude": "lat","longitude": "lon"}).drop("valid_time")
if var=='gh':
    ds = ds.sel(plev=plev).drop("plev").rename({"gh": "gh_"+str(plev)})
elif var=='v':
    ds = ds.sel(plev=plev).drop("plev").rename({"v": "v_"+str(plev)})
elif var=='u':
    ds = ds.sel(plev=plev).drop("plev").rename({"u": "u_"+str(plev)})
elif var=="ttr":
    ds = ds.rename({"ttr": "olr"}).drop("nominal_top")
ds
ds.lead
#Each model has different lead days available
if model=='ecmwf':
    ds = ds.sel(lead=slice("1 days","47 days")) # for ecmwf
elif model=='ncep':
    ds = ds.sel(lead=slice("2days","44 days")) # for ncep
ds = ds.sortby('init')
ds = ds.chunk({'init':-1,'lead':-1,'lon':'auto','lat':'auto','member':'auto'})<jupyter_output>/glade/u/home/jaye/.local/lib/python3.9/site-packages/xarray/core/indexing.py:1379: PerformanceWarning: Slicing with an out-of-order index is generating 20 times more chunks
  return self.array[key]<jupyter_text>Write to zarr<jupyter_code># If you are creating a new zarr use mode='w', if you are adding a variable to an existing zarr use mode='a'
%time ds.to_zarr('/glade/campaign/mmm/c3we/jaye/S2S_zarr/ECCC.uvolr.raw.daily.geospatial.zarr', mode="a", consolidated=True)<jupyter_output>CPU times: user 478 ms, sys: 34 ms, total: 512 ms
Wall time: 51.4 s
{'fraction_non_alphanumeric': 0.14720812182741116, 'fraction_numerical': 0.017766497461928935, 'mean_word_length': 5.839479392624729, 'pattern_counts': {'":': 11, '<': 18, '<?xml version=': 0, '>': 18, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '507798', 'license': 'MIT', 'n_tokens_mistral': 1150, 'n_tokens_neox': 1075, 'n_words': 294, 'path': '1_download_data.ipynb', 'repo_name': 'NCAR-ASP-2021/s2s_verif_and_data', 'size': '3152'}