<jupyter_start><jupyter_text>Hello World!Here's an example notebook with some documentation on how to access CMIP data.<jupyter_code>%matplotlib inline

import xarray as xr
import intake

# util.py is in the local directory
# it contains code that is common across project notebooks
# or routines that are too extensive and might otherwise clutter
# the notebook design
import util 
print('hello world!')<jupyter_output>hello world!<jupyter_text>Demonstrate how to use `intake-esm`[Intake-esm](https://intake-esm.readthedocs.io) is a data cataloging utility that facilitates access to CMIP data. It's pretty awesome.An `intake-esm` collection object establishes a link to a database that contains file locations and associated metadata (i.e., which experiement, model, the come from).  Opening a collectionFirst step is to open the collection by pointing the collection definition file, which is a JSON file that conforms to the [ESM Collection Specification](https://github.com/NCAR/esm-collection-spec). The collection JSON files are stored locally in this repository for purposes of reproducibility---and because Cheyenne compute nodes don't have Internet access. The primary source for these files is the [intake-esm-datastore](https://github.com/NCAR/intake-esm-datastore) repository. Any changes made to these files should be pulled from that repo. For instance, the Pangeo cloud collection is available [here](https://raw.githubusercontent.com/NCAR/intake-esm-datastore/master/catalogs/pangeo-cmip6.json).<jupyter_code>if util.is_ncar_host():
    col = intake.open_esm_datastore("../catalogs/glade-cmip6.json")
else:
    col = intake.open_esm_datastore("../catalogs/pangeo-cmip6.json")
col<jupyter_output><empty_output><jupyter_text>`intake-esm` is build on top of [pandas](https://pandas.pydata.org/pandas-docs/stable). It is possible to view the `pandas.DataFrame` as follows.<jupyter_code>col.df.head()<jupyter_output><empty_output><jupyter_text>It is possible to interact with the `DataFrame`; for instance, we can see what the "attributes" of the datasets are by printing the columns.<jupyter_code>col.df.columns<jupyter_output><empty_output><jupyter_text>Search and discovery Finding unique entriesLet's query the data to see what models ("source_id"), experiments ("experiment_id") and temporal frequencies ("table_id") are available.<jupyter_code>import pprint 
uni_dict = col.unique(['source_id', 'experiment_id', 'table_id'])
pprint.pprint(uni_dict, compact=True)<jupyter_output>{'experiment_id': {'count': 29,
                   'values': ['ssp370', 'esm-ssp585', '1pctCO2-bgc', 'hist-bgc',
                              '1pctCO2', 'abrupt-4xCO2', 'historical',
                              'piControl', 'amip', 'esm-hist', 'esm-piControl',
                              'hist-GHG', 'hist-aer', 'hist-nat', 'dcppA-assim',
                              'dcppA-hindcast', 'dcppC-hindcast-noAgung',
                              'dcppC-hindcast-noElChichon',
                              'dcppC-hindcast-noPinatubo', 'highresSST-present',
                              'control-1950', 'hist-1950', 'deforest-globe',
                              'esm-ssp585-ssp126Lu', 'omip1', 'lgm', 'ssp126',
                              'ssp245', 'ssp585']},
 'source_id': {'count': 48,
               'values': ['BCC-ESM1', 'BCC-CSM2-MR', 'CanESM5', 'CNRM-ESM2-1',
                          'UKESM1-0-LL', 'GISS-E2-1-G', 'CESM2', 'GFDL-ESM4',
                          'AWI-CM-1-1-MR', 'CAM[...]<jupyter_text>Searching for specific datasetsLet's find all the dissolved oxygen data at annual frequency from the ocean for the `historical` and `ssp585` experiments.<jupyter_code>cat = col.search(experiment_id=['historical', 'ssp585'], table_id='Oyr', variable_id='o2', grid_label='gn')
cat.df<jupyter_output><empty_output><jupyter_text>It might be desirable to get more specific. For instance, we may want to select only the models that have *both* `historical` and `ssp585` data. We coud do this as follows.<jupyter_code>models = set(uni_dict['source_id']['values']) # all the models

for experiment_id in ['historical', 'ssp585']:
    query = dict(experiment_id=experiment_id, table_id='Oyr', 
                 variable_id='o2', grid_label='gn')  
    cat = col.search(**query)
    models = models.intersection({model for model in cat.df.source_id.unique().tolist()})

# ensure the CESM2 models are not included (oxygen was erroneously submitted to the archive)
models = models - {'CESM2-WACCM', 'CESM2'}

models = list(models)
models
cat = col.search(experiment_id=['historical', 'ssp585'], table_id='Oyr', 
                 variable_id='o2', grid_label='gn', source_id=models)
cat.df<jupyter_output><empty_output><jupyter_text>Loading data`intake-esm` enables loading data directly into an [xarray.Dataset](http://xarray.pydata.org/en/stable/api.htmldataset).Note that data on the cloud are in [zarr](https://zarr.readthedocs.io/en/stable/) and data on [glade](https://www2.cisl.ucar.edu/resources/storage-and-file-systems/glade-file-spaces) are stored as [netCDF](https://www.unidata.ucar.edu/software/netcdf/) files. This is opaque to the user.`intake-esm` has rules for aggegating datasets; these rules are defined in the collection-specification file.<jupyter_code>dset_dict = cat.to_dataset_dict(zarr_kwargs={'consolidated': True, 'decode_times': False}, 
                                cdf_kwargs={'chunks': {}, 'decode_times': False})<jupyter_output>--> The keys in the returned dictionary of datasets are constructed as follows:
	'activity_id.institution_id.source_id.experiment_id.table_id.grid_label'

--> There will be 6 group(s)<jupyter_text>`dset_dict` is a dictionary of `xarray.Dataset`'s; its keys are constructed to refer to compatible groups.<jupyter_code>dset_dict.keys()<jupyter_output><empty_output><jupyter_text>We can access a particular dataset as follows.<jupyter_code>dset_dict['CMIP.CCCma.CanESM5.historical.Oyr.gn']<jupyter_output><empty_output><jupyter_text>More advanced queriesAs motivation for diving into more advanced manipulations with `intake-esm`, let's consider the task of getting access to grid information in the `Ofx` table_id.<jupyter_code>cat_fx = col.search(experiment_id=['historical', 'ssp585'], source_id=models, table_id='Ofx', grid_label='gn')
cat_fx.df<jupyter_output><empty_output><jupyter_text>This, however, comes with lots of redundant information.Additionally, it may be necessary to do more targeted manipulations of the search. For instance, we've found a handful of corrupted files on `glade` and might need to work around loading these. As an illustration of this, in the code below, we specify a list of to queries (in this case one) to eliminate.<jupyter_code>import numpy as np

# specify a list of queries to eliminate
corrupt_data = [dict(variable_id='areacello', source_id='IPSL-CM6A-LR',
                     experiment_id='historical', member_id='r2i1p1f1')
               ]


# copy the dataframe 
df = cat_fx.df.copy()

# eliminate data
for elim in corrupt_data:
    condition = np.ones(len(df), dtype=bool)
    for key, val in elim.items():
        condition = condition & (df[key] == val)
    df = df.loc[~condition]
df<jupyter_output><empty_output><jupyter_text>We then drop duplicates.<jupyter_code>df.drop_duplicates(subset=['source_id', 'variable_id'], inplace=True)<jupyter_output><empty_output><jupyter_text>Now, since we've only retained one ensemble member, we need to eliminate that column. If we omit this step, `intake-esm` will throw an error, complaining that different variables are present for each ensemble member. Setting the `member_id` column to NaN precludes attempts to join along the ensemble dimension.After this final manipulation, we copy the `DataFrame` back to the collection object and procede with loading the data.<jupyter_code>df['member_id'] = np.nan
cat_fx.df = df
fx_dsets = cat_fx.to_dataset_dict(zarr_kwargs={'consolidated': True}, cdf_kwargs={'chunks': {}})
fx_dsets.keys()
for key, ds in fx_dsets.items():
    print(ds.data_vars)<jupyter_output>Data variables:
    areacello           (j, i) float32 dask.array<chunksize=(291, 360), meta=np.ndarray>
    latitude            (j, i) float64 dask.array<chunksize=(291, 360), meta=np.ndarray>
    longitude           (j, i) float64 dask.array<chunksize=(291, 360), meta=np.ndarray>
    vertices_latitude   (j, i, vertices) float64 dask.array<chunksize=(291, 360, 4), meta=np.ndarray>
    vertices_longitude  (j, i, vertices) float64 dask.array<chunksize=(291, 360, 4), meta=np.ndarray>
    deptho              (j, i) float32 dask.array<chunksize=(291, 360), meta=np.ndarray>
    sftof               (j, i) float32 dask.array<chunksize=(291, 360), meta=np.ndarray>
    type                |S3 ...
    lev_bnds            (lev, bnds) float64 dask.array<chunksize=(45, 2), meta=np.ndarray>
    thkcello            (lev, j, i) float32 dask.array<chunksize=(45, 291, 360), meta=np.ndarray>
Data variables:
    area            (y, x) float32 dask.array<chunksize=(332, 362), meta=np.ndarray>
    areacello[...]<jupyter_text>Demonstrate how spin-up a dask clusterIf you expect to require Big Data capabilities, here's how you spin up a [dask](https://dask.org) cluster using [dask-jobqueue](https://dask-jobqueue.readthedocs.io/en/latest/).The syntax is different if on an NCAR machine versus the cloud.<jupyter_code>if util.is_ncar_host():
    from ncar_jobqueue import NCARCluster
    cluster = NCARCluster(project='UCGD0006')
    cluster.adapt(minimum_jobs=1, maximum_jobs=10)
else:
    from dask_kubernetes import KubeCluster
    cluster = KubeCluster()
    cluster.adapt(minimum=1, maximum=10)
cluster
from dask.distributed import Client
client = Client(cluster) # Connect this local process to remote workers
client<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11652563971862576, 'fraction_numerical': 0.01835049444387807, 'mean_word_length': 4.215311004784689, 'pattern_counts': {'":': 0, '<': 67, '<?xml version=': 0, '>': 69, 'https://': 10, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '4909', 'license': 'MIT', 'n_tokens_mistral': 3311, 'n_tokens_neox': 3124, 'n_words': 856, 'path': 'notebooks/hello_world.ipynb', 'repo_name': 'dhruvbalwada/cmip6hack-so-project', 'size': '9809'}