<jupyter_start><jupyter_text>Presenting and loading recipe output in a notebookThe goal of this notebook is to expand the functionality to run recipes directly from the notebook. Specifically, it demonstrates the API to retrieve the output, and display or load it in Python.See associated PR here: https://github.com/ESMValGroup/ESMValCore/pull/957<jupyter_code>%load_ext autoreload
%autoreload 2
from esmvalcore.experimental import CFG<jupyter_output>/home/stef/r/esmvalcore/esmvalcore/experimental/_warnings.py:18: UserWarning: 
  Thank you for trying out the new ESMValCore API.
  Note that this API is experimental and may be subject to change.
  More info: https://github.com/ESMValGroup/ESMValCore/issues/498
/home/stef/r/esmvalcore/esmvalcore/experimental/config/_config_validators.py:254: ESMValToolDeprecationWarning: `write_plots` will be removed in 2.2.0.
/home/stef/r/esmvalcore/esmvalcore/experimental/config/_config_validators.py:255: ESMValToolDeprecationWarning: `write_netcdf` will be removed in 2.2.0.
/home/stef/r/esmvalcore/esmvalcore/experimental/config/_config_validators.py:256: ESMValToolDeprecationWarning: `output_file_type` will be removed in 2.2.0.<jupyter_text>Getting a recipeRecipes can be obtained by their name, similar to what one may already be familiar with through `esmvaltool recipes list`. To do so, use `get_recipe`.<jupyter_code>from esmvalcore.experimental import get_recipe

recipe = get_recipe('examples/recipe_python.yml')
recipe<jupyter_output><empty_output><jupyter_text>Running a recipe and retrieving outputRun the recipe by using the `.run` method. The session can be defined as an arguement, but if left out, a new session is created automatically.Under the hood it calls `esmvalcore._recipe.Recipe` with the session config. The logger is redirected automatically to `main_log_debug.txt` / `main_log.txt` in the appropriate directory.<jupyter_code>recipe_output = recipe.run()<jupyter_output>{'auxiliary_data_dir': PosixPath('/home/stef/auxiliary_data'),
 'compress_netcdf': False,
 'config_developer_file': None,
 'config_file': PosixPath('/home/stef/.esmvaltool/config-user.yml'),
 'drs': {'CMIP5': 'default', 'CMIP6': 'default'},
 'exit_on_warning': False,
 'log_level': 'info',
 'max_parallel_tasks': None,
 'output_dir': PosixPath('/home/stef/esmvaltool_output'),
 'output_file_type': 'png',
 'plot_dir': PosixPath('/home/stef/esmvaltool_output/recipe_python_20210125_101426/plots'),
 'preproc_dir': PosixPath('/home/stef/esmvaltool_output/recipe_python_20210125_101426/preproc'),
 'profile_diagnostic': False,
 'remove_preproc_dir': True,
 'rootpath': {'CMIP5': [PosixPath('/home/stef/data')],
              'CMIP6': [PosixPath('/home/stef/data'),
                        PosixPath('/home/stef/r/esmvaltool_sample_data/esmvaltool_sample_data/data/timeseries/CMIP6')],
              'default': [PosixPath('/home/stef/data')]},
 'run_dir': PosixPath('/home/stef/esmvaltool_output/recipe_p[...]<jupyter_text>Recipe output can also be retrieved using the `.get_output` method.<jupyter_code>recipe_output = recipe.get_output()<jupyter_output><empty_output><jupyter_text>`recipe_output` is a mapping of the individual tasks and their output filenames (data and image files) with a set of attributes describing the data.<jupyter_code>recipe_output<jupyter_output><empty_output><jupyter_text>Output is sorted by the task that produced them.<jupyter_code>task_output = recipe_output['map/script1']

task_output<jupyter_output><empty_output><jupyter_text>The task output has a list of files associated with them, usually image (`.png`) or data files (`.nc`). To get a list of all files, use `.files`:<jupyter_code>print(task_output.files)<jupyter_output>(DataFile('CMIP6_BCC-ESM1_Amon_historical_r1i1p1f1_tas_2000-2000.nc'), ImageFile('CMIP5_CanESM2_Amon_historical_r1i1p1_tas_2000-2000.png'), DataFile('CMIP5_CanESM2_Amon_historical_r1i1p1_tas_2000-2000.nc'), ImageFile('CMIP6_BCC-ESM1_Amon_historical_r1i1p1f1_tas_2000-2000.png'))<jupyter_text>It is also possible to select the image files or data files only:<jupyter_code>for image_file in task_output.image_files:
    print(image_file)

for data_file in task_output.data_files:
    print(data_file)<jupyter_output>ImageFile('CMIP5_CanESM2_Amon_historical_r1i1p1_tas_2000-2000.png')
ImageFile('CMIP6_BCC-ESM1_Amon_historical_r1i1p1f1_tas_2000-2000.png')
DataFile('CMIP6_BCC-ESM1_Amon_historical_r1i1p1f1_tas_2000-2000.nc')
DataFile('CMIP5_CanESM2_Amon_historical_r1i1p1_tas_2000-2000.nc')<jupyter_text>Working with image filesOutput comes in two kinds, `DataFile`s, which correspond to data files in `.nc` format and `ImageFile`s which correspond to plots in `.png` format. Both object are derived from the same base class (`OutputFile`) and therfore share most of the functionality. Image output uses IPython magic to plot themselves in a notebook environment.<jupyter_code>from IPython.display import display

task = recipe_output['map/script1']

for image_file in task.image_files:
     display(image_file)<jupyter_output><empty_output><jupyter_text>The cell above specifically invokes the IPython magic using `display`, but this means you can do this:<jupyter_code>image_file = recipe_output['map/script1'].image_files[0]
image_file<jupyter_output><empty_output><jupyter_text>Working with data outputData items only print the path...<jupyter_code>data_files = recipe_output['timeseries/script1'].data_files

for data_file in task_output.data_files:
     print(data_file)<jupyter_output>DataFile('CMIP6_BCC-ESM1_Amon_historical_r1i1p1f1_tas_2000-2000.nc')
DataFile('CMIP5_CanESM2_Amon_historical_r1i1p1_tas_2000-2000.nc')<jupyter_text>But, they can be easily loaded using `xarray`:<jupyter_code>data_file = recipe_output['timeseries/script1'].data_files[0]

data = data_file.load_xarray()
type(data)<jupyter_output><empty_output><jupyter_text>Or `iris`:<jupyter_code>cube = data_file.load_iris()
type(cube)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10388954369422462, 'fraction_numerical': 0.03401246001010271, 'mean_word_length': 7.1931034482758625, 'pattern_counts': {'":': 0, '<': 48, '<?xml version=': 0, '>': 48, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '540172', 'license': 'Apache-2.0', 'n_tokens_mistral': 2223, 'n_tokens_neox': 2073, 'n_words': 431, 'path': 'Run recipe get output.ipynb', 'repo_name': 'stefsmeets/ESMValCore-Notebooks', 'size': '5939'}