<commit_before>{
 "metadata": {
  "name": "",
  "signature": "sha256:3621a254a7764cfbc723dc2a80fdbadbe2a39d4ecffd90bc47c4211aba8f3212"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Top: [Tutorials](Tutorials-0_Index.ipynb) | Next:"
     ]
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Handling CTM datasets"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "This tutorial shows how to load and save datasets, and also how it can be manipulated.\n",
      "\n",
      "The `pygchem.datafields` module provides functions to load and save CTM datasets: "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from pygchem import datafields"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 1
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "The Iris backend"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "PyGChem uses Iris (http://scitools.org.uk/iris/) for handling CTM datasets. Iris provides a data abstraction layer that is based on the Climate and Forecast (CF) data model (http://cfconventions.org/) and which allows to isolate analysis and visualisation code from data format specifics. It natively supports read/write access to the (CF)netCDF format. PyGChem currently adds support to the BPCH format and the netCDF files created with the GAMAP routines BPCH2NC and BPCH2COARDS (read access only).\n",
      "\n",
      "With Iris, a data field (data + metadata) relative to a particular phenomenon is represented by a `iris.cube.Cube` object (more info on Iris data strutures [here](http://scitools.org.uk/iris/docs/latest/userguide/iris_cubes.html)). Iris also supports basic data manipulation operations, such as arithmetic, interpolation, and statistics."
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Datasets used in this tutorial"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "This tutorial uses the datasets created by the 1-year GEOS-Chem benchmark simulations. These are available for download via anonymous FTP from <ftp://ftp.as.harvard.edu/gcgrid/geos-chem/>.\n",
      "\n",
      "For running the code cells below, you must specify the root folder of your local copy of these benchmarks: "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "bmk_root = '/home/benbovy/Aldebaran/ScientificResearch/geoschem'"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 2
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Loading datasets"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The function `pygchem.datafields.load` allows to load the content of a BPCH file or a netCDF file into a list of `iris.cube.Cube` objects. This function is just an alias of the `iris.load` function. Note that it doesn't load all the data into memory but enough information to select, manipulate and retreive the data.\n",
      "\n",
      "See the Iris documentation: [loading iris cubes](http://scitools.org.uk/iris/docs/latest/userguide/loading_iris_cubes.html) for more details."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "cd into the 1-year benchmark (v10-01c Run1)"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%cd {bmk_root}/1yr_benchmarks/v10-01/v10-01c/Run1"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "/mnt/hgfs/Aldebaran/ScientificResearch/geoschem/1yr_benchmarks/v10-01/v10-01c/Run1\n"
       ]
      }
     ],
     "prompt_number": 3
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Simple (unconstrained) loading"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The line below loads the BPCH file corresponding to the first month of the simulation:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "filename = 'bpch/ctm.bpch.v10-01c-geosfp-Run1.20120801'\n",
      "dataset = datafields.load(filename)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 4
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The line below print the list of the 20 lasts data fields of the list (name, units, dimensions and coordinates):"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print dataset[-20:]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0: DAO_FLDS__Z0 / (m)                  (longitude: 72; latitude: 46)\n",
        "1: DAO_FLDS__PBL / (m)                 (longitude: 72; latitude: 46)\n",
        "2: DAO_FLDS__CLDFRC / (1)              (longitude: 72; latitude: 46)\n",
        "3: DAO_FLDS__U10M / (m/s)              (longitude: 72; latitude: 46)\n",
        "4: DAO_FLDS__V10M / (m/s)              (longitude: 72; latitude: 46)\n",
        "5: DAO_FLDS__PS_PBL / (hPa)            (longitude: 72; latitude: 46)\n",
        "6: DAO_FLDS__ALBD / (1)                (longitude: 72; latitude: 46)\n",
        "7: DAO_FLDS__PHIS / (m)                (longitude: 72; latitude: 46)\n",
        "8: DAO_FLDS__CLDTOP / (1s)             (longitude: 72; latitude: 46)\n",
        "9: DAO_FLDS__TROPP / (hPa)             (longitude: 72; latitude: 46)\n",
        "10: DAO_FLDS__SLP / (hPa)               (longitude: 72; latitude: 46)\n",
        "11: DAO_FLDS__TSKIN / (K)               (longitude: 72; latitude: 46)\n",
        "12: DAO_FLDS__PARDF / (W/m2)            (longitude: 72; latitude: 46)\n",
        "13: DAO_FLDS__PARDR / (W/m2)            (longitude: 72; latitude: 46)\n",
        "14: DAO_FLDS__GWET / (1)                (longitude: 72; latitude: 46)\n",
        "15: DAO_FLDS__EFLUX / (W/m2)            (longitude: 72; latitude: 46)\n",
        "16: BXHGHT_S__BXHEIGHT / (m)            (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "17: BXHGHT_S__AD / (kg)                 (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "18: BXHGHT_S__AVGW / (1)                (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "19: BXHGHT_S__N(AIR) / (molec/m3)       (longitude: 72; latitude: 46; model_level_number: 72)\n"
       ]
      }
     ],
     "prompt_number": 5
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The same function can be used for loading the content of the netCDF files. The netCDF files available in this simulation were created by the 'BPCH2COARDS' GAMAP routine. As there are differences between the COARDS Conventions and the CF Conventions, we have to use a callback function 'gamap_bpch2coards' to properly load the content of the netCDF file into Iris cubes: "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "filename = 'netcdf/v10-01c-geosfp-Run1.20120801.nc'\n",
      "clb = datafields.load_callbacks['gamap_bpch2coards']\n",
      "dataset = datafields.load(filename, callback=clb)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 22
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print dataset[-20:]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0: OD_MAP_S__SBC / (cm2/cm3)           (-- : 59; latitude: 46; longitude: 72)\n",
        "1: OD_MAP_S__SD / (cm2/cm3)            (-- : 59; latitude: 46; longitude: 72)\n",
        "2: OD_MAP_S__SOC / (cm2/cm3)           (-- : 59; latitude: 46; longitude: 72)\n",
        "3: OD_MAP_S__SSO4 / (cm2/cm3)          (-- : 59; latitude: 46; longitude: 72)\n",
        "4: OD_MAP_S__SSSa / (cm2/cm3)          (-- : 59; latitude: 46; longitude: 72)\n",
        "5: OD_MAP_S__SSSc / (cm2/cm3)          (-- : 59; latitude: 46; longitude: 72)\n",
        "6: PEDGE_S__PSURF / (mb)               (Eta Edges: 73; latitude: 46; longitude: 72)\n",
        "7: PL_BC_S__BLKC / (kg)                (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "8: PL_OC_S__ORGC / (kg)                (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "9: SALTSRCE__SALA / (kg)               (latitude: 46; longitude: 72)\n",
        "10: SALTSRCE__SALC / (kg)               (latitude: 46; longitude: 72)\n",
        "11: SO2_AC_S__SO2 / (kg S)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "12: SO2_AN_S__SO2 / (kg S)              (-- : 2; latitude: 46; longitude: 72)\n",
        "13: SO2_BIOB__SO2 / (kg S)              (latitude: 46; longitude: 72)\n",
        "14: SO2_BIOF__SO2 / (kg S)              (latitude: 46; longitude: 72)\n",
        "15: SO2_EV_S__SO2 / (kg S)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "16: SO2_NV_S__SO2 / (kg S)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "17: SO2_SHIP__SO2 / (kg S)              (latitude: 46; longitude: 72)\n",
        "18: SO4_AN_S__SO4 / (kg S)              (-- : 2; latitude: 46; longitude: 72)\n",
        "19: SO4_BIOF__SO4 / (kg S)              (latitude: 46; longitude: 72)\n"
       ]
      }
     ],
     "prompt_number": 23
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "PyGChem also provides a callback for loading netCDF files created by the GAMAP routine 'BPCH2NC':"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "% cd {bmk_root}/1yr_benchmarks/v9-02/v9-02r/geos5/Run0"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "/mnt/hgfs/Aldebaran/ScientificResearch/geoschem/1yr_benchmarks/v9-02/v9-02r/geos5/Run0\n"
       ]
      }
     ],
     "prompt_number": 8
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "filename = 'netcdf/v9-02r-geos5-Run0.20050101.nc'\n",
      "clb = datafields.load_callbacks['gamap_bpch2nc']\n",
      "dataset = datafields.load(filename, callback=clb)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 9
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print dataset[-20:]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0: OD-MAP-S__OPSSc / (1)               (-- : 38; latitude: 46; longitude: 72)\n",
        "1: OD-MAP-S__OPTD / (1)                (-- : 38; latitude: 46; longitude: 72)\n",
        "2: OD-MAP-S__SBC / (cm2/cm3)           (-- : 38; latitude: 46; longitude: 72)\n",
        "3: OD-MAP-S__SD / (cm2/cm3)            (-- : 38; latitude: 46; longitude: 72)\n",
        "4: OD-MAP-S__SOC / (cm2/cm3)           (-- : 38; latitude: 46; longitude: 72)\n",
        "5: OD-MAP-S__SSO4 / (cm2/cm3)          (-- : 38; latitude: 46; longitude: 72)\n",
        "6: OD-MAP-S__SSSa / (cm2/cm3)          (-- : 38; latitude: 46; longitude: 72)\n",
        "7: OD-MAP-S__SSSc / (cm2/cm3)          (-- : 38; latitude: 46; longitude: 72)\n",
        "8: PEDGE-S__PSURF / (mb)               (-- : 48; latitude: 46; longitude: 72)\n",
        "9: SALTSRCE__SALA / (kg)               (latitude: 46; longitude: 72)\n",
        "10: SALTSRCE__SALC / (kg)               (latitude: 46; longitude: 72)\n",
        "11: SO2-AC-S__SO2 / (kg S)              (model_level_number: 47; latitude: 46; longitude: 72)\n",
        "12: SO2-AN-S__SO2 / (kg S)              (-- : 2; latitude: 46; longitude: 72)\n",
        "13: SO2-BIOB__SO2 / (kg S)              (latitude: 46; longitude: 72)\n",
        "14: SO2-BIOF__SO2 / (kg S)              (latitude: 46; longitude: 72)\n",
        "15: SO2-EV-S__SO2 / (kg S)              (model_level_number: 47; latitude: 46; longitude: 72)\n",
        "16: SO2-NV-S__SO2 / (kg S)              (model_level_number: 47; latitude: 46; longitude: 72)\n",
        "17: SO2-SHIP__SO2 / (kg S)              (latitude: 46; longitude: 72)\n",
        "18: SO4-AN-S__SO4 / (kg S)              (-- : 2; latitude: 46; longitude: 72)\n",
        "19: SO4-BIOF__SO4 / (kg S)              (latitude: 46; longitude: 72)\n"
       ]
      }
     ],
     "prompt_number": 10
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%cd {bmk_root}/1yr_benchmarks/v10-01/v10-01c/Run1"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "/mnt/hgfs/Aldebaran/ScientificResearch/geoschem/1yr_benchmarks/v10-01/v10-01c/Run1\n"
       ]
      }
     ],
     "prompt_number": 11
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Constrained loading"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "For large datasets, it is possible to constrain the load to match specific metadata or to load only a subset of the data. It is recommended as the loading is faster with given constraints."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The following example loads only the \"IJ_AVG_S__O3\" variable: "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "filename = 'netcdf/v10-01c-geosfp-Run1.20120801.nc'\n",
      "clb = datafields.load_callbacks['gamap_bpch2coards']\n",
      "dataset = datafields.load(filename, \"IJ_AVG_S__O3\",\n",
      "                          callback=clb)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 12
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print dataset"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0: IJ_AVG_S__O3 / (ppbv)               (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n"
       ]
      }
     ],
     "prompt_number": 13
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "We can specify multiple variables:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "dataset = datafields.load(filename, [\"IJ_AVG_S__O3\", \"IJ_AVG_S__NO\"],\n",
      "                          callback=clb)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 14
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print dataset"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0: IJ_AVG_S__O3 / (ppbv)               (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "1: IJ_AVG_S__NO / (ppbv)               (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n"
       ]
      }
     ],
     "prompt_number": 15
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "It is also possible to define more advanced constraints. For example, to load all \"IJ-AVG-$\" diagnostics:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "check_ij_avg = lambda cube: cube.name().startswith(\"IJ_AVG_S\")\n",
      "ij_avg = datafields.Constraint(cube_func=check_ij_avg)\n",
      "\n",
      "dataset = datafields.load(filename, ij_avg,\n",
      "                          callback=clb)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 16
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print dataset"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0: IJ_AVG_S__ACET / (ppb)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "1: IJ_AVG_S__ALD2 / (ppb)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "2: IJ_AVG_S__ALK4 / (ppb)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "3: IJ_AVG_S__BCPI / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "4: IJ_AVG_S__BCPO / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "5: IJ_AVG_S__Br / (ppbv)               (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "6: IJ_AVG_S__Br2 / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "7: IJ_AVG_S__BrCl / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "8: IJ_AVG_S__BrNO2 / (ppbv)            (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "9: IJ_AVG_S__BrNO3 / (ppbv)            (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "10: IJ_AVG_S__BrO / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "11: IJ_AVG_S__C2H6 / (ppb)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "12: IJ_AVG_S__C3H8 / (ppb)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "13: IJ_AVG_S__CCl4 / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "14: IJ_AVG_S__CFC11 / (ppbv)            (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "15: IJ_AVG_S__CFC12 / (ppbv)            (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "16: IJ_AVG_S__CFCX / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "17: IJ_AVG_S__CH2Br2 / (ppbv)           (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "18: IJ_AVG_S__CH2O / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "19: IJ_AVG_S__CH3Br / (ppbv)            (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "20: IJ_AVG_S__CH3CCl3 / (ppbv)          (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "21: IJ_AVG_S__CH3Cl / (ppbv)            (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "22: IJ_AVG_S__CH4 / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "23: IJ_AVG_S__CHBr3 / (ppbv)            (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "24: IJ_AVG_S__CO / (ppbv)               (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "25: IJ_AVG_S__Cl / (ppbv)               (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "26: IJ_AVG_S__Cl2 / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "27: IJ_AVG_S__Cl2O2 / (ppbv)            (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "28: IJ_AVG_S__ClNO2 / (ppbv)            (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "29: IJ_AVG_S__ClNO3 / (ppbv)            (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "30: IJ_AVG_S__ClO / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "31: IJ_AVG_S__ClOO / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "32: IJ_AVG_S__DMS / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "33: IJ_AVG_S__DST1 / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "34: IJ_AVG_S__DST2 / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "35: IJ_AVG_S__DST3 / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "36: IJ_AVG_S__DST4 / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "37: IJ_AVG_S__GLYC / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "38: IJ_AVG_S__H1211 / (ppbv)            (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "39: IJ_AVG_S__H1301 / (ppbv)            (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "40: IJ_AVG_S__H2402 / (ppbv)            (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "41: IJ_AVG_S__H2O / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "42: IJ_AVG_S__H2O2 / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "43: IJ_AVG_S__HAC / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "44: IJ_AVG_S__HBr / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "45: IJ_AVG_S__HCFC22 / (ppbv)           (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "46: IJ_AVG_S__HCFCX / (ppbv)            (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "47: IJ_AVG_S__HCl / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "48: IJ_AVG_S__HNO2 / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "49: IJ_AVG_S__HNO3 / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "50: IJ_AVG_S__HNO4 / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "51: IJ_AVG_S__HOBr / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "52: IJ_AVG_S__HOCl / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "53: IJ_AVG_S__IEPOX / (ppbv)            (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "54: IJ_AVG_S__ISOP / (ppb)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "55: IJ_AVG_S__ISOPN / (ppbv)            (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "56: IJ_AVG_S__MACR / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "57: IJ_AVG_S__MAP / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "58: IJ_AVG_S__MEK / (ppb)               (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "59: IJ_AVG_S__MMN / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "60: IJ_AVG_S__MOBA / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "61: IJ_AVG_S__MP / (ppbv)               (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "62: IJ_AVG_S__MPN / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "63: IJ_AVG_S__MSA / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "64: IJ_AVG_S__MVK / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "65: IJ_AVG_S__N2O / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "66: IJ_AVG_S__N2O5 / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "67: IJ_AVG_S__NH3 / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "68: IJ_AVG_S__NH4 / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "69: IJ_AVG_S__NIT / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "70: IJ_AVG_S__NITs / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "71: IJ_AVG_S__NO / (ppbv)               (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "72: IJ_AVG_S__NO2 / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "73: IJ_AVG_S__NO3 / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "74: IJ_AVG_S__O3 / (ppbv)               (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "75: IJ_AVG_S__OCPI / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "76: IJ_AVG_S__OCPO / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "77: IJ_AVG_S__OCS / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "78: IJ_AVG_S__OClO / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "79: IJ_AVG_S__PAN / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "80: IJ_AVG_S__PMN / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "81: IJ_AVG_S__PPN / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "82: IJ_AVG_S__PROPNN / (ppbv)           (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "83: IJ_AVG_S__PRPE / (ppb)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "84: IJ_AVG_S__R4N2 / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "85: IJ_AVG_S__RCHO / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "86: IJ_AVG_S__RIP / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "87: IJ_AVG_S__SALA / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "88: IJ_AVG_S__SALC / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "89: IJ_AVG_S__SO2 / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "90: IJ_AVG_S__SO4 / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "91: IJ_AVG_S__SO4s / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n"
       ]
      }
     ],
     "prompt_number": 17
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "A more advanced example, combining constraints and extracting data subsets:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def lon_subset(cell):\n",
      "   \"\"\"\n",
      "   return True or False as to whether the cell\n",
      "   center in question should be kept\n",
      "   \"\"\"\n",
      "   return cell > 0. and cell < 20.\n",
      "\n",
      "lon_cst = datafields.Constraint(longitude=lon_subset)\n",
      "\n",
      "dataset = datafields.load(filename,\n",
      "                          \"IJ_AVG_S__O3\" & lon_cst,\n",
      "                          callback=clb)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 18
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print dataset\n",
      "# note the reduced grid-size for the longitude"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0: IJ_AVG_S__O3 / (ppbv)               (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 3)\n"
       ]
      }
     ],
     "prompt_number": 19
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Loading multiple files"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "It is possible to load multiple files at once. The `pygchem.datafields.load` function will try to merge the fields when it is possible, i.e., when the multiple fields describing the same phenomenom overlap in space and/or time, so that it reduces at the minimum the number of loaded fields."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The following example load the \"IJ_AVG_S__O3\" diagnostic for the entire 1-year simulation:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# note the wildcard character in the filename\n",
      "# (UNIX expressions are supported)\n",
      "filename = 'bpch/ctm.bpch.v10-01c-geosfp-Run1.*'\n",
      "diagnostics = [\"BXHGHT_S__BXHEIGHT\",\n",
      "               \"BXHGHT_S__N(AIR)\",\n",
      "               \"IJ_AVG_S__O3\"]\n",
      "dataset = datafields.load(filename, diagnostics)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 24
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print dataset\n",
      "# note the additional time dimension"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0: BXHGHT_S__BXHEIGHT / (m)            (time: 12; longitude: 72; latitude: 46; model_level_number: 72)\n",
        "1: BXHGHT_S__N(AIR) / (molec/m3)       (time: 12; longitude: 72; latitude: 46; model_level_number: 72)\n",
        "2: IJ_AVG_S__O3 / (ppbv)               (time: 12; longitude: 72; latitude: 46; model_level_number: 72)\n"
       ]
      }
     ],
     "prompt_number": 25
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Merging fields may take a long time. If speed matters, it is still possible to load the fields without any merging: "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "dataset_nomerge = datafields.load_raw(filename, diagnostics)\n",
      "print dataset_nomerge"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0: BXHGHT_S__BXHEIGHT / (m)            (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "1: BXHGHT_S__BXHEIGHT / (m)            (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "2: BXHGHT_S__BXHEIGHT / (m)            (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "3: BXHGHT_S__BXHEIGHT / (m)            (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "4: BXHGHT_S__BXHEIGHT / (m)            (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "5: BXHGHT_S__BXHEIGHT / (m)            (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "6: BXHGHT_S__BXHEIGHT / (m)            (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "7: BXHGHT_S__BXHEIGHT / (m)            (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "8: BXHGHT_S__BXHEIGHT / (m)            (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "9: BXHGHT_S__BXHEIGHT / (m)            (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "10: BXHGHT_S__BXHEIGHT / (m)            (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "11: BXHGHT_S__BXHEIGHT / (m)            (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "12: BXHGHT_S__N(AIR) / (molec/m3)       (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "13: BXHGHT_S__N(AIR) / (molec/m3)       (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "14: BXHGHT_S__N(AIR) / (molec/m3)       (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "15: BXHGHT_S__N(AIR) / (molec/m3)       (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "16: BXHGHT_S__N(AIR) / (molec/m3)       (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "17: BXHGHT_S__N(AIR) / (molec/m3)       (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "18: BXHGHT_S__N(AIR) / (molec/m3)       (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "19: BXHGHT_S__N(AIR) / (molec/m3)       (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "20: BXHGHT_S__N(AIR) / (molec/m3)       (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "21: BXHGHT_S__N(AIR) / (molec/m3)       (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "22: BXHGHT_S__N(AIR) / (molec/m3)       (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "23: BXHGHT_S__N(AIR) / (molec/m3)       (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "24: IJ_AVG_S__O3 / (ppbv)               (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "25: IJ_AVG_S__O3 / (ppbv)               (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "26: IJ_AVG_S__O3 / (ppbv)               (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "27: IJ_AVG_S__O3 / (ppbv)               (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "28: IJ_AVG_S__O3 / (ppbv)               (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "29: IJ_AVG_S__O3 / (ppbv)               (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "30: IJ_AVG_S__O3 / (ppbv)               (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "31: IJ_AVG_S__O3 / (ppbv)               (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "32: IJ_AVG_S__O3 / (ppbv)               (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "33: IJ_AVG_S__O3 / (ppbv)               (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "34: IJ_AVG_S__O3 / (ppbv)               (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "35: IJ_AVG_S__O3 / (ppbv)               (longitude: 72; latitude: 46; model_level_number: 72)\n"
       ]
      }
     ],
     "prompt_number": 29
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Apply contraints to the field list after loading"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "We can extract data subsets and specific fields from a loaded list of fields:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "dataset_lon_subset = dataset.extract(lon_cst)\n",
      "print dataset_lon_subset"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0: BXHGHT_S__BXHEIGHT / (m)            (time: 12; longitude: 3; latitude: 46; model_level_number: 72)\n",
        "1: BXHGHT_S__N(AIR) / (molec/m3)       (time: 12; longitude: 3; latitude: 46; model_level_number: 72)\n",
        "2: IJ_AVG_S__O3 / (ppbv)               (time: 12; longitude: 3; latitude: 46; model_level_number: 72)\n"
       ]
      }
     ],
     "prompt_number": 31
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "To select only one field, the `extract_strict` method can be used: "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "ox_avg = dataset.extract_strict(\"IJ_AVG_S__O3\")\n",
      "print ox_avg"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "IJ_AVG_S__O3 / (ppbv)               (time: 12; longitude: 72; latitude: 46; model_level_number: 72)\n",
        "     Dimension coordinates:\n",
        "          time                           x              -             -                       -\n",
        "          longitude                      -              x             -                       -\n",
        "          latitude                       -              -             x                       -\n",
        "          model_level_number             -              -             -                       x\n",
        "     Attributes:\n",
        "          carbon_weight: 1\n",
        "          category: IJ-AVG-$\n",
        "          chemical: True\n",
        "          ctm_units: ppbv\n",
        "          full_name: O3 tracer\n",
        "          hydrocarbon: False\n",
        "          modelname: GEOSFP\n",
        "          molecular_weight: 0.048\n",
        "          name: O3\n",
        "          number: 2\n",
        "          resolution: (5.0, 4.0)\n",
        "          scale: 1000000000.0\n"
       ]
      }
     ],
     "prompt_number": 32
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}<commit_msg>updated the tutorials<commit_after>{
 "metadata": {
  "name": "",
  "signature": "sha256:4c3c7bc2b83e65008301a761d893286d7864f3a7240200108d928af05acb8124"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Top: [Tutorials](Tutorials-0_Index.ipynb) | Next:"
     ]
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Handling CTM datasets"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "This tutorial shows how to load and save datasets, and also how it can be manipulated.\n",
      "\n",
      "The `pygchem.datafields` module provides functions to load and save CTM datasets: "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from pygchem import datafields"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 1
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "The Iris backend"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "PyGChem uses Iris (http://scitools.org.uk/iris/) for handling CTM datasets. Iris provides a data abstraction layer that is based on the Climate and Forecast (CF) data model (http://cfconventions.org/) and which allows to isolate analysis and visualisation code from data format specifics. It natively supports read/write access to the (CF)netCDF format. PyGChem currently adds support to the BPCH format and the netCDF files created with the GAMAP routines BPCH2NC and BPCH2COARDS (read access only).\n",
      "\n",
      "With Iris, a data field (data + metadata) relative to a particular phenomenon is represented by a `iris.cube.Cube` object (more info on Iris data strutures [here](http://scitools.org.uk/iris/docs/latest/userguide/iris_cubes.html)). Iris also supports basic data manipulation operations, such as arithmetic, interpolation, and statistics."
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Datasets used in this tutorial"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "This tutorial uses the datasets created by the 1-year GEOS-Chem benchmark simulations. These are available for download via anonymous FTP from <ftp://ftp.as.harvard.edu/gcgrid/geos-chem/>.\n",
      "\n",
      "For running the code cells below, you must specify the root folder of your local copy of these benchmarks: "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "bmk_root = '/home/benbovy/Aldebaran/ScientificResearch/geoschem'"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 2
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Loading datasets"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The function `pygchem.datafields.load` allows to load the content of a BPCH file or a netCDF file into a list of `iris.cube.Cube` objects. This function is just an alias of the `iris.load` function. Note that it doesn't load all the data into memory but enough information to select, manipulate and retreive the data.\n",
      "\n",
      "See the Iris documentation: [loading iris cubes](http://scitools.org.uk/iris/docs/latest/userguide/loading_iris_cubes.html) for more details."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "cd into the 1-year benchmark (v10-01c Run1)"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%cd {bmk_root}/1yr_benchmarks/v10-01/v10-01c/Run1"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "/mnt/hgfs/Aldebaran/ScientificResearch/geoschem/1yr_benchmarks/v10-01/v10-01c/Run1\n"
       ]
      }
     ],
     "prompt_number": 3
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Simple (unconstrained) loading"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The line below loads the BPCH file corresponding to the first month of the simulation:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "filename = 'bpch/ctm.bpch.v10-01c-geosfp-Run1.20120801'\n",
      "dataset = datafields.load(filename)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 4
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The line below print the list of the 20 lasts data fields of the list (name, units, dimensions and coordinates):"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print dataset[-20:]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0: DAO_FLDS__Z0 / (m)                  (longitude: 72; latitude: 46)\n",
        "1: DAO_FLDS__PBL / (m)                 (longitude: 72; latitude: 46)\n",
        "2: DAO_FLDS__CLDFRC / (1)              (longitude: 72; latitude: 46)\n",
        "3: DAO_FLDS__U10M / (m/s)              (longitude: 72; latitude: 46)\n",
        "4: DAO_FLDS__V10M / (m/s)              (longitude: 72; latitude: 46)\n",
        "5: DAO_FLDS__PS_PBL / (hPa)            (longitude: 72; latitude: 46)\n",
        "6: DAO_FLDS__ALBD / (1)                (longitude: 72; latitude: 46)\n",
        "7: DAO_FLDS__PHIS / (m)                (longitude: 72; latitude: 46)\n",
        "8: DAO_FLDS__CLDTOP / (1s)             (longitude: 72; latitude: 46)\n",
        "9: DAO_FLDS__TROPP / (hPa)             (longitude: 72; latitude: 46)\n",
        "10: DAO_FLDS__SLP / (hPa)               (longitude: 72; latitude: 46)\n",
        "11: DAO_FLDS__TSKIN / (K)               (longitude: 72; latitude: 46)\n",
        "12: DAO_FLDS__PARDF / (W/m2)            (longitude: 72; latitude: 46)\n",
        "13: DAO_FLDS__PARDR / (W/m2)            (longitude: 72; latitude: 46)\n",
        "14: DAO_FLDS__GWET / (1)                (longitude: 72; latitude: 46)\n",
        "15: DAO_FLDS__EFLUX / (W/m2)            (longitude: 72; latitude: 46)\n",
        "16: BXHGHT_S__BXHEIGHT / (m)            (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "17: BXHGHT_S__AD / (kg)                 (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "18: BXHGHT_S__AVGW / (1)                (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "19: BXHGHT_S__N(AIR) / (molec/m3)       (longitude: 72; latitude: 46; model_level_number: 72)\n"
       ]
      }
     ],
     "prompt_number": 5
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The same function can be used for loading the content of the netCDF files. The netCDF files available in this simulation were created by the 'BPCH2COARDS' GAMAP routine. As there are differences between the COARDS Conventions and the CF Conventions, we have to use a callback function 'gamap_bpch2coards' to properly load the content of the netCDF file into Iris cubes: "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "filename = 'netcdf/v10-01c-geosfp-Run1.20120801.nc'\n",
      "clb = datafields.load_callbacks['gamap_bpch2coards']\n",
      "dataset = datafields.load(filename, callback=clb)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 22
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print dataset[-20:]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0: OD_MAP_S__SBC / (cm2/cm3)           (-- : 59; latitude: 46; longitude: 72)\n",
        "1: OD_MAP_S__SD / (cm2/cm3)            (-- : 59; latitude: 46; longitude: 72)\n",
        "2: OD_MAP_S__SOC / (cm2/cm3)           (-- : 59; latitude: 46; longitude: 72)\n",
        "3: OD_MAP_S__SSO4 / (cm2/cm3)          (-- : 59; latitude: 46; longitude: 72)\n",
        "4: OD_MAP_S__SSSa / (cm2/cm3)          (-- : 59; latitude: 46; longitude: 72)\n",
        "5: OD_MAP_S__SSSc / (cm2/cm3)          (-- : 59; latitude: 46; longitude: 72)\n",
        "6: PEDGE_S__PSURF / (mb)               (Eta Edges: 73; latitude: 46; longitude: 72)\n",
        "7: PL_BC_S__BLKC / (kg)                (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "8: PL_OC_S__ORGC / (kg)                (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "9: SALTSRCE__SALA / (kg)               (latitude: 46; longitude: 72)\n",
        "10: SALTSRCE__SALC / (kg)               (latitude: 46; longitude: 72)\n",
        "11: SO2_AC_S__SO2 / (kg S)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "12: SO2_AN_S__SO2 / (kg S)              (-- : 2; latitude: 46; longitude: 72)\n",
        "13: SO2_BIOB__SO2 / (kg S)              (latitude: 46; longitude: 72)\n",
        "14: SO2_BIOF__SO2 / (kg S)              (latitude: 46; longitude: 72)\n",
        "15: SO2_EV_S__SO2 / (kg S)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "16: SO2_NV_S__SO2 / (kg S)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "17: SO2_SHIP__SO2 / (kg S)              (latitude: 46; longitude: 72)\n",
        "18: SO4_AN_S__SO4 / (kg S)              (-- : 2; latitude: 46; longitude: 72)\n",
        "19: SO4_BIOF__SO4 / (kg S)              (latitude: 46; longitude: 72)\n"
       ]
      }
     ],
     "prompt_number": 23
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "PyGChem also provides a callback for loading netCDF files created by the GAMAP routine 'BPCH2NC':"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "% cd {bmk_root}/1yr_benchmarks/v9-02/v9-02r/geos5/Run0"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "/mnt/hgfs/Aldebaran/ScientificResearch/geoschem/1yr_benchmarks/v9-02/v9-02r/geos5/Run0\n"
       ]
      }
     ],
     "prompt_number": 8
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "filename = 'netcdf/v9-02r-geos5-Run0.20050101.nc'\n",
      "clb = datafields.load_callbacks['gamap_bpch2nc']\n",
      "dataset = datafields.load(filename, callback=clb)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 9
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print dataset[-20:]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0: OD-MAP-S__OPSSc / (1)               (-- : 38; latitude: 46; longitude: 72)\n",
        "1: OD-MAP-S__OPTD / (1)                (-- : 38; latitude: 46; longitude: 72)\n",
        "2: OD-MAP-S__SBC / (cm2/cm3)           (-- : 38; latitude: 46; longitude: 72)\n",
        "3: OD-MAP-S__SD / (cm2/cm3)            (-- : 38; latitude: 46; longitude: 72)\n",
        "4: OD-MAP-S__SOC / (cm2/cm3)           (-- : 38; latitude: 46; longitude: 72)\n",
        "5: OD-MAP-S__SSO4 / (cm2/cm3)          (-- : 38; latitude: 46; longitude: 72)\n",
        "6: OD-MAP-S__SSSa / (cm2/cm3)          (-- : 38; latitude: 46; longitude: 72)\n",
        "7: OD-MAP-S__SSSc / (cm2/cm3)          (-- : 38; latitude: 46; longitude: 72)\n",
        "8: PEDGE-S__PSURF / (mb)               (-- : 48; latitude: 46; longitude: 72)\n",
        "9: SALTSRCE__SALA / (kg)               (latitude: 46; longitude: 72)\n",
        "10: SALTSRCE__SALC / (kg)               (latitude: 46; longitude: 72)\n",
        "11: SO2-AC-S__SO2 / (kg S)              (model_level_number: 47; latitude: 46; longitude: 72)\n",
        "12: SO2-AN-S__SO2 / (kg S)              (-- : 2; latitude: 46; longitude: 72)\n",
        "13: SO2-BIOB__SO2 / (kg S)              (latitude: 46; longitude: 72)\n",
        "14: SO2-BIOF__SO2 / (kg S)              (latitude: 46; longitude: 72)\n",
        "15: SO2-EV-S__SO2 / (kg S)              (model_level_number: 47; latitude: 46; longitude: 72)\n",
        "16: SO2-NV-S__SO2 / (kg S)              (model_level_number: 47; latitude: 46; longitude: 72)\n",
        "17: SO2-SHIP__SO2 / (kg S)              (latitude: 46; longitude: 72)\n",
        "18: SO4-AN-S__SO4 / (kg S)              (-- : 2; latitude: 46; longitude: 72)\n",
        "19: SO4-BIOF__SO4 / (kg S)              (latitude: 46; longitude: 72)\n"
       ]
      }
     ],
     "prompt_number": 10
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%cd {bmk_root}/1yr_benchmarks/v10-01/v10-01c/Run1"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "/mnt/hgfs/Aldebaran/ScientificResearch/geoschem/1yr_benchmarks/v10-01/v10-01c/Run1\n"
       ]
      }
     ],
     "prompt_number": 11
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Constrained loading"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "For large datasets, it is possible to constrain the load to match specific metadata or to load only a subset of the data. It is recommended as the loading is faster with given constraints."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The following example loads only the \"IJ_AVG_S__O3\" variable: "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "filename = 'netcdf/v10-01c-geosfp-Run1.20120801.nc'\n",
      "clb = datafields.load_callbacks['gamap_bpch2coards']\n",
      "dataset = datafields.load(filename, \"IJ_AVG_S__O3\",\n",
      "                          callback=clb)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 12
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print dataset"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0: IJ_AVG_S__O3 / (ppbv)               (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n"
       ]
      }
     ],
     "prompt_number": 13
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "We can specify multiple variables:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "dataset = datafields.load(filename, [\"IJ_AVG_S__O3\", \"IJ_AVG_S__NO\"],\n",
      "                          callback=clb)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 14
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print dataset"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0: IJ_AVG_S__O3 / (ppbv)               (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "1: IJ_AVG_S__NO / (ppbv)               (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n"
       ]
      }
     ],
     "prompt_number": 15
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "It is also possible to define more advanced constraints. For example, to load all \"IJ-AVG-$\" diagnostics:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "check_ij_avg = lambda cube: cube.name().startswith(\"IJ_AVG_S\")\n",
      "ij_avg = datafields.Constraint(cube_func=check_ij_avg)\n",
      "\n",
      "dataset = datafields.load(filename, ij_avg,\n",
      "                          callback=clb)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 16
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print dataset"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0: IJ_AVG_S__ACET / (ppb)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "1: IJ_AVG_S__ALD2 / (ppb)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "2: IJ_AVG_S__ALK4 / (ppb)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "3: IJ_AVG_S__BCPI / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "4: IJ_AVG_S__BCPO / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "5: IJ_AVG_S__Br / (ppbv)               (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "6: IJ_AVG_S__Br2 / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "7: IJ_AVG_S__BrCl / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "8: IJ_AVG_S__BrNO2 / (ppbv)            (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "9: IJ_AVG_S__BrNO3 / (ppbv)            (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "10: IJ_AVG_S__BrO / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "11: IJ_AVG_S__C2H6 / (ppb)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "12: IJ_AVG_S__C3H8 / (ppb)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "13: IJ_AVG_S__CCl4 / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "14: IJ_AVG_S__CFC11 / (ppbv)            (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "15: IJ_AVG_S__CFC12 / (ppbv)            (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "16: IJ_AVG_S__CFCX / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "17: IJ_AVG_S__CH2Br2 / (ppbv)           (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "18: IJ_AVG_S__CH2O / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "19: IJ_AVG_S__CH3Br / (ppbv)            (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "20: IJ_AVG_S__CH3CCl3 / (ppbv)          (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "21: IJ_AVG_S__CH3Cl / (ppbv)            (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "22: IJ_AVG_S__CH4 / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "23: IJ_AVG_S__CHBr3 / (ppbv)            (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "24: IJ_AVG_S__CO / (ppbv)               (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "25: IJ_AVG_S__Cl / (ppbv)               (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "26: IJ_AVG_S__Cl2 / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "27: IJ_AVG_S__Cl2O2 / (ppbv)            (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "28: IJ_AVG_S__ClNO2 / (ppbv)            (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "29: IJ_AVG_S__ClNO3 / (ppbv)            (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "30: IJ_AVG_S__ClO / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "31: IJ_AVG_S__ClOO / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "32: IJ_AVG_S__DMS / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "33: IJ_AVG_S__DST1 / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "34: IJ_AVG_S__DST2 / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "35: IJ_AVG_S__DST3 / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "36: IJ_AVG_S__DST4 / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "37: IJ_AVG_S__GLYC / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "38: IJ_AVG_S__H1211 / (ppbv)            (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "39: IJ_AVG_S__H1301 / (ppbv)            (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "40: IJ_AVG_S__H2402 / (ppbv)            (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "41: IJ_AVG_S__H2O / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "42: IJ_AVG_S__H2O2 / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "43: IJ_AVG_S__HAC / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "44: IJ_AVG_S__HBr / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "45: IJ_AVG_S__HCFC22 / (ppbv)           (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "46: IJ_AVG_S__HCFCX / (ppbv)            (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "47: IJ_AVG_S__HCl / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "48: IJ_AVG_S__HNO2 / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "49: IJ_AVG_S__HNO3 / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "50: IJ_AVG_S__HNO4 / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "51: IJ_AVG_S__HOBr / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "52: IJ_AVG_S__HOCl / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "53: IJ_AVG_S__IEPOX / (ppbv)            (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "54: IJ_AVG_S__ISOP / (ppb)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "55: IJ_AVG_S__ISOPN / (ppbv)            (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "56: IJ_AVG_S__MACR / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "57: IJ_AVG_S__MAP / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "58: IJ_AVG_S__MEK / (ppb)               (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "59: IJ_AVG_S__MMN / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "60: IJ_AVG_S__MOBA / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "61: IJ_AVG_S__MP / (ppbv)               (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "62: IJ_AVG_S__MPN / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "63: IJ_AVG_S__MSA / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "64: IJ_AVG_S__MVK / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "65: IJ_AVG_S__N2O / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "66: IJ_AVG_S__N2O5 / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "67: IJ_AVG_S__NH3 / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "68: IJ_AVG_S__NH4 / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "69: IJ_AVG_S__NIT / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "70: IJ_AVG_S__NITs / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "71: IJ_AVG_S__NO / (ppbv)               (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "72: IJ_AVG_S__NO2 / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "73: IJ_AVG_S__NO3 / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "74: IJ_AVG_S__O3 / (ppbv)               (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "75: IJ_AVG_S__OCPI / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "76: IJ_AVG_S__OCPO / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "77: IJ_AVG_S__OCS / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "78: IJ_AVG_S__OClO / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "79: IJ_AVG_S__PAN / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "80: IJ_AVG_S__PMN / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "81: IJ_AVG_S__PPN / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "82: IJ_AVG_S__PROPNN / (ppbv)           (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "83: IJ_AVG_S__PRPE / (ppb)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "84: IJ_AVG_S__R4N2 / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "85: IJ_AVG_S__RCHO / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "86: IJ_AVG_S__RIP / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "87: IJ_AVG_S__SALA / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "88: IJ_AVG_S__SALC / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "89: IJ_AVG_S__SO2 / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "90: IJ_AVG_S__SO4 / (ppbv)              (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n",
        "91: IJ_AVG_S__SO4s / (ppbv)             (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 72)\n"
       ]
      }
     ],
     "prompt_number": 17
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "A more advanced example, combining constraints and extracting data subsets:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def lon_subset(cell):\n",
      "   \"\"\"\n",
      "   return True or False as to whether the cell\n",
      "   center in question should be kept\n",
      "   \"\"\"\n",
      "   return cell > 0. and cell < 20.\n",
      "\n",
      "lon_cst = datafields.Constraint(longitude=lon_subset)\n",
      "\n",
      "dataset = datafields.load(filename,\n",
      "                          \"IJ_AVG_S__O3\" & lon_cst,\n",
      "                          callback=clb)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 18
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print dataset\n",
      "# note the reduced grid-size for the longitude"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0: IJ_AVG_S__O3 / (ppbv)               (atmosphere_hybrid_sigma_pressure_coordinate: 72; latitude: 46; longitude: 3)\n"
       ]
      }
     ],
     "prompt_number": 19
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Loading multiple files"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "It is possible to load multiple files at once. The `pygchem.datafields.load` function will try to merge the fields when it is possible, i.e., when the multiple fields describing the same phenomenom overlap in space and/or time, so that it reduces at the minimum the number of loaded fields."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The following example load the \"IJ_AVG_S__O3\" diagnostic for the entire 1-year simulation:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# note the wildcard character in the filename\n",
      "# (UNIX expressions are supported)\n",
      "filename = 'bpch/ctm.bpch.v10-01c-geosfp-Run1.*'\n",
      "diagnostics = [\"BXHGHT_S__BXHEIGHT\",\n",
      "               \"BXHGHT_S__N(AIR)\",\n",
      "               \"IJ_AVG_S__O3\"]\n",
      "dataset = datafields.load(filename, diagnostics)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 24
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print dataset\n",
      "# note the additional time dimension"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0: BXHGHT_S__BXHEIGHT / (m)            (time: 12; longitude: 72; latitude: 46; model_level_number: 72)\n",
        "1: BXHGHT_S__N(AIR) / (molec/m3)       (time: 12; longitude: 72; latitude: 46; model_level_number: 72)\n",
        "2: IJ_AVG_S__O3 / (ppbv)               (time: 12; longitude: 72; latitude: 46; model_level_number: 72)\n"
       ]
      }
     ],
     "prompt_number": 25
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Merging fields may take a long time. If speed matters, it is still possible to load the fields without any merging: "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "dataset_nomerge = datafields.load_raw(filename, diagnostics)\n",
      "print dataset_nomerge"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0: BXHGHT_S__BXHEIGHT / (m)            (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "1: BXHGHT_S__BXHEIGHT / (m)            (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "2: BXHGHT_S__BXHEIGHT / (m)            (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "3: BXHGHT_S__BXHEIGHT / (m)            (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "4: BXHGHT_S__BXHEIGHT / (m)            (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "5: BXHGHT_S__BXHEIGHT / (m)            (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "6: BXHGHT_S__BXHEIGHT / (m)            (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "7: BXHGHT_S__BXHEIGHT / (m)            (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "8: BXHGHT_S__BXHEIGHT / (m)            (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "9: BXHGHT_S__BXHEIGHT / (m)            (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "10: BXHGHT_S__BXHEIGHT / (m)            (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "11: BXHGHT_S__BXHEIGHT / (m)            (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "12: BXHGHT_S__N(AIR) / (molec/m3)       (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "13: BXHGHT_S__N(AIR) / (molec/m3)       (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "14: BXHGHT_S__N(AIR) / (molec/m3)       (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "15: BXHGHT_S__N(AIR) / (molec/m3)       (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "16: BXHGHT_S__N(AIR) / (molec/m3)       (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "17: BXHGHT_S__N(AIR) / (molec/m3)       (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "18: BXHGHT_S__N(AIR) / (molec/m3)       (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "19: BXHGHT_S__N(AIR) / (molec/m3)       (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "20: BXHGHT_S__N(AIR) / (molec/m3)       (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "21: BXHGHT_S__N(AIR) / (molec/m3)       (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "22: BXHGHT_S__N(AIR) / (molec/m3)       (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "23: BXHGHT_S__N(AIR) / (molec/m3)       (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "24: IJ_AVG_S__O3 / (ppbv)               (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "25: IJ_AVG_S__O3 / (ppbv)               (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "26: IJ_AVG_S__O3 / (ppbv)               (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "27: IJ_AVG_S__O3 / (ppbv)               (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "28: IJ_AVG_S__O3 / (ppbv)               (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "29: IJ_AVG_S__O3 / (ppbv)               (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "30: IJ_AVG_S__O3 / (ppbv)               (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "31: IJ_AVG_S__O3 / (ppbv)               (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "32: IJ_AVG_S__O3 / (ppbv)               (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "33: IJ_AVG_S__O3 / (ppbv)               (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "34: IJ_AVG_S__O3 / (ppbv)               (longitude: 72; latitude: 46; model_level_number: 72)\n",
        "35: IJ_AVG_S__O3 / (ppbv)               (longitude: 72; latitude: 46; model_level_number: 72)\n"
       ]
      }
     ],
     "prompt_number": 29
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Apply contraints to the field list after loading"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "We can extract data subsets and/or specific fields from a loaded list of fields:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "dataset_lon_subset = dataset.extract(lon_cst)\n",
      "print dataset_lon_subset"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0: BXHGHT_S__BXHEIGHT / (m)            (time: 12; longitude: 3; latitude: 46; model_level_number: 72)\n",
        "1: BXHGHT_S__N(AIR) / (molec/m3)       (time: 12; longitude: 3; latitude: 46; model_level_number: 72)\n",
        "2: IJ_AVG_S__O3 / (ppbv)               (time: 12; longitude: 3; latitude: 46; model_level_number: 72)\n"
       ]
      }
     ],
     "prompt_number": 31
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "To select only one field, the `extract_strict` method can be used: "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "ox_avg = dataset.extract_strict(\"IJ_AVG_S__O3\")\n",
      "print ox_avg"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "IJ_AVG_S__O3 / (ppbv)               (time: 12; longitude: 72; latitude: 46; model_level_number: 72)\n",
        "     Dimension coordinates:\n",
        "          time                           x              -             -                       -\n",
        "          longitude                      -              x             -                       -\n",
        "          latitude                       -              -             x                       -\n",
        "          model_level_number             -              -             -                       x\n",
        "     Attributes:\n",
        "          carbon_weight: 1\n",
        "          category: IJ-AVG-$\n",
        "          chemical: True\n",
        "          ctm_units: ppbv\n",
        "          full_name: O3 tracer\n",
        "          hydrocarbon: False\n",
        "          modelname: GEOSFP\n",
        "          molecular_weight: 0.048\n",
        "          name: O3\n",
        "          number: 2\n",
        "          resolution: (5.0, 4.0)\n",
        "          scale: 1000000000.0\n"
       ]
      }
     ],
     "prompt_number": 32
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}
{'fraction_non_alphanumeric': 0.1465017588222776, 'fraction_numerical': 0.04813289435799857, 'mean_word_length': 2.2066473988439306, 'pattern_counts': {'":': 632, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 36, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4135214', 'n_tokens_mistral': 32726, 'n_tokens_neox': 27686, 'n_words': 5990}