<commit_before><commit_msg>Draft of netcdf with xarray<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Working With netCDF files - xarray"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Alternative to plain netCDF4 access from python. \n",
    "\n",
    "* Brings the power of pandas to the physical sciences, by providing N-dimensional variants of the core pandas data structures:\n",
    "\n",
    "Series and DataFrame <--> DataArray and Dataset.\n",
    "\n",
    "The approach adopts the Common Data Model for self-describing scientific data in widespread use in the Earth sciences (e.g., netCDF and OPeNDAP): xray.Dataset is an in-memory representation of a netCDF file.\n",
    "\n",
    "    HTML documentation: http://xray.readthedocs.org\n",
    "\n",
    "The main advantages of using xarray versus plain netCDF4 are:\n",
    "\n",
    "* intelligent selection along labelled dimensions (and also indexes)\n",
    "* groupby operations\n",
    "* data alignment\n",
    "* IO (netcdf)\n",
    "* conversion from and to Pandas.DataFrames\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import xarray as xr\n",
    "import matplotlib.pyplot as plt\n",
    "from mpl_toolkits.basemap import Basemap as bm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<xarray.Dataset>\n",
       "Dimensions:     (latc: 360, level: 5, lonc: 396, time: 6)\n",
       "Coordinates:\n",
       "  * latc        (latc) float64 45.4 45.45 45.5 45.55 45.6 45.65 45.7 45.75 ...\n",
       "  * level       (level) float64 1.0 6.0 11.0 16.0 21.0\n",
       "  * lonc        (lonc) float64 -17.5 -17.42 -17.34 -17.26 -17.18 -17.1 ...\n",
       "  * time        (time) datetime64[ns] 1996-02-01T01:00:00 ...\n",
       "Data variables:\n",
       "    bathymetry  (latc, lonc) float64 nan nan nan nan nan nan nan nan nan nan ...\n",
       "    h           (time, level, latc, lonc) float64 nan nan nan nan nan nan ...\n",
       "    temp        (time, level, latc, lonc) float64 nan nan nan nan nan nan ...\n",
       "Attributes:\n",
       "    title: North West European Shelf 3nm\n",
       "    history: Sun Nov 20 00:54:48 2016: ncks -v time,latc,lonc,level,temp,bathymetry,h -d time,0,120,24 -d level,1,25,5 /gpfs/afmcefas/share/nwes/36x36_old/timebase/nwes-3d-199602.nc cefas_GETM_nwes.nc\n",
       "Mon May  9 18:50:36 2016: ncatted -a units,time,o,c,seconds since 1996-01-01 00:00:00 /gpfs/afmcefas/share/nwes/36x36_old//timebase//nwes-3d-199602.nc\n",
       "Mon May  9 17:12:29 2016: ncap2 -s time=time+2678400 /gpfs/afmcefas/share/nwes/36x36_old//nwes-3d-199602.nc /gpfs/afmcefas/share/nwes/36x36_old//timebase//nwe...\n",
       "    NCO: 4.4.8\n",
       "    nco_openmp_thread_number: 1"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "GETM = xr.open_dataset('../data/cefas_GETM_nwes.nc')\n",
    "GETM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[u'latc', u'level', u'lonc', u'time']\n"
     ]
    }
   ],
   "source": [
    "print(GETM.dims.keys())\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[u'title', u'history', u'NCO', u'nco_openmp_thread_number']"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "GETM.attrs.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Accessing variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<xarray.DataArray 'temp' (time: 6, level: 5, latc: 360, lonc: 396)>\n",
       "[4276800 values with dtype=float64]\n",
       "Coordinates:\n",
       "  * latc     (latc) float64 45.4 45.45 45.5 45.55 45.6 45.65 45.7 45.75 45.8 ...\n",
       "  * level    (level) float64 1.0 6.0 11.0 16.0 21.0\n",
       "  * lonc     (lonc) float64 -17.5 -17.42 -17.34 -17.26 -17.18 -17.1 -17.02 ...\n",
       "  * time     (time) datetime64[ns] 1996-02-01T01:00:00 1996-02-02T01:00:00 ...\n",
       "Attributes:\n",
       "    units: degC\n",
       "    long_name: temperature\n",
       "    valid_range: [ -2.  40.]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bathymetry = GETM['bathymetry']\n",
    "temp=GETM['temp']\n",
    "temp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define selection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "KeyError",
     "evalue": "50.0",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-66-2e47575180d6>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[0mGETM\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msel\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mlevel\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;36m1\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'temp'\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 2\u001b[1;33m \u001b[0msection\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mGETM\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msel\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mlonc\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;36m50\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'temp'\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;32mC:\\Users\\TAMS00\\AppData\\Local\\Continuum\\Anaconda2\\lib\\site-packages\\xarray\\core\\dataset.pyc\u001b[0m in \u001b[0;36msel\u001b[1;34m(self, method, tolerance, **indexers)\u001b[0m\n\u001b[0;32m    967\u001b[0m         \"\"\"\n\u001b[0;32m    968\u001b[0m         pos_indexers, new_indexes = indexing.remap_label_indexers(\n\u001b[1;32m--> 969\u001b[1;33m             \u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mindexers\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mmethod\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mmethod\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mtolerance\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mtolerance\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    970\u001b[0m         )\n\u001b[0;32m    971\u001b[0m         \u001b[1;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0misel\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m**\u001b[0m\u001b[0mpos_indexers\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_replace_indexes\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mnew_indexes\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mC:\\Users\\TAMS00\\AppData\\Local\\Continuum\\Anaconda2\\lib\\site-packages\\xarray\\core\\indexing.pyc\u001b[0m in \u001b[0;36mremap_label_indexers\u001b[1;34m(data_obj, indexers, method, tolerance)\u001b[0m\n\u001b[0;32m    227\u001b[0m         \u001b[0mindex\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mdata_obj\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mdim\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mto_index\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    228\u001b[0m         idxr, new_idx = convert_label_indexer(index, label,\n\u001b[1;32m--> 229\u001b[1;33m                                               dim, method, tolerance)\n\u001b[0m\u001b[0;32m    230\u001b[0m         \u001b[0mpos_indexers\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mdim\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0midxr\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    231\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[0mnew_idx\u001b[0m \u001b[1;32mis\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[0mNone\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mC:\\Users\\TAMS00\\AppData\\Local\\Continuum\\Anaconda2\\lib\\site-packages\\xarray\\core\\indexing.pyc\u001b[0m in \u001b[0;36mconvert_label_indexer\u001b[1;34m(index, label, index_name, method, tolerance)\u001b[0m\n\u001b[0;32m    204\u001b[0m                 \u001b[0mindexer\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mnew_index\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mindex\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mget_loc_level\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mlabel\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mitem\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mlevel\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    205\u001b[0m             \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 206\u001b[1;33m                 \u001b[0mindexer\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mindex\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mget_loc\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mlabel\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mitem\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    207\u001b[0m         \u001b[1;32melif\u001b[0m \u001b[0mlabel\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdtype\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mkind\u001b[0m \u001b[1;33m==\u001b[0m \u001b[1;34m'b'\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    208\u001b[0m             \u001b[0mindexer\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mnonzero\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mlabel\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mC:\\Users\\TAMS00\\AppData\\Local\\Continuum\\Anaconda2\\lib\\site-packages\\pandas\\indexes\\numeric.pyc\u001b[0m in \u001b[0;36mget_loc\u001b[1;34m(self, key, method, tolerance)\u001b[0m\n\u001b[0;32m    367\u001b[0m             \u001b[1;32mpass\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    368\u001b[0m         return super(Float64Index, self).get_loc(key, method=method,\n\u001b[1;32m--> 369\u001b[1;33m                                                  tolerance=tolerance)\n\u001b[0m\u001b[0;32m    370\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    371\u001b[0m     \u001b[1;33m@\u001b[0m\u001b[0mproperty\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mC:\\Users\\TAMS00\\AppData\\Local\\Continuum\\Anaconda2\\lib\\site-packages\\pandas\\indexes\\base.pyc\u001b[0m in \u001b[0;36mget_loc\u001b[1;34m(self, key, method, tolerance)\u001b[0m\n\u001b[0;32m   1945\u001b[0m                 \u001b[1;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_engine\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mget_loc\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mkey\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1946\u001b[0m             \u001b[1;32mexcept\u001b[0m \u001b[0mKeyError\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1947\u001b[1;33m                 \u001b[1;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_engine\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mget_loc\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_maybe_cast_indexer\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mkey\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1948\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1949\u001b[0m         \u001b[0mindexer\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mget_indexer\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mkey\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mmethod\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mmethod\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mtolerance\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mtolerance\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mpandas\\index.pyx\u001b[0m in \u001b[0;36mpandas.index.IndexEngine.get_loc (pandas\\index.c:4154)\u001b[1;34m()\u001b[0m\n",
      "\u001b[1;32mpandas\\index.pyx\u001b[0m in \u001b[0;36mpandas.index.IndexEngine.get_loc (pandas\\index.c:4018)\u001b[1;34m()\u001b[0m\n",
      "\u001b[1;32mpandas\\hashtable.pyx\u001b[0m in \u001b[0;36mpandas.hashtable.Float64HashTable.get_item (pandas\\hashtable.c:9667)\u001b[1;34m()\u001b[0m\n",
      "\u001b[1;32mpandas\\hashtable.pyx\u001b[0m in \u001b[0;36mpandas.hashtable.Float64HashTable.get_item (pandas\\hashtable.c:9611)\u001b[1;34m()\u001b[0m\n",
      "\u001b[1;31mKeyError\u001b[0m: 50.0"
     ]
    }
   ],
   "source": [
    "GETM.sel(level=1)['temp']\n",
    "section=GETM.sel(lonc=50)['temp']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calculate average along a dimension"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "tave=GETM[['temp']].mean('time')\n",
    "\n",
    "plt.contourf(tave['temp'][0,:,:],50)\n",
    "plt.colorbar()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dimension lookup \tIndex lookup \tDataArray syntax \tDataset syntax\n",
    "Positional \tBy integer \tarr[:, 0] \tnot available\n",
    "Positional \tBy label \tarr.loc[:, 'IA'] \tnot available\n",
    "By name \tBy integer \tarr.isel(space=0) or\n",
    "arr[dict(space=0)] \tds.isel(space=0) or\n",
    "ds[dict(space=0)]\n",
    "By name \tBy label \tarr.sel(space='IA') or\n",
    "arr.loc[dict(space='IA')] \tds.sel(space='IA') or\n",
    "ds.loc[dict(space='IA')]"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [Root]",
   "language": "python",
   "name": "Python [Root]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
{'fraction_non_alphanumeric': 0.20744291325477665, 'fraction_numerical': 0.19685773250782237, 'mean_word_length': 3.233934611048478, 'pattern_counts': {'":': 101, '<': 8, '<?xml version=': 0, '>': 14, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4426103', 'n_tokens_mistral': 9145, 'n_tokens_neox': 7565, 'n_words': 711}