<commit_before>{
 "metadata": {
  "name": "",
  "signature": "sha256:73885c546e90eaedd2b2be2372f3ac0eb9e31b012175e34b11539841e555a557"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Working with data from AnyBody"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "There are several ways to output data from AnyBody. The prvious tutorials used the 'Dump' classoperaiton to export specific variables from in a macro command. Another option is to have AnyBody write specific variables to a file by adding the 'AnyOutputFile' class to the AnyBody model."
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Working with AnyOutputFiles"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "To come..."
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Working with HDF5 fiels"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Sometimes, it can be convenient to save the entire model along with all its data (although this can be several hundred megabytes). It is useful if we later want to analyse other output variables from the model. It can also be useful if we want to load the data in the GUI application and replay the result.\n",
      "\n",
      "AnyBody has a feature to save the output of a study to a HDF5 file. And like most things in AnyBody, this can also be done with a macro command. \n",
      "\n",
      "Let us try this with the model from the previous tutorials."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "macrolist = [[\n",
      "  'load \"Knee.any\"',\n",
      "  'operation Main.MyStudy.Kinematics',\n",
      "  'run',\n",
      "  'classoperation Main.MyStudy.Output \"Save data\" --type=\"Deep\" --file=\"output.anydata.h5\"',\n",
      "  'exit']] "
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 1
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      " Here we have added a `Save data` classoperation to the macro. "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from anypytools.abcutils import AnyPyProcess \n",
      "app = AnyPyProcess()\n",
      "\n",
      "app.start_macro(macrolist);"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\r",
        " [****************100%******************]  1 of 1 complete"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n"
       ]
      },
      {
       "html": [
        "Total time: 3 seconds"
       ],
       "metadata": {},
       "output_type": "display_data",
       "text": [
        "<IPython.core.display.HTML object>"
       ]
      }
     ],
     "prompt_number": 2
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The data stored in the file **output.anydata.h5** can be re-loaded in the AnyBody GUI application. To do this; load the model, then right click the `Main.MyStudy.Output` folder and select \"Load data\". \n",
      "\n",
      "Moreover, since we set **`type=\"Deep\"`** in the macro we can also load the data into external programs, e.g. Matlab or python. In python this is done using the *`h5py`* module"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import h5py\n",
      "h5file = h5py.File('output.anydata.h5')\n",
      "data = np.array( h5file['/Output/MomentArm'] )\n",
      "h5file.close()\n",
      "print data"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "[ 0.05740825  0.05665018  0.05586448  0.05505892  0.05424032  0.05341468\n",
        "  0.05258717  0.05176219  0.05094339  0.05013373  0.04933551  0.04855041\n",
        "  0.04777948  0.04702321  0.04628154  0.04555385  0.04483899  0.0441353\n",
        "  0.04344059  0.04275212  0.04206663  0.04138032  0.04068878  0.03998705\n",
        "  0.03926952  0.03852994  0.03776138  0.03695617  0.03610588  0.03520129]\n"
       ]
      }
     ],
     "prompt_number": 19
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The data structure of the HDF5 files can unfortunately be very confusing. AnyBody does not save duplicate copies of the same data. If there are multiple references to the same folder, only one will be present in the HDF5 file. \n",
      "In our model `Knee.any` we have a reference to the **`Knee`** joint folder just before the **`Model`** folder in the study section. Thus, all variables inside the **`Knee`** folder cannot be accessed with the path '/Output/Model/Knee/...', but only through the path of the reference '/Output/kneeref/...'.\n",
      "\n",
      "We can see the problem in the following code snippet:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "with h5py.File('output.anydata.h5') as f:\n",
      "    print( '/Output/Model/Knee/Pos' in f )\n",
      "    print( '/Output/kneeref/Pos' in f )\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "False\n",
        "True\n"
       ]
      }
     ],
     "prompt_number": 14
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "This makes it difficult to find the correct path in large models with many references. AnyPyTools contains a wrapper for the h5py module, which automatically locates the right data, no matter what path is used. Using this module, we can easily locate the data. "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import anypytools.h5py_wrapper as h5py2\n",
      "with h5py2.File('output.anydata.h5') as f:\n",
      "    print( '/Output/Model/Knee/Pos' in f )\n",
      "    print( '/Output/kneeref/Pos' in f )"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "True\n",
        "True\n"
       ]
      }
     ],
     "prompt_number": 2
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The h5py wrapper will also let us use the AnyScript variable names directly so we don't have to replace every . (dot) with a / (slash), and remove the stuff before the Output folder. "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "with h5py2.File('output.anydata.h5') as f:\n",
      "    momentarm = np.array(f['Output.MomentArm'] )                   # dot notation\n",
      "    kneeangle = np.array(f['Main.MyStudy.Output.Model.Knee.Pos'] ) # dot notation with full path\n",
      "    \n",
      "plt.plot(np.degrees(kneeangle),100*momentarm)\n",
      "plt.xlabel('Knee flexion (deg)')\n",
      "plt.ylabel('Moment arm (cm)');"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "display_data",
       "png": "<KEY>//<KEY>",
       "text": [
        "<matplotlib.figure.Figure at 0x77174e0>"
       ]
      }
     ],
     "prompt_number": 25
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}<commit_msg>Fixed missing numpy import in tutorial notebook<commit_after>{
 "metadata": {
  "name": "",
  "signature": "sha256:73885c546e90eaedd2b2be2372f3ac0eb9e31b012175e34b11539841e555a557"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Working with data from AnyBody"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "There are several ways to output data from AnyBody. The prvious tutorials used the 'Dump' classoperaiton to export specific variables from in a macro command. Another option is to have AnyBody write specific variables to a file by adding the 'AnyOutputFile' class to the AnyBody model."
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Working with AnyOutputFiles"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "To come..."
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Working with HDF5 fiels"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Sometimes, it can be convenient to save the entire model along with all its data (although this can be several hundred megabytes). It is useful if we later want to analyse other output variables from the model. It can also be useful if we want to load the data in the GUI application and replay the result.\n",
      "\n",
      "AnyBody has a feature to save the output of a study to a HDF5 file. And like most things in AnyBody, this can also be done with a macro command. \n",
      "\n",
      "Let us try this with the model from the previous tutorials."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "macrolist = [[\n",
      "  'load \"Knee.any\"',\n",
      "  'operation Main.MyStudy.Kinematics',\n",
      "  'run',\n",
      "  'classoperation Main.MyStudy.Output \"Save data\" --type=\"Deep\" --file=\"output.anydata.h5\"',\n",
      "  'exit']] "
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 1
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      " Here we have added a `Save data` classoperation to the macro. "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from anypytools.abcutils import AnyPyProcess \n",
      "app = AnyPyProcess()\n",
      "\n",
      "app.start_macro(macrolist);"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\r",
        " [****************100%******************]  1 of 1 complete"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n"
       ]
      },
      {
       "html": [
        "Total time: 3 seconds"
       ],
       "metadata": {},
       "output_type": "display_data",
       "text": [
        "<IPython.core.display.HTML object>"
       ]
      }
     ],
     "prompt_number": 2
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The data stored in the file **output.anydata.h5** can be re-loaded in the AnyBody GUI application. To do this; load the model, then right click the `Main.MyStudy.Output` folder and select \"Load data\". \n",
      "\n",
      "Moreover, since we set **`type=\"Deep\"`** in the macro we can also load the data into external programs, e.g. Matlab or python. In python this is done using the *`h5py`* module"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import numpy as np\n",
      "import h5py\n",
      "\n",
      "h5file = h5py.File('output.anydata.h5')\n",
      "data = np.array( h5file['/Output/MomentArm'] )\n",
      "h5file.close()\n",
      "print data"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "[ 0.05740825  0.05665018  0.05586448  0.05505892  0.05424032  0.05341468\n",
        "  0.05258717  0.05176219  0.05094339  0.05013373  0.04933551  0.04855041\n",
        "  0.04777948  0.04702321  0.04628154  0.04555385  0.04483899  0.0441353\n",
        "  0.04344059  0.04275212  0.04206663  0.04138032  0.04068878  0.03998705\n",
        "  0.03926952  0.03852994  0.03776138  0.03695617  0.03610588  0.03520129]\n"
       ]
      }
     ],
     "prompt_number": 19
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The data structure of the HDF5 files can unfortunately be very confusing. AnyBody does not save duplicate copies of the same data. If there are multiple references to the same folder, only one will be present in the HDF5 file. \n",
      "In our model `Knee.any` we have a reference to the **`Knee`** joint folder just before the **`Model`** folder in the study section. Thus, all variables inside the **`Knee`** folder cannot be accessed with the path '/Output/Model/Knee/...', but only through the path of the reference '/Output/kneeref/...'.\n",
      "\n",
      "We can see the problem in the following code snippet:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "with h5py.File('output.anydata.h5') as f:\n",
      "    print( '/Output/Model/Knee/Pos' in f )\n",
      "    print( '/Output/kneeref/Pos' in f )\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "False\n",
        "True\n"
       ]
      }
     ],
     "prompt_number": 14
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "This makes it difficult to find the correct path in large models with many references. AnyPyTools contains a wrapper for the h5py module, which automatically locates the right data, no matter what path is used. Using this module, we can easily locate the data. "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import anypytools.h5py_wrapper as h5py2\n",
      "with h5py2.File('output.anydata.h5') as f:\n",
      "    print( '/Output/Model/Knee/Pos' in f )\n",
      "    print( '/Output/kneeref/Pos' in f )"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "True\n",
        "True\n"
       ]
      }
     ],
     "prompt_number": 2
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The h5py wrapper will also let us use the AnyScript variable names directly so we don't have to replace every . (dot) with a / (slash), and remove the stuff before the Output folder. "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "with h5py2.File('output.anydata.h5') as f:\n",
      "    momentarm = np.array(f['Output.MomentArm'] )                   # dot notation\n",
      "    kneeangle = np.array(f['Main.MyStudy.Output.Model.Knee.Pos'] ) # dot notation with full path\n",
      "    \n",
      "plt.plot(np.degrees(kneeangle),100*momentarm)\n",
      "plt.xlabel('Knee flexion (deg)')\n",
      "plt.ylabel('Moment arm (cm)');"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "display_data",
       "png": "<KEY>",
       "text": [
        "<matplotlib.figure.Figure at 0x77174e0>"
       ]
      }
     ],
     "prompt_number": 25
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}
{'fraction_non_alphanumeric': 0.16354173198720762, 'fraction_numerical': 0.04652912773562425, 'mean_word_length': 2.075202468183571, 'pattern_counts': {'":': 230, '<': 10, '<?xml version=': 0, '>': 10, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3307943', 'n_tokens_mistral': 5802, 'n_tokens_neox': 5306, 'n_words': 1451}