<commit_before>{
 "metadata": {
  "name": "",
  "signature": "sha256:e08f952b935f783b3c29f194094ae38d19a5dbbbb1dd21b4ca6377702eb94cea"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# An introduction to Storage"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Introduction"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "All we need is contained in the `openpathsampling` package"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import openpathsampling as paths"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 1
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The storage itself is mainly a netCDF file and can also be used as such. Technically it is a subclass of `netCDF4.Dataset` and can use all of its functions in case we want to add additional tables to the file besides what we store using stores. You can of course also add new stores to the storage. Using `Storage()` will automatically create a set of needed storages when a new file is created. "
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "netCDF files are very generic while our Storage is more tuned to needs we have. It support etc native support for simtk.units, and can recursively store nested objects using JSON pickling. But we will get to that."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Open the output from the 'alanine.ipynb' notebook to have something to work with"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "storage = paths.storage.Storage('trajectory.nc')\n",
      "storage"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 2,
       "text": [
        "Storage @ 'trajectory.nc'"
       ]
      }
     ],
     "prompt_number": 2
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "and have a look at what stores are available"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print storage.list_stores()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "['trajectory', 'snapshot', 'configuration', 'momentum', 'sample', 'sampleset', 'orderparameter', 'pathmover', 'movedetails', 'shootingpoint', 'shootingpointselector', 'dynamicsengine', 'calculation', 'volume', 'ensemble', 'movepath']\n"
       ]
      }
     ],
     "prompt_number": 3
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "and we can access all of these using"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "snapshot_store = storage.snapshot"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 4
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Stores are lists"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "In general it is useful to think about the storage as a set of lists. Each of these lists contain objects of the same type, e.g. `Sample`, `Trajectory`, `Ensemble`, `Volume`, ... The class instances used to access elements from the storage are called a store. Imagine you go into a store to *buy* and *sell* objects (luckily our stores are free). All the stores share the same storage space, which is a netCDF file on disc.\n",
      "\n",
      "Still, a store is not really a list or subclassed from a list, but it almost acts like one."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print 'We have %d snapshots in our storage' % len(storage.snapshot)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "We have 1148 snapshots in our storage\n"
       ]
      }
     ],
     "prompt_number": 5
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Loading objects"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      " In the same way we access lists we can also access these lists using slicing, and even lists of indices."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Load by slicing"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print storage.sample[2:4]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "[<Sample @ 0x11268a510>, <Sample @ 0x11268a490>]\n"
       ]
      }
     ],
     "prompt_number": 6
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Load by name"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print storage.ensemble['Interface 1'].name"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Interface 1\n"
       ]
      }
     ],
     "prompt_number": 7
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Load by list of indices"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print storage.ensemble[[0,1,'Interface 3']]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "[<openpathsampling.ensemble.SequentialEnsemble object at 0x110196cd0>, <openpathsampling.ensemble.SequentialEnsemble object at 0x1068fb8d0>, <openpathsampling.ensemble.SequentialEnsemble object at 0x1068fbd90>]\n"
       ]
      }
     ],
     "prompt_number": 8
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Indexing"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Each loaded object is equipped with a `.idx` attribute which is a dictionary that contains the index for a specific storage. This is necessary since we can - in theory - store an object in several different stores at once and these might have different indices. Note that idx is NOT a function, but a dictionary, hence the square brackets."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print storage.sample[5].idx\n",
      "print storage.sample[2].idx[storage]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "{Storage @ 'trajectory.nc': 5}\n",
        "2\n"
       ]
      }
     ],
     "prompt_number": 9
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Saving"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Saving is somehow special, since we try to deal exclusively with immutable objects. That means that once an object is saved, it cannot be changed. This is not completely true, since the netCDF file allow changing, but we try not to do it. The only exchange are collective variables, these can store their cached values and we want to store intermediate states so we add new values once we have computed these. This should be the only exception and we use the `.sync` command to update the status of a once saved orderparameter"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Saving is easy. Just use `.save` on the store "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# storage.sample.save(my_sample)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 10
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "and it will add the object to the end of our store list or do nothing, if the object has already been stored. It is important to note, that each object knows, if it has been stored already. This allows to write nice recursive saving without worrying that we save the same object several times."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "You can also store directly using the storage. Both is fine and the storage just delegates the task to the appropriate store."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# storage.save(my_sample)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 11
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "I mentioned recursive saving. This does the following. Imagine a sample `snapshot` which itself has a `Configuration` and a `Momentum` object. If you store the snapshot it also store the content using the approriate stores. This can be arbitrarily complex. And most object can be either stored in a special way or get converted into a JSON string that we can turn into an object again. Python has something like this build it, which works similar, but we needed something that add the recursive storage connection and uses JSON. If you are curious, the json string can be accessed for some objects using `.json` but is only available for loaded or saved objects. It will not be computed unless it is used."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "volume = storage.volume[3]\n",
      "volume.json"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 12,
       "text": [
        "u'{\"_cls\": \"OrVolume\", \"_dict\": {\"volume1\": {\"_cls\": \"LambdaVolumePeriodic\", \"_dict\": {\"period_min\": -3.14159, \"lambda_min\": -2.094393333333333, \"orderparameter\": {\"_idx\": 0, \"_cls\": \"OP_MD_Function\"}, \"lambda_max\": -0.5235983333333332, \"period_max\": 3.14159}}, \"volume2\": {\"_cls\": \"OrVolume\", \"_dict\": {\"volume1\": {\"_cls\": \"LambdaVolumePeriodic\", \"_dict\": {\"period_min\": -3.14159, \"lambda_min\": -2.094393333333333, \"orderparameter\": {\"_idx\": 0, \"_cls\": \"OP_MD_Function\"}, \"lambda_max\": -0.5235983333333332, \"period_max\": 3.14159}}, \"volume2\": {\"_cls\": \"LambdaVolumePeriodic\", \"_dict\": {\"period_min\": -3.14159, \"lambda_min\": 1.7453277777777778, \"orderparameter\": {\"_idx\": 0, \"_cls\": \"OP_MD_Function\"}, \"lambda_max\": -3.14159, \"period_max\": 3.14159}}}}}}'"
       ]
      }
     ],
     "prompt_number": 12
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Iterators"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "A list is iterable and so is a store. Lets load all ensembles and list their names"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print [ens.name for ens in storage.ensemble]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "[u'Interface 0', u'Interface 1', u'Interface 2', u'Interface 3', u'Interface 4', u'Interface 5', u'Interface 6']\n"
       ]
      }
     ],
     "prompt_number": 13
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Maybe you have realized that some command run slower the first time. This is because we use caching and once an object is loaded it stays in memory and can be accessed much faster."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Searching for objects"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "One way to find objects is to use their name, which I mentioned before, but in general there are no search functions, but we can use python notation in the usual way to load what we need. *List comprehensions* is the magic word.\n",
      "Say, we want to get all snapshots that are reversed. We could just load all of these and filter them, but there is a more elegant way to do that, or let's say a more elegant way of writing it in python, because the underlying code does just that."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "reversed_samples = [snapshot for snapshot in storage.snapshot if snapshot.reversed]\n",
      "print 'We found %d reversed snapshots among %d total ones' % (len(reversed_samples), len(storage.snapshot))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "We found 542 reversed snapshots among 1148 total ones\n"
       ]
      }
     ],
     "prompt_number": 14
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Lets do something more useful: For TIS ensemble we want statistics on pathlengths associated with sampled trajectories `Sample` objects that are sampled for a specific ensemble. And we one want samples that have been generated in our production runs and are present in a `SampleSet`\n",
      "\n",
      "> TODO: add a way to select only specific SampleSets"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print storage.sampleset[0]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "<openpathsampling.sample.SampleSet object at 0x1123e6fd0>\n"
       ]
      }
     ],
     "prompt_number": 15
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "my_ensemble = storage.ensemble['Interface 2']\n",
      "relevant_samples = [\n",
      "    sample \n",
      "    for sampleset in storage.sampleset \n",
      "    for sample in sampleset \n",
      "    if sample.ensemble is my_ensemble\n",
      "]\n",
      "print len(relevant_samples)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "129\n"
       ]
      }
     ],
     "prompt_number": 16
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "and finally compute the average length"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "list_of_path_lengths = [\n",
      "    len(sample.trajectory)\n",
      "    for sampleset in storage.sampleset \n",
      "    for sample in sampleset \n",
      "    if sample.ensemble is my_ensemble\n",
      "]\n",
      "print list_of_path_lengths\n",
      "if len(list_of_path_lengths) > 0:\n",
      "    mean = float(sum(list_of_path_lengths))/len(list_of_path_lengths)\n",
      "else:\n",
      "    mean = 0.0 # actually, it is not defined, so we just set it to zero\n",
      "print mean"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "[8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8]\n",
        "8.0\n"
       ]
      }
     ],
     "prompt_number": 41
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Allright, we loaded from a bootstrapping sampling algorithm and the analysis is pointless, but still it is rather short considering what we just did."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Generator expression"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "There is another very cool feature about python that is worth noting: generator expressions. Before we used list comprehensions to generate a list of all that we need, but what, if we don't want the whole list at once? Maybe that is impossible because of too much memory and also not desirable? We can do the same thing as above using a generator (although it would only be useful if we had to average over billions of samples). So assume the list of lengths is too large for memory. The summing does not mind to use little pieces so we construct a function that always gives us the next element. These functions are called iterators and to make these iteratore there is syntactic way to create them easily: Instead of square brackets in in list comprehensions use round brackets. So the above example would look like this"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "iterator_over_path_lengths = (\n",
      "    len(sample.trajectory)\n",
      "    for sampleset in storage.sampleset \n",
      "    for sample in sampleset \n",
      "    if sample.ensemble is my_ensemble\n",
      ")\n",
      "print iterator_over_path_lengths\n",
      "total = float(sum(iterator_over_path_lengths))\n",
      "print total"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "<generator object <genexpr> at 0x1127573c0>\n",
        "1032.0\n"
       ]
      }
     ],
     "prompt_number": 18
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Note that we now have a generator and no computed values yet. If we iterator using our iterator called generator it will pass one value at a time and we can use it in sum as we did before. There are two important things to note. Once an iteratore is used, it is consumed and we cannot just be run again so we need to change the code again. I assume there are other ways to do that, too"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "iterator_over_path_lengths = (\n",
      "    len(sample.trajectory)\n",
      "    for sampleset in storage.sampleset \n",
      "    for sample in sampleset \n",
      "    if sample.ensemble is my_ensemble\n",
      ")\n",
      "total = 0\n",
      "count = 0\n",
      "for length in iterator_over_path_lengths:\n",
      "    total += length\n",
      "    count += 1\n",
      "    \n",
      "if count > 0:\n",
      "    mean = float(total)/count\n",
      "else:\n",
      "    mean = 0.0 # actually, it is not defined, so we just set it to zero\n",
      "print mean"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "8.0\n"
       ]
      }
     ],
     "prompt_number": 42
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Voil\u00e0, this time without computing all length before!"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "A last example that will be interesting is the statistics on acceptance. Each sample knows which mover was involved in its creation. This is stored in `.details.mover` in the `.details` attribute. Let us try to look at only forward moves"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "ff_movers = filter(lambda self : self.__class__ == 'ForwardShootMover', storage.pathmover)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 39
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "mover = ff_movers[2]\n",
      "print \"Use a '%s' for ensemble(s) '%s'\" % ( mover.cls, ','.join(ens.name for ens in mover.ensembles) )"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Use a 'ForwardShootMover' for ensemble(s) 'Interface 2'\n"
       ]
      }
     ],
     "prompt_number": 44
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "I use a little trick here, notice that we use a list comprehension inside of a function call, this actually uses the generator expression and passes the resulting iterator to the `.join` function.\n",
      "\n",
      "Now to get statistics on acceptances"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "mover = ff_movers[1]\n",
      "print \"Use a '%s' for ensemble(s) '%s'\" % ( mover.cls, ','.join(ens.name for ens in mover.ensembles) )\n",
      "acceptances = [\n",
      "    1.0 if sample.details.accepted else 0.0\n",
      "    for sample in storage.sample \n",
      "    if sample.details.mover is mover\n",
      "]\n",
      "print acceptances \n",
      "print 'Acceptance is about %d %%' % int(100 * float(sum(acceptances))/len(acceptances))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Use a 'ForwardShootMover' for ensemble(s) 'Interface 1'\n",
        "[1.0, 1.0, 1.0, 1.0, 1.0]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "Acceptance is about 100 %\n"
       ]
      }
     ],
     "prompt_number": 45
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "mover = ff_movers[3]\n",
      "print \"Use a '%s' for ensemble(s) '%s'\" % ( mover.cls, ','.join(ens.name for ens in mover.ensembles) )\n",
      "acceptances = [\n",
      "    1.0 if sample.details.accepted else 0.0\n",
      "    for sample in storage.sample \n",
      "    if sample.details.mover is mover\n",
      "]\n",
      "print acceptances \n",
      "print 'Acceptance is about %d %%' % int(100 * float(sum(acceptances))/len(acceptances))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Use a 'ForwardShootMover' for ensemble(s) 'Interface 3'\n",
        "[0.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.0, 1.0, 1.0, 1.0, 1.0, 1.0]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "Acceptance is about 87 %\n"
       ]
      }
     ],
     "prompt_number": 46
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Collective Variables"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "To be complete, you can update the storage to include the newest set of cached values. Just use"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# my_collectivevariable.sync()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 47
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Fun with movepaths"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "> This is experimental!"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Okay, this is more some idea to explain the intricacies with movepaths and their differences to deal just with a single mover that produces samples. \n",
      "\n",
      "Let us start with a simple example a `OneWayShootingMover`. Nice and convenient. It shoots either forward or backward so its acceptance ratio is different from each sub mover. But we can still ask the question : \"What is the acceptance of this joined mover?\" Now it gets tricky. Imagine we made an error and set up the forward and backward mover for different ensembles then the question we can ask is : \"What is the acceptance to either shoot forward in ensemble #1 or backward in ensemble #2 is shot?\" \n",
      "Lets make it more complicated and do two shooting moves in a row. What is now our acceptance? Is it the question that either or the two acceptes a move or both? How to we count if two samples are generated? What does it mean when a forward sample in ensemble #1 AND a backward sample in ensemble #2 is generated?\n",
      "\n",
      "The point I want to make is that while acceptance makes sense for a sample it does not necessarily in the replica context. To be clear: We ALWAYS generate a sample in a Monte Carlo move but it can be exactly the old one.\n",
      "So what is the difference between accepting a `Shooting` and accepting a `PathMove`?\n",
      "\n",
      "Acceptance does not mean we do not generate, it means the MC move does not move to a new point in our sample space and so convergence will not improve. I assume that the main point is to move around in the space to be samples. So, I could propose (or look in the literature) some things. Some observations that may or may not be true\n",
      "\n",
      "1. When combining several moves into one the probability should stay the same. So we should normalize according to internal number of attempts. \n",
      "\n",
      "2. We should check this per ensemble. So how often did a submove change the samples in a particular ensemble.\n",
      "\n",
      "3. In complicated moves we might have to differentiate between likely and unlikely combinations. In a PartialAcceptance move e.g. it is unlikely that all move pass but some will and so the number of attempts is not fixed. Assume we attempt 5 times the same forward mover and continue if one succeded. If we accept the first and reject the second, what is the acceptance probability? It is not 50%, it should be lower. But maybe that does not matter, since number 1 accepted is relative to all other moves and so we could say it has speed one change per step?\n",
      "\n",
      "4. Can we treat moving back to an original one? Like doing 2 times the same RepEx move does this still count as a move?\n",
      "\n",
      "Many question, which we can play around with."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Some more examples with the storage"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print storage.movepath[2]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Restrict to last sample : True : 1 samples\n",
        " +- PartialAcceptanceMove : True : 1 samples\n",
        " |   +- RandomChoice :\n",
        " |   |   +- SampleMove : BackwardShootMover : True : 1 samples [<Sample @ 0x10fa9e390>]\n",
        " |   +- SampleMove : EnsembleHopMover : False :[]\n"
       ]
      }
     ],
     "prompt_number": 48
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Which sample uses which mover"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print [sample.details.mover.idx[storage] for sample in storage.sample]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "[0, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 3, 2, 3, 2, 3, 2, 1, 2, 3, 2, 1, 2, 3, 2, 3, 2, 3, 2, 1, 2, 1, 2, 1, 2, 3, 2, 3, 2, 1, 2, 3, 2, 1, 2, 1, 2, 1, 2, 1, 2, 3, 2, 1, 2, 1, 2, 1, 2, 3, 2, 1, 2, 1, 2, 1, 2, 3, 2, 3, 2, 1, 2, 3, 2, 1, 2, 3, 2, 1, 2, 1, 2, 1, 2, 3, 2, 1, 2, 3, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 3, 2, 1, 2, 4, 2, 1, 5, 6, 7, 6, 7, 6, 7, 6, 5, 6, 5, 6, 7, 6, 5, 6, 8, 6, 7, 9, 10, 9, 10, 12, 10, 11, 13, 14, 15, 14, 13, 14, 15, 14, 13, 14, 15, 14, 15, 14, 15, 14, 15, 14, 15, 14, 13, 14, 15, 14, 13, 14, 15, 14, 15, 14, 15, 14, 15, 14, 15, 14, 13, 14, 13, 14, 13, 14, 15, 14, 15, 14, 16, 14, 15, 17, 18, 17, 18, 19, 18, 17, 18, 17, 18, 17, 18, 19, 18, 17, 18, 17, 18, 17, 18, 17, 18, 17, 18, 17, 18, 17, 18, 19, 18, 17, 18, 17, 18, 20, 18, 19, 21, 22, 21, 22, 23, 22, 23, 22, 21, 22, 23, 22, 21, 22, 23, 22, 23, 22, 21, 22, 23, 22, 23, 22, 23, 22, 23, 22, 23, 22, 23, 22, 23, 22, 23, 22, 23, 22, 21, 22, 21, 22, 23, 22, 23, 22, 21, 22, 23, 22, 23, 22, 21, 22, 21, 22, 21, 22, 23, 22, 23, 22, 21, 22, 23, 22, 21, 22, 23, 22, 21, 22, 23, 22, 23, 22, 23, 22, 21, 22, 21, 22, 23, 22, 23, 22, 23, 22, 21, 22, 23, 22, 21, 22, 23, 22, 21, 22, 21, 22, 23, 22, 23, 22, 23, 22, 21, 22, 21, 22, 23, 22, 23, 22, 23, 22, 21, 22, 23, 22, 23, 22, 23, 22, 21, 22, 21, 22, 21, 22, 23, 22, 23, 22, 21, 22, 21, 22, 23, 22, 21, 22, 21, 22, 23, 22, 23, 22, 21, 22, 21, 22, 21, 22, 21, 22, 21, 22, 23, 22, 23, 22, 23, 22, 24, 22, 21]\n"
       ]
      }
     ],
     "prompt_number": 49
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "How often was a HopMove accepted independent of the Ensemble?"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "results = [\n",
      "    1 if sample.details.accepted else 0\n",
      "    for sample in storage.sample \n",
      "    if type(sample.details.mover) == paths.EnsembleHopMover \n",
      "]\n",
      "print float(sum(results))/len(results)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0.03125\n"
       ]
      }
     ],
     "prompt_number": 50
    }
   ],
   "metadata": {}
  }
 ]
}<commit_msg>fixed storage tutorial<commit_after>{
 "metadata": {
  "name": "",
  "signature": "sha256:a1a6f4d4c603f6be29169ccf30300495918ab342ba85f41be4b6120af4954a7b"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# An introduction to Storage"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Introduction"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "All we need is contained in the `openpathsampling` package"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import openpathsampling as paths"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 1
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The storage itself is mainly a netCDF file and can also be used as such. Technically it is a subclass of `netCDF4.Dataset` and can use all of its functions in case we want to add additional tables to the file besides what we store using stores. You can of course also add new stores to the storage. Using `Storage()` will automatically create a set of needed storages when a new file is created. "
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "netCDF files are very generic while our Storage is more tuned to needs we have. It support etc native support for simtk.units, and can recursively store nested objects using JSON pickling. But we will get to that."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Open the output from the 'alanine.ipynb' notebook to have something to work with"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "storage = paths.storage.Storage('trajectory.nc')\n",
      "storage"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 2,
       "text": [
        "Storage @ 'trajectory.nc'"
       ]
      }
     ],
     "prompt_number": 2
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "and have a look at what stores are available"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print storage.list_stores()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "['trajectory', 'snapshot', 'configuration', 'momentum', 'sample', 'sampleset', 'orderparameter', 'pathmover', 'movedetails', 'shootingpoint', 'shootingpointselector', 'dynamicsengine', 'calculation', 'volume', 'ensemble', 'movepath']\n"
       ]
      }
     ],
     "prompt_number": 3
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "and we can access all of these using"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "snapshot_store = storage.snapshot"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 4
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Stores are lists"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "In general it is useful to think about the storage as a set of lists. Each of these lists contain objects of the same type, e.g. `Sample`, `Trajectory`, `Ensemble`, `Volume`, ... The class instances used to access elements from the storage are called a store. Imagine you go into a store to *buy* and *sell* objects (luckily our stores are free). All the stores share the same storage space, which is a netCDF file on disc.\n",
      "\n",
      "Still, a store is not really a list or subclassed from a list, but it almost acts like one."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print 'We have %d snapshots in our storage' % len(storage.snapshot)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "We have 414 snapshots in our storage\n"
       ]
      }
     ],
     "prompt_number": 5
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Loading objects"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      " In the same way we access lists we can also access these lists using slicing, and even lists of indices."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Load by slicing"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print storage.sample[2:4]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "[<Sample @ 0x112677410>, <Sample @ 0x1126bd450>]\n"
       ]
      }
     ],
     "prompt_number": 6
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Load by name"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print storage.ensemble['Interface 1'].name"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Interface 1\n"
       ]
      }
     ],
     "prompt_number": 7
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Load by list of indices"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print storage.ensemble[[0,1,'Interface 3']]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "[<openpathsampling.ensemble.SequentialEnsemble object at 0x1126bd4d0>, <openpathsampling.ensemble.SequentialEnsemble object at 0x105caad90>, <openpathsampling.ensemble.SequentialEnsemble object at 0x105caad50>]\n"
       ]
      }
     ],
     "prompt_number": 8
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Indexing"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Each loaded object is equipped with a `.idx` attribute which is a dictionary that contains the index for a specific storage. This is necessary since we can - in theory - store an object in several different stores at once and these might have different indices. Note that idx is NOT a function, but a dictionary, hence the square brackets."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print storage.sample[5].idx\n",
      "print storage.sample[2].idx[storage]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "{Storage @ 'trajectory.nc': 5}\n",
        "2\n"
       ]
      }
     ],
     "prompt_number": 9
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Saving"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Saving is somehow special, since we try to deal exclusively with immutable objects. That means that once an object is saved, it cannot be changed. This is not completely true, since the netCDF file allow changing, but we try not to do it. The only exchange are collective variables, these can store their cached values and we want to store intermediate states so we add new values once we have computed these. This should be the only exception and we use the `.sync` command to update the status of a once saved orderparameter"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Saving is easy. Just use `.save` on the store "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# storage.sample.save(my_sample)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 10
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "and it will add the object to the end of our store list or do nothing, if the object has already been stored. It is important to note, that each object knows, if it has been stored already. This allows to write nice recursive saving without worrying that we save the same object several times."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "You can also store directly using the storage. Both is fine and the storage just delegates the task to the appropriate store."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# storage.save(my_sample)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 11
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "I mentioned recursive saving. This does the following. Imagine a sample `snapshot` which itself has a `Configuration` and a `Momentum` object. If you store the snapshot it also store the content using the approriate stores. This can be arbitrarily complex. And most object can be either stored in a special way or get converted into a JSON string that we can turn into an object again. Python has something like this build it, which works similar, but we needed something that add the recursive storage connection and uses JSON. If you are curious, the json string can be accessed for some objects using `.json` but is only available for loaded or saved objects. It will not be computed unless it is used."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "volume = storage.volume[3]\n",
      "volume.json"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 12,
       "text": [
        "u'{\"_cls\": \"OrVolume\", \"_dict\": {\"volume1\": {\"_cls\": \"LambdaVolumePeriodic\", \"_dict\": {\"period_min\": -3.14159, \"lambda_min\": -2.094393333333333, \"orderparameter\": {\"_idx\": 0, \"_cls\": \"OP_MD_Function\"}, \"lambda_max\": -0.5235983333333332, \"period_max\": 3.14159}}, \"volume2\": {\"_cls\": \"OrVolume\", \"_dict\": {\"volume1\": {\"_cls\": \"LambdaVolumePeriodic\", \"_dict\": {\"period_min\": -3.14159, \"lambda_min\": -2.094393333333333, \"orderparameter\": {\"_idx\": 0, \"_cls\": \"OP_MD_Function\"}, \"lambda_max\": -0.5235983333333332, \"period_max\": 3.14159}}, \"volume2\": {\"_cls\": \"LambdaVolumePeriodic\", \"_dict\": {\"period_min\": -3.14159, \"lambda_min\": 1.7453277777777778, \"orderparameter\": {\"_idx\": 0, \"_cls\": \"OP_MD_Function\"}, \"lambda_max\": -3.14159, \"period_max\": 3.14159}}}}}}'"
       ]
      }
     ],
     "prompt_number": 12
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Iterators"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "A list is iterable and so is a store. Lets load all ensembles and list their names"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print [ens.name for ens in storage.ensemble]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "[u'Interface 0', u'Interface 1', u'Interface 2', u'Interface 3', u'Interface 4', u'Interface 5', u'Interface 6']\n"
       ]
      }
     ],
     "prompt_number": 13
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Maybe you have realized that some command run slower the first time. This is because we use caching and once an object is loaded it stays in memory and can be accessed much faster."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Searching for objects"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "One way to find objects is to use their name, which I mentioned before, but in general there are no search functions, but we can use python notation in the usual way to load what we need. *List comprehensions* is the magic word.\n",
      "Say, we want to get all snapshots that are reversed. We could just load all of these and filter them, but there is a more elegant way to do that, or let's say a more elegant way of writing it in python, because the underlying code does just that."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "reversed_samples = [snapshot for snapshot in storage.snapshot if snapshot.reversed]\n",
      "print 'We found %d reversed snapshots among %d total ones' % (len(reversed_samples), len(storage.snapshot))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "We found 192 reversed snapshots among 414 total ones\n"
       ]
      }
     ],
     "prompt_number": 14
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Lets do something more useful: For TIS ensemble we want statistics on pathlengths associated with sampled trajectories `Sample` objects that are sampled for a specific ensemble. And we one want samples that have been generated in our production runs and are present in a `SampleSet`\n",
      "\n",
      "> TODO: add a way to select only specific SampleSets"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print storage.sampleset[0]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "<openpathsampling.sample.SampleSet object at 0x11269da10>\n"
       ]
      }
     ],
     "prompt_number": 15
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "my_ensemble = storage.ensemble['Interface 2']\n",
      "relevant_samples = [\n",
      "    sample \n",
      "    for sampleset in storage.sampleset \n",
      "    for sample in sampleset \n",
      "    if sample.ensemble is my_ensemble\n",
      "]\n",
      "print len(relevant_samples)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "35\n"
       ]
      }
     ],
     "prompt_number": 16
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "and finally compute the average length"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "list_of_path_lengths = [\n",
      "    len(sample.trajectory)\n",
      "    for sampleset in storage.sampleset \n",
      "    for sample in sampleset \n",
      "    if sample.ensemble is my_ensemble\n",
      "]\n",
      "print list_of_path_lengths\n",
      "if len(list_of_path_lengths) > 0:\n",
      "    mean = float(sum(list_of_path_lengths))/len(list_of_path_lengths)\n",
      "else:\n",
      "    mean = 0.0 # actually, it is not defined, so we just set it to zero\n",
      "print mean"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "[17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17]\n",
        "17.0\n"
       ]
      }
     ],
     "prompt_number": 17
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Allright, we loaded from a bootstrapping sampling algorithm and the analysis is pointless, but still it is rather short considering what we just did."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Generator expression"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "There is another very cool feature about python that is worth noting: generator expressions. Before we used list comprehensions to generate a list of all that we need, but what, if we don't want the whole list at once? Maybe that is impossible because of too much memory and also not desirable? We can do the same thing as above using a generator (although it would only be useful if we had to average over billions of samples). So assume the list of lengths is too large for memory. The summing does not mind to use little pieces so we construct a function that always gives us the next element. These functions are called iterators and to make these iteratore there is syntactic way to create them easily: Instead of square brackets in in list comprehensions use round brackets. So the above example would look like this"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "iterator_over_path_lengths = (\n",
      "    len(sample.trajectory)\n",
      "    for sampleset in storage.sampleset \n",
      "    for sample in sampleset \n",
      "    if sample.ensemble is my_ensemble\n",
      ")\n",
      "print iterator_over_path_lengths\n",
      "total = float(sum(iterator_over_path_lengths))\n",
      "print total"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "<generator object <genexpr> at 0x112757a00>\n",
        "595.0\n"
       ]
      }
     ],
     "prompt_number": 18
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Note that we now have a generator and no computed values yet. If we iterator using our iterator called generator it will pass one value at a time and we can use it in sum as we did before. There are two important things to note. Once an iteratore is used, it is consumed and we cannot just be run again so we need to change the code again. I assume there are other ways to do that, too"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "iterator_over_path_lengths = (\n",
      "    len(sample.trajectory)\n",
      "    for sampleset in storage.sampleset \n",
      "    for sample in sampleset \n",
      "    if sample.ensemble is my_ensemble\n",
      ")\n",
      "total = 0\n",
      "count = 0\n",
      "for length in iterator_over_path_lengths:\n",
      "    total += length\n",
      "    count += 1\n",
      "    \n",
      "if count > 0:\n",
      "    mean = float(total)/count\n",
      "else:\n",
      "    mean = 0.0 # actually, it is not defined, so we just set it to zero\n",
      "print mean"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "17.0\n"
       ]
      }
     ],
     "prompt_number": 19
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Voil\u00e0, this time without computing all length before!"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "A last example that will be interesting is the statistics on acceptance. Each sample knows which mover was involved in its creation. This is stored in `.details.mover` in the `.details` attribute. Let us try to look at only forward moves"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "ff_movers = filter(lambda self : type(self) == paths.ForwardShootMover, storage.pathmover)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 24
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "ff_movers"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 25,
       "text": [
        "[<openpathsampling.pathmover.ForwardShootMover at 0x111ce8110>,\n",
        " <openpathsampling.pathmover.ForwardShootMover at 0x1121e5c90>,\n",
        " <openpathsampling.pathmover.ForwardShootMover at 0x1121e5810>,\n",
        " <openpathsampling.pathmover.ForwardShootMover at 0x111cd5cd0>]"
       ]
      }
     ],
     "prompt_number": 25
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "mover = ff_movers[2]\n",
      "print \"Use a '%s' for ensemble(s) '%s'\" % ( mover.cls, ','.join(ens.name for ens in mover.ensembles) )"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Use a 'ForwardShootMover' for ensemble(s) 'Interface 4'\n"
       ]
      }
     ],
     "prompt_number": 26
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "I use a little trick here, notice that we use a list comprehension inside of a function call, this actually uses the generator expression and passes the resulting iterator to the `.join` function.\n",
      "\n",
      "Now to get statistics on acceptances"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "mover = ff_movers[1]\n",
      "print \"Use a '%s' for ensemble(s) '%s'\" % ( mover.cls, ','.join(ens.name for ens in mover.ensembles) )\n",
      "acceptances = [\n",
      "    1.0 if sample.details.accepted else 0.0\n",
      "    for sample in storage.sample \n",
      "    if sample.details.mover is mover\n",
      "]\n",
      "print acceptances \n",
      "print 'Acceptance is about %d %%' % int(100 * float(sum(acceptances))/len(acceptances))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Use a 'ForwardShootMover' for ensemble(s) 'Interface 3'\n",
        "[1.0]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "Acceptance is about 100 %\n"
       ]
      }
     ],
     "prompt_number": 27
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "mover = ff_movers[3]\n",
      "print \"Use a '%s' for ensemble(s) '%s'\" % ( mover.cls, ','.join(ens.name for ens in mover.ensembles) )\n",
      "acceptances = [\n",
      "    1.0 if sample.details.accepted else 0.0\n",
      "    for sample in storage.sample \n",
      "    if sample.details.mover is mover\n",
      "]\n",
      "print acceptances \n",
      "print 'Acceptance is about %d %%' % int(100 * float(sum(acceptances))/len(acceptances))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Use a 'ForwardShootMover' for ensemble(s) 'Interface 5'\n",
        "[0.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]\n",
        "Acceptance is about 90 %\n"
       ]
      }
     ],
     "prompt_number": 28
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Collective Variables"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "To be complete, you can update the storage to include the newest set of cached values. Just use"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# my_collectivevariable.sync()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 29
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Fun with movepaths"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "> This is experimental!"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Okay, this is more some idea to explain the intricacies with movepaths and their differences to deal just with a single mover that produces samples. \n",
      "\n",
      "Let us start with a simple example a `OneWayShootingMover`. Nice and convenient. It shoots either forward or backward so its acceptance ratio is different from each sub mover. But we can still ask the question : \"What is the acceptance of this joined mover?\" Now it gets tricky. Imagine we made an error and set up the forward and backward mover for different ensembles then the question we can ask is : \"What is the acceptance to either shoot forward in ensemble #1 or backward in ensemble #2 is shot?\" \n",
      "Lets make it more complicated and do two shooting moves in a row. What is now our acceptance? Is it the question that either or the two acceptes a move or both? How to we count if two samples are generated? What does it mean when a forward sample in ensemble #1 AND a backward sample in ensemble #2 is generated?\n",
      "\n",
      "The point I want to make is that while acceptance makes sense for a sample it does not necessarily in the replica context. To be clear: We ALWAYS generate a sample in a Monte Carlo move but it can be exactly the old one.\n",
      "So what is the difference between accepting a `Shooting` and accepting a `PathMove`?\n",
      "\n",
      "Acceptance does not mean we do not generate, it means the MC move does not move to a new point in our sample space and so convergence will not improve. I assume that the main point is to move around in the space to be samples. So, I could propose (or look in the literature) some things. Some observations that may or may not be true\n",
      "\n",
      "1. When combining several moves into one the probability should stay the same. So we should normalize according to internal number of attempts. \n",
      "\n",
      "2. We should check this per ensemble. So how often did a submove change the samples in a particular ensemble.\n",
      "\n",
      "3. In complicated moves we might have to differentiate between likely and unlikely combinations. In a PartialAcceptance move e.g. it is unlikely that all move pass but some will and so the number of attempts is not fixed. Assume we attempt 5 times the same forward mover and continue if one succeded. If we accept the first and reject the second, what is the acceptance probability? It is not 50%, it should be lower. But maybe that does not matter, since number 1 accepted is relative to all other moves and so we could say it has speed one change per step?\n",
      "\n",
      "4. Can we treat moving back to an original one? Like doing 2 times the same RepEx move does this still count as a move?\n",
      "\n",
      "Many question, which we can play around with."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Some more examples with the storage"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print storage.movepath[2]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Restrict to last sample : True : 1 samples\n",
        " +- PartialAcceptanceMove : True : 3 samples\n",
        " |   +- RandomChoice :\n",
        " |   |   +- SampleMove : ForwardShootMover : True : 1 samples [<Sample @ 0x111ce8890>]\n",
        " |   +- SampleMove : EnsembleHopMover : True : 1 samples [<Sample @ 0x1121f6750>]\n",
        " |   +- SampleMove : ReplicaIDChangeMover : True : 1 samples [<Sample @ 0x112666810>]\n"
       ]
      }
     ],
     "prompt_number": 30
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Which sample uses which mover"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print [sample.details.mover.idx[storage] for sample in storage.sample]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "[0, 3, 2, 1, 6, 5, 4, 9, 8, 7, 12, 11, 10, 13, 14, 15, 14, 15, 14, 13, 14, 13, 14, 15, 14, 13, 14, 13, 14, 15, 14, 15, 14, 13, 14, 16, 14, 13, 17, 18, 19, 18, 17, 18, 19, 18, 19, 18, 17, 18, 19, 18, 19, 18, 17, 18, 17, 18, 19, 18, 19, 18, 17, 18, 19, 18, 17, 18, 19, 18, 19, 18, 17, 18, 19, 18, 20, 18, 17]\n"
       ]
      }
     ],
     "prompt_number": 31
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "How often was a HopMove accepted independent of the Ensemble?"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "results = [\n",
      "    1 if sample.details.accepted else 0\n",
      "    for sample in storage.sample \n",
      "    if type(sample.details.mover) == paths.EnsembleHopMover \n",
      "]\n",
      "print float(sum(results))/len(results)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0.166666666667\n"
       ]
      }
     ],
     "prompt_number": 32
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}
{'fraction_non_alphanumeric': 0.15153462165768794, 'fraction_numerical': 0.03148322048139179, 'mean_word_length': 2.226663831596853, 'pattern_counts': {'":': 890, '<': 27, '<?xml version=': 0, '>': 35, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 11, 'word_list_counts': {'cursed_substrings.json': 110, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5655172', 'n_tokens_mistral': 21049, 'n_tokens_neox': 19140, 'n_words': 6466}