<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# AdaptiveMD\n",
    "\n",
    "## Example 1 - Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 0. Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import sys, os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alright, let's load the package and pick the `Project` since we want to start a project"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from adaptivemd import Project"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's open a project with a UNIQUE name. This will be the name used in the DB so make sure it is new and not too short. Opening a project will always create a non-existing project and reopen an exising one. You cannot chose between opening types as you would with a file. This is a precaution to not accidentally delete your project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Use this to completely remove the example-worker project from the database.\n",
    "Project.delete('tutorial')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "project = Project('tutorial')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have a handle for our project. First thing is to set it up to work on a resource."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Set the resource"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is a resource? A `Resource` specifies a shared filesystem with one or more clusteres attached to it. This can be your local machine or just a regular cluster or even a group of cluster that can access the same FS (like Titan, Eos and Rhea do).\n",
    "\n",
    "Once you have chosen your place to store your results this way it is set for the project and can (at least should) not be altered since all file references are made to match this resource. Currently you can use the Fu Berlin Allegro Cluster or run locally. There are two specific local adaptations that include already the path to your conda installation. This simplifies the use of `openmm` or `pyemma`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us pick a local resource on a laptop for now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from adaptivemd import LocalCluster, AllegroCluster"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "first pick your resource -- where you want to run your simulation. Local or on Allegro"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "resource = LocalCluster()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you can add some additional paths, conda environment, etc, before we setup the project. This works by setting a special task `.wrapper` (see notebook 4 for more things to do)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<adaptivemd.task.DummyTask at 0x111adf1d0>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "resource.wrapper"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In a nutshell, this dummy task has a `.pre` and `.post` list of commands you can add any command you want to be executed before every task you run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "resource.wrapper.pre.append('echo \"Hello World\"')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A task can also automatically add to the `PATH` variable, set environment variables and you can add conda environments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "resource.wrapper.add_conda_env('my_env_python_27')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "resource.wrapper.add_path('/x/y/z')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "resource.wrapper.environment['CONDA'] = 'True'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Task: DummyTask\n",
      "<pre>\n",
      "export PATH=/x/y/z:$PATH\n",
      "export CONDA=True\n",
      "echo \"Hello World\"\n",
      "</pre>\n",
      "<main />\n",
      "<post>\n",
      "</post>\n"
     ]
    }
   ],
   "source": [
    "print resource.wrapper.description"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's reset that now and just add a little comment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "resource = LocalCluster()\n",
    "resource.wrapper.pre.append('# This is part of the adaptivemd tutorial')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and finally initialize the project with this specific resource. This is done once for a project and should not be altered."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "project.initialize(resource)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Add `TaskGenerators`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TaskGenerators are instances whose purpose is to create tasks to be executed. This is similar to the\n",
    "way Kernels work. A TaskGenerator will generate `Task` objects for you which will be translated into a `ComputeUnitDescription` and executed. In simple terms:\n",
    "\n",
    "**The task generator creates the bash scripts for you that run a simulation or run pyemma.**\n",
    "\n",
    "A task generator will be initialized with all parameters needed to make it work and it will now what needs to be staged to be used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from adaptivemd.engine.openmm import OpenMMEngine\n",
    "from adaptivemd.analysis.pyemma import PyEMMAAnalysis\n",
    "\n",
    "from adaptivemd import File, Directory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The engine"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A task generator that will create jobs to run simulations. Currently it uses a little python script that will excute OpenMM. It requires conda to be added to the PATH variable or at least openmm to be installed on the cluster. If you setup your resource correctly then this should all happen automatically."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we define a `File` object. These are used to represent files anywhere, on the cluster or your local application. `File` like any complex object in adaptivemd can have a `.name` attribute that makes them easier to find later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "pdb_file = File('file://../files/alanine/alanine.pdb').named('initial_pdb').load()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we used a special prefix that can point to specific locations. \n",
    "\n",
    "- `file://` points to files on your local machine. \n",
    "- `unit://` specifies files on the current working directory of the executing node. Usually these are temprary files for a single execution.\n",
    "- `shared://` specifies the root shared FS directory (e.g. `NO_BACKUP/` on Allegro) Use this to import and export files that are already on the cluster.\n",
    "- `staging://` a special scheduler specific directory where files are moved after they are completed on a node and should be used for later. Use this to relate to files that should be stored or reused. After you one excution is done you usually move all important files to this place.\n",
    "- `sandbox://` this should not concern you and is a special RP folder where all pilot/session folders are located."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `.load()` at the end is important. It causes the `File` object to load the content of the file and if you save the `File` object, the actual file is stored with it. This way it can simply be rewritten on the cluster or anywhere else."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So let's do an example for an OpenMM engine. This is simply a small python script that makes OpenMM look like a executable. It run a simulation by providing an initial frame, OpenMM specific system.xml and integrator.xml files and some additional parameters like the platform name, how often to store simulation frames, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "engine = OpenMMEngine(\n",
    "    pdb_file=pdb_file,\n",
    "    system_file=File('file://../files/alanine/system.xml').load(),\n",
    "    integrator_file=File('file://../files/alanine/integrator.xml').load(),\n",
    "    args='-r --report-interval 1 -p CPU --store-interval 1'\n",
    ").named('openmm')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "engine.add_output_type('master', 'master.dcd', stride=10)\n",
    "engine.add_output_type('protein', 'protein.dcd', stride=1, selection='protein')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To explain this we have now an OpenMMEngine which uses the previously made pdb `File` object and uses the location defined in there. The same some Files for the OpenMM XML files and some args to store each frame (to keep it fast) and run using the `CPU` kernel.\n",
    "\n",
    "Last we name the engine `openmm` to find it later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'openmm'"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "engine.name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The modeller"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The instance to compute an MSM model of existing trajectories that you pass it. It is initialized with a `.pdb` file that is used to create features between the $c_\\alpha$ atoms. This implementaton requires a PDB but in general this is not necessay. It is specific to my PyEMMAAnalysis show case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "modeller = PyEMMAAnalysis(\n",
    "    engine=engine,\n",
    "    outtype='protein',\n",
    "    features={'add_inverse_distances': {'select_Backbone': None}}\n",
    ").named('pyemma')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again we name it `pyemma` for later reference."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Add generators to project"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next step is to add these to the project for later usage. We pick the `.generators` store and just add it. Consider a store to work like a `set()` in python. It contains objects only once and is not ordered. Therefore we need a name to find the objects later. Of course you can always iterate over all objects, but the order is not given.\n",
    "\n",
    "To be precise there is an order in the time of creation of the object, but it is only accurate to seconds and it really is the time it was created and not stored."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "project.generators.add(engine)\n",
    "project.generators.add(modeller)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note, that you cannot add the same engine twice. But if you create a new engine it will be considered different and hence you can store it again. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Create one intial trajectory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally we are ready to run a first trajectory that we will store as a point of reference in the project. Also it is nice to see how it works in general."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are using a _Worker_ approach. This means simply that someone (in our case the user from inside a script or a notebook) creates a list of tasks to be done and some other instance (the worker) will actually do the work."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create a `Trajectory` object"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we create the parameters for the engine to run the simulation. Since it seemed appropriate we use a `Trajectory` object (a special `File` with initial frame and length) as the input. You could of course pass these things separately, but this way, we can actualy reference the no yet existing trajectory and do stuff with it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A Trajectory should have a unique name and so there is a project function to get you one. It uses numbers and makes sure that this number has not been used yet in the project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Trajectory('alanine.pdb' >> [0..100])"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trajectory = project.new_trajectory(engine['pdb_file'], 100)\n",
    "trajectory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This says, initial is `alanine.pdb` run for 100 frames and is named `xxxxxxxx.dcd`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "#### Create a `Task` object"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we want that this trajectory actually exists so we have to make it. This requires a `Task` object that _knows_ to describe a simulation. Since `Task` objects are very flexible and can be complex there are helper functions (i.e. factories) to get these in an easy manner, like the ones we already created just before. Let's use the openmm engine to create an openmm task now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "task = engine.run(trajectory)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's it, just take a trajectory description and turn it into a task that contains the shell commands and needed files, etc. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Submit the task to the queue"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally we need to add this task to the things we want to be done. This is easy and only requires saving the task to the project. This is done to the `project.tasks` bundle and once it has been stored it can be picked up by any worker to execute it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "project.queue(task)  # shortcut for project.tasks.add(task)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That is all we can do from here. To execute the tasks you need to run a worker using\n",
    "\n",
    "```bash\n",
    "adaptivemdworker -l tutorial --verbose\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once this is done, come back here and check your results. If you want you can execute the next cell which will block until the task has been completed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<StoredBundle for with 20 file(s) @ 0x111adf0d0>\n",
      "<ViewBundle for with 4 file(s) @ 0x111adf290>\n"
     ]
    }
   ],
   "source": [
    "print project.files\n",
    "print project.trajectories"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and close the project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<adaptivemd.task.PythonTask object at 0x111bc6890>\n"
     ]
    }
   ],
   "source": [
    "task = modeller.task_run_msm_files(list(project.trajectories))\n",
    "print task\n",
    "project.queue(task)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "u'created'"
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "task.state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "11:03:35 [worker:3] stdout from running task\n",
      "22-03-17 11:03:35 pyemma.coordinates.data.featurization.featurizer.MDFeaturizer[0] WARNING  The 1D arrays input for add_inverse_distances() have been sorted, and index duplicates have been eliminated.\n",
      "Check the output of describe() to see the actual order of the features\n",
      "#trajectories : 3\n",
      "22-03-17 11:03:35 pyemma.coordinates.clustering.kmeans.KmeansClustering[3] INFO     Algorithm did not reach convergence criterion of 1e-05 in 10 iterations. Consider increasing max_iter.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print task.stdout"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {<commit_msg>nb update<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# AdaptiveMD\n",
    "\n",
    "## Example 1 - Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 0. Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import sys, os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alright, let's load the package and pick the `Project` since we want to start a project"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from adaptivemd import Project"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's open a project with a UNIQUE name. This will be the name used in the DB so make sure it is new and not too short. Opening a project will always create a non-existing project and reopen an exising one. You cannot chose between opening types as you would with a file. This is a precaution to not accidentally delete your project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Use this to completely remove the example-worker project from the database.\n",
    "Project.delete('tutorial')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "project = Project('tutorial')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have a handle for our project. First thing is to set it up to work on a resource."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Set the resource"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is a resource? A `Resource` specifies a shared filesystem with one or more clusteres attached to it. This can be your local machine or just a regular cluster or even a group of cluster that can access the same FS (like Titan, Eos and Rhea do).\n",
    "\n",
    "Once you have chosen your place to store your results this way it is set for the project and can (at least should) not be altered since all file references are made to match this resource. Currently you can use the Fu Berlin Allegro Cluster or run locally. There are two specific local adaptations that include already the path to your conda installation. This simplifies the use of `openmm` or `pyemma`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us pick a local resource on a laptop for now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from adaptivemd import LocalCluster, AllegroCluster"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "first pick your resource -- where you want to run your simulation. Local or on Allegro"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "resource = LocalCluster()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you can add some additional paths, conda environment, etc, before we setup the project. This works by setting a special task `.wrapper` (see notebook 4 for more things to do)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<adaptivemd.task.DummyTask at 0x114c04850>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "resource.wrapper"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In a nutshell, this dummy task has a `.pre` and `.post` list of commands you can add any command you want to be executed before every task you run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "resource.wrapper.pre.append('echo \"Hello World\"')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A task can also automatically add to the `PATH` variable, set environment variables and you can add conda environments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "resource.wrapper.add_conda_env('my_env_python_27')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "resource.wrapper.add_path('/x/y/z')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "resource.wrapper.environment['CONDA'] = 'True'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Task: DummyTask\n",
      "<pre>\n",
      "export PATH=/x/y/z:$PATH\n",
      "export CONDA=True\n",
      "echo \"Hello World\"\n",
      "</pre>\n",
      "<main />\n",
      "<post>\n",
      "</post>\n"
     ]
    }
   ],
   "source": [
    "print resource.wrapper.description"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's reset that now and just add a little comment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "resource = LocalCluster()\n",
    "resource.wrapper.pre.append('# This is part of the adaptivemd tutorial')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and finally initialize the project with this specific resource. This is done once for a project and should not be altered."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "project.initialize(resource)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Add `TaskGenerators`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TaskGenerators are instances whose purpose is to create tasks to be executed. This is similar to the\n",
    "way Kernels work. A TaskGenerator will generate `Task` objects for you which will be translated into a `ComputeUnitDescription` and executed. In simple terms:\n",
    "\n",
    "**The task generator creates the bash scripts for you that run a simulation or run pyemma.**\n",
    "\n",
    "A task generator will be initialized with all parameters needed to make it work and it will now what needs to be staged to be used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from adaptivemd.engine.openmm import OpenMMEngine\n",
    "from adaptivemd.analysis.pyemma import PyEMMAAnalysis\n",
    "\n",
    "from adaptivemd import File, Directory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The engine"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A task generator that will create jobs to run simulations. Currently it uses a little python script that will excute OpenMM. It requires conda to be added to the PATH variable or at least openmm to be installed on the cluster. If you setup your resource correctly then this should all happen automatically."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we define a `File` object. These are used to represent files anywhere, on the cluster or your local application. `File` like any complex object in adaptivemd can have a `.name` attribute that makes them easier to find later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "pdb_file = File('file://../files/alanine/alanine.pdb').named('initial_pdb').load()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we used a special prefix that can point to specific locations. \n",
    "\n",
    "- `file://` points to files on your local machine. \n",
    "- `unit://` specifies files on the current working directory of the executing node. Usually these are temprary files for a single execution.\n",
    "- `shared://` specifies the root shared FS directory (e.g. `NO_BACKUP/` on Allegro) Use this to import and export files that are already on the cluster.\n",
    "- `staging://` a special scheduler specific directory where files are moved after they are completed on a node and should be used for later. Use this to relate to files that should be stored or reused. After you one excution is done you usually move all important files to this place.\n",
    "- `sandbox://` this should not concern you and is a special RP folder where all pilot/session folders are located."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `.load()` at the end is important. It causes the `File` object to load the content of the file and if you save the `File` object, the actual file is stored with it. This way it can simply be rewritten on the cluster or anywhere else."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So let's do an example for an OpenMM engine. This is simply a small python script that makes OpenMM look like a executable. It run a simulation by providing an initial frame, OpenMM specific system.xml and integrator.xml files and some additional parameters like the platform name, how often to store simulation frames, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "engine = OpenMMEngine(\n",
    "    pdb_file=pdb_file,\n",
    "    system_file=File('file://../files/alanine/system.xml').load(),\n",
    "    integrator_file=File('file://../files/alanine/integrator.xml').load(),\n",
    "    args='-r --report-interval 1 -p CPU --store-interval 1'\n",
    ").named('openmm')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we need to "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "engine.add_output_type('master', 'master.dcd', stride=10)\n",
    "engine.add_output_type('protein', 'protein.dcd', stride=1, selection='protein')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To explain this we have now an OpenMMEngine which uses the previously made pdb `File` object and uses the location defined in there. The same some Files for the OpenMM XML files and some args to store each frame (to keep it fast) and run using the `CPU` kernel.\n",
    "\n",
    "Last we name the engine `openmm` to find it later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'openmm'"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "engine.name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The modeller"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The instance to compute an MSM model of existing trajectories that you pass it. It is initialized with a `.pdb` file that is used to create features between the $c_\\alpha$ atoms. This implementaton requires a PDB but in general this is not necessay. It is specific to my PyEMMAAnalysis show case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "modeller = PyEMMAAnalysis(\n",
    "    engine=engine,\n",
    "    outtype='protein',\n",
    "    features={'add_inverse_distances': {'select_Backbone': None}}\n",
    ").named('pyemma')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again we name it `pyemma` for later reference."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Add generators to project"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next step is to add these to the project for later usage. We pick the `.generators` store and just add it. Consider a store to work like a `set()` in python. It contains objects only once and is not ordered. Therefore we need a name to find the objects later. Of course you can always iterate over all objects, but the order is not given.\n",
    "\n",
    "To be precise there is an order in the time of creation of the object, but it is only accurate to seconds and it really is the time it was created and not stored."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "project.generators.add(engine)\n",
    "project.generators.add(modeller)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note, that you cannot add the same engine twice. But if you create a new engine it will be considered different and hence you can store it again. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Create one intial trajectory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally we are ready to run a first trajectory that we will store as a point of reference in the project. Also it is nice to see how it works in general."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are using a _Worker_ approach. This means simply that someone (in our case the user from inside a script or a notebook) creates a list of tasks to be done and some other instance (the worker) will actually do the work."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create a `Trajectory` object"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we create the parameters for the engine to run the simulation. Since it seemed appropriate we use a `Trajectory` object (a special `File` with initial frame and length) as the input. You could of course pass these things separately, but this way, we can actualy reference the no yet existing trajectory and do stuff with it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A Trajectory should have a unique name and so there is a project function to get you one. It uses numbers and makes sure that this number has not been used yet in the project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Trajectory('alanine.pdb' >> [0..100])"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trajectory = project.new_trajectory(engine['pdb_file'], 100)\n",
    "trajectory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This says, initial is `alanine.pdb` run for 100 frames and is named `xxxxxxxx.dcd`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "#### Create a `Task` object"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we want that this trajectory actually exists so we have to make it. This requires a `Task` object that _knows_ to describe a simulation. Since `Task` objects are very flexible and can be complex there are helper functions (i.e. factories) to get these in an easy manner, like the ones we already created just before. Let's use the openmm engine to create an openmm task now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "task = engine.run(trajectory)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's it, just take a trajectory description and turn it into a task that contains the shell commands and needed files, etc. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Submit the task to the queue"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally we need to add this task to the things we want to be done. This is easy and only requires saving the task to the project. This is done to the `project.tasks` bundle and once it has been stored it can be picked up by any worker to execute it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "project.queue(task)  # shortcut for project.tasks.add(task)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That is all we can do from here. To execute the tasks you need to run a worker using\n",
    "\n",
    "```bash\n",
    "adaptivemdworker -l tutorial --verbose\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once this is done, come back here and check your results. If you want you can execute the next cell which will block until the task has been completed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<StoredBundle for with 20 file(s) @ 0x111adf0d0>\n",
      "<ViewBundle for with 4 file(s) @ 0x111adf290>\n"
     ]
    }
   ],
   "source": [
    "print project.files\n",
    "print project.trajectories"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and close the project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<adaptivemd.task.PythonTask object at 0x111bc6890>\n"
     ]
    }
   ],
   "source": [
    "task = modeller.task_run_msm_files(list(project.trajectories))\n",
    "print task\n",
    "project.queue(task)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "u'success'"
      ]
     },
     "execution_count": 121,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "task.state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "11:14:52 [worker:3] stdout from running task\n",
      "22-03-17 11:14:51 pyemma.coordinates.data.featurization.featurizer.MDFeaturizer[0] WARNING  The 1D arrays input for add_inverse_distances() have been sorted, and index duplicates have been eliminated.\n",
      "Check the output of describe() to see the actual order of the features\n",
      "#trajectories : 4\n",
      "22-03-17 11:14:51 pyemma.coordinates.clustering.kmeans.KmeansClustering[3] INFO     Cluster centers converged after 10 steps.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print task.stdout"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {
{'fraction_non_alphanumeric': 0.15275921801534273, 'fraction_numerical': 0.007943578322197475, 'mean_word_length': 2.7087922173274595, 'pattern_counts': {'":': 677, '<': 21, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 8}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5620907', 'n_tokens_mistral': 13669, 'n_tokens_neox': 13071, 'n_words': 4265}