<commit_before>{
 "metadata": {
  "name": "",
  "signature": "sha256:52c403be5d640e7a556feaf493840e9752b3372f51d5c385996a6b9a250de8cd"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Filter and pick OTUs"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "* License: BSD\n",
      "* Copyright: American Gut Project, 2014"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Early in the American Gut Project, it was observed that some organisms bloomed likely as a result of increased shipping time and delay between when samples were collected and when they were put on ice (more detail can be found [here](http://americangut.org/?page_id=277)). The purpose of this notebook is to apply the filter developed in order to bioinformatically subtract these observed bloom sequences from fecal samples. It is important to apply this filter when combining data with the American Gut as to remove a potential study-effect bias as all fecal data in the American Gut has had this filter applied. The specific steps covered are:\n",
      "\n",
      "* Filter demultiplexed sequence data to only fecal samples\n",
      "* Determine what sequences in the fecal samples recruit to the observed bloom sequences\n",
      "* Remove the recruited bloom sequences from the demultiplexed sequence data\n",
      "* Trim the filtered demultiplexed data (to reduce study bias when combining with short reads, such as the data in [Yatsunenko et al 2012](http://www.nature.com/nature/journal/v486/n7402/abs/nature11053.html))\n",
      "* Pick OTUs against Greengenes 13_8 for the trimmed and full length filtered demultiplexed data\n",
      "\n",
      "The notebook is designed to support bloom filtering multiple datasets. \n",
      "\n",
      "The filtering is only intended to be applied to fecal data. As such, this notebook allows you to describe what metadata column and value to use so that only fecal samples are used."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%run cluster_utils.ipy\n",
      "import os\n",
      "from functools import partial\n",
      "\n",
      "current_home = os.environ['HOME']\n",
      "basedir = os.path.join(current_home, \"ICU\")\n",
      "working_dir = os.path.join(basedir, 'blooming')\n",
      "prj_name = 'blooming'\n",
      "trim_length = 100\n",
      "\n",
      "# define a submit function against qsub\n",
      "def submit(cmd_prefix, prj_name, queue, extra_args, cmd):\n",
      "    \"\"\"Submission wrapper to submit to qsub\"\"\"\n",
      "    cmd = \"%s; %s\" % (cmd_prefix, cmd)\n",
      "    return submit_qsub(cmd, job_name=prj_name, queue=queue,\n",
      "                       extra_args=extra_args)\n",
      "\n",
      "# define any environment specific details for any running compute job\n",
      "cmd_prefix = 'source ~/.bash_profile; workon ag'\n",
      "\n",
      "# bootstrap the submit method\n",
      "submit_8gb = partial(submit, cmd_prefix, prj_name, 'short8gb', '-l walltime=23:59:59')\n",
      "submit_smr = partial(submit, cmd_prefix, prj_name, 'long8gb', '-l nodes=1:ppn=64 -l walltime=72:00:00')\n",
      "\n",
      "# setup a working directory\n",
      "if not os.path.exists(working_dir):\n",
      "    os.mkdir(working_dir)\n",
      "path_joiner = partial(os.path.join, working_dir)"<commit_msg>addressing @adamrp's comments<commit_after>{
 "metadata": {
  "name": "",
  "signature": "sha256:52c403be5d640e7a556feaf493840e9752b3372f51d5c385996a6b9a250de8cd"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Filter and pick OTUs"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "* License: BSD\n",
      "* Copyright: American Gut Project, 2014"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Early in the American Gut Project, it was observed that some organisms bloomed likely as a result of increased shipping time and delay between when samples were collected and when they were put on ice (more detail can be found [here](http://americangut.org/?page_id=277)). The purpose of this notebook is to apply the filter developed in order to bioinformatically subtract these observed bloom sequences from fecal samples. It is important to apply this filter when combining data with the American Gut as to remove a potential study-effect bias as all fecal data in the American Gut has had this filter applied. The specific steps covered are:\n",
      "\n",
      "* Filter demultiplexed sequence data to only fecal samples\n",
      "* Determine what sequences in the fecal samples recruit to the observed bloom sequences\n",
      "* Remove the recruited bloom sequences from the demultiplexed sequence data\n",
      "* Trim the filtered demultiplexed data (to reduce study bias when combining with short reads, such as the data in [Yatsunenko et al 2012](http://www.nature.com/nature/journal/v486/n7402/abs/nature11053.html))\n",
      "* Pick OTUs against Greengenes 13_8 for the trimmed and full length filtered demultiplexed data\n",
      "\n",
      "The notebook is designed to support bloom filtering multiple datasets. \n",
      "\n",
      "The filtering is only intended to be applied to fecal data. As such, this notebook allows you to describe what metadata column and value to use so that only fecal samples are used."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%run cluster_utils.ipy\n",
      "import os\n",
      "from functools import partial\n",
      "\n",
      "# modify the following variable to set a base directory below your home\n",
      "YOUR_BASE_DIRECTORY = "ICU"\n",
      "current_home = os.environ['HOME']\n",
      "basedir = os.path.join(current_home, YOUR_BASE_DIRECTORY)\n",
      "working_dir = os.path.join(basedir, 'blooming')\n",
      "prj_name = 'blooming'\n",
      "trim_length = 100\n",
      "\n",
      "# define a submit function against qsub\n",
      "def submit(cmd_prefix, prj_name, queue, extra_args, cmd):\n",
      "    \"\"\"Submission wrapper to submit to qsub\"\"\"\n",
      "    cmd = \"%s; %s\" % (cmd_prefix, cmd)\n",
      "    return submit_qsub(cmd, job_name=prj_name, queue=queue,\n",
      "                       extra_args=extra_args)\n",
      "\n",
      "# define any environment specific details for any running compute job.\n",
      "# it is very likely that these will need to be changed to fit your environment.\n",
      "cmd_prefix = 'source ~/.bash_profile; workon ag'\n",
      "\n",
      "# bootstrap the submit method\n",
      "submit_8gb = partial(submit, cmd_prefix, prj_name, 'short8gb', '-l walltime=23:59:59')\n",
      "submit_smr = partial(submit, cmd_prefix, prj_name, 'long8gb', '-l nodes=1:ppn=64 -l walltime=72:00:00')\n",
      "\n",
      "# setup a working directory\n",
      "if not os.path.exists(working_dir):\n",
      "    os.mkdir(working_dir)\n",
      "path_joiner = partial(os.path.join, working_dir)"
{'fraction_non_alphanumeric': 0.12725040916530278, 'fraction_numerical': 0.02591380250954719, 'mean_word_length': 2.9467168998923574, 'pattern_counts': {'":': 40, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7364964', 'n_tokens_mistral': 2425, 'n_tokens_neox': 2244, 'n_words': 746}