<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To train a model, you first need to convert your sequences and targets into the input HDF5 format. Check out my tutorials for how to do that; they're linked from the [main page](../README.md).\n",
    "\n",
    "For this tutorial, grab a small example HDF5 that I constructed here with 10% of the training sequences and only GM12878 targets for various DNase-seq, ChIP-seq, and CAGE experiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import os, subprocess\n",
    "\n",
    "if not os.path.isfile('data/heart_l131k.h5'):\n",
    "    subprocess.call('curl -o heart_l131k.h5 https://storage.googleapis.com/basenji_tutorial_data/heart_l131k.h5', shell=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, you need to decide what sort of architecture to use. This grammar probably needs work; my goal was to enable hyperparameter searches to write the parameters to file so that I could run parallel training jobs to explore the hyperparameter space. I included an example set of parameters that will work well with this data in models/params_small.txt."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, run [basenji_train.py](https://github.com/calico/basenji/blob/master/bin/basenji_train.py) to train a model. The program will offer training feedback via stdout and write the model output files to the prefix given by the *-s* parameter.\n",
    "\n",
    "The most relevant options here are:\n",
    "\n",
    "| Option/Argument | Value | Note |\n",
    "|:---|:---|:---|\n",
    "| --augment_rc | | Process even-numbered epochs as forward, odd-numbered as reverse complemented. |\n",
    "| --ensemble_rc | | Average forward and reverse complemented predictions on validation set. |\n",
    "| --augment_shifts | \"1,0,-1\" | Rotate epochs over small sequence shifts. |\n",
    "| --logdir | models/heart | Directory to save training logs and model checkpoints. |\n",
    "| --params | models/params_small.txt | Table of parameters to setup the model architecture and optimization. |\n",
    "| --data | data/heart_l131k.h5 | HDF5 file containing the training and validation input and output datasets as generated by [basenji_hdf5_single.py](https://github.com/calico/basenji/blob/master/bin/basenji_hdf5_single.py) |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to train, uncomment the following line and run it. Depending on your hardware, it may require several hours."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# ! basenji_train.py --augment_rc --ensemble_rc --augment_shifts \"1,0,-1\" --logdir models/heart --params models/params_small.txt --data data/heart_l131k.h5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, you can just download a trained model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "if not os.path.isdir('models/heart'):\n",
    "    os.mkdir('models/heart')\n",
    "if not os.path.isfile('models/heart/model_best.tf.meta'):\n",
    "    subprocess.call('curl -o models/heart/model_best.tf.index https://storage.googleapis.com/basenji_tutorial_data/model_best.tf.index', shell=True)\n",
    "    subprocess.call('curl -o models/heart/model_best.tf.meta https://storage.googleapis.com/basenji_tutorial_data/model_best.tf.meta', shell=True)\n",
    "    subprocess.call('curl -o models/heart/model_best.tf.data-00000-of-00001 https://storage.googleapis.com/basenji_tutorial_data/model_best.tf.data-00000-of-00001', shell=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "models/heart/model_best.tf will now specify the name of your saved model to be provided to other programs.\n",
    "\n",
    "To further benchmark the accuracy (e.g. computing significant \"peak\" accuracy), use [basenji_test.py](https://github.com/calico/basenji/blob/master/bin/basenji_test.py).\n",
    "\n",
    "The most relevant options here are:\n",
    "\n",
    "| Option/Argument | Value | Note |\n",
    "|:---|:---|:---|\n",
    "| --rc | | Average the forward and reverse complement to form prediction. |\n",
    "| -o | output/heart_test | Output directory. |\n",
    "| --ai | 0,1,2 | Make accuracy scatter plots for targets 0, 1, and 2. |\n",
    "| --ti | 3,4,5 | Make BigWig tracks for targets 3, 4, and 5. |\n",
    "| -t | data/heart.bed | BED file describing sequence regions for BigWig track output. |\n",
    "| params_file | models/params_small.txt | Table of parameters to setup the model architecture and optimization. |\n",
    "| model_file | models/heart/model_best.tf | Trained saved model prefix. |\n",
    "| data_file | data/heart_l131k.h5 | HDF5 file containing the test input and output datasets as generated by [basenji_hdf5_single.py](https://github.com/calico/basenji/blob/master/bin/basenji_hdf5_single.py) |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/Users/davidkelley/anaconda3/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: compiletime version 3.6 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.5\n",
      "  return f(*args, **kwds)\n",
      "{'adam_beta2': 0.98, 'cnn_dilation': [1, 1, 1, 1, 1, 2, 4, 8, 16, 32, 64, 1], 'optimizer': 'adam', 'num_targets': 3, 'cnn_pool': [2, 4, 4, 4, 1, 0, 0, 0, 0, 0, 0, 0], 'cnn_dropout': 0.1, 'cnn_dense': [0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0], 'loss': 'poisson', 'adam_beta1': 0.97, 'learning_rate': 0.002, 'link': 'softplus', 'cnn_filter_sizes': [20, 7, 7, 7, 3, 3, 3, 3, 3, 3, 3, 1], 'batch_buffer': 4096, 'cnn_filters': [128, 128, 192, 256, 256, 32, 32, 32, 32, 32, 32, 384], 'batch_size': 4, 'target_pool': 128}\n",
      "Targets pooled by 128 to length 1024\n",
      "Convolution w/ 3 384x1 filters to final targets\n",
      "Model building time 11s\n",
      "2018-05-15 22:46:09.654777: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA\n",
      "SeqNN test: 671s\n",
      "Compute stats: 0s\n",
      "Test Loss:         0.45207\n",
      "Test R2:           0.29181\n",
      "Test PearsonR:     0.54311\n",
      "Test log PearsonR: 0.49610\n",
      "Test AUROC:     0.88940\n",
      "Test AUPRC:     0.34590\n"
     ]
    }
   ],
   "source": [
    "! basenji_test.py --rc -o output/heart_test --ai 0,1,2 -t data/heart_l131k.bed --ti 0,1,2 --peaks models/params_small.txt models/heart/model_best.tf data/heart_l131k.h5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*data/gm12878_test/acc.txt* is a table specifiying the loss function value, R2, R2 after log2, and Spearman correlation for each dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {<commit_msg>path fixes<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To train a model, you first need to convert your sequences and targets into the input HDF5 format. Check out my tutorials for how to do that; they're linked from the [main page](../README.md).\n",
    "\n",
    "For this tutorial, grab a small example HDF5 that I constructed here with 10% of the training sequences and only GM12878 targets for various DNase-seq, ChIP-seq, and CAGE experiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import os, subprocess\n",
    "\n",
    "if not os.path.isfile('data/heart_l131k.h5'):\n",
    "    subprocess.call('curl -o data/heart_l131k.h5 https://storage.googleapis.com/basenji_tutorial_data/heart_l131k.h5', shell=True)\n",
    "    subprocess.call('curl -o data/heart_l131k.bed https://storage.googleapis.com/basenji_tutorial_data/heart_l131k.bed', shell=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, you need to decide what sort of architecture to use. This grammar probably needs work; my goal was to enable hyperparameter searches to write the parameters to file so that I could run parallel training jobs to explore the hyperparameter space. I included an example set of parameters that will work well with this data in models/params_small.txt."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, run [basenji_train.py](https://github.com/calico/basenji/blob/master/bin/basenji_train.py) to train a model. The program will offer training feedback via stdout and write the model output files to the prefix given by the *-s* parameter.\n",
    "\n",
    "The most relevant options here are:\n",
    "\n",
    "| Option/Argument | Value | Note |\n",
    "|:---|:---|:---|\n",
    "| --augment_rc | | Process even-numbered epochs as forward, odd-numbered as reverse complemented. |\n",
    "| --ensemble_rc | | Average forward and reverse complemented predictions on validation set. |\n",
    "| --augment_shifts | \"1,0,-1\" | Rotate epochs over small sequence shifts. |\n",
    "| --logdir | models/heart | Directory to save training logs and model checkpoints. |\n",
    "| --params | models/params_small.txt | Table of parameters to setup the model architecture and optimization. |\n",
    "| --data | data/heart_l131k.h5 | HDF5 file containing the training and validation input and output datasets as generated by [basenji_hdf5_single.py](https://github.com/calico/basenji/blob/master/bin/basenji_hdf5_single.py) |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to train, uncomment the following line and run it. Depending on your hardware, it may require several hours."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# ! basenji_train.py --augment_rc --ensemble_rc --augment_shifts \"1,0,-1\" --logdir models/heart --params models/params_small.txt --data data/heart_l131k.h5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, you can just download a trained model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "if not os.path.isdir('models/heart'):\n",
    "    os.mkdir('models/heart')\n",
    "if not os.path.isfile('models/heart/model_best.tf.meta'):\n",
    "    subprocess.call('curl -o models/heart/model_best.tf.index https://storage.googleapis.com/basenji_tutorial_data/model_best.tf.index', shell=True)\n",
    "    subprocess.call('curl -o models/heart/model_best.tf.meta https://storage.googleapis.com/basenji_tutorial_data/model_best.tf.meta', shell=True)\n",
    "    subprocess.call('curl -o models/heart/model_best.tf.data-00000-of-00001 https://storage.googleapis.com/basenji_tutorial_data/model_best.tf.data-00000-of-00001', shell=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "models/heart/model_best.tf will now specify the name of your saved model to be provided to other programs.\n",
    "\n",
    "To further benchmark the accuracy (e.g. computing significant \"peak\" accuracy), use [basenji_test.py](https://github.com/calico/basenji/blob/master/bin/basenji_test.py).\n",
    "\n",
    "The most relevant options here are:\n",
    "\n",
    "| Option/Argument | Value | Note |\n",
    "|:---|:---|:---|\n",
    "| --rc | | Average the forward and reverse complement to form prediction. |\n",
    "| -o | output/heart_test | Output directory. |\n",
    "| --ai | 0,1,2 | Make accuracy scatter plots for targets 0, 1, and 2. |\n",
    "| --ti | 3,4,5 | Make BigWig tracks for targets 3, 4, and 5. |\n",
    "| -t | data/heart_l131k.bed | BED file describing sequence regions for BigWig track output. |\n",
    "| params_file | models/params_small.txt | Table of parameters to setup the model architecture and optimization. |\n",
    "| model_file | models/heart/model_best.tf | Trained saved model prefix. |\n",
    "| data_file | data/heart_l131k.h5 | HDF5 file containing the test input and output datasets as generated by [basenji_hdf5_single.py](https://github.com/calico/basenji/blob/master/bin/basenji_hdf5_single.py) |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/Users/davidkelley/anaconda3/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: compiletime version 3.6 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.5\n",
      "  return f(*args, **kwds)\n",
      "{'adam_beta2': 0.98, 'adam_beta1': 0.97, 'target_pool': 128, 'cnn_filters': [128, 128, 192, 256, 256, 32, 32, 32, 32, 32, 32, 384], 'cnn_dilation': [1, 1, 1, 1, 1, 2, 4, 8, 16, 32, 64, 1], 'batch_size': 4, 'batch_buffer': 4096, 'loss': 'poisson', 'cnn_filter_sizes': [20, 7, 7, 7, 3, 3, 3, 3, 3, 3, 3, 1], 'cnn_dropout': 0.1, 'learning_rate': 0.002, 'cnn_dense': [0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0], 'optimizer': 'adam', 'num_targets': 3, 'link': 'softplus', 'cnn_pool': [2, 4, 4, 4, 1, 0, 0, 0, 0, 0, 0, 0]}\n",
      "Targets pooled by 128 to length 1024\n",
      "Convolution w/ 3 384x1 filters to final targets\n",
      "Model building time 13s\n",
      "2018-06-05 18:27:30.205759: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA\n"
     ]
    }
   ],
   "source": [
    "! basenji_test.py --rc -o output/heart_test --ai 0,1,2 -t data/heart_l131k.bed --ti 0,1,2 --peaks models/params_small.txt models/heart/model_best.tf data/heart_l131k.h5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*data/gm12878_test/acc.txt* is a table specifiying the loss function value, R2, R2 after log2, and Spearman correlation for each dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
{'fraction_non_alphanumeric': 0.16276683087027916, 'fraction_numerical': 0.03824575807334428, 'mean_word_length': 3.194261119081779, 'pattern_counts': {'":': 104, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 17, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5256595', 'n_tokens_mistral': 5547, 'n_tokens_neox': 4997, 'n_words': 1409}