<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intro. to Snorkel: Extracting Spouse Relations from the News"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part III: Training our End Extraction Model\n",
    "\n",
    "In this final section of the tutorial, we'll use the noisy training labels we generated in the last tutorial part to train our end extraction model.\n",
    "\n",
    "For this tutorial, we will be training a simple - but fairly effective - logistic regression model.  More generally, however, Snorkel plugs in with many ML libraries including [TensorFlow](https://www.tensorflow.org/), making it easy to use almost any state-of-the-art model as the end extractor!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline\n",
    "import os\n",
    "\n",
    "# TO USE A DATABASE OTHER THAN SQLITE, USE THIS LINE\n",
    "# Note that this is necessary for parallel execution amongst other things...\n",
    "# os.environ['SNORKELDB'] = 'postgres:///snorkel-intro'\n",
    "\n",
    "from snorkel import SnorkelSession\n",
    "session = SnorkelSession()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We repeat our definition of the `Spouse` `Candidate` subclass, and load the test set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from snorkel.models import candidate_subclass\n",
    "\n",
    "Spouse = candidate_subclass('Spouse', ['person1', 'person2'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use the training marginals to train a discriminative model that classifies each `Candidate` as a true or false mention. We'll use a random hyperparameter search, evaluated on the development set labels, to find the best hyperparameters for our model. To run a hyperparameter search, we need labels for a development set. If they aren't already available, we can manually create labels using the Viewer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from snorkel.annotations import load_marginals\n",
    "\n",
    "train_marginals = load_marginals(session, split=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_cands = session.query(Spouse).filter(Spouse.split == 0).order_by(Spouse.id).all()\n",
    "dev_cands   = session.query(Spouse).filter(Spouse.split == 1).order_by(Spouse.id).all()\n",
    "test_cands  = session.query(Spouse).filter(Spouse.split == 2).order_by(Spouse.id).all()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from snorkel.annotations import load_gold_labels\n",
    "\n",
    "L_gold_dev  = load_gold_labels(session, annotator_name='gold', split=1, load_as_array=True, zero_one=True)\n",
    "L_gold_test = load_gold_labels(session, annotator_name='gold', split=2, zero_one=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[reRNN] Training model\n",
      "[reRNN] n_train=17142  #epochs=10  batch size=256\n",
      "[reRNN] Epoch 0 (51.39s)\tAverage loss=0.585975\tDev F1=2.97\n",
      "[reRNN] Epoch 1 (106.58s)\tAverage loss=0.543870\tDev F1=32.67\n",
      "[reRNN] Epoch 2 (160.81s)\tAverage loss=0.538390\tDev F1=33.78\n",
      "[reRNN] Epoch 3 (223.37s)\tAverage loss=0.537752\tDev F1=34.86\n",
      "[reRNN] Epoch 4 (278.30s)\tAverage loss=0.537312\tDev F1=34.19\n",
      "[reRNN] Epoch 5 (332.84s)\tAverage loss=0.537003\tDev F1=36.06\n",
      "[reRNN] Epoch 6 (387.64s)\tAverage loss=0.537022\tDev F1=35.01\n",
      "[reRNN] Epoch 7 (442.61s)\tAverage loss=0.536780\tDev F1=35.68\n",
      "[reRNN] Model saved as <reRNN>\n",
      "[reRNN] Epoch 8 (498.82s)\tAverage loss=0.536516\tDev F1=35.75\n",
      "[reRNN] Model saved as <reRNN>\n",
      "[reRNN] Epoch 9 (555.09s)\tAverage loss=0.536319\tDev F1=37.62\n",
      "[reRNN] Model saved as <reRNN>\n",<commit_msg>Added pointer to GridSearch.<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intro. to Snorkel: Extracting Spouse Relations from the News"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part III: Training our End Extraction Model\n",
    "\n",
    "In this final section of the tutorial, we'll use the noisy training labels we generated in the last tutorial part to train our end extraction model.\n",
    "\n",
    "For this tutorial, we will be training a Bi-LSTM, a state-of-the-art deep neural network implemented in [TensorFlow](https://www.tensorflow.org/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline\n",
    "import os\n",
    "\n",
    "# TO USE A DATABASE OTHER THAN SQLITE, USE THIS LINE\n",
    "# Note that this is necessary for parallel execution amongst other things...\n",
    "# os.environ['SNORKELDB'] = 'postgres:///snorkel-intro'\n",
    "\n",
    "from snorkel import SnorkelSession\n",
    "session = SnorkelSession()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We repeat our definition of the `Spouse` `Candidate` subclass:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from snorkel.models import candidate_subclass\n",
    "\n",
    "Spouse = candidate_subclass('Spouse', ['person1', 'person2'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We reload the probabilistic training labels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from snorkel.annotations import load_marginals\n",
    "\n",
    "train_marginals = load_marginals(session, split=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also reload the candidates:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_cands = session.query(Spouse).filter(Spouse.split == 0).order_by(Spouse.id).all()\n",
    "dev_cands   = session.query(Spouse).filter(Spouse.split == 1).order_by(Spouse.id).all()\n",
    "test_cands  = session.query(Spouse).filter(Spouse.split == 2).order_by(Spouse.id).all()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we load gold labels for evaluation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from snorkel.annotations import load_gold_labels\n",
    "\n",
    "L_gold_dev  = load_gold_labels(session, annotator_name='gold', split=1, load_as_array=True, zero_one=True)\n",
    "L_gold_test = load_gold_labels(session, annotator_name='gold', split=2, zero_one=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can setup our discriminative model. Here we specify the model and learning hyperparameters.\n",
    "\n",
    "They can also be set automatically using a search based on the dev set with a [GridSearch](https://github.com/HazyResearch/snorkel/blob/master/snorkel/learning/utils.py) object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[reRNN] Training model\n",
      "[reRNN] n_train=17142  #epochs=10  batch size=256\n",
      "[reRNN] Epoch 0 (51.39s)\tAverage loss=0.585975\tDev F1=2.97\n",
      "[reRNN] Epoch 1 (106.58s)\tAverage loss=0.543870\tDev F1=32.67\n",
      "[reRNN] Epoch 2 (160.81s)\tAverage loss=0.538390\tDev F1=33.78\n",
      "[reRNN] Epoch 3 (223.37s)\tAverage loss=0.537752\tDev F1=34.86\n",
      "[reRNN] Epoch 4 (278.30s)\tAverage loss=0.537312\tDev F1=34.19\n",
      "[reRNN] Epoch 5 (332.84s)\tAverage loss=0.537003\tDev F1=36.06\n",
      "[reRNN] Epoch 6 (387.64s)\tAverage loss=0.537022\tDev F1=35.01\n",
      "[reRNN] Epoch 7 (442.61s)\tAverage loss=0.536780\tDev F1=35.68\n",
      "[reRNN] Model saved as <reRNN>\n",
      "[reRNN] Epoch 8 (498.82s)\tAverage loss=0.536516\tDev F1=35.75\n",
      "[reRNN] Model saved as <reRNN>\n",
      "[reRNN] Epoch 9 (555.09s)\tAverage loss=0.536319\tDev F1=37.62\n",
      "[reRNN] Model saved as <reRNN>\n",
{'fraction_non_alphanumeric': 0.17532962841887326, 'fraction_numerical': 0.04424103737604882, 'mean_word_length': 3.077298978231897, 'pattern_counts': {'":': 111, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 3, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6945642', 'n_tokens_mistral': 3693, 'n_tokens_neox': 3401, 'n_words': 779}