<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intro Tutorial on Relation Extraction from News Articles"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part III: Creating or Loading Evaluation Labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "from snorkel import SnorkelSession\n",
    "session = SnorkelSession()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part III(a): Creating Evaluation Labels in the `Viewer`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We repeat our definition of the `Spouse` `Candidate` subclass from Part II."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from snorkel.models import candidate_subclass\n",
    "\n",
    "Spouse = candidate_subclass('Spouse', ['person1', 'person2'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the development `CandidateSet`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will start by viewing the development `CandidateSet` we created in Part II in the `Viewer`.\n",
    "\n",
    "First we reload the development `CandidateSet`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from snorkel.models import CandidateSet\n",
    "\n",
    "cs = session.query(CandidateSet).filter(CandidateSet.name == 'News Development Candidates').one()\n",
    "cs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Labeling the `CandidateSet` in the `Viewer`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We create a `Viewer` to annotate them manually."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from snorkel.viewer import SentenceNgramViewer\n",
    "\n",
    "# NOTE: This if-then statement is only to avoid opening the viewer during automated testing of this notebook\n",
    "# You should ignore this!\n",
    "import os\n",
    "if 'CI' not in os.environ:\n",
    "    sv = SentenceNgramViewer(cs, session, annotator_name=\"Tutorial Part III User\")\n",
    "else:\n",
    "    sv = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now open the Viewer.\n",
    "\n",
    "You can mark each `Candidate` as true or false. Remember that <span style=\"color:red\">red</span> denotes the first argument (chemical) and <span style=\"color:blue\">blue</span> denotes the second (disease). Try it!\n",
    "\n",
    "These labels are automatically saved in the database backend, and can be accessed using the annotator's name ('Tutorial Part III User') as the AnnotationKey."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part III(b): Loading External Evaluation Labels\n",
    "\n",
    "In case you want to skip annotating the data yourself, this tutorial also includes labels for the development and test sets.\n",
    "\n",
    "Snorkel uses `AnnotationManager` objects to manage information associated with each `Candidate`. Types of annotations include `Label`, `Feature`, and `Prediction`. We will use annotations further in parts IV and V.\n",
    "\n",
    "First, we create an `AnnotationKeySet` and `AnnotationKey` for our labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from snorkel.queries import get_or_create_single_key_set\n",
    "\n",
    "gold_key_set, gold_key = get_or_create_single_key_set(session, 'News Gold Labels')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Next, we read in triples, two spans represented by their _stable ids_, representing a `Spouse` `Candidate` and a value, where -1 means False and 1 means True.\n",
    "\n",
    "Note that here we start to use more of the `SnorkelSession` querying syntax. Snorkel is built on [SQLAlchemy](http://www.sqlalchemy.org/), so their [querying API](http://docs.sqlalchemy.org/en/latest/orm/tutorial.html#querying) can be used in Snorkel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from snorkel.models import Span, Label\n",
    "import pandas as pd\n",
    "dev_labels = pd.read_csv('data/gold_labels.tsv', sep=\"\\t\")\n",
    "for index, row in dev_labels.iterrows():\n",
    "    person1 = session.query(Span).filter(Span.stable_id == row['person1']).one()\n",
    "    person2 = session.query(Span).filter(Span.stable_id == row['person2']).one()\n",
    "    candidate = session.query(Spouse).filter(Spouse.person1 == person1).filter(Spouse.person2 == person2).first()\n",
    "    if candidate is None:\n",
    "        candidate = session.query(Spouse).filter(Spouse.person1 == person2).filter(Spouse.person2 == person1).one()\n",
    "    \n",
    "    label = session.query(Label).filter(Label.candidate == candidate).one_or_none()\n",
    "    if label is None:\n",
    "        label = Label(candidate=candidate, key=gold_key, value=int(row['label']))\n",
    "        session.add(label)\n",
    "    else:\n",
    "        label.value = int(row['label'])\n",
    "session.commit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, in Part IV, we will build a model to predict these labels using data programming."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2<commit_msg>Added check for variations in CoreNLP output to intro tutorial 3.<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intro Tutorial on Relation Extraction from News Articles"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part III: Creating or Loading Evaluation Labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "from snorkel import SnorkelSession\n",
    "session = SnorkelSession()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part III(a): Creating Evaluation Labels in the `Viewer`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We repeat our definition of the `Spouse` `Candidate` subclass from Part II."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from snorkel.models import candidate_subclass\n",
    "\n",
    "Spouse = candidate_subclass('Spouse', ['person1', 'person2'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the development `CandidateSet`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will start by viewing the development `CandidateSet` we created in Part II in the `Viewer`.\n",
    "\n",
    "First we reload the development `CandidateSet`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from snorkel.models import CandidateSet\n",
    "\n",
    "cs = session.query(CandidateSet).filter(CandidateSet.name == 'News Development Candidates').one()\n",
    "cs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Labeling the `CandidateSet` in the `Viewer`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We create a `Viewer` to annotate them manually."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from snorkel.viewer import SentenceNgramViewer\n",
    "\n",
    "# NOTE: This if-then statement is only to avoid opening the viewer during automated testing of this notebook\n",
    "# You should ignore this!\n",
    "import os\n",
    "if 'CI' not in os.environ:\n",
    "    sv = SentenceNgramViewer(cs, session, annotator_name=\"Tutorial Part III User\")\n",
    "else:\n",
    "    sv = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now open the Viewer.\n",
    "\n",
    "You can mark each `Candidate` as true or false. Remember that <span style=\"color:red\">red</span> denotes the first argument (chemical) and <span style=\"color:blue\">blue</span> denotes the second (disease). Try it!\n",
    "\n",
    "These labels are automatically saved in the database backend, and can be accessed using the annotator's name ('Tutorial Part III User') as the AnnotationKey."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part III(b): Loading External Evaluation Labels\n",
    "\n",
    "In case you want to skip annotating the data yourself, this tutorial also includes labels for the development and test sets.\n",
    "\n",
    "Snorkel uses `AnnotationManager` objects to manage information associated with each `Candidate`. Types of annotations include `Label`, `Feature`, and `Prediction`. We will use annotations further in parts IV and V.\n",
    "\n",
    "First, we create an `AnnotationKeySet` and `AnnotationKey` for our labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from snorkel.queries import get_or_create_single_key_set\n",
    "\n",
    "gold_key_set, gold_key = get_or_create_single_key_set(session, 'News Gold Labels')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Next, we read in triples, two spans represented by their _stable ids_, representing a `Spouse` `Candidate` and a value, where -1 means False and 1 means True.\n",
    "\n",
    "Note that here we start to use more of the `SnorkelSession` querying syntax. Snorkel is built on [SQLAlchemy](http://www.sqlalchemy.org/), so their [querying API](http://docs.sqlalchemy.org/en/latest/orm/tutorial.html#querying) can be used in Snorkel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from snorkel.models import Span, Label\n",
    "from sqlalchemy.orm.exc import NoResultFound\n",
    "\n",
    "gold_labels = pd.read_csv('data/gold_labels.tsv', sep=\"\\t\")\n",
    "for index, row in gold_labels.iterrows():\n",
    "    try:\n",
    "        person1 = session.query(Span).filter(Span.stable_id == row['person1']).one()\n",
    "        person2 = session.query(Span).filter(Span.stable_id == row['person2']).one()\n",
    "    except NoResultFound as e:\n",
    "        if int(row['label']) == -1:\n",
    "            ### Due to variations in the NER output of CoreNLP, some of the included annotations for\n",
    "            ### false candidates might cover slightly different text spans when run on some systems,\n",
    "            ### so we must skip them.\n",
    "            continue\n",
    "        else:\n",
    "            raise e\n",
    "    candidate = session.query(Spouse).filter(Spouse.person1 == person1).filter(Spouse.person2 == person2).first()\n",
    "    if candidate is None:\n",
    "        candidate = session.query(Spouse).filter(Spouse.person1 == person2).filter(Spouse.person2 == person1).one()\n",
    "    \n",
    "    label = session.query(Label).filter(Label.candidate == candidate).one_or_none()\n",
    "    if label is None:\n",
    "        label = Label(candidate=candidate, key=gold_key, value=int(row['label']))\n",
    "        session.add(label)\n",
    "    else:\n",
    "        label.value = int(row['label'])\n",
    "session.commit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, in Part IV, we will build a model to predict these labels using data programming."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
{'fraction_non_alphanumeric': 0.17338770578462467, 'fraction_numerical': 0.0031717263253285004, 'mean_word_length': 2.8801640785232934, 'pattern_counts': {'":': 178, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3732976', 'n_tokens_mistral': 4592, 'n_tokens_neox': 4483, 'n_words': 1143}