<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intro Tutorial on Relation Extraction from News Articles"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part I: Preprocessing\n",
    "\n",
    "In this tutorial, we will walk through the process of using `Snorkel` to identify mentions of spouses in a corpus of news articles. The tutorial is broken up into 5 notebooks, each covering a step in the pipeline:\n",
    "1. Preprocessing\n",
    "2. Candidate Extraction\n",
    "3. Adding Labels\n",
    "4. Featurization & Training\n",
    "5. Evaluation\n",
    "\n",
    "In this notebook, we preprocess several documents using `Snorkel` utilities, parsing them into a simple hierarchy of component parts of our input data, which we refer to as _contexts_. We also extract standard linguistic features from each context which will be useful downstream using [CoreNLP](http://stanfordnlp.github.io/CoreNLP/), \n",
    "\n",
    "All of this preprocessed input data is saved to a database.  In Snorkel, if no database is specified, then a SQLite database is created by default--so no setup is needed here!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initializing a `SnorkelSession`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we initialize a `SnorkelSession`, which will enable us to save intermediate results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline\n",
    "\n",
    "from snorkel import SnorkelSession\n",
    "session = SnorkelSession()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the `Corpus`\n",
    "\n",
    "Next, we load and pre-process the corpus, storing it for convenience in a `Corpus` object."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Unarchive the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "os.system(\"tar -xzvf data/data.tar.gz\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configuring a `DocParser`\n",
    "\n",
    "We'll start by defining a `DocParser` class to read in Pubmed abstracts from [Pubtator]([Pubtator](http://www.ncbi.nlm.nih.gov/CBBresearch/Lu/Demo/PubTator/index.cgi)), where they are stored along with \"gold\" (i.e. hand-annotated by experts) *chemical* and *disease mention* annotations. We'll use the `XMLDocParser` class, which allows us to use [XPath queries](https://en.wikipedia.org/wiki/XPath) to specify the relevant sections of the XML format.\n",
    "\n",
    "_Note that we are newline-concatenating text from the title and abstract together for simplicity, but if we wanted to, we could easily extend the `DocParser` classes to preserve information about document structure._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from snorkel.parser import TSVDocParser\n",
    "doc_parser = TSVDocParser(path='data/articles-train.tsv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating a `SentenceParser`\n",
    "\n",
    "Next, we'll use an NLP preprocessing tool to split the `Document` objects into sentences, tokens, and provide annotations--part-of-speech tags, dependency parse structure, lemmatized word forms, etc.--for these sentences.  Here we use the default `SentenceParser` class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from snorkel.parser import SentenceParser\n",
    "\n",
    "sent_parser = SentenceParser()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pre-processing & loading the `Corpus`\n",
    "\n",
    "Finally, we'll put this all together using a `CorpusParser` object, which will execute the parsers and store the results as a `Corpus`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from snorkel.parser import CorpusParser\n",
    "\n",
    "cp = CorpusParser(doc_parser, sent_parser)\n",
    "%time corpus = cp.parse_corpus(session, 'News Training')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the printed stats are a property of the `Corpus` object, and can be printed again via the `corpus.stats()` method!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "doc = corpus.documents[0]\n",
    "doc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sent = doc.sentences[0]\n",
    "print unicode(sent)\n",
    "print unicode(sent.words)\n",
    "print sent.pos_tags"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Saving the `Corpus`\n",
    "Finally, we persist the parsed corpus in Snorkel's database backend:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "session.add(corpus)\n",
    "session.commit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Repeating for development and test corpora\n",
    "We will rerun the same operations for the other two News corpora: development and test. All we do is change the path that the `TSVDocParser` uses."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "for name, path in [('News Development', 'data/articles-dev.tsv'),\n",
    "                   ('News Test', 'data/articles-test.tsv')]:\n",
    "    doc_parser.path=path\n",
    "    %time corpus = cp.parse_corpus(session, name)\n",
    "    session.commit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, in Part 2, we will look at how to extract `Candidate` relations from our saved `Corpus`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "## This cell is just for speeding up automatic testing. You can safely ignore it!\n",
    "import os\n",
    "if 'CI' in os.environ:\n",
    "    from snorkel.models import Corpus\n",
    "    import random\n",
    "    for corpus_name in ['News Training', 'News Development', 'News Test']:\n",
    "        corpus = session.query(Corpus).filter(Corpus.name == corpus_name).one()\n",
    "        docs = set([d for d in corpus.documents])\n",
    "        for doc in docs:\n",
    "            if random.random() > .10:\n",
    "                corpus.remove(doc)\n",
    "    session.commit()"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
<commit_msg>Fix for unarchiving command.<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intro Tutorial on Relation Extraction from News Articles"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part I: Preprocessing\n",
    "\n",
    "In this tutorial, we will walk through the process of using `Snorkel` to identify mentions of spouses in a corpus of news articles. The tutorial is broken up into 5 notebooks, each covering a step in the pipeline:\n",
    "1. Preprocessing\n",
    "2. Candidate Extraction\n",
    "3. Adding Labels\n",
    "4. Featurization & Training\n",
    "5. Evaluation\n",
    "\n",
    "In this notebook, we preprocess several documents using `Snorkel` utilities, parsing them into a simple hierarchy of component parts of our input data, which we refer to as _contexts_. We also extract standard linguistic features from each context which will be useful downstream using [CoreNLP](http://stanfordnlp.github.io/CoreNLP/), \n",
    "\n",
    "All of this preprocessed input data is saved to a database.  In Snorkel, if no database is specified, then a SQLite database is created by default--so no setup is needed here!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initializing a `SnorkelSession`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we initialize a `SnorkelSession`, which will enable us to save intermediate results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline\n",
    "\n",
    "from snorkel import SnorkelSession\n",
    "session = SnorkelSession()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the `Corpus`\n",
    "\n",
    "Next, we load and pre-process the corpus, storing it for convenience in a `Corpus` object."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Unarchive the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import os\n",
    "os.system('cd data; tar -xzvf data.tar.gz')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configuring a `DocParser`\n",
    "\n",
    "We'll start by defining a `DocParser` class to read in Pubmed abstracts from [Pubtator]([Pubtator](http://www.ncbi.nlm.nih.gov/CBBresearch/Lu/Demo/PubTator/index.cgi)), where they are stored along with \"gold\" (i.e. hand-annotated by experts) *chemical* and *disease mention* annotations. We'll use the `XMLDocParser` class, which allows us to use [XPath queries](https://en.wikipedia.org/wiki/XPath) to specify the relevant sections of the XML format.\n",
    "\n",
    "_Note that we are newline-concatenating text from the title and abstract together for simplicity, but if we wanted to, we could easily extend the `DocParser` classes to preserve information about document structure._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from snorkel.parser import TSVDocParser\n",
    "doc_parser = TSVDocParser(path='data/articles-train.tsv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating a `SentenceParser`\n",
    "\n",
    "Next, we'll use an NLP preprocessing tool to split the `Document` objects into sentences, tokens, and provide annotations--part-of-speech tags, dependency parse structure, lemmatized word forms, etc.--for these sentences.  Here we use the default `SentenceParser` class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from snorkel.parser import SentenceParser\n",
    "\n",
    "sent_parser = SentenceParser()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pre-processing & loading the `Corpus`\n",
    "\n",
    "Finally, we'll put this all together using a `CorpusParser` object, which will execute the parsers and store the results as a `Corpus`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from snorkel.parser import CorpusParser\n",
    "\n",
    "cp = CorpusParser(doc_parser, sent_parser)\n",
    "%time corpus = cp.parse_corpus(session, 'News Training')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the printed stats are a property of the `Corpus` object, and can be printed again via the `corpus.stats()` method!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "doc = corpus.documents[0]\n",
    "doc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sent = doc.sentences[0]\n",
    "print unicode(sent)\n",
    "print unicode(sent.words)\n",
    "print sent.pos_tags"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Saving the `Corpus`\n",
    "Finally, we persist the parsed corpus in Snorkel's database backend:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "session.add(corpus)\n",
    "session.commit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Repeating for development and test corpora\n",
    "We will rerun the same operations for the other two News corpora: development and test. All we do is change the path that the `TSVDocParser` uses."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "for name, path in [('News Development', 'data/articles-dev.tsv'),\n",
    "                   ('News Test', 'data/articles-test.tsv')]:\n",
    "    doc_parser.path=path\n",
    "    %time corpus = cp.parse_corpus(session, name)\n",
    "    session.commit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, in Part 2, we will look at how to extract `Candidate` relations from our saved `Corpus`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "## This cell is just for speeding up automatic testing. You can safely ignore it!\n",
    "import os\n",
    "if 'CI' in os.environ:\n",
    "    from snorkel.models import Corpus\n",
    "    import random\n",
    "    for corpus_name in ['News Training', 'News Development', 'News Test']:\n",
    "        corpus = session.query(Corpus).filter(Corpus.name == corpus_name).one()\n",
    "        docs = set([d for d in corpus.documents])\n",
    "        for doc in docs:\n",
    "            if random.random() > .10:\n",
    "                corpus.remove(doc)\n",
    "    session.commit()"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}

{'fraction_non_alphanumeric': 0.16943287407882088, 'fraction_numerical': 0.0028196090996475487, 'mean_word_length': 2.857142857142857, 'pattern_counts': {'":': 236, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1443670', 'n_tokens_mistral': 5455, 'n_tokens_neox': 5262, 'n_words': 1443}