<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "We'll be applying naive bayes to determine whether a tweet was written by [<NAME>](https://twitter.com/realDonaldTrump) or [<NAME>](https://twitter.com/HillaryClinton). For a more in-depth theoretical explanation of the theory behind the classifier, see [our notes](https://github.com/MachinesWhoLearn/lectures/blob/master/2016-2017.Meetings/05.DIY_naive_bayes/naive_bayes_primer/naive_bayes_primer.pdf).\n",
    "\n",
    "### Features\n",
    "In our model, the features are the individual words. For example, we'd expect that the word \"wall\" would be more likely to appear in Trump tweets, so we want to account for that in our model. To start off with, we'll load the text data and then perform some basic tokenization to separate out all the words."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tokenizing and Counting the Words\n",
    "We want to transform our collection of tweets into something that the model can understand. A basic idea from natural language processing (NLP) is the bag of words approach (BOW). When we use the bag of words, we simply count the number of times a word occurs in a document, and divide it by the total number of words. Doing this for each word, we can get a probability distribution for the probability of a word occurring."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start by loading our training data (the raw tweets) and the associated labels (indicating authorship, \"0\" for Hillary and \"1\" for Trump). We'll then count up all the statistics we'll need to use later for calculating probabilities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# setting up some imports and some miscellaneous helper methods we'll be using\n",
    "import re\n",
    "import string\n",
    "from math import log\n",
    "from __future__ import print_function\n",
    "\n",
    "# our default smoothing value, we pretend we see each word at\n",
    "# least once so all our probabilities are well-formed \n",
    "# (so no multiplying by 0 if a word isn't in train set)\n",
    "SMOOTHING = 1.0\n",
    "\n",
    "# location of data file relative to this notebook path\n",
    "TRAIN_DATA_PATH = \"../data/tweets_train.txt\"\n",
    "TRAIN_LABELS_PATH = \"../data/labels_train.txt\"\n",
    "TEST_DATA_PATH = \"../data/tweets_test.txt\"\n",
    "TEST_LABELS_PATH = \"../data/labels_test.txt\"\n",
    "\n",
    "# strip punctuation from a string\n",
    "def remove_punctuation(input_str):\n",
    "    \"from http://stackoverflow.com/questions/265960/best-way-to-strip-punctuation-from-a-string-in-python\"\n",
    "    table = string.maketrans(\"\",\"\")\n",
    "    return input_str.translate(table, string.punctuation)\n",
    "\n",
    "# split a line into tokens, or a list of words (known as tokens) \n",
    "# that are separated by whitespace\n",
    "def tokenize(line):\n",
    "    line = remove_punctuation(line)\n",
    "    line = line.lower().strip()\n",
    "    # re.split has an odd tendency to add empty strings, remove those\n",
    "    return [token for token in re.split(\"\\W+\", line) if token != '']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Setting up some the data structures we'll use to keep track of counts\n",
    "\n",
    "# this dictionary maps the numeric label to the candidate name\n",
    "# the \"\\n\" after each label accounts for the newline character \n",
    "# that follows each one\n",
    "LABEL_MAP = {'0\\n': 'hillary', '1\\n': 'trump'}\n",
    "\n",
    "# this dictionary keeps track of the number of times\n",
    "# word appears in a trump or hillary tweet\n",
    "word_counts_per_candidate = {\n",
    "    \"trump\": {},\n",
    "    \"hillary\": {}\n",
    "}\n",
    "# this dictionary keeps track of how many trump tweets there are and how many\n",
    "# hillary tweets there are. These are our priors!\n",
    "total_tweet_count = {'trump': 0.0, 'hillary': 0.0}\n",
    "\n",
    "# counts words across all tweets\n",
    "vocabulary = {}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a moment to verify that `tokenize` does what we want it to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['i',\n",
       " 'am',\n",
       " 'growing',\n",
       " 'the',\n",
       " 'republican',\n",
       " 'party',\n",
       " 'tremendously',\n",
       " 'just',\n",
       " 'look',\n",
       " 'at',\n",
       " 'the',\n",
       " 'numbers',\n",
       " 'way',\n",
       " 'up',\n",
       " 'democrats',\n",
       " 'numbers',\n",
       " 'are',\n",
       " 'significantly',\n",
       " 'down',\n",
       " 'from',\n",
       " 'years',\n",
       " 'past']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_tweet = \"I am growing the Republican Party tremendously - just look at the numbers, way up! Democrats numbers are significantly down from years past.\"\n",
    "sample_tokens = tokenize(sample_tweet)\n",
    "sample_tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# open the file with train data and refer to it as \"data\", and\n",
    "# open the file with train labels and refer to it as \"labels\"\n",
    "with open(TRAIN_DATA_PATH, 'r') as data, open(TRAIN_LABELS_PATH, 'r') as labels:\n",
    "    # zip data and labels together to create a list of tuples of (tweet,label),\n",
    "    # which we then iterate over. the tweet variable refers to an individual tweet\n",
    "    # and the label variable refers to an individual label\n",
    "    for tweet, label in zip(data, labels):\n",
    "        # if it is a Hillary tweet, increment the \"hillary\" element of `total_tweet_count`\n",
    "        # if it is a Trump tweet, increment the \"trump\" element of `total_tweet_count`.\n",
    "        total_tweet_count[LABEL_MAP[label]] += 1.0\n",
    "        \n",
    "        tokens = tokenize(tweet)\n",
    "        \n",
    "        # iterate over the words we found in the tweet\n",
    "        for word in tokens:\n",
    "            # if we haven't seen a word yet, let's add it to our dictionary with a \n",
    "            # count of 2 * how much we smoothing up by (`SMOOTHING`),\n",
    "            # (since we presume that both hillary and trump have seen it)\n",
    "            # we'll be incrementing it later\n",
    "            if word not in vocabulary:\n",
    "                vocabulary[word] = 4 * SMOOTHING\n",
    "                # similarily, since we pretend that we saw an extra tweet \n",
    "                # with this unknown word and one without, we add 2*SMOOTHING\n",
    "                # to hillary and trump\n",
    "                total_tweet_count[\"trump\"] += 2*SMOOTHING\n",
    "                total_tweet_count[\"hillary\"] += 2*SMOOTHING\n",
    "                word_counts_per_candidate[\"trump\"][word] = 1\n",
    "                word_counts_per_candidate[\"hillary\"][word] = 1\n",
    "            # now add one to the global dictionary\n",
    "            # and the associated candidate's\n",
    "            # dictionary\n",
    "            vocabulary[word] += 1\n",
    "            word_counts_per_candidate[LABEL_MAP[label]][word] += 1\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at our output / a bit of our output for the long dictionaries"
   ]<commit_msg>fix smoothing bug once again<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "We'll be applying naive bayes to determine whether a tweet was written by [<NAME>](https://twitter.com/realDonaldTrump) or [<NAME>](https://twitter.com/HillaryClinton). For a more in-depth theoretical explanation of the theory behind the classifier, see [our notes](https://github.com/MachinesWhoLearn/lectures/blob/master/2016-2017.Meetings/05.DIY_naive_bayes/naive_bayes_primer/naive_bayes_primer.pdf).\n",
    "\n",
    "### Features\n",
    "In our model, the features are the individual words. For example, we'd expect that the word \"wall\" would be more likely to appear in Trump tweets, so we want to account for that in our model. To start off with, we'll load the text data and then perform some basic tokenization to separate out all the words."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tokenizing and Counting the Words\n",
    "We want to transform our collection of tweets into something that the model can understand. A basic idea from natural language processing (NLP) is the bag of words approach (BOW). When we use the bag of words, we simply count the number of times a word occurs in a document, and divide it by the total number of words. Doing this for each word, we can get a probability distribution for the probability of a word occurring."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start by loading our training data (the raw tweets) and the associated labels (indicating authorship, \"0\" for Hillary and \"1\" for Trump). We'll then count up all the statistics we'll need to use later for calculating probabilities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# setting up some imports and some miscellaneous helper methods we'll be using\n",
    "import re\n",
    "import string\n",
    "from math import log\n",
    "from __future__ import print_function\n",
    "\n",
    "# our default smoothing value, we pretend we see each word at\n",
    "# least once so all our probabilities are well-formed \n",
    "# (so no multiplying by 0 if a word isn't in train set)\n",
    "SMOOTHING = 1.0\n",
    "\n",
    "# location of data file relative to this notebook path\n",
    "TRAIN_DATA_PATH = \"../data/tweets_train.txt\"\n",
    "TRAIN_LABELS_PATH = \"../data/labels_train.txt\"\n",
    "TEST_DATA_PATH = \"../data/tweets_test.txt\"\n",
    "TEST_LABELS_PATH = \"../data/labels_test.txt\"\n",
    "\n",
    "# strip punctuation from a string\n",
    "def remove_punctuation(input_str):\n",
    "    \"from http://stackoverflow.com/questions/265960/best-way-to-strip-punctuation-from-a-string-in-python\"\n",
    "    table = string.maketrans(\"\",\"\")\n",
    "    return input_str.translate(table, string.punctuation)\n",
    "\n",
    "# split a line into tokens, or a list of words (known as tokens) \n",
    "# that are separated by whitespace\n",
    "def tokenize(line):\n",
    "    line = remove_punctuation(line)\n",
    "    line = line.lower().strip()\n",
    "    # re.split has an odd tendency to add empty strings, remove those\n",
    "    return [token for token in re.split(\"\\W+\", line) if token != '']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Setting up some the data structures we'll use to keep track of counts\n",
    "\n",
    "# this dictionary maps the numeric label to the candidate name\n",
    "# the \"\\n\" after each label accounts for the newline character \n",
    "# that follows each one\n",
    "LABEL_MAP = {'0\\n': 'hillary', '1\\n': 'trump'}\n",
    "\n",
    "# this dictionary keeps track of the number of times\n",
    "# word appears in a trump or hillary tweet\n",
    "word_counts_per_candidate = {\n",
    "    \"trump\": {},\n",
    "    \"hillary\": {}\n",
    "}\n",
    "# this dictionary keeps track of how many trump tweets there are and how many\n",
    "# hillary tweets there are. These are our priors!\n",
    "total_tweet_count = {'trump': 0.0, 'hillary': 0.0}\n",
    "\n",
    "# counts words across all tweets\n",
    "vocabulary = {}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a moment to verify that `tokenize` does what we want it to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['i',\n",
       " 'am',\n",
       " 'growing',\n",
       " 'the',\n",
       " 'republican',\n",
       " 'party',\n",
       " 'tremendously',\n",
       " 'just',\n",
       " 'look',\n",
       " 'at',\n",
       " 'the',\n",
       " 'numbers',\n",
       " 'way',\n",
       " 'up',\n",
       " 'democrats',\n",
       " 'numbers',\n",
       " 'are',\n",
       " 'significantly',\n",
       " 'down',\n",
       " 'from',\n",
       " 'years',\n",
       " 'past']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_tweet = \"I am growing the Republican Party tremendously - just look at the numbers, way up! Democrats numbers are significantly down from years past.\"\n",
    "sample_tokens = tokenize(sample_tweet)\n",
    "sample_tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# open the file with train data and refer to it as \"data\", and\n",
    "# open the file with train labels and refer to it as \"labels\"\n",
    "with open(TRAIN_DATA_PATH, 'r') as data, open(TRAIN_LABELS_PATH, 'r') as labels:\n",
    "    # zip data and labels together to create a list of tuples of (tweet,label),\n",
    "    # which we then iterate over. the tweet variable refers to an individual tweet\n",
    "    # and the label variable refers to an individual label\n",
    "    for tweet, label in zip(data, labels):\n",
    "        # if it is a Hillary tweet, increment the \"hillary\" element of `total_tweet_count`\n",
    "        # if it is a Trump tweet, increment the \"trump\" element of `total_tweet_count`.\n",
    "        total_tweet_count[LABEL_MAP[label]] += 1.0\n",
    "        \n",
    "        tokens = tokenize(tweet)\n",
    "        \n",
    "        # iterate over the words we found in the tweet\n",
    "        for word in tokens:\n",
    "            # if we haven't seen a word yet, let's add it to our dictionary with a \n",
    "            # count of 2 * how much we smoothing up by (`SMOOTHING`),\n",
    "            # (since we presume that both hillary and trump have seen it)\n",
    "            # we'll be incrementing it later\n",
    "            if word not in vocabulary:\n",
    "                vocabulary[word] = 2 * SMOOTHING\n",
    "                # similarily, since we pretend that we saw an extra tweet \n",
    "                # with this unknown word and one without, we add 2*SMOOTHING\n",
    "                # to hillary and trump\n",
    "                total_tweet_count[\"trump\"] += 2*SMOOTHING\n",
    "                total_tweet_count[\"hillary\"] += 2*SMOOTHING\n",
    "                word_counts_per_candidate[\"trump\"][word] = 1\n",
    "                word_counts_per_candidate[\"hillary\"][word] = 1\n",
    "            # now add one to the global dictionary\n",
    "            # and the associated candidate's\n",
    "            # dictionary\n",
    "            vocabulary[word] += 1\n",
    "            word_counts_per_candidate[LABEL_MAP[label]][word] += 1\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at our output / a bit of our output for the long dictionaries"
   ]
{'fraction_non_alphanumeric': 0.15477750733320852, 'fraction_numerical': 0.005367284528490295, 'mean_word_length': 2.3999575641841715, 'pattern_counts': {'":': 94, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1122165', 'n_tokens_mistral': 5391, 'n_tokens_neox': 5143, 'n_words': 1628}