<commit_before><commit_msg>This notebook returns the email where a string first occurred<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "find first occurrence of ONE string in ONE or MORE mailing lists....PAM!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from bigbang.archive import load as load_archive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#insert ONE OR MORE urls of mailing lists\n",
    "urls = [\"http://lists.ncuc.org/pipermail/ncuc-discuss/\",\n",
    "       \"http://mm.icann.org/pipermail/wp4/\"]\n",
    "\n",
    "\n",
    "try:\n",
    "    arch_paths =[]\n",
    "    for url in urls:\n",
    "        arch_paths.append('../archives/'+url[:-1].replace('://','_/')+'.csv')\n",
    "    archives = [load_archive(arch_path).data for arch_path in arch_paths]\n",
    "except:\n",
    "    arch_paths =[]\n",
    "    for url in urls:\n",
    "        arch_paths.append('../archives/'+url[:-1].replace('//','/')+'.csv')\n",
    "    archives = [load_archive(arch_path).data for arch_path in arch_paths]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "checkword = 'Freedom of expression'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_first_occurrence(word,mailing_list):\n",
    "    for mail in mailing_list.iterrows():\n",
    "        if str.lower(word) in str.lower(mail[1][\"Body\"]):\n",
    "            return mail"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "http://lists.ncuc.org/pipermail/ncuc-discuss/\n",
      "2002-06-26 21:38:13\n",
      "CCHIU at aclu.org (<NAME>)\n",
      "A report entitled Success by Default suggests, among other things, that\n",
      "several changes should be made in the UDRP to protect freedom of expression\n",
      "online. For example, the study recommends that the \"standards for a finding\n",
      "of 'confusing similarity' to a trademark\" should be clarified, and that\n",
      "\"[d]omain names \"that signal criticism, parody, or commentary upon products\n",
      "or companies ... should not be classified as 'confusing' unless they are\n",
      "used in ways that actively promote fraud, deception or confusion.\" Success\n",
      "by Default is based on information contained in a special UDRP case tracking\n",
      "database, both of which were created by the Convergence Center of Syracuse\n",
      "University. \n",
      "\n",
      "See\n",
      "http://www.internetdemocracyproject.org/#highlights\n",
      "\n",
      "Sincerely,\n",
      "<NAME>\n",
      "Global Internet Liberty Campaign Organizer\n",
      "American Civil Liberties Union\n",
      "****************************************************************\n",
      "http://mm.icann.org/pipermail/wp4/\n",
      "2015-08-06 19:34:47\n",
      "gregshatanipc at gmail.com (<NAME>)\n",
      "I really don't think using a 500 year old colloquialism \"cheapens the\n",
      "discussion.\"  I also find it ironic to find this admonition in a discussion\n",
      "about, inter alia, freedom of expression.  Political correctness, trigger\n",
      "words, and other such stuff are a not inconsequential threat to freedom of\n",
      "expression and freedom of ideas -- starting with a kernel of good\n",
      "intentions, and then becoming quite damaging....\n",
      "\n",
      "On Thu, Aug 6, 2015 at 3:05 PM, <NAME> <avri at acm.org> wrote:\n",
      "\n",
      ">\n",
      ">\n",
      "> On 06-Aug-15 14:28, <NAME> wrote:\n",
      "> > Otherwise, we are just buying a pig in a poke.\n",
      ">\n",
      ">\n",
      "> I really have trouble with such a reference to human rights.  Please do\n",
      "> not cheapen this discussion.\n",
      ">\n",
      "> avri\n",
      ">\n",
      ">\n",
      "> ---\n",
      "> This email has been checked for viruses by Avast antivirus software.\n",
      "> https://www.avast.com/antivirus\n",
      ">\n",
      "> _______________________________________________\n",
      "> Wp4 mailing list\n",
      "> Wp4 at icann.org\n",
      "> https://mm.icann.org/mailman/listinfo/wp4\n",
      ">\n",
      "-------------- next part --------------\n",
      "An HTML attachment was scrubbed...\n",
      "URL: <http://mm.icann.org/pipermail/wp4/attachments/20150806/2289b836/attachment-0001.html>\n",
      "****************************************************************\n"
     ]
    }
   ],
   "source": [
    "for i in range(0,len(archives)):\n",
    "    first_mail = get_first_occurrence(checkword, archives[i])\n",
    "    print urls[i]\n",
    "    print first_mail[1][\"Date\"]\n",
    "    print first_mail[1][\"From\"]\n",
    "    print first_mail[1][\"Body\"]\n",
    "    print \"****************************************************************\"\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}

{'fraction_non_alphanumeric': 0.2152070604209097, 'fraction_numerical': 0.016463000678886626, 'mean_word_length': 2.5035671819262784, 'pattern_counts': {'":': 60, '<': 10, '<?xml version=': 0, '>': 32, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7313985', 'n_tokens_mistral': 2110, 'n_tokens_neox': 1972, 'n_words': 486}