<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Step 1. Run content extraction on html document"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the following example, we will run the `readability` extractor and the `title` extractor on the html document.  \n",
    "The extraction is controlled by an `extraction_config` and the input is json document with `html` document embedded in it.  \n",
    "\n",
    "Let's breakdown the extraction_config:  \n",
    "> `input_path`  \n",
    "\n",
    "specifies the path where the html content is in the json  \n",
    "> `extractors.readability`  \n",
    "\n",
    "sets up `etk` to run readability extractor on the `input_path`\n",
    "\n",
    "> strict: yes\n",
    "\n",
    "makes `readability` extractor more precise. ```strict: no``` will result in more content getting extracted from the html as main content  \n",
    "\n",
    "#### Output  \n",
    "The output will be under the field `content_extraction` in the input json document. It should have `content_relaxed` corresponding to ```strict: no``` and `content_strict` for ```strict: yes```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{   'content_relaxed': {   'text': u\"\\n \\n \\n \\n \\n \\n \\n smoothlegs24  28 \\n \\n \\n chrissy391  27 \\n \\n \\n My name is Helena height 160cms weight 55 kilos  contact me at <EMAIL>           <NAME>         age: 23 HrumpMeNow  28 \\n \\n \\n xxtradition  24 \\n \\n \\n jumblyjumb  26 \\n \\n \\n claudia77  26 \\n \\n \\n gushinPuss  28 \\n \\n \\n Littlexdit  25 \\n \\n \\n PinkSweets2  28 \\n \\n \\n withoutlimit  27 \\n \\n \\n bothOfUs3  28 \\n \\n \\n lovelylips  27 \\n \\n \\n killerbod  27 \\n \\n \\n Littlexdit  27 \\n \\n \\n azneyes  23 \\n \\n \\n \\n \\n \\n Escort's Phone: \\n \\n \\n323-452-2013  \\n \\n Escort's Location: \\nLos Angeles, California  \\n Escort's Age:   23   Date of Escort Post:   Jan 02nd 6:46am \\n REVIEWS:   \\n READ AND CREATE REVIEWS FOR THIS ESCORT   \\n \\n \\n \\n \\n \\nThere are  50  girls looking in  .\\n VIEW GIRLS \\n \\nHey I'm luna 3234522013 Let's explore , embrace and indulge in your favorite fantasy  % independent. discreet no drama Firm Thighs and Sexy. My Soft skin & Tight Grip is exactly what you deserve Call or text   Fetish friendly   Fantasy friendly   Party friendly 140 Hr SPECIALS 3234522013.\\xa0Call  323-452-2013 .  Me and my friends are on EZsex  soooo you can find us all on there if you want... skittlegirl \\n \\xa0\\xa0\\n \\n \\xa0\\xa0\\n \\n \\xa0\\xa0\\n Call me on my cell at 323-452-2013. Date of ad: 2017-01-02 06:46:00 \\n \\n \\n \\n\"},\n",
      "    'content_strict': {   'text': u\"\\n \\n \\n \\n \\n \\n smoothlegs24  28 \\n \\n \\n chrissy391  27 \\n \\n \\n My name is Helena height 160cms weight 55 kilos  contact me at <EMAIL>           jefferson ave         age: 23 HrumpMeNow  28 \\n \\n \\n xxtradition  24 \\n \\n \\n jumblyjumb  26 \\n \\n \\n claudia77  26 \\n \\n \\n gushinPuss  28 \\n \\n \\n Littlexdit  25 \\n \\n \\n PinkSweets2  28 \\n \\n \\n withoutlimit  27 \\n \\n \\n bothOfUs3  28 \\n \\n \\n lovelylips  27 \\n \\n \\n killerbod  27 \\n \\n \\n Littlexdit  27 \\n \\n \\n azneyes  23 \\n \\n \\n \\n \\n \\n Escort's Phone: \\n \\n \\n323-452-2013  \\n \\n Escort's Location: \\nLos Angeles, California  \\n Escort's Age:   23   Date of Escort Post:   Jan 02nd 6:46am \\n REVIEWS:   \\n READ AND CREATE REVIEWS FOR THIS ESCORT   \\n \\n \\n \\n \\n \\nThere are  50  girls looking in  .\\n VIEW GIRLS \\n \\nHey I'm luna 3234522013 Let's explore , embrace and indulge in your favorite fantasy  % independent. discreet no drama Firm Thighs and Sexy. My Soft skin & Tight Grip is exactly what you deserve Call or text   Fetish friendly   Fantasy friendly   Party friendly 140 Hr SPECIALS 3234522013.\\xa0Call  323-452-2013 .  Me and my friends are on EZsex  soooo you can find us all on there if you want... skittlegirl \\n \\xa0\\xa0\\n \\n \\xa0\\xa0\\n \\n \\xa0\\xa0\\n Call me on my cell at 323-452-2013. Date of ad: 2017-01-02 06:46:00 \\n \\n \\n\"}}\n"
     ]
    }
   ],
   "source": [
    "from etk.core import Core\n",
    "import  json\n",
    "import pprint\n",
    "\n",
    "e_config = {'content_extraction': {\n",
    "            \"input_path\": \"raw_content\",\n",
    "            \"extractors\": {\n",
    "              \"readability\": [\n",
    "                {\n",
    "                  \"strict\": \"yes\",\n",
    "                  \"extraction_policy\": \"keep_existing\"\n",
    "                },\n",
    "                {\n",
    "                  \"strict\": \"no\",\n",
    "                  \"extraction_policy\": \"keep_existing\",\n",
    "                  \"field_name\": \"content_relaxed\"\n",
    "                }\n",
    "              ]\n",
    "            }\n",
    "          }\n",
    "        }\n",
    "doc = {\"raw_content\": \"\\n<!DOCTYPE html PUBLIC \\\"-//W3C//DTD XHTML 1.0 Strict//EN\\\" \\\"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd\\\">\\n<html xmlns=\\\"http://www.w3.org/1999/xhtml\\\" lang=\\\"en\\\" xml:lang=\\\"en\\\">\\n<head>\\n<title>323-452-2013 ESCORT ALERT! - Luna The Hot Playmate (323) 452-2013 - 23</title>\\n<meta name=\\\"netinsert\\\" content=\\\"0.0.1.13.13.1\\\">\\n<meta name=\\\"description\\\" content=\\\"323-452-2013 Escort listings. View all of her listings at once. On this page you will see a history for 323-452-2013 and all of the escort ads placed with this phone number. Specifically in losangeles: 323-452-2013 ESCORT ALERT! - Luna The Hot Playmate (323) 452-2013 - 23\\\"/>\\n<meta name=\\\"keywords\\\" content=\\\"323-452-2013, escort listing, escort ad, escorts in Los angeles, 323-452-2013 in Los angeles, escorts in losangeles, call 323-452-2013, phone number 323-452-2013, craigslist escorts, backpage escorts, listings\\\"/>\\n<script type=\\\"text/javascript\\\">\\n//<![CDATA[\\ntry{if (!window.CloudFlare) {var CloudFlare=[{verbose:0,p:0,byc:0,owlid:\\\"cf\\\",bag2:1,mirage2:0,oracle:0,paths:{cloudflare:\\\"/cdn-cgi/nexp/dok3v=1613a3a185/\\\"},atok:\\\"e6dd48a0eec39904cfe3144aa9a603eb\\\",petok:\\\"070d45bb85df778c1b97f07da69638091ee5d4ed-1483380159-1800\\\",zone:\\\"eroticmugshots.com\\\",rocket:\\\"a\\\",apps:{}}];document.write('<script type=\\\"text/javascript\\\" src=\\\"//ajax.cloudflare.com/cdn-cgi/nexp/dok3v=f2befc48d1/cloudflare.min.js\\\"><'+'\\\\/script>');}}catch(e){};\\n//]]>\\n</script>\\n<link rel=\\\"stylesheet\\\" type=\\\"text/css\\\" href=\\\"/css/www-main.css\\\"/>\\n<script type=\\\"text/rocketscript\\\">\\n  var _paq = _paq || [];\\n  _paq.push([\\\"setDomains\\\", [\\\"*.eroticmugshots.com\\\"]]);\\n  _paq.push(['trackPageView']);\\n  _paq.push(['enableLinkTracking']);\\n  (function() {\\n    var u=\\\"//a.blr.pw/\\\";\\n    _paq.push(['setTrackerUrl', u+'piwik.php']);\\n    _paq.push(['setSiteId', 10]);\\n    var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0];\\n    g.type='text/javascript'; g.async=true; g.defer=true; g.src=u+'piwik.js'; s.parentNode.insertBefore(g,s);\\n  })();\\n</script>\\n<noscript><p><img src=\\\"//a.blr.pw/piwik.php?idsite=10\\\" style=\\\"border:0;\\\" alt=\\\"\\\"/></p></noscript>\\n</head>\\n<body>\\n<div id=\\\"PageContainer\\\">\\n<div id=\\\"logo\\\">\\n<div style=\\\"float:left;\\\">\\n<a href=\\\"/index.php?cities=1\\\"><img src=\\\"/images/ems.png\\\" alt=\\\"All Cities\\\"/></a>\\n</div>\\n<div class='city_name'>\\n<h2><strong>323-452-2013</strong> Los Angeles Escort Listing</h2><a href='/index.php?cities=1'>View Escorts in other cities</a> </div>\\n</div>\\n<div class=\\\"profileNav\\\">\\n</div>\\n<script language=\\\"javascript\\\" type=\\\"text/rocketscript\\\">\\nfunction alertContents(){\\n\\tif (http_request.readyState == 4){\\n\\t\\tif(http_request.status == 200){\\n\\t\\t\\tresult = http_request.responseText;\\n\\t\\t\\talert(result);\\n\\t\\t\\tif(result == 'favorite-added'){\\n\\t\\t\\t\\tdocument.getElementById('favimg').src = '/images/addedto_favorite.png';\\n\\t\\t\\t\\tdocument.getElementById('favimg').onclick = function() { add_fav('viewfavorites'); };\\n\\t\\t\\t}else if(result == 'registration-needed'){\\n\\t\\t\\t\\twindow.location = \\\"http://eroticmugshots.com/registration.php\\\";\\n\\t\\t\\t}else if(result == 'error'){\\n\\t\\t\\t\\t/* alert('There was an error with your request. Please contact the webmaster at <EMAIL>.'); */\\n\\t\\t\\t}else{\\n\\t\\t\\t\\twindow.location = \\\"http://eroticmugshots.com/404page.php\\\";\\n\\t\\t\\t}\\n\\t\\t}\\n\\t}\\n}\\n\\nfunction delete_post(pid,ban){\\n\\tvar poststr = \\\"pid=\\\" + encodeURI(pid) + \\\"&p=\\\" + encodeURI('D1003D') + \\\"&ban=\\\" + encodeURI(ban);\\n\\tbeing_xed = pid;\\n\\tmakePOSTRequest('/m5/deletePostm5.php', poststr);\\n}\\n\\t\\nfunction add_fav(bpid){\\n\\tif(bpid=='viewfavorites'){\\n\\t\\twindow.location = \\\"http://eroticmugshots.com/favorites.php\\\";\\n\\t\\texit();\\n\\t}\\n\\tvar poststr = \\\"bpid=\\\" + encodeURI(bpid);\\n\\tmakePOSTRequest('/add_favorite.php', poststr);\\n}\\n\\n</script>\\n<script language=\\\"javascript\\\" data-rocketsrc=\\\"/ajax/http_request.ajax.js\\\" type=\\\"text/rocketscript\\\"></script>\\n<style>.largeHeadline{font-family:times,Times New Roman,times-roman,georgia,serif;color:#444;margin:0;padding:0px;font-size:31px;line-height:44px;letter-spacing:-2px;font-weight:bold;}.fav{cursor:pointer;}</style>\\n<div class='listingsAdvertContainer' style=\\\"clear:none;\\\">\\n<div class='container'>\\n<div style='clear:left;'>\\n<div class='adclass ps'><a href='/otra.php?c=14&a=postthumbs' rel='nofollow'><img src='http://www.eroticmugshots.com/adthumbs/87.jpg' width='110px' height='110px'/><br/>smoothlegs24</a> 28<br/></div>\\n</div>\\n<div style='clear:left;'>\\n<div class='adclass ps'><a href='/otra.php?c=14&a=postthumbs' rel='nofollow'><img src='http://www.eroticmugshots.com/adthumbs/30.jpg' width='110px' height='110px'/><br/>chrissy391</a> 27<br/></div>\\n</div>\\n<div style='clear:left;'>\\n<div class='adclass ps'><a href='/otra.php?c=14&a=postthumbs' rel='nofollow'><img src='http://www.eroticmugshots.com/adthumbs/47.jpg' width='110px' height='110px'/>My name is Helena height 160cms weight 55 kilos  contact me at <EMAIL>           jefferson ave         age: 23<br/>HrumpMeNow</a> 28<br/></div>\\n</div>\\n<div style='clear:left;'>\\n<div class='adclass ps'><a href='/otra.php?c=14&a=postthumbs' rel='nofollow'><img src='http://www.eroticmugshots.com/adthumbs/50.jpg' width='110px' height='110px'/><br/>xxtradition</a> 24<br/></div>\\n</div>\\n<div style='clear:left;'>\\n<div class='adclass ps'><a href='/otra.php?c=14&a=postthumbs' rel='nofollow'><img src='http://www.eroticmugshots.com/adthumbs/115.jpg' width='110px' height='110px'/><br/>jumblyjumb</a> 26<br/></div>\\n</div>\\n<div style='clear:left;'>\\n<div class='adclass ps'><a href='/otra.php?c=14&a=postthumbs' rel='nofollow'><img src='http://www.eroticmugshots.com/adthumbs/199.jpg' width='110px' height='110px'/><br/>claudia77</a> 26<br/></div>\\n</div>\\n<div style='clear:left;'>\\n<div class='adclass ps'><a href='/otra.php?c=14&a=postthumbs' rel='nofollow'><img src='http://www.eroticmugshots.com/adthumbs/11.jpg' width='110px' height='110px'/><br/>gushinPuss</a> 28<br/></div>\\n</div>\\n<div style='clear:left;'>\\n<div class='adclass ps'><a href='/otra.php?c=14&a=postthumbs' rel='nofollow'><img src='http://www.eroticmugshots.com/adthumbs/256.jpg' width='110px' height='110px'/><br/>Littlexdit</a> 25<br/></div>\\n</div>\\n<div style='clear:left;'>\\n<div class='adclass ps'><a href='/otra.php?c=14&a=postthumbs' rel='nofollow'><img src='http://www.eroticmugshots.com/adthumbs/169.jpg' width='110px' height='110px'/><br/>PinkSweets2</a> 28<br/></div>\\n</div>\\n<div style='clear:left;'>\\n<div class='adclass ps'><a href='/otra.php?c=14&a=postthumbs' rel='nofollow'><img src='http://www.eroticmugshots.com/adthumbs/185.jpg' width='110px' height='110px'/><br/>withoutlimit</a> 27<br/></div>\\n</div>\\n<div style='clear:left;'>\\n<div class='adclass ps'><a href='/otra.php?c=14&a=postthumbs' rel='nofollow'><img src='http://www.eroticmugshots.com/adthumbs/14.jpg' width='110px' height='110px'/><br/>bothOfUs3</a> 28<br/></div>\\n</div>\\n<div style='clear:left;'>\\n<div class='adclass ps'><a href='/otra.php?c=14&a=postthumbs' rel='nofollow'><img src='http://www.eroticmugshots.com/adthumbs/53.jpg' width='110px' height='110px'/><br/>lovelylips</a> 27<br/></div>\\n</div>\\n<div style='clear:left;'>\\n<div class='adclass ps'><a href='/otra.php?c=14&a=postthumbs' rel='nofollow'><img src='http://www.eroticmugshots.com/adthumbs/235.jpg' width='110px' height='110px'/><br/>killerbod</a> 27<br/></div>\\n</div>\\n<div style='clear:left;'>\\n<div class='adclass ps'><a href='/otra.php?c=14&a=postthumbs' rel='nofollow'><img src='http://www.eroticmugshots.com/adthumbs/95.jpg' width='110px' height='110px'/><br/>Littlexdit</a> 27<br/></div>\\n</div>\\n<div style='clear:left;'>\\n<div class='adclass ps'><a href='/otra.php?c=14&a=postthumbs' rel='nofollow'><img src='http://www.eroticmugshots.com/adthumbs/255.jpg' width='110px' height='110px'/><br/>azneyes</a> 23<br/></div>\\n</div>\\n</div>\\n</div>\\n<div class='listingsThumbnailContainer' style=\\\"clear:none;\\\">\\n<span class=\\\"mugshotTitle\\\"><h1 class=\\\"mugshot_title_h1\\\"><a href=\\\"?pid=89701497\\\">323-452-2013 ESCORT ALERT! - Luna The Hot Playmate (323) 452-2013 - 23</a></h1></span>\\n<span class=\\\"mugshotDataCat\\\">Escort's Phone:</span>\\n<span class=\\\"mugshotData\\\">\\n<h2 style=\\\"padding:none; margin:none; display:inline;\\\">\\n323-452-2013 </h2>\\n</span>\\n<br/>\\n<span class=\\\"mugshotDataCat\\\">Escort's Location:</span><span class=\\\"mugshotData\\\">\\nLos Angeles, California </span>\\n<br/><span class=\\\"mugshotDataCat\\\">Escort's Age:</span> <span class=\\\"mugshotData\\\">23</span> <br/><span class=\\\"mugshotDataCat\\\">Date of Escort Post:</span> <span class=\\\"mugshotData\\\">Jan 02nd 6:46am</span>\\n<br/><span class=\\\"mugshotDataCat\\\">REVIEWS:</span> <span class=\\\"mugshotData\\\">\\n<a href='/otra.php?c=14&a=overview' rel='nofollow'>READ AND CREATE REVIEWS FOR THIS ESCORT</a> </span>\\n<br/>\\n<span class=\\\"mugshotDataCat\\\">\\n<img id='favimg' onclick=\\\"add_fav('89701497');\\\" class='fav' src='/images/add-favorite.png' alt='Add Escort to Favorites'/>\\n</span>\\n<a style=\\\"text-decoration:none;\\\" href=\\\"/otra.php?c=14&a=NewYearsImage\\\"><img src=\\\"/images/new-years-2012.gif\\\" alt=\\\"The New Years Is Here. Get Lucky in 2017\\\"/></a>\\n<br/>\\n<span class=\\\"largeHeadline\\\">\\nThere are <span style=\\\"font-size:36px;color:#6CB500;\\\">50</span> girls looking in <strong></strong>.\\n<a href=\\\"/otra.php?c=14&a=viewgirls\\\">VIEW GIRLS</a>\\n</span><br/><br/>\\nHey I'm luna 3234522013<br/>Let's explore , embrace and indulge in your favorite fantasy<br/> % independent. discreet no drama<br/>Firm Thighs and Sexy. My Soft skin & Tight Grip is exactly what you deserve Call or text <br/> Fetish friendly <br/> Fantasy friendly <br/> Party friendly 140 Hr SPECIALS 3234522013.&nbsp;Call <u>323-452-2013</u>. <a href='/otra.php?c=14&a=inline' rel='nofollow'>Me and my friends are on EZsex</a> soooo you can find us all on there if you want... skittlegirl<br/><br/><a href='/otra.php?c=14&a=image' rel='nofollow' target=\\\"_self\\\">\\n<img src='http://www.eroticmugshots.com/cities/33/large/89701497-1.png' style='border:1px solid #00f;' width='384' height='683' alt='Erotic Escort Image 2'></a>&nbsp;&nbsp;\\n<a href='/otra.php?c=14&a=image' rel='nofollow' target=\\\"_self\\\">\\n<img src='http://www.eroticmugshots.com/cities/33/large/89701497-2.png' style='border:1px solid #00f;' width='384' height='682' alt='Erotic Escort Image 3'></a>&nbsp;&nbsp;\\n<a href='/otra.php?c=14&a=image' rel='nofollow' target=\\\"_self\\\">\\n<img src='http://www.eroticmugshots.com/cities/33/large/89701497-3.png' style='border:1px solid #00f;' width='640' height='359' alt='Erotic Escort Image 4'></a>&nbsp;&nbsp;\\n<br/>Call me on my cell at 323-452-2013.<br/>Date of ad: 2017-01-02 06:46:00<br/><div class='moreposts'>\\n<ul style='font-size:12px; margin:0px;'><font style='font-size:28px;'>More posts from <strong>323-452-2013</strong></font><li><a href='/losangeles-escorts/323-452-2013/?pid=14184121' target='_self'>\\n<strong>323-452-2013</strong> <font color='#DD6F00'>Nov 04, 2016</font> Guess whose New with 80Specials (323) 452-2013 - 21</a>\\n</li>\\n</ul>\\n</div>\\n</div>\\n<div id=\\\"footer-extra\\\">\\n<div style=\\\"float:left;\\\">\\n<div class=\\\"footer-info-cont\\\">\\n<ul class=\\\"recentPosts\\\"><h3>Recent Escort Classifieds</h3>\\n<li style='border:none;'><span>North Jersey, New Jersey</span> (732-621-4443) <br/>\\n<a href='http://eroticmugshots.com/northjersey-escorts/732-621-4443/?pid=16700232&city=North-Jersey'>:*: G O O D G I R L :*: G O N E *:**: B A D ;) LATINA - 21</a></li>\\n<li><span>Chicago, Illinois</span> (773-412-2044) <br/>\\n<a href='http://eroticmugshots.com/chicago-escorts/773-412-2044/?pid=19572029&city=Chicago'>( LAtE NiGHt ) UNRUShEd (ULTiMAtE) PLEASURE (*AmAziNg Azz*) CHOOSE..W...</a></li>\\n<li><span>Chicago, Illinois</span> (414-914-3777) <br/>\\n<a href='http://eroticmugshots.com/chicago-escorts/414-914-3777/?pid=19571307&city=Chicago'>Petite. and Sweet. Super new and Ready... in out call - 21</a></li>\\n<li><span>Chicago, Illinois</span> (312-600-8628) <br/>\\n<a href='http://eroticmugshots.com/chicago-escorts/312-600-8628/?pid=19571369&city=Chicago'> CM Get PAMPERED - 22</a></li>\\n<li><span>Ft Lauderdale, Florida</span> (786-504-1860) <br/>\\n<a href='http://eroticmugshots.com/ftlauderdale-escorts/786-504-1860/?pid=24132737&city=Ft-Lauderdale'>you will ME. GORGEOUSLY sexy LATIN - 21</a></li>\\n<li><span>Atlanta, Georgia</span> (401-324-9388) <br/>\\n<a href='http://eroticmugshots.com/atlanta-escorts/401-324-9388/?pid=20684382&city=Atlanta'>WoW. MuSt TaKe A LoOk At ThIs. - 21</a></li>\\n<li><span>Atlanta, Georgia</span> (347-940-1982) <br/>\\n<a href='http://eroticmugshots.com/atlanta-escorts/347-940-1982/?pid=20684487&city=Atlanta'> SMOKiNG HOT Specials BuSTy BaBe (( 5 SeRvICe )) Pretty 36DDDs ( ) ( ...</a></li>\\n<li><span>Atlanta, Georgia</span> (305-849-8140) <br/>\\n<a href='http://eroticmugshots.com/atlanta-escorts/305-849-8140/?pid=20648883&city=Atlanta'>Here to please all your desiresCALL ME 100% Real Curvy White Girl. - 21</a></li>\\n<li><span>Houston, Texas</span> (832-914-9667) <br/>\\n<a href='http://eroticmugshots.com/houston-escorts/832-914-9667/?pid=20182393&city=Houston'>Beautiful Salvadorean The One And Only(: - 21</a></li>\\n<li><span>Phoenix, Arizona</span> (623-500-7076) <br/>\\n<a href='http://eroticmugshots.com/phoenix-escorts/623-500-7076/?pid=25716280&city=Phoenix'>NEW GIRL PERSIAN Gem EXotIC Blend - 21</a></li>\\n<li><span>Toronto, Ontario</span> (416-554-3337) <br/>\\n<a href='http://eroticmugshots.com/toronto-escorts/416-554-3337/?pid=19645707&city=Toronto'>(L) (L) ~~~Special 80 for 20 min:) 22YeAr oLd $$exyy LaTiNa BoMbSheLL~~(L...</a></li>\\n<li><span>Toronto, Ontario</span> (416-520-5198) <br/>\\n<a href='http://eroticmugshots.com/toronto-escorts/416-520-5198/?pid=19660304&city=Toronto'>**21 years old * $80 **real pictures ** A sian Kathy *** - 21</a></li>\\n<li><span>Toronto, Ontario</span> (647-702-6825) <br/>\\n<a href='http://eroticmugshots.com/toronto-escorts/647-702-6825/?pid=18874635&city=Toronto'>**: :** 18 Yr Old Mercedes **: :** Incalls/Outcalls (647)702-6825 - 1...</a></li>\\n<li><span>Toronto, Ontario</span> (416-275-1296) <br/>\\n<a href='http://eroticmugshots.com/toronto-escorts/416-275-1296/?pid=19660322&city=Toronto'>100% real pictures ** True college sweet heart ** Taiwan Kelly ** 416-275...</a></li>\\n<li><span>Northern Virginia, DC</span> (206-922-9303) <br/>\\n<a href='http://eroticmugshots.com/nova-escorts/206-922-9303/?pid=13266655&city=Northern-Virginia'>SPECIALs 60/90/150 FuN & WiLd BrUnEtTe ThIcK & BuSty JUSt THe WAy U LIKe ...</a></li>\\n<li><span>Chicago, Illinois</span> (608-609-5899) <br/>\\n<a href='http://eroticmugshots.com/chicago-escorts/608-609-5899/?pid=19571089&city=Chicago'> New To Town Dont Miss Me 100$ Specials - 18</a></li>\\n<li><span>Dallas, Texas</span> (469-510-5849) <br/>\\n<a href='http://eroticmugshots.com/dallas-escorts/469-510-5849/?pid=24887000&city=Dallas'>JENNY ...A New RedHeaded Firecracker. YeeHaw. - 21</a></li>\\n<li><span>New York, New York</span> (442-222-0227) <br/>\\n<a href='http://eroticmugshots.com/newyork-escorts/442-222-0227/?pid=42902367&city=New-York'> F R E A K Y __ O B S E S S I O N UPSCALE UNRUSHED-100%...</a></li>\\n<li><span>New York, New York</span> (917-676-1333) <br/>\\n<a href='http://eroticmugshots.com/newyork-escorts/917-676-1333/?pid=44835334&city=New-York'> SuPeR SpEciALs Red BoNe ThiCk Incalls - 19</a></li>\\n<li><span>Houston, Texas</span> (301-000-6018) <br/>\\n<a href='http://eroticmugshots.com/houston-escorts/301-000-6018/?pid=20182064&city=Houston'>(Jamaican Ebony Goddess) petite sexy spontaneous UPSCALE discreet dates M...</a></li>\\n</ul>\\n</div>\\n<div class=\\\"footer-info-cont\\\" style=\\\"clear:left;\\\">\\n<ul class=\\\"recentPosts\\\"><h3>Site Map</h3>\\n<li style='border:none;'>Call this Escort at <strong>323-452-2013</strong></li>\\n<li><a href='/index.php?cities=1' target='_self' rel='follow'>View escorts in other cities</a></li>\\n<li><a href=\\\"/registration.php\\\">Register Here</a></li>\\n<li><a href=\\\"/login.php\\\">Login to your account</a></li>\\n<li><a href=\\\"/acronyms.php\\\">Key for Escort Acronyms</a></li>\\n<li><a href=\\\"/top-10-escort-practices.php\\\">Top 10 Escort Practices</a></li>\\n<li><a href=\\\"/prostitution-laws.html\\\" rel=\\\"follow\\\">Prostitution Laws Around the World</a></li>\\n<li><a href=\\\"/state-prostitution-laws.html\\\" rel=\\\"follow\\\">Prostitution Laws In Your State</a></li>\\n</ul>\\n</div>\\n</div>\\n<div style=\\\"float:left;\\\">\\n<div class=\\\"footer-info-cont\\\">\\n<ul class=\\\"recentPosts\\\"><h3>Top Escort Cities</h3>\\n<li style=\\\"border:none;\\\"><a href='/newyork-escorts/'>New York, New York</a></li>\\n<li><a href='/toronto-escorts/'>Toronto, Ontario</a></li>\\n<li><a href='/dallas-escorts/'>Dallas, Texas</a></li>\\n<li><a href='/chicago-escorts/'>Chicago, Illinois</a></li>\\n<li><a href='/atlanta-escorts/'>Atlanta, Georgia</a></li>\\n<li><a href='/northjersey-escorts/'>North Jersey, New Jersey</a></li>\\n<li><a href='/detroit-escorts/'>Detroit, Michigan</a></li>\\n<li><a href='/losangeles-escorts/'>Los Angeles, California</a></li>\\n<li><a href='/sandiego-escorts/'>San Diego, California</a></li>\\n<li><a href='/orangecounty-escorts/'>Orange County, California</a></li>\\n<li><a href='/houston-escorts/'>Houston, Texas</a></li>\\n<li><a href='/phoenix-escorts/'>Phoenix, Arizona</a></li>\\n<li><a href='/philadelphia-escorts/'>Philadelphia, Pennsylvania</a></li>\\n<li><a href='/boston-escorts/'>Boston, Massachusetts</a></li>\\n<li><a href='/washingtondc-escorts/'>Washington DC, DC</a></li>\\n<li><a href='/orlando-escorts/'>Orlando, Florida</a></li>\\n<li><a href='/lasvegas-escorts/'>Las Vegas, Nevada</a></li>\\n<li><a href='/miami-escorts/'>Miami, Florida</a></li>\\n</ul>\\n</div>\\n<div class=\\\"footer-info-cont\\\" style=\\\"clear:left;\\\">\\n<ul class=\\\"recentPosts\\\"><h3>Recent Blog Posts</h3>\\n<li style=\\\"border:none;\\\"><a href=\\\"/blog/index.php?id=36\\\" rel=\\\"follow\\\">Sheriff candidate Minister and Detective <NAME>s arrested in prostitution bust</a></li><li><a href=\\\"/blog/index.php?id=35\\\" rel=\\\"follow\\\">Man gets 35 years for impersonating cop to get free sex from hooker</a></li><li><a href=\\\"/blog/index.php?id=34\\\" rel=\\\"follow\\\"><NAME>: Psychologist by Day, Pimp by Night</a></li><li><a href=\\\"/blog/index.php?id=32\\\" rel=\\\"follow\\\">Surfside Beach, SC Prostitution BUST: Video</a></li><li><a href=\\\"/blog/index.php?id=31\\\" rel=\\\"follow\\\">College Park, Maryland Prostitution Sting: 30 arrested</a></li><li><a href=\\\"/blog/index.php?id=30\\\" rel=\\\"follow\\\">SMU Players Robbed By Escort</a></li><li><a href=\\\"/blog/index.php?id=29\\\" rel=\\\"follow\\\">Study: 1 in 6 Sex Offenders Live Double Life</a></li><li><a href=\\\"/blog/index.php?id=28\\\" rel=\\\"follow\\\">Dark Knight Rises Suspect Killer James Holmes Bought Prostitutes Online</a></li><li><a href=\\\"/blog/index.php?id=27\\\" rel=\\\"follow\\\">Hookers Picked Up Off the Street</a></li><li><a href=\\\"/blog/index.php?id=26\\\" rel=\\\"follow\\\">Hookers paid extra for photos</a></li> </ul>\\n</div>\\n<div class=\\\"footer-info-cont\\\" style=\\\"clear:left;\\\">\\n<ul class=\\\"recentPosts\\\"><h3>Links</h3>\\n<li><a href=\\\"/losangeles-escorts/cams/\\\" rel=\\\"nofollow\\\">Escorts on Webcam</a></li>\\n<li><strong><a href=\\\"/contact.php\\\">Ad Removal Form</a></strong></li>\\n<li><a href=\\\"/contact.php\\\">Contact Us</a></li>\\n</ul>\\n</div>\\n</div>\\n</div>\\n</div>\\n<div style=\\\"margin:0px; padding:0px; position:absolute; top:10px; right:20px; float:right;\\\" align=\\\"right\\\">\\n<div style=\\\"padding:5px 5px 0 5px; font-family:'times new roman'; font-size:18px; color: #E412CC;\\\">\\n<center>Search Box</center>\\n<input type=\\\"text\\\" style=\\\"margin-bottom:5px;\\\" value=\\\"\\\" size=\\\"18\\\"><br/>\\n<button style=\\\"margin:none; padding:none;\\\" onclick=\\\"window.location='/otra.php?c=14&a=searchbutton'\\\">Search For Profiles</button>\\n<ul class=\\\"recentPosts\\\" style=\\\"padding-bottom:0px; margin-bottom:0px;\\\">\\n<li><a href=\\\"/registration.php\\\">Register Here</a></li>\\n<li><a href=\\\"/login.php\\\">Login to your account</a></li>\\n<li><a href='/losangeles-escorts/323-452-2013/?pid=89701497&v=nm' target='_self'>Non Mobile Version</a></li>\\n<li><a href=\\\"/blog/\\\" rel=\\\"follow\\\">Escort Blog</a></li>\\n<li><a href=\\\"/acronyms.php\\\" rel=\\\"follow\\\">Key for Escort Acronyms</a></li>\\n<li><a href=\\\"/top-10-escort-practices.php\\\" rel=\\\"follow\\\">Top 10 Escort Practices</a></li>\\n<li><a href=\\\"/otra.php?c=14&a=sidebar\\\" rel=\\\"nofollow\\\">Escort Reviews</a></li>\\n<li><a href=\\\"/otra.php?c=14&a=sidebar\\\" rel=\\\"nofollow\\\">See Escorts on Webcam</a></li>\\n<li><a href=\\\"/state-prostitution-laws.html\\\" rel=\\\"follow\\\">Prostitution Laws</a></li>\\n<div style='border:2px solid #E4E4E4; background-color:#FDFDFD; padding:4px; text-align:center;'>\\n<strong>Most<br/>Recently Viewed</strong><br/>\\n<font size='-1'>Today at 10:02am Pacific</font><br/>\\n<a style='border:0px solid #00F;' href=\\\"/killeen-escorts/346-908-0622/?pid=16849944\\\" target='_blank'>\\n<img src='http://www.eroticmugshots.com/cities/313/thumbnail/16849944-1.jpg' alt=\\\"346-908-0622 - killeen\\\">\\n</a>\\n<br/>\\n<a style='border:0px solid #00F;' href=\\\"/killeen-escorts/346-908-0622/?pid=16849944\\\" target='_blank'>\\n346-908-0622 </a>\\n<br/>\\n<a style='border:0px solid #00F;' href=\\\"/killeen-escorts/346-908-0622/?pid=16849944\\\" target='_blank'>\\nKilleen </a>\\n</div>\\n</ul>\\n<br/>\\n</div>\\n</div>\\n</div>\\n \\n</body>\\n</html>\\n\"}\n",
    "c = Core(extraction_config=e_config)\n",
    "r = c.process(doc)\n",
    "\n",
    "pp = pprint.PrettyPrinter(indent=4)\n",
    "pp.pprint(r['content_extraction'])\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda env:etk_env]",
   "language": "python",
   "name": "conda-env-etk_env-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"<commit_msg>read the json doc from file<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Step 1. Run content extraction on html document"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the following example, we will run the `readability` extractor and the `title` extractor on the html document.  \n",
    "The extraction is controlled by an `extraction_config` and the input is json document with `html` document embedded in it.  \n",
    "\n",
    "Let's breakdown the extraction_config:  \n",
    "> `input_path`  \n",
    "\n",
    "specifies the path where the html content is in the json  \n",
    "> `extractors.readability`  \n",
    "\n",
    "sets up `etk` to run readability extractor on the `input_path`\n",
    "\n",
    "> strict: yes\n",
    "\n",
    "makes `readability` extractor more precise. ```strict: no``` will result in more content getting extracted from the html as main content  \n",
    "\n",
    "#### Output  \n",
    "The output will be under the field `content_extraction` in the input json document. It should have `content_relaxed` corresponding to ```strict: no``` and `content_strict` for ```strict: yes```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{   'content_relaxed': {   'text': u\"\\n \\n \\n \\n \\n \\n \\n smoothlegs24  28 \\n \\n \\n chrissy391  27 \\n \\n \\n My name is Helena height 160cms weight 55 kilos  contact me at <EMAIL>           <NAME>         age: 23 HrumpMeNow  28 \\n \\n \\n xxtradition  24 \\n \\n \\n jumblyjumb  26 \\n \\n \\n claudia77  26 \\n \\n \\n gushinPuss  28 \\n \\n \\n Littlexdit  25 \\n \\n \\n PinkSweets2  28 \\n \\n \\n withoutlimit  27 \\n \\n \\n bothOfUs3  28 \\n \\n \\n lovelylips  27 \\n \\n \\n killerbod  27 \\n \\n \\n Littlexdit  27 \\n \\n \\n azneyes  23 \\n \\n \\n \\n \\n \\n Escort's Phone: \\n \\n \\n323-452-2013  \\n \\n Escort's Location: \\nLos Angeles, California  \\n Escort's Age:   23   Date of Escort Post:   Jan 02nd 6:46am \\n REVIEWS:   \\n READ AND CREATE REVIEWS FOR THIS ESCORT   \\n \\n \\n \\n \\n \\nThere are  50  girls looking in  .\\n VIEW GIRLS \\n \\nHey I'm luna 3234522013 Let's explore , embrace and indulge in your favorite fantasy  % independent. discreet no drama Firm Thighs and Sexy. My Soft skin & Tight Grip is exactly what you deserve Call or text   Fetish friendly   Fantasy friendly   Party friendly 140 Hr SPECIALS 3234522013.\\xa0Call  323-452-2013 .  Me and my friends are on EZsex  soooo you can find us all on there if you want... skittlegirl \\n \\xa0\\xa0\\n \\n \\xa0\\xa0\\n \\n \\xa0\\xa0\\n Call me on my cell at 323-452-2013. Date of ad: 2017-01-02 06:46:00 \\n \\n \\n \\n\"},\n",
      "    'content_strict': {   'text': u\"\\n \\n \\n \\n \\n \\n smoothlegs24  28 \\n \\n \\n chrissy391  27 \\n \\n \\n My name is Helena height 160cms weight 55 kilos  contact me at <EMAIL>           <NAME>         age: 23 HrumpMeNow  28 \\n \\n \\n xxtradition  24 \\n \\n \\n jumblyjumb  26 \\n \\n \\n claudia77  26 \\n \\n \\n gushinPuss  28 \\n \\n \\n Littlexdit  25 \\n \\n \\n PinkSweets2  28 \\n \\n \\n withoutlimit  27 \\n \\n \\n bothOfUs3  28 \\n \\n \\n lovelylips  27 \\n \\n \\n killerbod  27 \\n \\n \\n Littlexdit  27 \\n \\n \\n azneyes  23 \\n \\n \\n \\n \\n \\n Escort's Phone: \\n \\n \\n323-452-2013  \\n \\n Escort's Location: \\nLos Angeles, California  \\n Escort's Age:   23   Date of Escort Post:   Jan 02nd 6:46am \\n REVIEWS:   \\n READ AND CREATE REVIEWS FOR THIS ESCORT   \\n \\n \\n \\n \\n \\nThere are  50  girls looking in  .\\n VIEW GIRLS \\n \\nHey I'm luna 3234522013 Let's explore , embrace and indulge in your favorite fantasy  % independent. discreet no drama Firm Thighs and Sexy. My Soft skin & Tight Grip is exactly what you deserve Call or text   Fetish friendly   Fantasy friendly   Party friendly 140 Hr SPECIALS 3234522013.\\xa0Call  323-452-2013 .  Me and my friends are on EZsex  soooo you can find us all on there if you want... skittlegirl \\n \\xa0\\xa0\\n \\n \\xa0\\xa0\\n \\n \\xa0\\xa0\\n Call me on my cell at 323-452-2013. Date of ad: 2017-01-02 06:46:00 \\n \\n \\n\"}}\n"
     ]
    }
   ],
   "source": [
    "from etk.core import Core\n",
    "import  json\n",
    "import pprint\n",
    "import codecs\n",
    "\n",
    "extraction_config = {'content_extraction': {\n",
    "            \"input_path\": \"raw_content\",\n",
    "            \"extractors\": {\n",
    "              \"readability\": [\n",
    "                {\n",
    "                  \"strict\": \"yes\",\n",
    "                  \"extraction_policy\": \"keep_existing\"\n",
    "                },\n",
    "                {\n",
    "                  \"strict\": \"no\",\n",
    "                  \"extraction_policy\": \"keep_existing\",\n",
    "                  \"field_name\": \"content_relaxed\"\n",
    "                }\n",
    "              ]\n",
    "            }\n",
    "          }\n",
    "        }\n",
    "# read the json document from disk\n",
    "doc = json.load(codecs.open('etk/unit_tests/ground_truth/1.jl', 'r'))\n",
    "c = Core(extraction_config=extraction_config)\n",
    "r = c.process(doc)\n",
    "\n",
    "pp = pprint.PrettyPrinter(indent=4)\n",
    "pp.pprint(r['content_extraction'])\n"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda env:etk_env]",
   "language": "python",
   "name": "conda-env-etk_env-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
{'fraction_non_alphanumeric': 0.24699505067169455, 'fraction_numerical': 0.06180768324298845, 'mean_word_length': 3.174763251752552, 'pattern_counts': {'":': 81, '<': 699, '<?xml version=': 0, '>': 705, 'https://': 0, 'lorem ipsum': 0, 'www.': 22, 'xml': 2}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '907766', 'n_tokens_mistral': 14813, 'n_tokens_neox': 13101, 'n_words': 2578}