<commit_before><!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <meta name="generator" content="pandoc">
  <title>Hacking Oregon's Hidden Political Connections</title>
  <meta name="apple-mobile-web-app-capable" content="yes" />
  <meta name="apple-mobile-web-app-status-bar-style" content="black-translucent" />
  <meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no">
  <link rel="stylesheet" href="css/reveal.css"/>
  <style type="text/css">code{white-space: pre;}</style>
  <link rel="stylesheet" href="css/theme/moon.css" id="theme">
  <!-- If the query includes 'print-pdf', include the PDF print sheet -->
  <script>
    if( window.location.search.match( /print-pdf/gi ) ) {
      var link = document.createElement( 'link' );
      link.rel = 'stylesheet';
      link.type = 'text/css';
      link.href = 'css/print/pdf.css';
      document.getElementsByTagName( 'head' )[0].appendChild( link );
    }
  </script>
  <!--[if lt IE 9]>
  <script src="lib/js/html5shiv.js"></script>
  <![endif]-->
</head>
<body>
  <div class="reveal">
    <div class="slides">

<section>
    <h1 class="title">Hacking Oregon's Hidden Political Connections</h1>
    <h3 class="date"></h3>
</section>

<section id="hacking-oregons-hidden-political-connections" class="slide level1">
<h1><a href="http://totalgood.github.io/talks/2015-10-27-Hacking-Oregon-Hidden-Political-Connections.html">Hacking Oregon's Hidden Political Connections</a></h1>
<ul>
<li>iPython Notebooks: bit.ly/<a href="https://github.com/totalgood/talks/tree/master/notebooks">totalgood</a></li>
<li>by <script type="text/javascript">
<!--
h='&#116;&#x6f;&#116;&#x61;&#108;&#x67;&#x6f;&#x6f;&#100;&#46;&#x63;&#x6f;&#x6d;&#x3f;&#x53;&#x75;&#98;&#106;&#x65;&#x63;&#116;&#x3d;&#72;&#x61;&#x63;&#x6b;&#x25;&#50;&#48;&#x4f;&#114;&#x65;&#x67;&#x6f;&#110;';a='&#64;';n='&#104;&#x6f;&#98;&#x73;';e=n+a+h;
document.write('<a h'+'ref'+'="ma'+'ilto'+':'+e+'">'+'Hobs'+'<\/'+'a'+'>');
// -->
</script><noscript>&#72;&#x6f;&#98;&#x73;&#32;&#40;&#104;&#x6f;&#98;&#x73;&#32;&#x61;&#116;&#32;&#116;&#x6f;&#116;&#x61;&#108;&#x67;&#x6f;&#x6f;&#100;&#32;&#100;&#x6f;&#116;&#32;&#x63;&#x6f;&#x6d;&#x3f;&#x53;&#x75;&#98;&#106;&#x65;&#x63;&#116;&#x3d;&#72;&#x61;&#x63;&#x6b;&#x25;&#50;&#48;&#x4f;&#114;&#x65;&#x67;&#x6f;&#110;&#x29;</noscript></li>
</ul>
</section>
<section id="agenda" class="slide level1">
<h1>Agenda</h1>
<ol type="1">
<li>Using Pandas to turn CSVs in to a relational DB
<ul>
<li>Identify key relationships (FKs) automatically</li>
<li>Using FKs to do join SQL-like queries in Pandas</li>
</ul></li>
</ol>
</section>
<section id="agenda-1" class="slide level1">
<h1>Agenda</h1>
<ol start="2" type="1">
<li>Intersect large sets in Pandas
<ul>
<li>Are there AM emails in BehindTheCurtain DB?</li>
<li>Convert 10 GB mysql dump into dozens of CSVs</li>
<li>Load 50M emails into memory efficiently</li>
<li><h2>Intersect email set with public record emails</h2></li>
</ul></li>
</ol>
</section>
<section id="agenda-2" class="slide level1">
<h1>Agenda</h1>
<ol start="3" type="1">
<li>Destructure a DB -- stringify and tokenize
<ul>
<li>Why?</li>
<li>How</li>
<li>Restructure (TFIDF)
<ul>
<li>Raw python</li>
<li>Sklearn</li>
</ul></li>
</ul></li>
</ol>
</section>
<section id="agenda-3" class="slide level1">
<h1>Agenda</h1>
<ol start="4" type="1">
<li>Use a sparse TFIDF to detect similarity between records/documents</li>
</ol>
<ul>
<li>cluster Oregon PACs by their &quot;mission&quot; or &quot;measure&quot; or &quot;description&quot;</li>
<li>d3 force-directed graph of PAC similarity</li>
<li>compare to DAG of financial transactions between PACs</li>
</ul>
</section>
<section id="agenda-4" class="slide level1">
<h1>Agenda</h1>
<ol start="5" type="1">
<li>Measure the similarity between similarity matrices/graphs</li>
</ol>
<ul>
<li>PAC NLP similarity vs financial connections</li>
</ul>
</section>
<section id="why-destructure-a-db-why" class="slide level1">
<h1>3. Why Destructure a DB: Why?</h1>
<ul>
<li>Why squish multiple fields into a string?</li>
<li>You will vectorize later anyway, right?</li>
</ul>
<h2 id="because">Because</h2>
<ul>
<li>Dimensions are vaguely defined/understood</li>
<li>Information &quot;smear&quot; across fields/dimensions</li>
</ul>
</section>
<section id="destructure-a-db-how" class="slide level1">
<h1>3. Destructure a DB: How?</h1>
<ol type="1">
<li>Ignore numbers/dates</li>
<li>Stringify each field</li>
<li>Stem words</li>
<li>Ignore words (are you sure?)</li>
<li>Concatenate</li>
<li>Split</li>
<li>Vectorize/Count</li>
</ol>
</section>
<section id="restructure-the-db-tfidf" class="slide level1">
<h1>3. Restructure the DB: TFIDF</h1>
<p>Compute sparse TFIDF</p>
<ul>
<li>Explicit python builtins: <code>Counter</code>, <code>defaultdict</code></li>
<li>sklearn</li>
</ul>
</section>
<section id="thank-you" class="slide level1">
<h1>Thank You!</h1>
<ul>
<li><script type="text/javascript">
<!--
h='&#x67;&#x6d;&#x61;&#x69;&#108;&#46;&#x63;&#x6f;&#x6d;&#x3f;&#x53;&#x75;&#98;&#106;&#x65;&#x63;&#116;&#x3d;&#72;&#x61;&#x63;&#x6b;&#x25;&#50;&#48;&#x4f;&#114;&#x65;&#x67;&#x6f;&#110;';a='&#64;';n='&#x6d;&#x65;&#108;&#x61;&#110;&#x67;&#x65;&#46;&#x61;&#x75;&#46;&#98;&#108;&#x65;&#x75;';e=n+a+h;
document.write('<a h'+'ref'+'="ma'+'ilto'+':'+e+'">'+'Thunder'+'<\/'+'a'+'>');
// -->
</script><noscript>&#84;&#104;&#x75;&#110;&#100;&#x65;&#114;&#32;&#40;&#x6d;&#x65;&#108;&#x61;&#110;&#x67;&#x65;&#46;&#x61;&#x75;&#46;&#98;&#108;&#x65;&#x75;&#32;&#x61;&#116;&#32;&#x67;&#x6d;&#x61;&#x69;&#108;&#32;&#100;&#x6f;&#116;&#32;&#x63;&#x6f;&#x6d;&#x3f;&#x53;&#x75;&#98;&#106;&#x65;&#x63;&#116;&#x3d;&#72;&#x61;&#x63;&#x6b;&#x25;&#50;&#48;&#x4f;&#114;&#x65;&#x67;&#x6f;&#110;&#x29;</noscript></li>
<li><a href="http://hackoregon.org">Hack Oregon</a>
<ul>
<li>Pizza, data, and a cause!</li>
</ul></li>
<li><a href="http://jeremytanner.com/"><NAME></a>
<ul>
<li>Awesome <a href="pydx.org">PyDx</a> talk on &quot;Python for Evil&quot;</li>
</ul></li>
<li><script type="text/javascript">
<!--
h='&#x67;&#114;&#x69;&#x6d;&#x6d;&#x73;&#x63;&#x69;&#x65;&#110;&#x63;&#x65;&#46;&#x63;&#x6f;&#x6d;&#x3f;&#x53;&#x75;&#98;&#106;&#x65;&#x63;&#116;&#x3d;&#72;&#x61;&#x63;&#x6b;&#x25;&#50;&#48;&#x4f;&#114;&#x65;&#x67;&#x6f;&#110;';a='&#64;';n='&#x61;&#108;&#108;&#x65;&#110;';e=n+a+h;
document.write('<a h'+'ref'+'="ma'+'ilto'+':'+e+'">'+'Grimm'+'<\/'+'a'+'>');
// -->
</script><noscript>&#x47;&#114;&#x69;&#x6d;&#x6d;&#32;&#40;&#x61;&#108;&#108;&#x65;&#110;&#32;&#x61;&#116;&#32;&#x67;&#114;&#x69;&#x6d;&#x6d;&#x73;&#x63;&#x69;&#x65;&#110;&#x63;&#x65;&#32;&#100;&#x6f;&#116;&#32;&#x63;&#x6f;&#x6d;&#x3f;&#x53;&#x75;&#98;&#106;&#x65;&#x63;&#116;&#x3d;&#72;&#x61;&#x63;&#x6b;&#x25;&#50;&#48;&#x4f;&#114;&#x65;&#x67;&#x6f;&#110;&#x29;</noscript></li><commit_msg>auto-commit new build  Tue Oct 27 13:43:25 PDT 2015<commit_after><!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <meta name="generator" content="pandoc">
  <title>Hacking Oregon's Hidden Political Connections</title>
  <meta name="apple-mobile-web-app-capable" content="yes" />
  <meta name="apple-mobile-web-app-status-bar-style" content="black-translucent" />
  <meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no">
  <link rel="stylesheet" href="css/reveal.css"/>
  <style type="text/css">code{white-space: pre;}</style>
  <link rel="stylesheet" href="css/theme/moon.css" id="theme">
  <!-- If the query includes 'print-pdf', include the PDF print sheet -->
  <script>
    if( window.location.search.match( /print-pdf/gi ) ) {
      var link = document.createElement( 'link' );
      link.rel = 'stylesheet';
      link.type = 'text/css';
      link.href = 'css/print/pdf.css';
      document.getElementsByTagName( 'head' )[0].appendChild( link );
    }
  </script>
  <!--[if lt IE 9]>
  <script src="lib/js/html5shiv.js"></script>
  <![endif]-->
</head>
<body>
  <div class="reveal">
    <div class="slides">

<section>
    <h1 class="title">Hacking Oregon's Hidden Political Connections</h1>
    <h3 class="date"></h3>
</section>

<section id="hacking-oregons-hidden-political-connections" class="slide level1">
<h1><a href="http://totalgood.github.io/talks/2015-10-27-Hacking-Oregon-Hidden-Political-Connections.html">Hacking Oregon's Hidden Political Connections</a></h1>
<ul>
<li>iPython Notebooks: bit.ly/<a href="https://github.com/totalgood/talks/tree/master/notebooks">totalgood</a></li>
<li>by <script type="text/javascript">
<!--
h='&#116;&#x6f;&#116;&#x61;&#108;&#x67;&#x6f;&#x6f;&#100;&#46;&#x63;&#x6f;&#x6d;&#x3f;&#x53;&#x75;&#98;&#106;&#x65;&#x63;&#116;&#x3d;&#72;&#x61;&#x63;&#x6b;&#x25;&#50;&#48;&#x4f;&#114;&#x65;&#x67;&#x6f;&#110;';a='&#64;';n='&#104;&#x6f;&#98;&#x73;';e=n+a+h;
document.write('<a h'+'ref'+'="ma'+'ilto'+':'+e+'">'+'Hobs'+'<\/'+'a'+'>');
// -->
</script><noscript>&#72;&#x6f;&#98;&#x73;&#32;&#40;&#104;&#x6f;&#98;&#x73;&#32;&#x61;&#116;&#32;&#116;&#x6f;&#116;&#x61;&#108;&#x67;&#x6f;&#x6f;&#100;&#32;&#100;&#x6f;&#116;&#32;&#x63;&#x6f;&#x6d;&#x3f;&#x53;&#x75;&#98;&#106;&#x65;&#x63;&#116;&#x3d;&#72;&#x61;&#x63;&#x6b;&#x25;&#50;&#48;&#x4f;&#114;&#x65;&#x67;&#x6f;&#110;&#x29;</noscript></li>
</ul>
</section>
<section class="slide level1">

<h2 id="agenda">Agenda</h2>
<ol type="1">
<li>Pandas as a relational DB</li>
</ol>
<ul>
<li>Identify foreign keys automatically</li>
<li>Use FKs to do join SQL-like queries</li>
</ul>
</section>
<section class="slide level1">

<h2 id="agenda-1">Agenda</h2>
<ol start="2" type="1">
<li>Intersect large sets</li>
</ol>
<ul>
<li>AM emails in BehindTheCurtain DB?</li>
<li>10 GB mysql dump &gt;&gt; dozens of CSVs</li>
<li>Load 50M emails efficiently</li>
<li>Intersect emails with public records</li>
</ul>
</section>
<section id="agenda-2" class="slide level1">
<h1>Agenda</h1>
<ol start="3" type="1">
<li>Destructure and Restructure DB</li>
</ol>
<ul>
<li>Why?</li>
<li>How?</li>
<li>Restructure (TFIDF)
<ul>
<li>Raw python</li>
<li>Sklearn</li>
</ul></li>
</ul>
</section>
<section id="agenda-3" class="slide level1">
<h1>Agenda</h1>
<ol start="4" type="1">
<li>TFIDF to detect similarity between records</li>
</ol>
<ul>
<li>cluster Oregon PACs by their &quot;mission&quot;</li>
<li>d3 force-directed graph of PAC similarity</li>
<li>compare to DAG of financial transactions</li>
</ul>
</section>
<section id="agenda-4" class="slide level1">
<h1>Agenda</h1>
<ol start="5" type="1">
<li>Measure similarity between similarity matrices</li>
</ol>
<p>Say: NLP Similarity</p>
<p>vs.</p>
<p>Do: Financial Transactions</p>
</section>
<section id="why-destructure-a-db-why" class="slide level1">
<h1>3. Why Destructure a DB: Why?</h1>
<ul>
<li>Why squish multiple fields into a string?</li>
<li>You will vectorize later anyway, right?</li>
</ul>
<h2 id="because">Because</h2>
<ul>
<li>Dimensions are vaguely defined/understood</li>
<li>Information &quot;smear&quot; across fields/dimensions</li>
</ul>
</section>
<section id="destructure-a-db-how" class="slide level1">
<h1>3. Destructure a DB: How?</h1>
<ol type="1">
<li>Ignore numbers/dates</li>
<li>Stringify each field</li>
<li>Stem words</li>
<li>Ignore words (are you sure?)</li>
<li>Concatenate</li>
<li>Split</li>
<li>Vectorize/Count</li>
</ol>
</section>
<section id="restructure-the-db-tfidf" class="slide level1">
<h1>3. Restructure the DB: TFIDF</h1>
<p>Compute sparse TFIDF</p>
<ul>
<li>Explicit python builtins: <code>Counter</code>, <code>defaultdict</code></li>
<li>sklearn</li>
</ul>
</section>
<section id="tfidf-similarity" class="slide level1">
<h1>4. TFIDF Similarity</h1>
<p>Large dimensions are scary</p>
<ul>
<li>Everything is far apart</li>
<li>Euclidean distance is meaningless</li>
<li>Our brains fail</li>
</ul>
</section>
<section id="tfidf-similarity-1" class="slide level1">
<h1>4. TFIDF Similarity</h1>
<p>Vector distances</p>
<ul>
<li><strong>L_1</strong>, L_2, [L_0, L_inf, L_sup]</li>
<li><a href="http://citeseer.ist.psu.edu/viewdoc/download?doi=10.1.1.23.7409&amp;rep=rep1&amp;type=pdf">Fractional Distance</a></li>
</ul>
</section>
<section id="tfidf-similarity-2" class="slide level1">
<h1>4. TFIDF Similarity</h1>
<h2 id="cosine-similarity-similarity-1-distance">Cosine Similarity (similarity = 1 / distance)</h2>
<ul>
<li>Equivalent:
<ul>
<li>Pierson Correlation</li>
<li>| <strong>v_1</strong> dot <strong>v_2</strong> | (projection)</li>
<li>angle between <strong>v1</strong> and <strong>v2</strong></li>
</ul></li>
<li>Bounded: [-1, +1]</li>
</ul>
</section>
<section class="slide level1">

<p>cluster Oregon PACs by their &quot;mission&quot; - d3 force-directed graph of PAC similarity - compare to DAG of financial transactions</p>
</section>
<section id="thank-you" class="slide level1">
<h1>Thank You!</h1>
<ul>
<li><script type="text/javascript">
<!--
h='&#x67;&#x6d;&#x61;&#x69;&#108;&#46;&#x63;&#x6f;&#x6d;&#x3f;&#x53;&#x75;&#98;&#106;&#x65;&#x63;&#116;&#x3d;&#72;&#x61;&#x63;&#x6b;&#x25;&#50;&#48;&#x4f;&#114;&#x65;&#x67;&#x6f;&#110;';a='&#64;';n='&#x6d;&#x65;&#108;&#x61;&#110;&#x67;&#x65;&#46;&#x61;&#x75;&#46;&#98;&#108;&#x65;&#x75;';e=n+a+h;
document.write('<a h'+'ref'+'="ma'+'ilto'+':'+e+'">'+'Thunder'+'<\/'+'a'+'>');
// -->
</script><noscript>&#84;&#104;&#x75;&#110;&#100;&#x65;&#114;&#32;&#40;&#x6d;&#x65;&#108;&#x61;&#110;&#x67;&#x65;&#46;&#x61;&#x75;&#46;&#98;&#108;&#x65;&#x75;&#32;&#x61;&#116;&#32;&#x67;&#x6d;&#x61;&#x69;&#108;&#32;&#100;&#x6f;&#116;&#32;&#x63;&#x6f;&#x6d;&#x3f;&#x53;&#x75;&#98;&#106;&#x65;&#x63;&#116;&#x3d;&#72;&#x61;&#x63;&#x6b;&#x25;&#50;&#48;&#x4f;&#114;&#x65;&#x67;&#x6f;&#110;&#x29;</noscript></li>
<li><a href="http://hackoregon.org">Hack Oregon</a>
<ul>
<li>Pizza, data, and a cause!</li>
</ul></li>
<li><a href="http://jeremytanner.com/"><NAME></a>
<ul>
<li>Awesome <a href="pydx.org">PyDx</a> talk on &quot;Python for Evil&quot;</li>
</ul></li>
<li><script type="text/javascript">
<!--
h='&#x67;&#114;&#x69;&#x6d;&#x6d;&#x73;&#x63;&#x69;&#x65;&#110;&#x63;&#x65;&#46;&#x63;&#x6f;&#x6d;&#x3f;&#x53;&#x75;&#98;&#106;&#x65;&#x63;&#116;&#x3d;&#72;&#x61;&#x63;&#x6b;&#x25;&#50;&#48;&#x4f;&#114;&#x65;&#x67;&#x6f;&#110;';a='&#64;';n='&#x61;&#108;&#108;&#x65;&#110;';e=n+a+h;
document.write('<a h'+'ref'+'="ma'+'ilto'+':'+e+'">'+'Grimm'+'<\/'+'a'+'>');
// -->
</script><noscript>&#x47;&#114;&#x69;&#x6d;&#x6d;&#32;&#40;&#x61;&#108;&#108;&#x65;&#110;&#32;&#x61;&#116;&#32;&#x67;&#114;&#x69;&#x6d;&#x6d;&#x73;&#x63;&#x69;&#x65;&#110;&#x63;&#x65;&#32;&#100;&#x6f;&#116;&#32;&#x63;&#x6f;&#x6d;&#x3f;&#x53;&#x75;&#98;&#106;&#x65;&#x63;&#116;&#x3d;&#72;&#x61;&#x63;&#x6b;&#x25;&#50;&#48;&#x4f;&#114;&#x65;&#x67;&#x6f;&#110;&#x29;</noscript></li>
{'fraction_non_alphanumeric': 0.3019964446875427, 'fraction_numerical': 0.09831806372213865, 'mean_word_length': 7.609181871689229, 'pattern_counts': {'":': 0, '<': 527, '<?xml version=': 0, '>': 527, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2889643', 'n_tokens_mistral': 7431, 'n_tokens_neox': 6073, 'n_words': 914}