<commit_before><commit_msg>added kmeans clustering example for workshop<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Clustering NIST headlines and descriptions\n",
    "\n",
    "_adapted from https://github.com/StarCYing/open_data_day_dc_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction:\n",
    "In this workshop we show you an example of a workflow in data science from initial data ingestion, cleaning, modeling, and ultimately clustering. In this example we scrape the news feed of the National Institute of Standards and Technology ([NIST](www.nist.gov)). For those not in the know, NIST is comprised of multiple research centers which include: \n",
    "* Center for Nanoscale Science and Technology (CNST)\n",
    "* Engineering Laboratory (EL)\n",
    "* Information Technology Laboratory (ITL)\n",
    "* NIST Center for Neutron Research (NCNR)\n",
    "* Material Measurement Laboratory (MML)\n",
    "* Physical Measurement Laboratory (PML)\n",
    "\n",
    "This makes it an easy target for topic modeling, a way of identifying patterns in a corpus that uses __clustering__."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from lxml import html\n",
    "import requests\n",
    "from __future__ import print_function\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.cluster import KMeans, MiniBatchKMeans\n",
    "from time import time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Building the list of headlines and descriptions\n",
    "\n",
    "We request NIST news based on the following URL, 'http://www.nist.gov/allnews.cfm?s=01-01-2014&e=12-31-2014'. For this workshop, we look at only 2014 news articles posted on the NIST website. \n",
    "\n",
    "We then pass that retrieved content to our HTML parser and search for a specific div class, \"select_portal_module_wrapper\" which is assigned to every headline and every description (headlines receive a strong tag and descriptions receive a p tag).\n",
    "\n",
    "We then merge both the headline and description into one entry in the list because we don't need to differentiate between title and description."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(\"Retrieving data from NIST...\")\n",
    "\n",
    "# Retrieve the data from the web page.\n",
    "page = requests.get('http://www.nist.gov/allnews.cfm?s=01-01-2014&e=12-31-2014') \n",
    "\n",
    "# Use html module to parse it out and store in tree.\n",
    "tree = html.fromstring(page.content)\n",
    "\n",
    "# Create list of news headlines and descriptions. \n",
    "# This required obtaining the xpath of the elements by examining the web page.\n",
    "list_of_headlines = tree.xpath('//div[@class=\"select_portal_module_wrapper\"]/a/strong/text()')\n",
    "list_of_descriptions = tree.xpath('//div[@class=\"select_portal_module_wrapper\"]/p/text()')\n",
    "\n",
    "#Combine each headline and description into one value in a list\n",
    "news=[]\n",
    "for each_headline in list_of_headlines:\n",
    "    for each_description in list_of_descriptions:\n",
    "        news.append(each_headline+each_description)\n",
    "\n",
    "print(\"Last item in list retrieved: %s\" % news[-1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Term Frequency-Inverse Document Frequency\n",
    "\n",
    "- The weight of a term that occurs in a document is proportional to the term frequency.    \n",
    "- Term frequency is the number of times a term occurs in a document.        \n",
    "- Inverse document frequency diminishes the weight of terms that occur very frequently in the document set and increases the weight of terms that occur rarely.\n",
    " \n",
    "![TFIDF](figures/TFIDF-FIG-01.JPG) \n",
    "\n",
    "\n",
    "### Convert collection of documents to TF-IDF matrix\n",
    "We now call a TF-IDF vectorizer to create a sparse matrix with term frequency-inverse document frequency weights:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print(\"Extracting features from the training dataset using a sparse vectorizer\")\n",
    "t0 = time()\n",
    "\n",
    "# Create a sparse word occurrence frequency matrix of the most frequent words\n",
    "# with the following parameters:\n",
    "# Maximum document frequency = half the total documents\n",
    "# Minimum document frequency = two documents\n",
    "# Toss out common English stop words.\n",
    "vectorizer = TfidfVectorizer(input=news, max_df=0.5, min_df=2, stop_words='english')\n",
    "\n",
    "# This calculates the counts \n",
    "X = vectorizer.fit_transform(news) \n",
    "\n",
    "print(\"done in %fs\" % (time() - t0))\n",
    "print(\"n_samples: %d, n_features: %d\" % X.shape)\n",
    "print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's do some clustering\n",
    "\n",
    "![Kmeans clustering](figures/voronoi.png)    \n",
    "\n",
    "I happen to know there are 15 [subject areas](http://www.nist.gov/subject_areas.cfm) at NIST:    \n",
    "    - Bioscience & Health\n",
    "    - Building and Fire Research\n",
    "    - Chemistry\n",
    "    - Electronics & Telecommunications\n",
    "    - Energy\n",
    "    - Environment/Climate\n",
    "    - Information Technology\n",
    "    - Manufacturing\n",
    "    - Materials Science\n",
    "    - Math\n",
    "    - Nanotechnology\n",
    "    - Physics\n",
    "    - Public Safety & Security\n",
    "    - Quality\n",
    "    - Transportation\n",
    "\n",
    "So, why don't we cheat and set the number of clusters to 15?     \n",
    "\n",
    "Then we call the KMeans clustering model from sklearn and set an upper bound to the number of iterations for fitting the data to the model.    \n",
    "\n",
    "Finally we list out each centroid and the top 10 terms associated with each centroid."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Set the number of clusters to 15\n",
    "k = 15\n",
    "\n",
    "# Initialize the kMeans cluster model.\n",
    "km = KMeans(n_clusters=k, init='k-means++', max_iter=100)\n",
    "\n",
    "print(\"Clustering sparse data with %s\" % km)\n",
    "t0 = time()\n",
    "\n",
    "# Pass the model our sparse matrix with the TF-IDF counts.\n",
    "km.fit(X)\n",
    "print(\"done in %0.3fs\" % (time() - t0))\n",
    "print()\n",
    "\n",
    "print(\"Top terms per cluster:\")\n",
    "order_centroids = km.cluster_centers_.argsort()[:, ::-1]\n",
    "\n",
    "terms = vectorizer.get_feature_names()\n",
    "for i in range(k):\n",
    "    print(\"Cluster %d:\" % (i+1), end='')\n",
    "    for ind in order_centroids[i, :10]:\n",
    "        print(' %s' % terms[ind], end='')\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "How do the results compare to NIST's listed [subject areas](http://www.nist.gov/subject_areas.cfm)?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}

{'fraction_non_alphanumeric': 0.16271442986881937, 'fraction_numerical': 0.009460141271442986, 'mean_word_length': 2.8397094430992738, 'pattern_counts': {'":': 62, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 1, 'lorem ipsum': 0, 'www.': 5, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4015419', 'n_tokens_mistral': 2739, 'n_tokens_neox': 2612, 'n_words': 775}