<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following instructions have been tried on OSX 10.11.5.\n",
    "Setup on other platforms should be quite similar.\n",
    "\n",
    "# Requirements\n",
    "\n",
    "* python 3.5\n",
    "* mongodb >= 3.2.3\n",
    "* anakonda >= 4.0.8 for python 3.5 [download](https://www.continuum.io/downloads)\n",
    "* other python libraries: delorean >= 0.6.0, persistent >= 4.2.1\n",
    "These libraries are installed with the following command.\n",
    "\n",
    "> pip install ...\n",
    "\n",
    "If you do not have pip, you can install it using these [official instructions](https://pip.pypa.io/en/stable/installing/).\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "## Data\n",
    "\n",
    "### Load data into mongodb\n",
    "\n",
    "With mongodb running, import the data with the following two commands.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2016-06-14T18:03:42.832+0300\tconnected to: localhost\n",
      "2016-06-14T18:03:43.154+0300\timported 900 documents\n",
      "2016-06-14T18:03:43.166+0300\tconnected to: localhost\n",
      "2016-06-14T18:03:43.279+0300\timported 332 documents\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "mongoimport -d firenze_sample -c checkins --file data/firenze_checkins.json --upsert\n",
    "mongoimport -d firenze_sample -c venues --file data/firenze_venues.json --upsert"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download other data\n",
    "\n",
    "\n",
    "## Train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Process is terminated.\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "python3.5 -W ignore train.py -k_min 1 -k_step 1 -k_max 15 --runs 10 -lambdas \"1\" --iter 100 --rel_change 0.00001 \\\n",
    "    --plot --save mongo --dbname firenze_sample --query '{\"city\":\"Firenze\"}'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "The above outputs files of the following form,\n",
    "* [date]-logl_across_k.pdf\n",
    "* [date].desc\n",
    "* [date].mdl\n",
    "* [date].scaler\n",
    "* [date].unigrams\n",
    "\n",
    "where [date] is the timestamp when the program finished."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }<commit_msg>more setup instructions<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following instructions have been tried on OSX 10.11.5.\n",
    "Setup on other platforms should be quite similar.\n",
    "\n",
    "# Requirements\n",
    "\n",
    "Below we list the software required to run the code, along with the version for which it was run without problems.\n",
    "\n",
    "* python 3.5\n",
    "* mongodb 3.2.3 [download the community version](https://www.mongodb.com/download-center?jmp=nav#community)\n",
    "* anakonda 4.0.8 for python 3.5 [download](https://www.continuum.io/downloads)\n",
    "* other python libraries: delorean 0.6.0, persistent 4.2.1\n",
    "These libraries are installed with the following command.\n",
    "\n",
    "> pip install delorean persistent\n",
    "\n",
    "If you do not have pip, you can install it using these [official instructions](https://pip.pypa.io/en/stable/installing/).\n",
    "\n",
    "\n",
    "\n",
    "## Data\n",
    "\n",
    "We provide a sample dataset with Foursquare data from Florence (Firenze).\n",
    "The two data files are data/firenze_checkins.json and data/firenze_venues.json with checkins and venues from that city.\n",
    "\n",
    "### Data schema\n",
    "\n",
    "The checkins in the data have the following schema,\n",
    "```\n",
    "{\n",
    "\t\"_id\" : \"558ae31a498e73ad37c4ad1e\",\n",
    "\t\"foursquareURL\" : \"https://foursquare.com/vincentjuggler/checkin/558ae31a498e73ad37c4ad1e\",\n",
    "\t\"venueId\" : \"4c7ce6167a856dcb9651e3a7\",\n",
    "\t\"timestamp\" : 1435165466,\n",
    "\t\"foursquareUserId\" : \"40232529\",\n",
    "\t\"timeZoneOffset\" : 120,\n",
    "\t\"swarmURL\" : \"https://www.swarmapp.com/c/cM24svI46Ow\"\n",
    "},\n",
    "```\n",
    "where\n",
    "* '\\_id' is a unique identifier for the checkin,\n",
    "* 'foursquareURL' is the public Foursquare URL for the checkin,\n",
    "* 'venueId' is a unique identifier for the venue where the checkin was performed,\n",
    "* 'timestamp' is the posix timestamp of the checkin,\n",
    "* 'foursquareUserId' is a unique identifier for the user,\n",
    "* 'timeZoneOffset' the number of minutes offset from Greenwich time,\n",
    "* 'swarmULR' is the public Swarm URL for the checkin.\n",
    "\n",
    "The venues in the data have the following schema,\n",
    "```\n",
    "{\n",
    "\t\"_id\" : \"4c7ce6167a856dcb9651e3a7\",\n",
    "\t\"numUsers\" : 271,\n",
    "\t\"state\" : \"Toscana\",\n",
    "\t\"gadmCity\" : \"Florence\",\n",
    "\t\"foursquareURL\" : \"https://foursquare.com/v/giardini-di-lungarno-del-tempio/4c7ce6167a856dcb9651e3a7\",\n",
    "\t\"coordinates\" : [\n",
    "\t\t11.272625494550192,\n",
    "\t\t43.76587635749327\n",
    "\t],\n",
    "\t\"city\" : \"Firenze\",\n",
    "\t\"country\" : \"Italia\",\n",
    "\t\"numTips\" : 2,\n",
    "\t\"numCheckins\" : 546,\n",
    "\t\"name\" : \"<NAME>\",\n",
    "\t\"categories\" : [\n",
    "\t\t{\n",
    "\t\t\t\"primary\" : true,\n",
    "\t\t\t\"id\" : \"4bf58dd8d48988d162941735\",\n",
    "\t\t\t\"name\" : \"Other Great Outdoors\"\n",
    "\t\t}\n",
    "\t],\n",
    "\t\"postalCode\" : \"50121\"\n",
    "},\n",
    "```\n",
    "where\n",
    "* '\\_id' is the unique identifier for the venue (referenced from `checkins`),\n",
    "* 'coordinates' are the latitude and longitude of the venue location,\n",
    "* 'name' is the name of the venue,\n",
    "* 'categories' are descriptors for the venue (i.e., what kind of venue this is, e.g., restaurant, café, etc).\n",
    "\n",
    "\n",
    "### Load data into mongodb\n",
    "\n",
    "With mongodb running, import the data into a MongoDB database with the following two commands.\n",
    "\n",
    "> mongoimport -d firenze_db -c checkins --file data/firenze_checkins.json --upsert\n",
    "\n",
    "> mongoimport -d firenze_db -c venues --file data/firenze_venues.json --upsert\n",
    "\n",
    "The data are stored in a database named 'firenze_db' and in two collections, 'checkins' and 'venues'.\n",
    "\n",
    "### Download other data\n",
    "\n",
    "TODO later if needed.\n",
    "\n",
    "\n",
    "## Training\n",
    "\n",
    "To train a model on the data, issue a command like the following.\n",
    "> python3.5 -W ignore train.py -k_min 1 -k_step 1 -k_max 15 --runs 10 --iter 100 \\\n",
    ">    --save mongo --dbname firenze_db --query '{\"city\":\"Firenze\"}'\n",
    "\n",
    "The specified parameters have the following meaning.\n",
    "* '-W ignore': do not print warnings. If not included, several numberical warnings are printed, which are however dealt with in the code.\n",
    "* '-k_min', '-k_max', '-k_step': the minimum, maximum, and increment of the number $k$ of regions to be learned by the model.\n",
    "* '-runs': the number of random parameter initializations tried for each value of $k$,\n",
    "* '-iter': the number of EM iterations,\n",
    "* '--save': directive to save the results in a file\n",
    "* '--dbname': the name of the database that stores the Foursquare data\n",
    "* '--query': the subset of the checkins on which to run the model.\n",
    "\n",
    "\n",
    "The above outputs files with filenames of the following form,\n",
    "* [date].desc: a summary **description** of the results,\n",
    "* [date].mdl: the learned **model**,\n",
    "* [date].scaler: the serialization of a 'scikit-learn' Scaler object, to scale the learned geographic regions back to their original scale (the *model* saves them in normalized form),\n",
    "* [date].unigrams: the values of the various features, as encountered in the dataset.\n",
    "\n",
    "where [date] is the timestamp when the program terminated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
{'fraction_non_alphanumeric': 0.1974823882665435, 'fraction_numerical': 0.03741771567155561, 'mean_word_length': 2.5829540752999587, 'pattern_counts': {'":': 47, '<': 4, '<?xml version=': 0, '>': 14, 'https://': 8, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3454202', 'n_tokens_mistral': 3345, 'n_tokens_neox': 3131, 'n_words': 838}