<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Sentiment Classification on Large Movie Reviews\n",
    "\n",
    "[Sentiment Analysis](https://en.wikipedia.org/wiki/Sentiment_analysis) is understood as a classic natural language processing problem. In this example, a large moview review dataset was chosen from IMDB to do a sentiment classification task with some deep learning approaches. The labeled data set consists of 50,000 [IMDB](http://www.imdb.com/) movie reviews (good or bad), in which 25000 highly polar movie reviews for training, and 25,000 for testing. The dataset is originally collected by Stanford researchers and was used in a [2011 paper](http://ai.stanford.edu/~amaas/papers/wvSent_acl2011.pdf), and the highest accuray of 88.33% was achieved without using the unbalanced data. This example illustrates some deep learning approaches to do the sentiment classification with [BigDL](https://github.com/intel-analytics/BigDL) python API.\n",
    "\n",
    "### Load the IMDB Dataset\n",
    "The IMDB dataset need to be loaded into BigDL, note that the dataset has been pre-processed, and each review was encoded as a sequence of integers. Each integer represents the index of the overall frequency of dataset, for instance, '5' means the 5-th most frequent words occured in the data. It is very convinient to filter the words by some conditions, for example, to filter only the top 5,000 most common word and/or eliminate the top 30 most common words. Let's define functions to load the pre-processed data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using /home/arda/chentao/apache/spark-2.2.0-bin-hadoop2.6\nAdding /home/arda/chentao/MyPython/lib/python2.7/site-packages/bigdl/share/lib/bigdl-0.5.0-jar-with-dependencies.jar to BIGDL_JARS\nPrepending /home/arda/chentao/MyPython/lib/python2.7/site-packages/bigdl/share/conf/spark-bigdl.conf to sys.path\nProcessing text dataset\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "finished processing text\n"
     ]
    }
   ],
   "source": [
    "from bigdl.dataset import base\n",
    "import numpy as np\n",
    "\n",
    "def download_imdb(dest_dir):\n",
    "    \"\"\"Download pre-processed IMDB movie review data\n",
    "\n",
    "    :argument\n",
    "        dest_dir: destination directory to store the data\n",
    "\n",
    "    :return\n",
    "        The absolute path of the stored data\n",
    "    \"\"\"\n",
    "    file_name = \"imdb.npz\"\n",
    "    file_abs_path = base.maybe_download(file_name,\n",
    "                                        dest_dir,\n",
    "                                        'https://s3.amazonaws.com/text-datasets/imdb.npz')\n",
    "    return file_abs_path\n",
    "\n",
    "def load_imdb(dest_dir='/tmp/.bigdl/dataset'):\n",
    "    \"\"\"Load IMDB dataset.\n",
    "\n",
    "    :argument\n",
    "        dest_dir: where to cache the data (relative to `~/.bigdl/dataset`).\n",
    "\n",
    "    :return\n",
    "        the train, test separated IMDB dataset.\n",
    "    \"\"\"\n",
    "    path = download_imdb(dest_dir)\n",
    "    f = np.load(path)\n",
    "    x_train = f['x_train']\n",
    "    y_train = f['y_train']\n",
    "    x_test = f['x_test']\n",
    "    y_test = f['y_test']\n",
    "    f.close()\n",
    "\n",
    "    return (x_train, y_train), (x_test, y_test)\n",
    "\n",
    "print('Processing text dataset')\n",
    "(x_train, y_train), (x_test, y_test) = load_imdb()\n",
    "print('finished processing text')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "In order to set a proper max sequence length, we need to go througth the property of the data and see the length distribution of each sentence in the dataset. A box and whisker plot is shown below for reviewing the length distribution in words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Review length: \nMean 233.76 words (172.911495)\n"
     ]
    }
   ],
   "source": [
    "\n",
    "# Summarize review length\n",
    "from matplotlib import pyplot\n",
    "\n",
    "print(\"Review length: \")\n",
    "X = np.concatenate((x_train, x_test), axis=0)\n",
    "result = [len(x) for x in X]\n",
    "print(\"Mean %.2f words (%f)\" % (np.mean(result), np.std(result)))\n",
    "# plot review length\n",
    "# Create a figure instance\n",
    "fig = pyplot.figure(1, figsize=(6, 6))\n",
    "pyplot.boxplot(result)\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Looking the box and whisker plot, the max length of a sample in words is 500, and the mean and median are below 250. According to the plot, we can probably cover the mass of the distribution with a clipped length of 400 to 500. Here we set the max sequence length of each sample as 500."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The corresponding vocabulary sorted by frequency is also required, for further embedding the words with pre-trained vectors. The downloaded vocabulary is in {word: index}, where each word as a key and the index as a value. It needs to be transformed into {index: word} format.\n",
    "\n",
    "Let's define a function to obtain the vocabulary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Processing vocabulary\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "finished processing vocabulary\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "def get_word_index(dest_dir='/tmp/.bigdl/dataset', ):\n",
    "    \"\"\"Retrieves the dictionary mapping word indices back to words.\n",
    "\n",
    "    :argument\n",
    "        path: where to cache the data (relative to `~/.bigdl/dataset`).\n",
    "\n",
    "    :return\n",
    "        The word index dictionary.\n",
    "    \"\"\"\n",
    "    file_name = \"imdb_word_index.json\"\n",
    "    path = base.maybe_download(file_name,\n",
    "                               dest_dir,\n",
    "                               source_url='https://s3.amazonaws.com/text-datasets/imdb_word_index.json')\n",
    "    f = open(path)\n",
    "    data = json.load(f)\n",
    "    f.close()\n",
    "    return data\n",
    "\n",
    "print('Processing vocabulary')\n",
    "word_idx = get_word_index()\n",
    "idx_word = {v:k for k,v in word_idx.items()}\n",
    "print('finished processing vocabulary')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Text pre-processing\n",
    "\n",
    "Before we train the network, some pre-processing steps need to be applied to the dataset. \n",
    "\n",
    "Next let's go through the mechanisms that used to be applied to the data.\n",
    "\n",
    "* We insert a `start_char` at the beginning of each sentence to mark the start point. We set it as `2` here, and each other word index will plus a constant `index_from` to differentiate some 'helper index' (eg. `start_char`, `oov_char`, etc.).\n",
    "\n",
    "* A `max_words` variable is defined as the maximum index number (the least frequent word) included in the sequence. If the word index number is larger than `max_words`, it will be replaced by a out-of-vocabulary number `oov_char`, which is `3` here.\n",
    "\n",
    "* Each word index sequence is restricted to the same length. We used left-padding here, which means the right (end) of the sequence will be keep as many as possible and drop the left (head) of the sequence if its length is more than pre-defined `sequence_len`, or padding the left (head) of the sequence with `padding_value`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "start transformation\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "finish transformation\n"
     ]
    }
   ],
   "source": [
    "def replace_oov(x, oov_char, max_words):\n",
    "    \"\"\"\n",
    "    Replace the words out of vocabulary with `oov_char`\n",
    "    :param x: a sequence\n",
    "    :param max_words: the max number of words to include\n",
    "    :param oov_char: words out of vocabulary because of exceeding the `max_words`\n",
    "        limit will be replaced by this character\n",
    "\n",
    "    :return: The replaced sequence\n",
    "    \"\"\"\n",
    "    return [oov_char if w >= max_words else w for w in x]\n",
    "\n",
    "def pad_sequence(x, fill_value, length):\n",
    "    \"\"\"\n",
    "    Pads each sequence to the same length\n",
    "    :param x: a sequence\n",
    "    :param fill_value: pad the sequence with this value\n",
    "    :param length: pad sequence to the length\n",
    "\n",
    "    :return: the padded sequence\n",
    "    \"\"\"\n",
    "    if len(x) >= length:\n",
    "        return x[(len(x) - length):]\n",
    "    else:\n",
    "        return [fill_value] * (length - len(x)) + x\n",
    "\n",
    "def to_sample(features, label):\n",
    "    \"\"\"\n",
    "    Wrap the `features` and `label` to a training sample object\n",
    "    :param features: features of a sample\n",
    "    :param label: label of a sample\n",
    "    \n",
    "    :return: a sample object including features and label\n",
    "    \"\"\"\n",
    "    return Sample.from_ndarray(np.array(features, dtype='float'), np.array(label))\n",
    "\n",
    "padding_value = 1\n",
    "start_char = 2\n",
    "oov_char = 3\n",
    "index_from = 3\n",
    "max_words = 5000\n",
    "sequence_len = 500\n",
    "\n",
    "print('start transformation')\n",
    "\n",
    "from bigdl.util.common import get_spark_context\n",
    "sc = get_spark_context()\n",
    "\n",
    "train_rdd = sc.parallelize(zip(x_train, y_train), 2) \\\n",
    "        .map(lambda (x, y): ([start_char] + [w + index_from for w in x] , y))\\\n",
    "        .map(lambda (x, y): (replace_oov(x, oov_char, max_words), y))\\\n",
    "        .map(lambda (x, y): (pad_sequence(x, padding_value, sequence_len), y))\\\n",
    "        .map(lambda (x, y): to_sample(x, y))\n",
    "test_rdd = sc.parallelize(zip(x_test, y_test), 2) \\\n",
    "        .map(lambda (x, y): ([start_char] + [w + index_from for w in x], y))\\\n",
    "        .map(lambda (x, y): (replace_oov(x, oov_char, max_words), y))\\\n",
    "        .map(lambda (x, y): (pad_sequence(x, padding_value, sequence_len), y))\\\n",
    "        .map(lambda (x, y): to_sample(x, y))\n",
    "        \n",
    "print('finish transformation')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Word Embedding\n",
    "\n",
    "[Word embedding](https://en.wikipedia.org/wiki/Word_embedding) is a recent breakthrough in natural language field. The key idea is to encode words and phrases into distributed representations in the format of word vectors, which means each word is represented as a vector. There are two widely used word vector training alogirhms, one is published by Google called [word to vector](https://arxiv.org/abs/1310.4546), the other is published by Standford called [Glove](https://nlp.stanford.edu/projects/glove/). In this example, pre-trained glove is loaded into a lookup table and will be fine-tuned during the training process. BigDL provides a method to download and load glove in `news20` package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loading glove\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "finish loading glove\n"
     ]
    }
   ],
   "source": [
    "from bigdl.dataset import news20\n",
    "import itertools\n",
    "\n",
    "embedding_dim = 100\n",
    "\n",
    "print('loading glove')\n",
    "glove = news20.get_glove_w2v(source_dir='/tmp/.bigdl/dataset', dim=embedding_dim)\n",
    "print('finish loading glove')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "For each word whose index less than the `max_word` should try to match its embedding and store in an array.\n",
    "\n",
    "With regard to those words which can not be found in glove, we randomly sample it from a [-0.05, 0.05] uniform distribution.\n",
    "\n",
    "BigDL usually use a `LookupTable` layer to do word embedding, so the matrix will be loaded to the LookupTable by seting the weight."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "processing glove\nfinish processing glove\n"
     ]
    }
   ],
   "source": [
    "print('processing glove')\n",
    "w2v = [glove.get(idx_word.get(i - index_from), np.random.uniform(-0.05, 0.05, embedding_dim))\n",
    "        for i in xrange(1, max_words + 1)]\n",
    "w2v = np.array(list(itertools.chain(*np.array(w2v, dtype='float'))), dtype='float') \\\n",
    "        .reshape([max_words, embedding_dim])\n",
    "print('finish processing glove')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Build models\n",
    "\n",
    "Next, let's build some deep learning models for the sentiment classification. \n",
    "\n",
    "As an example, several deep learning models are illustrated for tutorial, comparison and demonstration.\n",
    "\n",
    "**LSTM**, **GRU**, **Bi-LSTM**, **CNN** and **CNN + LSTM** models are implemented as options. To decide which model to use, just assign model_type the corresponding string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from bigdl.nn.layer import *\n",
    "from bigdl.util.common import *\n",
    "\n",
    "p = 0.2\n",
    "\n",
    "def build_model(w2v):\n",
    "    model = Sequential()\n",
    "\n",
    "    embedding = LookupTable(max_words, embedding_dim)\n",
    "    embedding.set_weights([w2v])\n",
    "    model.add(embedding)\n",
    "    if model_type.lower() == \"gru\":\n",
    "        model.add(Recurrent()\n",
    "                .add(GRU(embedding_dim, 128, p))) \\\n",
    "            .add(Select(2, -1))\n",
    "    elif model_type.lower() == \"lstm\":\n",
    "        model.add(Recurrent()\n",
    "                  .add(LSTM(embedding_dim, 128, p)))\\\n",
    "            .add(Select(2, -1))\n",
    "    elif model_type.lower() == \"bi_lstm\":\n",
    "        model.add(BiRecurrent(CAddTable())\n",
    "                  .add(LSTM(embedding_dim, 128, p)))\\\n",
    "            .add(Select(2, -1))\n",
    "    elif model_type.lower() == \"cnn\":\n",
    "        model.add(Transpose([(2, 3)]))\\\n",
    "            .add(Dropout(p))\\\n",
    "            .add(Reshape([embedding_dim, 1, sequence_len]))\\\n",
    "            .add(SpatialConvolution(embedding_dim, 128, 5, 1))\\\n",
    "            .add(ReLU())\\\n",
    "            .add(SpatialMaxPooling(sequence_len - 5 + 1, 1, 1, 1))\\\n",
    "            .add(Reshape([128]))\n",
    "    elif model_type.lower() == \"cnn_lstm\":\n",
    "        model.add(Transpose([(2, 3)]))\\\n",
    "            .add(Dropout(p))\\\n",
    "            .add(Reshape([embedding_dim, 1, sequence_len])) \\\n",
    "            .add(SpatialConvolution(embedding_dim, 64, 5, 1)) \\\n",
    "            .add(ReLU()) \\\n",
    "            .add(SpatialMaxPooling(4, 1, 1, 1)) \\\n",
    "            .add(Squeeze(3)) \\\n",
    "            .add(Transpose([(2, 3)])) \\\n",
    "            .add(Recurrent()\n",
    "                 .add(LSTM(64, 128, p))) \\\n",
    "            .add(Select(2, -1))\n",
    "\n",
    "    model.add(Linear(128, 100))\\\n",
    "        .add(Dropout(0.2))\\\n",
    "        .add(ReLU())\\\n",
    "        .add(Linear(100, 1))\\\n",
    "        .add(Sigmoid())\n",
    "\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Optimization\n",
    "`Optimizer` need to be created to optimise the model.\n",
    "\n",
    "Here we use the `CNN` model.\n",
    "\n",
    "More details about optimizer in BigDL, please refer to [Programming Guide](https://github.com/intel-analytics/BigDL/wiki/Programming-Guide#optimizer)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "creating: createSequential\ncreating: createLookupTable\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "creating: createRecurrent\ncreating: createTanh\ncreating: createSigmoid\ncreating: createGRU\ncreating: createSelect\ncreating: createLinear\ncreating: createDropout\ncreating: createReLU\ncreating: createLinear\ncreating: createSigmoid\ncreating: createBCECriterion\ncreating: createMaxEpoch\ncreating: createAdam\ncreating: createDistriOptimizer\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "creating: createEveryEpoch\ncreating: createTop1Accuracy\n"
     ]
    }
   ],
   "source": [
    "from bigdl.optim.optimizer import *\n",
    "from bigdl.nn.criterion import *\n",
    "\n",
    "max_epoch = 4\n",
    "batch_size = 64\n",
    "model_type = 'gru'\n",
    "\n",
    "init_engine()\n",
    "\n",
    "optimizer = Optimizer(\n",
    "        model=build_model(w2v),\n",
    "        training_rdd=train_rdd,\n",
    "        criterion=BCECriterion(),\n",
    "        end_trigger=MaxEpoch(max_epoch),\n",
    "        batch_size=batch_size,\n",
    "        optim_method=Adam())\n",
    "\n",
    "optimizer.set_validation(\n",
    "        batch_size=batch_size,\n",
    "        val_rdd=test_rdd,\n",
    "        trigger=EveryEpoch(),\n",
    "        val_method=Top1Accuracy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "To make the training process be visualized by TensorBoard, training summaries should be saved as a format of logs.\n",
    "\n",
    "With regard to the usage of TensorBoard in BigDL, please refer to [BigDL Wiki](https://github.com/intel-analytics/BigDL/wiki/Visualization-with-TensorBoard)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "creating: createTrainSummary\ncreating: createSeveralIteration\ncreating: createValidationSummary\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<bigdl.optim.optimizer.Optimizer at 0x7fdd52119fd0>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import datetime as dt\n",
    "\n",
    "logdir = '/tmp/.bigdl/'\n",
    "app_name = 'adam-' + dt.datetime.now().strftime(\"%Y%m%d-%H%M%S\")\n",
    "\n",
    "train_summary = TrainSummary(log_dir=logdir, app_name=app_name)\n",
    "train_summary.set_summary_trigger(\"Parameters\", SeveralIteration(50))\n",
    "val_summary = ValidationSummary(log_dir=logdir, app_name=app_name)\n",
    "optimizer.set_train_summary(train_summary)\n",
    "optimizer.set_val_summary(val_summary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now, let's start training!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization Done.\nCPU times: user 178 ms, sys: 61.5 ms, total: 239 ms\nWall time: 37min 35s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "show_bigdl_info_logs()\n",
    "redire_spark_logs()\n",
    "train_model = optimizer.optimize()\n",
    "print \"Optimization Done.\"\n",
    "# model = build_model(w2v)\n",
    "# print train_rdd.count()\n",
    "# model.forward(train_rdd.map())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Test\n",
    "Validation accuracy is shown in the training log, here let's get the accuracy on validation set by hand.\n",
    "\n",
    "Predict the `test_rdd` (validation set data), and obtain the predicted label and ground truth label in the list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "predictions = train_model.predict(test_rdd)\n",
    "\n",
    "def map_predict_label(l):\n",
    "    if l > 0.5:\n",
    "        return 1\n",
    "    else:\n",
    "        return 0\n",
    "def map_groundtruth_label(l):\n",
    "    return l.to_ndarray()[0]\n",
    "\n",
    "y_pred = np.array([ map_predict_label(s) for s in predictions.collect()])\n",
    "\n",
    "y_true = np.array([map_groundtruth_label(s.label) for s in test_rdd.collect()])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Then let's see the prediction accuracy on validation set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prediction accuracy on validation set is:  0.89312\n"
     ]
    }
   ],
   "source": [
    "correct = 0\n",
    "for i in xrange(0, y_pred.size):\n",
    "    if (y_pred[i] == y_true[i]):\n",
    "        correct += 1\n",
    "\n",
    "accuracy = float(correct) / y_pred.size\n",
    "print 'Prediction accuracy on validation set is: ', accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Show the confusion matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7fdd3275ab10>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "<KEY>",
      "text/plain": [
       "<Figure size 360x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sn\n",
    "import pandas as pd\n",
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "cm = confusion_matrix(y_true, y_pred)\n",
    "cm.shape\n",
    "\n",
    "df_cm = pd.DataFrame(cm)\n",
    "plt.figure(figsize = (5,4))\n",
    "sn.heatmap(df_cm, annot=True,fmt='d')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "Because of the limitation of ariticle length, not all the results of optional models can be shown respectively. Please try other provided optional models to see the results. If you are interested in optimizing the results, try different training parameters which may make inpacts on the result, such as the max sequence length, batch size, training epochs, preprocessing schemes, optimization methods and so on. Among the models, CNN training would be much quicker. Note that the LSTM and it variants (eg. GRU) are difficult to train, even a unsuitable batch size may cause the model not converge. In addition it is prone to overfitting, please try different dropout threshold and/or add regularizers (abouth how to add regularizers in BigDL please see [BigDL Wiki](https://github.com/intel-analytics/BigDL/wiki/Programming-Guide#regularizers)).\n",
    "\n",
    "### Summary\n",
    "In this example, you learned how to use BigDL to develop deep learning models for sentiment analysis including:\n",
    "\n",
    "* How to load and review the IMDB dataset\n",
    "* How to do word embedding with Glove\n",
    "* How to build a CNN model for NLP with BigDL\n",
    "* How to build a LSTM model for NLP with BigDL\n",
    "* How to build a GRU model for NLP with BigDL\n",
    "* How to build a Bi-LSTM model for NLP with BigDL\n",
    "* How to build a CNN-LSTM model for NLP with BigDL\n",
    "* How to train deep learning models with BigDL\n",
    "\n",
    "Thanks for your reading, please enjoy the trip on using BigDL to build deep learning models."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
<commit_msg>fix some bugs in sentimentAnalysis<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Sentiment Classification on Large Movie Reviews\n",
    "\n",
    "[Sentiment Analysis](https://en.wikipedia.org/wiki/Sentiment_analysis) is understood as a classic natural language processing problem. In this example, a large moview review dataset was chosen from IMDB to do a sentiment classification task with some deep learning approaches. The labeled data set consists of 50,000 [IMDB](http://www.imdb.com/) movie reviews (good or bad), in which 25000 highly polar movie reviews for training, and 25,000 for testing. The dataset is originally collected by Stanford researchers and was used in a [2011 paper](http://ai.stanford.edu/~amaas/papers/wvSent_acl2011.pdf), and the highest accuray of 88.33% was achieved without using the unbalanced data. This example illustrates some deep learning approaches to do the sentiment classification with [BigDL](https://github.com/intel-analytics/BigDL) python API.\n",
    "\n",
    "### Load the IMDB Dataset\n",
    "The IMDB dataset need to be loaded into BigDL, note that the dataset has been pre-processed, and each review was encoded as a sequence of integers. Each integer represents the index of the overall frequency of dataset, for instance, '5' means the 5-th most frequent words occured in the data. It is very convinient to filter the words by some conditions, for example, to filter only the top 5,000 most common word and/or eliminate the top 30 most common words. Let's define functions to load the pre-processed data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Processing text dataset\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "finished processing text\n"
     ]
    }
   ],
   "source": [
    "from bigdl.dataset import base\n",
    "import numpy as np\n",
    "\n",
    "def download_imdb(dest_dir):\n",
    "    \"\"\"Download pre-processed IMDB movie review data\n",
    "\n",
    "    :argument\n",
    "        dest_dir: destination directory to store the data\n",
    "\n",
    "    :return\n",
    "        The absolute path of the stored data\n",
    "    \"\"\"\n",
    "    file_name = \"imdb.npz\"\n",
    "    file_abs_path = base.maybe_download(file_name,\n",
    "                                        dest_dir,\n",
    "                                        'https://s3.amazonaws.com/text-datasets/imdb.npz')\n",
    "    return file_abs_path\n",
    "\n",
    "def load_imdb(dest_dir='/tmp/.bigdl/dataset'):\n",
    "    \"\"\"Load IMDB dataset.\n",
    "\n",
    "    :argument\n",
    "        dest_dir: where to cache the data (relative to `~/.bigdl/dataset`).\n",
    "\n",
    "    :return\n",
    "        the train, test separated IMDB dataset.\n",
    "    \"\"\"\n",
    "    path = download_imdb(dest_dir)\n",
    "    f = np.load(path)\n",
    "    x_train = f['x_train']\n",
    "    y_train = f['y_train']\n",
    "    x_test = f['x_test']\n",
    "    y_test = f['y_test']\n",
    "    f.close()\n",
    "\n",
    "    return (x_train, y_train), (x_test, y_test)\n",
    "\n",
    "print('Processing text dataset')\n",
    "(x_train, y_train), (x_test, y_test) = load_imdb()\n",
    "print('finished processing text')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "In order to set a proper max sequence length, we need to go througth the property of the data and see the length distribution of each sentence in the dataset. A box and whisker plot is shown below for reviewing the length distribution in words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Review length: \nMean 233.76 words (172.911495)\n"
     ]
    },
    {
     "data": {
      "image/png": "<KEY>",
      "text/plain": [
       "<Figure size 432x432 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "# Summarize review length\n",
    "from matplotlib import pyplot\n",
    "\n",
    "print(\"Review length: \")\n",
    "X = np.concatenate((x_train, x_test), axis=0)\n",
    "result = [len(x) for x in X]\n",
    "print(\"Mean %.2f words (%f)\" % (np.mean(result), np.std(result)))\n",
    "# plot review length\n",
    "# Create a figure instance\n",
    "fig = pyplot.figure(1, figsize=(6, 6))\n",
    "pyplot.boxplot(result)\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Looking the box and whisker plot, the max length of a sample in words is 500, and the mean and median are below 250. According to the plot, we can probably cover the mass of the distribution with a clipped length of 400 to 500. Here we set the max sequence length of each sample as 500."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The corresponding vocabulary sorted by frequency is also required, for further embedding the words with pre-trained vectors. The downloaded vocabulary is in {word: index}, where each word as a key and the index as a value. It needs to be transformed into {index: word} format.\n",
    "\n",
    "Let's define a function to obtain the vocabulary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Processing vocabulary\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "finished processing vocabulary\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "def get_word_index(dest_dir='/tmp/.bigdl/dataset', ):\n",
    "    \"\"\"Retrieves the dictionary mapping word indices back to words.\n",
    "\n",
    "    :argument\n",
    "        path: where to cache the data (relative to `~/.bigdl/dataset`).\n",
    "\n",
    "    :return\n",
    "        The word index dictionary.\n",
    "    \"\"\"\n",
    "    file_name = \"imdb_word_index.json\"\n",
    "    path = base.maybe_download(file_name,\n",
    "                               dest_dir,\n",
    "                               source_url='https://s3.amazonaws.com/text-datasets/imdb_word_index.json')\n",
    "    f = open(path)\n",
    "    data = json.load(f)\n",
    "    f.close()\n",
    "    return data\n",
    "\n",
    "print('Processing vocabulary')\n",
    "word_idx = get_word_index()\n",
    "idx_word = {v:k for k,v in word_idx.items()}\n",
    "print('finished processing vocabulary')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Text pre-processing\n",
    "\n",
    "Before we train the network, some pre-processing steps need to be applied to the dataset. \n",
    "\n",
    "Next let's go through the mechanisms that used to be applied to the data.\n",
    "\n",
    "* We insert a `start_char` at the beginning of each sentence to mark the start point. We set it as `2` here, and each other word index will plus a constant `index_from` to differentiate some 'helper index' (eg. `start_char`, `oov_char`, etc.).\n",
    "\n",
    "* A `max_words` variable is defined as the maximum index number (the least frequent word) included in the sequence. If the word index number is larger than `max_words`, it will be replaced by a out-of-vocabulary number `oov_char`, which is `3` here.\n",
    "\n",
    "* Each word index sequence is restricted to the same length. We used left-padding here, which means the right (end) of the sequence will be keep as many as possible and drop the left (head) of the sequence if its length is more than pre-defined `sequence_len`, or padding the left (head) of the sequence with `padding_value`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "start transformation\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "finish transformation\n"
     ]
    }
   ],
   "source": [
    "def replace_oov(x, oov_char, max_words):\n",
    "    \"\"\"\n",
    "    Replace the words out of vocabulary with `oov_char`\n",
    "    :param x: a sequence\n",
    "    :param max_words: the max number of words to include\n",
    "    :param oov_char: words out of vocabulary because of exceeding the `max_words`\n",
    "        limit will be replaced by this character\n",
    "\n",
    "    :return: The replaced sequence\n",
    "    \"\"\"\n",
    "    return [oov_char if w >= max_words else w for w in x]\n",
    "\n",
    "def pad_sequence(x, fill_value, length):\n",
    "    \"\"\"\n",
    "    Pads each sequence to the same length\n",
    "    :param x: a sequence\n",
    "    :param fill_value: pad the sequence with this value\n",
    "    :param length: pad sequence to the length\n",
    "\n",
    "    :return: the padded sequence\n",
    "    \"\"\"\n",
    "    if len(x) >= length:\n",
    "        return x[(len(x) - length):]\n",
    "    else:\n",
    "        return [fill_value] * (length - len(x)) + x\n",
    "\n",
    "def to_sample(features, label):\n",
    "    \"\"\"\n",
    "    Wrap the `features` and `label` to a training sample object\n",
    "    :param features: features of a sample\n",
    "    :param label: label of a sample\n",
    "    \n",
    "    :return: a sample object including features and label\n",
    "    \"\"\"\n",
    "    return Sample.from_ndarray(np.array(features, dtype='float'), np.array(label))\n",
    "\n",
    "padding_value = 1\n",
    "start_char = 2\n",
    "oov_char = 3\n",
    "index_from = 3\n",
    "max_words = 5000\n",
    "sequence_len = 500\n",
    "\n",
    "print('start transformation')\n",
    "\n",
    "from bigdl.util.common import get_spark_context\n",
    "sc = get_spark_context()\n",
    "\n",
    "train_rdd = sc.parallelize(zip(x_train, y_train), 2) \\\n",
    "        .map(lambda (x, y): ([start_char] + [w + index_from for w in x] , y))\\\n",
    "        .map(lambda (x, y): (replace_oov(x, oov_char, max_words), y))\\\n",
    "        .map(lambda (x, y): (pad_sequence(x, padding_value, sequence_len), y))\\\n",
    "        .map(lambda (x, y): to_sample(x, y))\n",
    "test_rdd = sc.parallelize(zip(x_test, y_test), 2) \\\n",
    "        .map(lambda (x, y): ([start_char] + [w + index_from for w in x], y))\\\n",
    "        .map(lambda (x, y): (replace_oov(x, oov_char, max_words), y))\\\n",
    "        .map(lambda (x, y): (pad_sequence(x, padding_value, sequence_len), y))\\\n",
    "        .map(lambda (x, y): to_sample(x, y))\n",
    "        \n",
    "print('finish transformation')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Word Embedding\n",
    "\n",
    "[Word embedding](https://en.wikipedia.org/wiki/Word_embedding) is a recent breakthrough in natural language field. The key idea is to encode words and phrases into distributed representations in the format of word vectors, which means each word is represented as a vector. There are two widely used word vector training alogirhms, one is published by Google called [word to vector](https://arxiv.org/abs/1310.4546), the other is published by Standford called [Glove](https://nlp.stanford.edu/projects/glove/). In this example, pre-trained glove is loaded into a lookup table and will be fine-tuned during the training process. BigDL provides a method to download and load glove in `news20` package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loading glove\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "finish loading glove\n"
     ]
    }
   ],
   "source": [
    "from bigdl.dataset import news20\n",
    "import itertools\n",
    "\n",
    "embedding_dim = 100\n",
    "\n",
    "print('loading glove')\n",
    "glove = news20.get_glove_w2v(source_dir='/tmp/.bigdl/dataset', dim=embedding_dim)\n",
    "print('finish loading glove')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "For each word whose index less than the `max_word` should try to match its embedding and store in an array.\n",
    "\n",
    "With regard to those words which can not be found in glove, we randomly sample it from a [-0.05, 0.05] uniform distribution.\n",
    "\n",
    "BigDL usually use a `LookupTable` layer to do word embedding, so the matrix will be loaded to the LookupTable by seting the weight."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "processing glove\nfinish processing glove\n"
     ]
    }
   ],
   "source": [
    "print('processing glove')\n",
    "w2v = [glove.get(idx_word.get(i - index_from), np.random.uniform(-0.05, 0.05, embedding_dim))\n",
    "        for i in xrange(1, max_words + 1)]\n",
    "w2v = np.array(list(itertools.chain(*np.array(w2v, dtype='float'))), dtype='float') \\\n",
    "        .reshape([max_words, embedding_dim])\n",
    "print('finish processing glove')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Build models\n",
    "\n",
    "Next, let's build some deep learning models for the sentiment classification. \n",
    "\n",
    "As an example, several deep learning models are illustrated for tutorial, comparison and demonstration.\n",
    "\n",
    "**LSTM**, **GRU**, **Bi-LSTM**, **CNN** and **CNN + LSTM** models are implemented as options. To decide which model to use, just assign model_type the corresponding string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from bigdl.nn.layer import *\n",
    "from bigdl.util.common import *\n",
    "\n",
    "p = 0.2\n",
    "\n",
    "def build_model(w2v):\n",
    "    model = Sequential()\n",
    "\n",
    "    embedding = LookupTable(max_words, embedding_dim)\n",
    "    embedding.set_weights([w2v])\n",
    "    model.add(embedding)\n",
    "    if model_type.lower() == \"gru\":\n",
    "        model.add(Recurrent()\n",
    "                .add(GRU(embedding_dim, 128, p))) \\\n",
    "            .add(Select(2, -1))\n",
    "    elif model_type.lower() == \"lstm\":\n",
    "        model.add(Recurrent()\n",
    "                  .add(LSTM(embedding_dim, 128, p)))\\\n",
    "            .add(Select(2, -1))\n",
    "    elif model_type.lower() == \"bi_lstm\":\n",
    "        model.add(BiRecurrent(CAddTable())\n",
    "                  .add(LSTM(embedding_dim, 128, p)))\\\n",
    "            .add(Select(2, -1))\n",
    "    elif model_type.lower() == \"cnn\":\n",
    "        model.add(Transpose([(2, 3)]))\\\n",
    "            .add(Dropout(p))\\\n",
    "            .add(Reshape([embedding_dim, 1, sequence_len]))\\\n",
    "            .add(SpatialConvolution(embedding_dim, 128, 5, 1))\\\n",
    "            .add(ReLU())\\\n",
    "            .add(SpatialMaxPooling(sequence_len - 5 + 1, 1, 1, 1))\\\n",
    "            .add(Reshape([128]))\n",
    "    elif model_type.lower() == \"cnn_lstm\":\n",
    "        model.add(Transpose([(2, 3)]))\\\n",
    "            .add(Dropout(p))\\\n",
    "            .add(Reshape([embedding_dim, 1, sequence_len])) \\\n",
    "            .add(SpatialConvolution(embedding_dim, 64, 5, 1)) \\\n",
    "            .add(ReLU()) \\\n",
    "            .add(SpatialMaxPooling(4, 1, 1, 1)) \\\n",
    "            .add(Squeeze(3)) \\\n",
    "            .add(Transpose([(2, 3)])) \\\n",
    "            .add(Recurrent()\n",
    "                 .add(LSTM(64, 128, p))) \\\n",
    "            .add(Select(2, -1))\n",
    "\n",
    "    model.add(Linear(128, 100))\\\n",
    "        .add(Dropout(0.2))\\\n",
    "        .add(ReLU())\\\n",
    "        .add(Linear(100, 1))\\\n",
    "        .add(Sigmoid())\n",
    "\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Optimization\n",
    "`Optimizer` need to be created to optimise the model.\n",
    "\n",
    "Here we use the `CNN` model.\n",
    "\n",
    "More details about optimizer in BigDL, please refer to [Programming Guide](https://github.com/intel-analytics/BigDL/wiki/Programming-Guide#optimizer)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "creating: createSequential\ncreating: createLookupTable\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "creating: createRecurrent\ncreating: createTanh\ncreating: createSigmoid\ncreating: createGRU\ncreating: createSelect\ncreating: createLinear\ncreating: createDropout\ncreating: createReLU\ncreating: createLinear\ncreating: createSigmoid\ncreating: createBCECriterion\ncreating: createMaxEpoch\ncreating: createAdam\ncreating: createDistriOptimizer\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "creating: createEveryEpoch\ncreating: createTop1Accuracy\n"
     ]
    }
   ],
   "source": [
    "from bigdl.optim.optimizer import *\n",
    "from bigdl.nn.criterion import *\n",
    "\n",
    "max_epoch = 4\n",
    "batch_size = 64\n",
    "model_type = 'gru'\n",
    "\n",
    "init_engine()\n",
    "\n",
    "optimizer = Optimizer(\n",
    "        model=build_model(w2v),\n",
    "        training_rdd=train_rdd,\n",
    "        criterion=BCECriterion(),\n",
    "        end_trigger=MaxEpoch(max_epoch),\n",
    "        batch_size=batch_size,\n",
    "        optim_method=Adam())\n",
    "\n",
    "optimizer.set_validation(\n",
    "        batch_size=batch_size,\n",
    "        val_rdd=test_rdd,\n",
    "        trigger=EveryEpoch(),\n",
    "        val_method=Top1Accuracy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "To make the training process be visualized by TensorBoard, training summaries should be saved as a format of logs.\n",
    "\n",
    "With regard to the usage of TensorBoard in BigDL, please refer to [BigDL Wiki](https://github.com/intel-analytics/BigDL/wiki/Visualization-with-TensorBoard)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "creating: createTrainSummary\ncreating: createSeveralIteration\ncreating: createValidationSummary\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<bigdl.optim.optimizer.Optimizer at 0x7fdd52119fd0>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import datetime as dt\n",
    "\n",
    "logdir = '/tmp/.bigdl/'\n",
    "app_name = 'adam-' + dt.datetime.now().strftime(\"%Y%m%d-%H%M%S\")\n",
    "\n",
    "train_summary = TrainSummary(log_dir=logdir, app_name=app_name)\n",
    "train_summary.set_summary_trigger(\"Parameters\", SeveralIteration(50))\n",
    "val_summary = ValidationSummary(log_dir=logdir, app_name=app_name)\n",
    "optimizer.set_train_summary(train_summary)\n",
    "optimizer.set_val_summary(val_summary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now, let's start training!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization Done.\nCPU times: user 178 ms, sys: 61.5 ms, total: 239 ms\nWall time: 37min 35s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "show_bigdl_info_logs()\n",
    "redire_spark_logs()\n",
    "train_model = optimizer.optimize()\n",
    "print \"Optimization Done.\"\n",
    "# model = build_model(w2v)\n",
    "# print train_rdd.count()\n",
    "# model.forward(train_rdd.map())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Test\n",
    "Validation accuracy is shown in the training log, here let's get the accuracy on validation set by hand.\n",
    "\n",
    "Predict the `test_rdd` (validation set data), and obtain the predicted label and ground truth label in the list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "predictions = train_model.predict(test_rdd)\n",
    "\n",
    "def map_predict_label(l):\n",
    "    if l > 0.5:\n",
    "        return 1\n",
    "    else:\n",
    "        return 0\n",
    "def map_groundtruth_label(l):\n",
    "    return l.to_ndarray()[0]\n",
    "\n",
    "y_pred = np.array([ map_predict_label(s) for s in predictions.collect()])\n",
    "\n",
    "y_true = np.array([map_groundtruth_label(s.label) for s in test_rdd.collect()])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Then let's see the prediction accuracy on validation set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prediction accuracy on validation set is:  0.89312\n"
     ]
    }
   ],
   "source": [
    "correct = 0\n",
    "for i in xrange(0, y_pred.size):\n",
    "    if (y_pred[i] == y_true[i]):\n",
    "        correct += 1\n",
    "\n",
    "accuracy = float(correct) / y_pred.size\n",
    "print 'Prediction accuracy on validation set is: ', accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Show the confusion matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7fdd3275ab10>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "<KEY>",
      "text/plain": [
       "<Figure size 360x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sn\n",
    "import pandas as pd\n",
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "cm = confusion_matrix(y_true, y_pred)\n",
    "cm.shape\n",
    "\n",
    "df_cm = pd.DataFrame(cm)\n",
    "plt.figure(figsize = (5,4))\n",
    "sn.heatmap(df_cm, annot=True,fmt='d')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "Because of the limitation of ariticle length, not all the results of optional models can be shown respectively. Please try other provided optional models to see the results. If you are interested in optimizing the results, try different training parameters which may make inpacts on the result, such as the max sequence length, batch size, training epochs, preprocessing schemes, optimization methods and so on. Among the models, CNN training would be much quicker. Note that the LSTM and it variants (eg. GRU) are difficult to train, even a unsuitable batch size may cause the model not converge. In addition it is prone to overfitting, please try different dropout threshold and/or add regularizers (abouth how to add regularizers in BigDL please see [BigDL Wiki](https://github.com/intel-analytics/BigDL/wiki/Programming-Guide#regularizers)).\n",
    "\n",
    "### Summary\n",
    "In this example, you learned how to use BigDL to develop deep learning models for sentiment analysis including:\n",
    "\n",
    "* How to load and review the IMDB dataset\n",
    "* How to do word embedding with Glove\n",
    "* How to build a CNN model for NLP with BigDL\n",
    "* How to build a LSTM model for NLP with BigDL\n",
    "* How to build a GRU model for NLP with BigDL\n",
    "* How to build a Bi-LSTM model for NLP with BigDL\n",
    "* How to build a CNN-LSTM model for NLP with BigDL\n",
    "* How to train deep learning models with BigDL\n",
    "\n",
    "Thanks for your reading, please enjoy the trip on using BigDL to build deep learning models."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}

{'fraction_non_alphanumeric': 0.16962870410085223, 'fraction_numerical': 0.010760308076758108, 'mean_word_length': 2.458274216432349, 'pattern_counts': {'":': 539, '<': 13, '<?xml version=': 0, '>': 19, 'https://': 20, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 26, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6195696', 'n_tokens_mistral': 19208, 'n_tokens_neox': 18510, 'n_words': 4785}